Kernel-based Virtual Machine
The Linux SCSI Target Wiki
 Screenshot of qemu/KVM running NetBSD, OpenSolaris and Kubuntu on an Linux host. | |
| Developer(s) | Red Hat, Inc. |
|---|---|
| Written in | C (programming language) |
| Operating system | Linux kernel |
| Type | Platform virtualization |
| License | GNU General Public License or GNU Lesser General Public License |
| Website | linux-kvm.org (unofficial) |
Kernel-based Virtual Machine (KVM) is a Linux kernel virtualization infrastructure. KVM supports native virtualization on x86 processors that provide Intel VT-x or AMD-V extensions; it has also been ported to S/390,[1] PowerPC,[2] IA-64, and an ARM port[3] is in progress.
Using KVM, it is possible to use both Datera, Inc. SCSI initiators and LinuxIOs inside of KVM guests at high bandwith speeds using PCI Express (PCIe) device passthrough.
Although doing PCIe device passthrough is not a requirement for using initiators and targets in KVM guests, this page focuses on 10 Gb/sec Ethernet hardware capable of single function operation, multi-function operation, and future PCIsig SR-IOV (included in v2.6.30-rc) and MR-IOV capable networking hardware.
Target KVM
FILEIO and RAMDISK storages objects are exported from KVM guests to other Linux instances via using iSCSI in Demo Mode. RAMDISK storage objects are using a 64 slot queue depth.
LinuxIO can also bring legacy PCIe networking and storage HBAs into KVM guests, e.g.:
The struct socket->sock_ops->sendpage() has been disabled for outgoing bulk struct scatterlist TX traffic (eg: iSCSI READ) in LIO code (it is usually enabled by default). This was done because sendpage() currently triggers a bug with the X3110 adapter in the KVM guest, and has been disabled for these tests.
The KVM guest is started with:
qemu-system-x86_64 -m 3072 -smp 8 -pcidevice host=06:00.0 lenny64guest1-orig.img
Initiators KVM
Both Core-iSCSI and Open-iSCSI Initiators can be used on kernel v2.6.29.2 inside of KVM guest.
The iSCSI Initiator KVM guest is started with:
qemu-system-x86_64 -m 4096 -smp 8 -pcidevice host=07:00.0 lenny64guest0-orig.img
Core-iSCSI
Core-iSCSI is using between 1 and 4 iSCSI connections per iSCSI session.
Both Initiators are using digests disabled with:
-
MaxRecvDataSegmentLength=131072 -
MaxBurstLength=262144 -
FirstBurstLength=65536 -
ImmediateData=Yes -
InitialR2T=Yes
Also, struct socket->sock_ops->sendpage() triggers a bug in KVM guest with vxge, so unforutately it is not possible to obtain Open-iSCSI WRITE and WRITE/READ numbers yet..
Core-iSCSI is passing:
-
iscsi_can_queue=128 -
iscsi_cmd_per_lun=64
These correspond to struct scsi_host_template->can_queue and struct scsi_host_template->can_queue->cmd_per_lun settings.
Open-iSCSI
The Open-iSCSI module is from upstream linux-2.6.29.y.git. Open-iSCSI is using 1 iSCSI connection per iSCSI session.
Open-iSCSI is using its default values for can_queue and cmd_per_lun which are 128 and 32 respectively.
Test environment
The first tests focus on using RAMDISK and FILEIO storage objects using 10 GbE adapters. Of course, IBLOCK storage objects (SAS, SATA, SSD, etc.) work across the 10 GbE and iSCSI fabric as well.
It is also possible to use Linux dm-multipath for scaling across multiple iSCSI sessions to the same LinuxIO and iSCSI Target Port using both Open-iSCSI and Core-iSCSI. However, tests here focus on iSCSI MC/S and SC/S to evaluate which configuration provides maximum bandwith to the underlying Linux/SCSI struct scsi_device referencing an iSCSI LUN in a KVM guest.
Configuration
- CPU RAM: The KVM Host used was a Tyan S7010 with a Intel IOH-5520 chipset using dual Xeon E5520 microprocessors and 8 GB of 1033 Mhz DDR-3 ECC memory.
- Ethernet: two Neterion X3110 10 GbE PCIe Single-function/Multi-Function capable adapters are being passed using PCIe device passthrough into KVM/QEMU guests.
- OS: The KVM host runs linux-2.6.git (2.6.30-rc3) on Fedora 11 x86_64 preview.
- KVM: The
kvm.koandkvm_intel.koand qemu-system-x86_64 kernel modules are from the kvm-85 tarball with Marcelo's patch series forsmp_send_reschedule() - Patches (
smp_send_rescheduleinvcpu_kick): assigned dev host intx race fix (1/4)(2/4) (3/4) (4/4)
Setup
Each KVM guest is assigned a Neterion X3110 10 GbE adapter running in single-function (SF) mode. Each of these adapters has a total of 8 Vpath pairs (a TX and RX MSI-X interrupt pair for each Vpath) and each Linux KVM guest is using 4 of the 8 VPaths available for each adapter.
Both KVM guests providing LinuxIO and initiator functionality below are using a 9000 byte MTU, and the following sysctl settings:
### IPV4 specific settings
# turns TCP timestamp support off, default 1, reduces CPU use
net.ipv4.tcp_timestamps = 0
# turn SACK support off, default on
net.ipv4.tcp_sack = 0
# on systems with a VERY fast bus -> memory interface this is the big gainer
# sets min/default/max TCP read buffer, default 4096 87380 174760
net.ipv4.tcp_rmem = 10000000 10000000 10000000
# sets min/pressure/max TCP write buffer, default 4096 16384 131072
net.ipv4.tcp_wmem = 10000000 10000000 10000000
# sets min/pressure/max TCP buffer space, default 31744 32256 32768
net.ipv4.tcp_mem = 10000000 10000000 10000000
### CORE settings (mostly for socket and UDP effect)
# maximum receive socket buffer size, default 131071
net.core.rmem_max = 524287
# maximum send socket buffer size, default 131071
net.core.wmem_max = 524287
# default receive socket buffer size, default 65535
net.core.rmem_default = 524287
# default send socket buffer size, default 65535
net.core.wmem_default = 524287
# maximum amount of option memory buffers, default 10240
net.core.optmem_max = 524287
# number of unprocessed input packets before kernel starts dropping them, default 300
net.core.netdev_max_backlog = 300000
Benchmarking
The benchmarking tool was disktest from Linux-Test-Project (LTP), using pthread and O_DIRECT to RAMDISK and FILEIO backstore objects. Results summary:
- Core-iSCSI read/write (bi-directional) throughput with MC/S scales a single Linux/iSCSI LUN to 10 Gb/sec line-rate speeds (>1200MB/s).
- Core-iSCSI read (uni-directional) throughput with MC/S average is about 6-7 Gb/sec.
- Core-iSCSI write (uni-directional) throughput with MC/S average is about 5 Gb/sec.
KVM I/O bandwith
The first test is using disktest for iSCSI READs, then with iSCSI WRITEs, then with both iSCSI WRITEs/READs. This is done using the following LIP disktest parameters with a 256K block size and 64 pthreads with O_DIRECT access to struct block_device enabled:
disktest -K64 -ID -PTX -h3 -t30 -r -b262144 /dev/sda disktest -K64 -ID -PTX -h3 -t30 -w -b262144 /dev/sda disktest -K64 -ID -PTX -h3 -t30 -wr -b262144 /dev/sda
Open-iSCSI Initiator
Read: SC/S
| 2009/05/14-18:55:20 | START | 2767 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0) | 2009/05/14-18:55:20 | INFO | 2767 | v1.4.2 | /dev/sda | Starting pass | 2009/05/14-18:56:20 | STAT | 2767 | v1.4.2 | /dev/sda | Total bytes read in 98669 transfers: 25865486336 | 2009/05/14-18:56:20 | STAT | 2767 | v1.4.2 | /dev/sda | Total read throughput: 431091438.9B/s (411.12MB/s), IOPS 1644.5/s. | 2009/05/14-18:56:20 | END | 2767 | v1.4.2 | /dev/sda | Test Done (Passed)
Write: SC/S
This test could not effectively be performed because an Open-iSCSI problem: Unable to use sendpage() for iSCSI WRITE.
Read/Write: SC/S
This test could not effectively be performed because an Open-iSCSI problem: Unable to use sendpage() for iSCSI WRITE.
Core-iSCSI Initiator
Read: SC/S
| 2009/05/14-17:58:31 | START | 3334 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0) | 2009/05/14-17:59:31 | STAT | 3334 | v1.4.2 | /dev/sda | Total bytes read in 176951 transfers: 46386642944 | 2009/05/14-17:59:31 | STAT | 3334 | v1.4.2 | /dev/sda | Total read throughput: 773110715.7B/s (737.30MB/s), IOPS 2949.2/s. | 2009/05/14-17:59:31 | END | 3334 | v1.4.2 | /dev/sda | Test Done (Passed)
Read: MC/S with 2 connections
| 2009/05/14-18:03:33 | START | 3406 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0) | 2009/05/14-18:04:34 | STAT | 3406 | v1.4.2 | /dev/sda | Total bytes read in 211214 transfers: 55368482816 | 2009/05/14-18:04:34 | STAT | 3406 | v1.4.2 | /dev/sda | Total read throughput: 922808046.9B/s (880.06MB/s), IOPS 3520.2/s. | 2009/05/14-18:04:34 | END | 3406 | v1.4.2 | /dev/sda | Test Done (Passed)
Read: MC/S with 3 connections
| 2009/05/14-18:06:32 | START | 3478 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0) | 2009/05/14-18:07:32 | STAT | 3478 | v1.4.2 | /dev/sda | Total bytes read in 194460 transfers: 50976522240 | 2009/05/14-18:07:32 | STAT | 3478 | v1.4.2 | /dev/sda | Total read throughput: 849608704.0B/s (810.25MB/s), IOPS 3241.0/s. | 2009/05/14-18:07:32 | END | 3478 | v1.4.2 | /dev/sda | Test Done (Passed)
Read: MC/S with 4 connections
| 2009/05/14-18:12:56 | START | 3551 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0) | 2009/05/14-18:13:56 | STAT | 3551 | v1.4.2 | /dev/sda | Total bytes read in 194242 transfers: 50919374848 | 2009/05/14-18:13:56 | STAT | 3551 | v1.4.2 | /dev/sda | Total read throughput: 848656247.5B/s (809.34MB/s), IOPS 3237.4/s. | 2009/05/14-18:13:56 | END | 3551 | v1.4.2 | /dev/sda | Test Done (Passed)
Write: SC/S
| 2009/05/14-18:19:52 | START | 3929 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0) | 2009/05/14-18:20:53 | STAT | 3929 | v1.4.2 | /dev/sda | Total bytes written in 124977 transfers: 32761970688 | 2009/05/14-18:20:53 | STAT | 3929 | v1.4.2 | /dev/sda | Total write throughput: 546032844.8B/s (520.74MB/s), IOPS 2082.9/s. | 2009/05/14-18:20:53 | END | 3929 | v1.4.2 | /dev/sda | Test Done (Passed)
Write: MC/S with 2 connections
| 2009/05/14-18:21:51 | START | 3999 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0) | 2009/05/14-18:22:52 | STAT | 3999 | v1.4.2 | /dev/sda | Total bytes written in 151405 transfers: 39689912320 | 2009/05/14-18:22:52 | STAT | 3999 | v1.4.2 | /dev/sda | Total write throughput: 661498538.7B/s (630.85MB/s), IOPS 2523.4/s. | 2009/05/14-18:22:52 | END | 3999 | v1.4.2 | /dev/sda | Test Done (Passed)
Write: MC/S with 3 connections
| 2009/05/14-18:23:12 | START | 4072 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0) | 2009/05/14-18:24:13 | STAT | 4072 | v1.4.2 | /dev/sda | Total bytes written in 131623 transfers: 34504179712 | 2009/05/14-18:24:13 | STAT | 4072 | v1.4.2 | /dev/sda | Total write throughput: 575069661.9B/s (548.43MB/s), IOPS 2193.7/s. | 2009/05/14-18:24:13 | END | 4072 | v1.4.2 | /dev/sda | Test Done (Passed)
Write: MC/S with 4 connections
| 2009/05/14-18:26:24 | START | 4145 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0) | 2009/05/14-18:27:24 | STAT | 4145 | v1.4.2 | /dev/sda | Total bytes written in 133475 transfers: 34989670400 | 2009/05/14-18:27:24 | STAT | 4145 | v1.4.2 | /dev/sda | Total write throughput: 583161173.3B/s (556.15MB/s), IOPS 2224.6/s. | 2009/05/14-18:27:24 | END | 4145 | v1.4.2 | /dev/sda | Test Done (Passed)
Read/Write: SC/S
| 2009/05/14-17:50:16 | START | 2815 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0) | 2009/05/14-17:51:16 | STAT | 2815 | v1.4.2 | /dev/sda | Total bytes read in 62973 transfers: 16507994112 | 2009/05/14-17:51:16 | STAT | 2815 | v1.4.2 | /dev/sda | Total bytes written in 62973 transfers: 16507994112 | 2009/05/14-17:51:16 | STAT | 2815 | v1.4.2 | /dev/sda | Total read throughput: 550266470.4B/s (524.77MB/s), IOPS 2099.1/s. | 2009/05/14-17:51:16 | STAT | 2815 | v1.4.2 | /dev/sda | Total write throughput: 550266470.4B/s (524.77MB/s), IOPS 2099.1/s. | 2009/05/14-17:51:16 | END | 2815 | v1.4.2 | /dev/sda | Test Done (Passed)
Read/Write: MC/S with 2 connections
| 2009/05/14-17:51:30 | START | 2891 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0) | 2009/05/14-17:52:30 | STAT | 2891 | v1.4.2 | /dev/sda | Total bytes read in 82009 transfers: 21498167296 | 2009/05/14-17:52:30 | STAT | 2891 | v1.4.2 | /dev/sda | Total bytes written in 82010 transfers: 21498429440 | 2009/05/14-17:52:30 | STAT | 2891 | v1.4.2 | /dev/sda | Total read throughput: 716605576.5B/s (683.41MB/s), IOPS 2733.6/s. | 2009/05/14-17:52:30 | STAT | 2891 | v1.4.2 | /dev/sda | Total write throughput: 716614314.7B/s (683.42MB/s), IOPS 2733.7/s. | 2009/05/14-17:52:30 | END | 2891 | v1.4.2 | /dev/sda | Test Done (Passed)
Read/Write: MC/S with 3 connections
| 2009/05/14-17:52:41 | START | 2964 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0) | 2009/05/14-17:53:41 | STAT | 2964 | v1.4.2 | /dev/sda | Total bytes read in 83225 transfers: 21816934400 | 2009/05/14-17:53:41 | STAT | 2964 | v1.4.2 | /dev/sda | Total bytes written in 83225 transfers: 21816934400 | 2009/05/14-17:53:41 | STAT | 2964 | v1.4.2 | /dev/sda | Total read throughput: 727231146.7B/s (693.54MB/s), IOPS 2774.2/s. | 2009/05/14-17:53:41 | STAT | 2964 | v1.4.2 | /dev/sda | Total write throughput: 727231146.7B/s (693.54MB/s), IOPS 2774.2/s. | 2009/05/14-17:53:41 | END | 2964 | v1.4.2 | /dev/sda | Test Done (Passed)
Read/Write: MC/S with 4 connections
| 2009/05/14-17:55:27 | START | 3176 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0) | 2009/05/14-17:56:28 | STAT | 3176 | v1.4.2 | /dev/sda | Total bytes read in 84816 transfers: 22234005504 | 2009/05/14-17:56:28 | STAT | 3176 | v1.4.2 | /dev/sda | Total bytes written in 84817 transfers: 22234267648 | 2009/05/14-17:56:28 | STAT | 3176 | v1.4.2 | /dev/sda | Total read throughput: 741133516.8B/s (706.80MB/s), IOPS 2827.2/s. | 2009/05/14-17:56:28 | STAT | 3176 | v1.4.2 | /dev/sda | Total write throughput: 741142254.9B/s (706.81MB/s), IOPS 2827.2/s. | 2009/05/14-17:56:28 | END | 3176 | v1.4.2 | /dev/sda | Test Done (Passed)
See also
Notes
- ↑ Sergey Kiselev (2007-04-26). "kvm-20 on SLES 10 fails". gmane.comp.emulators.kvm.devel.
- ↑ Igor Serebryany (2010-11-20). "OOM kernel panic on bootup in debian/ubuntu guests". gmane.comp.emulators.kvm.devel.
- ↑ "KVM for ARM wiki".
External links
- Official website
- KVM Wikipedia entry
- QEMU TCM Overview
- Linux-Test-Project (LTP)
