vagrant
一个基于ruby的部署虚拟化开发环境的工具，用virtualbox作为VIM支撑（即命令行中的provider，目前也支持libvirt，暂未测试），chef作为自动化环境创建。官方的SPDK代码中提供的Vagrantfile提供了一个挂载了NVMe设备的虚机，以及对应实例代码hello_world，用来帮助开发者进行SPDK开发环境的快速搭建。
**upgrade vagrant **
wget https://releases.hashicorp.com/vagrant/1.9.1/vagrant_1.9.1_x86_64.deb
dpkg -i vagrant_1.9.1_x86_64.deb
vagrant plugin install vagrant-libvirt
vagrant plugin install vagrant-proxyconf
默认情况虚机分配两个CPU，4G内存。可以通过环境变量ENV['SPDK_VAGRANT_VMCPU']和ENV['SPDK_VAGRANT_VMRAM']来进行设置
default option: virtualbox作为provider
apt-get install virtualbox
apt-get install virtualbox-ext-pack
在spdk/scripts/vagrant下执行
- vagrant up
option 2: libvirt作为provider
apt-get install libvirt-bin
apt-get install libvirt-dev
vagrant up --provider=libvirt
虚机执行完毕后基本的输出如下：
Ignoring ruby-libvirt-0.7.1 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.1
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'generic/fedora26'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: vagrant_default_1534148841675_59957
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: hostonly
==> default: Forwarding ports...
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
==> default: Running provisioner: file...
ssh进入该虚机，默认目录是/home/vagrant。理论上host的spdk目录会共享到虚机当中，避免在虚机上重新clone，配置和编译spdk代码。如果发现该目录是空的，需要重新进行一遍host的clone，配置和编译的步骤。
- vagrant ssh
Ignoring ruby-libvirt-0.7.1 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.1
执行示例代码（hello world）
以下是虚机内操作，可以看到NVMe devices已经挂载可用
[vagrant@localhost ~]$ pwd
/home/vagrant
[vagrant@localhost spdk]$ sudo HUGEMEM=2048 scripts/setup.sh
[vagrant@localhost spdk]$ ./scripts/setup.sh status
Hugepages
node hugesize free / total
node0 2048kB 1024 / 1024
NVMe devices
BDF Numa Node Driver name Device name
**0000:00:0e.0 0 uio_pci_generic **-
I/OAT DMA
BDF Numa Node Driver Name
virtio
BDF Numa Node Driver Name Device Name
[vagrant@localhost spdk]$ sudo examples/nvme/hello_world/hello_world
Starting SPDK v18.10-pre / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: hello_world -c 0x1 --legacy-mem --file-prefix=spdk0 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/spdk0/mp_socket
EAL: Probing VFIO support...
EAL: WARNING! Base virtual address hint (0x20040008f000 != 0x7f69b830d000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x2008000f0000 != 0x7f69b82ac000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x200c00151000 != 0x7f69b824b000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
Initializing NVMe Controllers
EAL: PCI device 0000:00:0e.0 on NUMA socket 0
EAL: probe driver: 80ee:4e56 spdk_nvme
Attaching to 0000:00:0e.0
Attached to 0000:00:0e.0
Using controller ORCL-VBOX-NVME-VER12 (VB1234-56789 ) with 1 namespaces.
Namespace ID: 1 size: 1GB
Initialization complete.
INFO: using host memory buffer for IO
Hello world!
执行示例代码（FIO）
退出到重启VM释放大页内存
vagrant reload
vagrant ssh
部署FIO
git clone https://github.com/axboe/fio
cd fio && git checkout fio-3.3
make
cd
./configure --with-fio=/home/vagrant/spdk/fio/
vim examples/nvme/fio_plugin/example_config.fio 并加入最后一行配置指定PCI地址
- sudo LD_PRELOAD=/home/vagrant/spdk/examples/nvme/fio_plugin/fio_plugin /home/vagrant/spdk/fio/fio /home/vagrant/spdk/examples/nvme/fio_plugin/example_config.fio
执行普通FIO，添加配置文件中size=10G
- sudo /home/vagrant/spdk/fio/fio /home/vagrant/spdk/examples/nvme/fio_plugin/example_config.fio
更多参见https://github.com/spdk/spdk/tree/master/examples/nvme/fio_plugin
执行示例代码（perf****）
参见http://www.spdk.io/doc/nvme.html
注意执行时需要加上sudo确保root权限执行
[vagrant@localhost spdk]$ sudo ./examples/nvme/perf/perf -q 128 -s 4096 -w randread -r 'trtype:PCIe traddr:0000:00:0e.0' -t 300
Starting SPDK v18.10-pre / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --legacy-mem --file-prefix=spdk_pid4352 ]
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/spdk_pid4352/mp_socket
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:00:0e.0 on NUMA socket 0
EAL: probe driver: 80ee:4e56 spdk_nvme
Attaching to NVMe Controller at 0000:00:0e.0
Attached to NVMe Controller at 0000:00:0e.0 [80ee:4e56]
Associating ORCL-VBOX-NVME-VER12 (VB1234-56789 ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
ORCL-VBOX-NVME-VER12 (VB1234-56789 ) from core 0: 62271.46 243.25 2055.47 982.67 14446.51
========================================================
Total : 62271.46 243.25 2055.47 982.67 14446.51