[SPDK] VM crashes while testing SPDK hotplug

Chang, Cunyin cunyin.chang at intel.com
Tue May 9 17:28:36 PDT 2017


Hi Isaac,

Daniel is right, the 3.10 is too old, I have tested hotplug on centos 7 for some times, it will cause system crash from time to time,
so just use the kernel Daniel mentioned.

Thanks,
Cunyin

From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of Verkamp, Daniel
Sent: Wednesday, May 10, 2017 6:33 AM
To: Storage Performance Development Kit <spdk at lists.01.org>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug

Hi Isaac,

Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5.  I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels).

Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue?

Thanks,
-- Daniel

From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Tuesday, May 9, 2017 2:11 PM
To: spdk at lists.01.org<mailto:spdk at lists.01.org>
Subject: [SPDK] VM crashes while testing SPDK hotplug

I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.

1.   VM creation is as follows

IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso

[root at host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root at host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso

qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
                -hda $IMAGE \
                -net nic,model=virtio \
                -net bridge,br=br1 \
                -m ${MEM} \
                -pidfile "/tmp/qemu_pid2.txt" \
                --enable-kvm \
                -cpu host \
                -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
                -mon chardev=mon0,mode=readline \
                -cdrom $ISO

2.    Without  running the SPDK  ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/

         echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
         echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
         echo  "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none"  | nc localhost 4449
         echo  "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none"  | nc localhost 4449

         echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0"  |nc localhost 4449
         echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
        echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2"  |nc localhost 4449
         echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449

        Also, commands to delete the devices work without crashing the VM
        echo "device_del nvme0" | nc localhost 4449
        echo "device_del nvme1" | nc localhost 4449
        echo "device_del nvme2" | nc localhost 4449
        echo "device_del nvme3" | nc localhost 4449

3.   However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.

1.   VM creation is as follows

IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso

[root at host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root at host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso

qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
                -hda $IMAGE \
                -net nic,model=virtio \
                -net bridge,br=br1 \
                -m ${MEM} \
                -pidfile "/tmp/qemu_pid2.txt" \
                --enable-kvm \
                -cpu host \
                -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
                -mon chardev=mon0,mode=readline \
                -cdrom $ISO

2.    Without  running the SPDK  ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/

         echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
         echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
         echo  "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none"  | nc localhost 4449
         echo  "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none"  | nc localhost 4449

         echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0"  |nc localhost 4449
         echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
        echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2"  |nc localhost 4449
         echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449

        Also, commands to delete the devices work without crashing the VM
        echo "device_del nvme0" | nc localhost 4449
        echo "device_del nvme1" | nc localhost 4449
        echo "device_del nvme2" | nc localhost 4449
        echo "device_del nvme3" | nc localhost 4449

3.   However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.

Isaac











-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.01.org/pipermail/spdk/attachments/20170510/34a4883a/attachment.html>


More information about the SPDK mailing list