[SPDK] VM crashes while testing SPDK hotplug

Isaac Otsiabah IOtsiabah at us.fujitsu.com
Mon May 15 17:08:35 PDT 2017


Daniel, the attached hotplug.sh is modified to skip the VM creation and spdk copying sections. I ran it twice, and it oops the second time with the script output redirected to /tmp/hotplug.out4 file. The oops trace is similar to the one sent this morning.

[root at host1k]# ./test/lib/nvme/hotplug.sh 2>&1 |tee /tmp/hotplug.out3

[root at host1]# ./test/lib/nvme/hotplug.sh 2>&1 |tee /tmp/hotplug.out4

Isaac
From: Isaac Otsiabah
Sent: Monday, May 15, 2017 1:08 PM
To: Storage Performance Development Kit <spdk at lists.01.org>
Cc: Isaac Otsiabah <IOtsiabah at us.fujitsu.com>; Paul Von-Stamwitz <PVonStamwitz at us.fujitsu.com>; Edward Yang <eyang at us.fujitsu.com>
Subject: RE: VM crashes while testing SPDK hotplug

Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm   which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9.

[root at host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64  -version
QEMU emulator version 2.9.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag  "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is  is different. The qemu-system-86_64 man page on -netdev flag is as follows:
-netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr]
         [,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr]
         [,restrict=on|off][,hostname=host][,dhcpstart=addr]
         [,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir]
         [,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]]
                configure a user mode network backend with ID 'str',
                its DHCP server and optional services

It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \

>From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command.

So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times?

Isaac
From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of Verkamp, Daniel
Sent: Tuesday, May 09, 2017 3:33 PM
To: Storage Performance Development Kit <spdk at lists.01.org<mailto:spdk at lists.01.org>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug

Hi Isaac,

Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5.  I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels).

Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue?

Thanks,
-- Daniel

From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Tuesday, May 9, 2017 2:11 PM
To: spdk at lists.01.org<mailto:spdk at lists.01.org>
Subject: [SPDK] VM crashes while testing SPDK hotplug

I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.

1.   VM creation is as follows

IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso

[root at host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root at host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso

qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
                -hda $IMAGE \
                -net nic,model=virtio \
                -net bridge,br=br1 \
                -m ${MEM} \
                -pidfile "/tmp/qemu_pid2.txt" \
                --enable-kvm \
                -cpu host \
                -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
                -mon chardev=mon0,mode=readline \
                -cdrom $ISO

2.    Without  running the SPDK  ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/

         echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
         echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
         echo  "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none"  | nc localhost 4449
         echo  "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none"  | nc localhost 4449

         echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0"  |nc localhost 4449
         echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
        echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2"  |nc localhost 4449
         echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449

        Also, commands to delete the devices work without crashing the VM
        echo "device_del nvme0" | nc localhost 4449
        echo "device_del nvme1" | nc localhost 4449
        echo "device_del nvme2" | nc localhost 4449
        echo "device_del nvme3" | nc localhost 4449

3.   However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.

1.   VM creation is as follows

IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso

[root at host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root at host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso

qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
                -hda $IMAGE \
                -net nic,model=virtio \
                -net bridge,br=br1 \
                -m ${MEM} \
                -pidfile "/tmp/qemu_pid2.txt" \
                --enable-kvm \
                -cpu host \
                -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
                -mon chardev=mon0,mode=readline \
                -cdrom $ISO

2.    Without  running the SPDK  ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/

         echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
         echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
         echo  "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none"  | nc localhost 4449
         echo  "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none"  | nc localhost 4449

         echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0"  |nc localhost 4449
         echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
        echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2"  |nc localhost 4449
         echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449

        Also, commands to delete the devices work without crashing the VM
        echo "device_del nvme0" | nc localhost 4449
        echo "device_del nvme1" | nc localhost 4449
        echo "device_del nvme2" | nc localhost 4449
        echo "device_del nvme3" | nc localhost 4449

3.   However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.

Isaac











-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.01.org/pipermail/spdk/attachments/20170515/b4372ce2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hotplug.sh
Type: application/octet-stream
Size: 3147 bytes
Desc: hotplug.sh
URL: <http://lists.01.org/pipermail/spdk/attachments/20170515/b4372ce2/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hotplug.out3
Type: application/octet-stream
Size: 19469 bytes
Desc: hotplug.out3
URL: <http://lists.01.org/pipermail/spdk/attachments/20170515/b4372ce2/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hotplug.out4
Type: application/octet-stream
Size: 14564 bytes
Desc: hotplug.out4
URL: <http://lists.01.org/pipermail/spdk/attachments/20170515/b4372ce2/attachment-0002.obj>


More information about the SPDK mailing list