12:09 p.m.
FYI, we noticed the following commit (built with gcc-7):
commit: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into
__blk_release_queue")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: trinity
with following parameters:
runtime: 300s
test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 2G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+-------------------------------------------------+------------+------------+
| | 31cb1d64da | 47cdee29ef |
+-------------------------------------------------+------------+------------+
| boot_successes | 3 | 0 |
| boot_failures | 13 | 8 |
| BUG:kernel_reboot-without-warning_in_test_stage | 13 | |
| BUG:kernel_NULL_pointer_dereference,address | 0 | 8 |
| Oops:#[##] | 0 | 8 |
| RIP:blk_mq_free_rqs | 0 | 8 |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 8 |
+-------------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
[ 6.560544] BUG: kernel NULL pointer dereference, address: 0000000000000020
[ 6.561658] #PF: supervisor read access in kernel mode
[ 6.562495] #PF: error_code(0x0000) - not-present page
[ 6.563277] PGD 0 P4D 0
[ 6.563277] Oops: 0000 [#1] PTI
[ 6.563277] CPU: 0 PID: 147 Comm: kworker/0:2 Tainted: G T
5.2.0-rc1-00387-g47cdee29 #1
[ 6.563277] Workqueue: events __blk_release_queue
[ 6.563277] RIP: 0010:blk_mq_free_rqs+0x2c/0xaf
[ 6.563277] Code: 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fd 53 48 89 f3 41
89 d7 48 83 bb 90 00 00 00 00 74 0e 49 8b 45 38 45 31 e4 <48> 83 78 20 00 75 3b 4c
8d ab a0 00 00 00 eb 39 48 8b 83 98 00 00
[ 6.563277] RSP: 0000:ffff888071f6bdb0 EFLAGS: 00010246
[ 6.563277] RAX: 0000000000000000 RBX: ffff88807f1a76b8 RCX: 000000000011000b
[ 6.563277] RDX: 0000000000000000 RSI: ffff88807f1a76b8 RDI: ffff88807f0012c8
[ 6.563277] RBP: ffff888071f6bdd8 R08: ffff888071f6bf48 R09: 0000000000000000
[ 6.563277] R10: ffff888071f6bdd8 R11: 0000000000000030 R12: 0000000000000000
[ 6.563277] R13: ffff88807f0012c8 R14: ffff88807f0012c8 R15: 0000000000000000
[ 6.563277] FS: 0000000000000000(0000) GS:ffffffff8263d000(0000)
knlGS:0000000000000000
[ 6.563277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6.563277] CR2: 0000000000000020 CR3: 000000000262a000 CR4: 00000000000406b0
[ 6.563277] Call Trace:
[ 6.563277] blk_mq_sched_tags_teardown+0x46/0x6a
[ 6.563277] blk_mq_exit_sched+0x6f/0x81
[ 6.563277] elevator_exit+0x38/0x50
[ 6.563277] __blk_release_queue+0x6b/0xdc
[ 6.563277] process_one_work+0x174/0x26a
[ 6.563277] ? rescuer_thread+0x262/0x262
[ 6.563277] worker_thread+0x1a8/0x267
[ 6.563277] ? rescuer_thread+0x262/0x262
[ 6.563277] kthread+0xff/0x107
[ 6.563277] ? kthread_create_worker_on_cpu+0x5f/0x5f
[ 6.563277] ret_from_fork+0x24/0x30
[ 6.563277] Modules linked in:
[ 6.563277] CR2: 0000000000000020
[ 6.563277] _warn_unseeded_randomness: 1 callbacks suppressed
[ 6.563277] random: get_random_bytes called from init_oops_id+0x26/0x36 with
crng_init=0
[ 6.563277] ---[ end trace 1855f0c417feab1e ]---
To reproduce:
# build kernel
cd linux
cp config-5.2.0-rc1-00387-g47cdee29 .config
make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig
make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 prepare
make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 modules_prepare
make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 SHELL=/bin/bash
make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 bzImage
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email
Thanks,
Rong Chen
2:03 p.m.
Hi Rong Chen,
Thanks for your test & report!
On Tue, Jun 04, 2019 at 10:09:56AM +0800, kernel test robot wrote:
FYI, we noticed the following commit (built with gcc-7):
commit: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into
__blk_release_queue")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: trinity
with following parameters:
runtime: 300s
test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 2G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+-------------------------------------------------+------------+------------+
| | 31cb1d64da | 47cdee29ef |
+-------------------------------------------------+------------+------------+
| boot_successes | 3 | 0 |
| boot_failures | 13 | 8 |
| BUG:kernel_reboot-without-warning_in_test_stage | 13 | |
| BUG:kernel_NULL_pointer_dereference,address | 0 | 8 |
| Oops:#[##] | 0 | 8 |
| RIP:blk_mq_free_rqs | 0 | 8 |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 8 |
+-------------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
[ 6.560544] BUG: kernel NULL pointer dereference, address: 0000000000000020
[ 6.561658] #PF: supervisor read access in kernel mode
[ 6.562495] #PF: error_code(0x0000) - not-present page
[ 6.563277] PGD 0 P4D 0
[ 6.563277] Oops: 0000 [#1] PTI
[ 6.563277] CPU: 0 PID: 147 Comm: kworker/0:2 Tainted: G T
5.2.0-rc1-00387-g47cdee29 #1
[ 6.563277] Workqueue: events __blk_release_queue
[ 6.563277] RIP: 0010:blk_mq_free_rqs+0x2c/0xaf
Looks there is race between removing queue and switching elevator, and
which should be done by Trinity.
I guess that commit 47cdee29ef9d94e485eb08f962c74943023a5271 just
changes the timing and makes it easy to trigger.
Please test the following patch and see if difference can be made.
If the patch can't fix the issue, please enable KASAN and reproduce,
then more useful log may be got.
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 75b5281cc577..400a2102a4e4 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -848,11 +848,13 @@ static void blk_exit_queue(struct request_queue *q)
* perform I/O scheduler exit before disassociating from the block
* cgroup controller.
*/
+ mutex_lock(&q->sysfs_lock);
if (q->elevator) {
ioc_clear_queue(q);
elevator_exit(q, q->elevator);
q->elevator = NULL;
}
+ mutex_unlock(&q->sysfs_lock);
/*
* Remove all references to @q from the block cgroup controller before
Thanks,
Ming
7:06 p.m.
Hi,
On 6/4/19 12:03 PM, Ming Lei wrote:
Hi Rong Chen,
Thanks for your test & report!
On Tue, Jun 04, 2019 at 10:09:56AM +0800, kernel test robot wrote:
> FYI, we noticed the following commit (built with gcc-7):
>
> commit: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue
into __blk_release_queue")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: trinity
> with following parameters:
>
> runtime: 300s
>
> test-description: Trinity is a linux system call fuzz tester.
> test-url: http://codemonkey.org.uk/projects/trinity/
>
>
> on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 2G
>
> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>
>
> +-------------------------------------------------+------------+------------+
> | | 31cb1d64da | 47cdee29ef |
> +-------------------------------------------------+------------+------------+
> | boot_successes | 3 | 0 |
> | boot_failures | 13 | 8 |
> | BUG:kernel_reboot-without-warning_in_test_stage | 13 | |
> | BUG:kernel_NULL_pointer_dereference,address | 0 | 8 |
> | Oops:#[##] | 0 | 8 |
> | RIP:blk_mq_free_rqs | 0 | 8 |
> | Kernel_panic-not_syncing:Fatal_exception | 0 | 8 |
> +-------------------------------------------------+------------+------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <rong.a.chen(a)intel.com>
>
>
> [ 6.560544] BUG: kernel NULL pointer dereference, address: 0000000000000020
> [ 6.561658] #PF: supervisor read access in kernel mode
> [ 6.562495] #PF: error_code(0x0000) - not-present page
> [ 6.563277] PGD 0 P4D 0
> [ 6.563277] Oops: 0000 [#1] PTI
> [ 6.563277] CPU: 0 PID: 147 Comm: kworker/0:2 Tainted: G T
5.2.0-rc1-00387-g47cdee29 #1
> [ 6.563277] Workqueue: events __blk_release_queue
> [ 6.563277] RIP: 0010:blk_mq_free_rqs+0x2c/0xaf
Looks there is race between removing queue and switching elevator, and
which should be done by Trinity.
I guess that commit 47cdee29ef9d94e485eb08f962c74943023a5271 just
changes the timing and makes it easy to trigger.
Please test the following patch and see if difference can be made.
If the patch can't fix the issue, please enable KASAN and reproduce,
then more useful log may be got.
The patch doesn't work, Attached please find the dmesg file with KASAN
enabled.
Best Regards,
Rong Chen
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 75b5281cc577..400a2102a4e4 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -848,11 +848,13 @@ static void blk_exit_queue(struct request_queue *q)
* perform I/O scheduler exit before disassociating from the block
* cgroup controller.
*/
+ mutex_lock(&q->sysfs_lock);
if (q->elevator) {
ioc_clear_queue(q);
elevator_exit(q, q->elevator);
q->elevator = NULL;
}
+ mutex_unlock(&q->sysfs_lock);
/*
* Remove all references to @q from the block cgroup controller before
Thanks,
Ming
8:43 p.m.
On Tue, Jun 04, 2019 at 05:06:44PM +0800, Rong Chen wrote:
Hi,
On 6/4/19 12:03 PM, Ming Lei wrote:
> Hi Rong Chen,
>
> Thanks for your test & report!
>
> On Tue, Jun 04, 2019 at 10:09:56AM +0800, kernel test robot wrote:
> > FYI, we noticed the following commit (built with gcc-7):
> >
> > commit: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move
blk_exit_queue into __blk_release_queue")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >
> > in testcase: trinity
> > with following parameters:
> >
> > runtime: 300s
> >
> > test-description: Trinity is a linux system call fuzz tester.
> > test-url: http://codemonkey.org.uk/projects/trinity/
> >
> >
> > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 2G
> >
> > caused below changes (please refer to attached dmesg/kmsg for entire
log/backtrace):
> >
> >
> > +-------------------------------------------------+------------+------------+
> > | | 31cb1d64da | 47cdee29ef |
> > +-------------------------------------------------+------------+------------+
> > | boot_successes | 3 | 0 |
> > | boot_failures | 13 | 8 |
> > | BUG:kernel_reboot-without-warning_in_test_stage | 13 | |
> > | BUG:kernel_NULL_pointer_dereference,address | 0 | 8 |
> > | Oops:#[##] | 0 | 8 |
> > | RIP:blk_mq_free_rqs | 0 | 8 |
> > | Kernel_panic-not_syncing:Fatal_exception | 0 | 8 |
> > +-------------------------------------------------+------------+------------+
> >
> >
> > If you fix the issue, kindly add following tag
> > Reported-by: kernel test robot <rong.a.chen(a)intel.com>
> >
> >
> > [ 6.560544] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > [ 6.561658] #PF: supervisor read access in kernel mode
> > [ 6.562495] #PF: error_code(0x0000) - not-present page
> > [ 6.563277] PGD 0 P4D 0
> > [ 6.563277] Oops: 0000 [#1] PTI
> > [ 6.563277] CPU: 0 PID: 147 Comm: kworker/0:2 Tainted: G T
5.2.0-rc1-00387-g47cdee29 #1
> > [ 6.563277] Workqueue: events __blk_release_queue
> > [ 6.563277] RIP: 0010:blk_mq_free_rqs+0x2c/0xaf
>
> Looks there is race between removing queue and switching elevator, and
> which should be done by Trinity.
>
> I guess that commit 47cdee29ef9d94e485eb08f962c74943023a5271 just
> changes the timing and makes it easy to trigger.
>
> Please test the following patch and see if difference can be made.
> If the patch can't fix the issue, please enable KASAN and reproduce,
> then more useful log may be got.
The patch doesn't work, Attached please find the dmesg file with KASAN
enabled.
Thanks for your test.
I think I can understand the issue now, it is because blk_mq_free_rqs()
needs tag_set, however tag_set may have been freed.
In theory, we don't need tagset for freeing scheduler tags which is
per-request-queue, not like driver tags.
However, the big trouble is that .exit_request() needs tagset, and this
one is a generic issue, not limited to ide.
Give me a little time, I will investigate and see if good solution can be
figured out. Otherwise, we may have to revert that commit.
Thanks,
Ming
11:11 p.m.
On Tue, Jun 04, 2019 at 06:43:27PM +0800, Ming Lei wrote:
On Tue, Jun 04, 2019 at 05:06:44PM +0800, Rong Chen wrote:
> Hi,
>
> On 6/4/19 12:03 PM, Ming Lei wrote:
> > Hi Rong Chen,
> >
> > Thanks for your test & report!
> >
> > On Tue, Jun 04, 2019 at 10:09:56AM +0800, kernel test robot wrote:
> > > FYI, we noticed the following commit (built with gcc-7):
> > >
> > > commit: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move
blk_exit_queue into __blk_release_queue")
> > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > >
> > > in testcase: trinity
> > > with following parameters:
> > >
> > > runtime: 300s
> > >
> > > test-description: Trinity is a linux system call fuzz tester.
> > > test-url: http://codemonkey.org.uk/projects/trinity/
> > >
> > >
> > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m
2G
> > >
> > > caused below changes (please refer to attached dmesg/kmsg for entire
log/backtrace):
> > >
> > >
> > >
+-------------------------------------------------+------------+------------+
> > > | | 31cb1d64da |
47cdee29ef |
> > >
+-------------------------------------------------+------------+------------+
> > > | boot_successes | 3 | 0
|
> > > | boot_failures | 13 | 8
|
> > > | BUG:kernel_reboot-without-warning_in_test_stage | 13 |
|
> > > | BUG:kernel_NULL_pointer_dereference,address | 0 | 8
|
> > > | Oops:#[##] | 0 | 8
|
> > > | RIP:blk_mq_free_rqs | 0 | 8
|
> > > | Kernel_panic-not_syncing:Fatal_exception | 0 | 8
|
> > >
+-------------------------------------------------+------------+------------+
> > >
> > >
> > > If you fix the issue, kindly add following tag
> > > Reported-by: kernel test robot <rong.a.chen(a)intel.com>
> > >
> > >
> > > [ 6.560544] BUG: kernel NULL pointer dereference, address:
0000000000000020
> > > [ 6.561658] #PF: supervisor read access in kernel mode
> > > [ 6.562495] #PF: error_code(0x0000) - not-present page
> > > [ 6.563277] PGD 0 P4D 0
> > > [ 6.563277] Oops: 0000 [#1] PTI
> > > [ 6.563277] CPU: 0 PID: 147 Comm: kworker/0:2 Tainted: G
T 5.2.0-rc1-00387-g47cdee29 #1
> > > [ 6.563277] Workqueue: events __blk_release_queue
> > > [ 6.563277] RIP: 0010:blk_mq_free_rqs+0x2c/0xaf
> >
> > Looks there is race between removing queue and switching elevator, and
> > which should be done by Trinity.
> >
> > I guess that commit 47cdee29ef9d94e485eb08f962c74943023a5271 just
> > changes the timing and makes it easy to trigger.
> >
> > Please test the following patch and see if difference can be made.
> > If the patch can't fix the issue, please enable KASAN and reproduce,
> > then more useful log may be got.
>
> The patch doesn't work, Attached please find the dmesg file with KASAN
> enabled.
Thanks for your test.
I think I can understand the issue now, it is because blk_mq_free_rqs()
needs tag_set, however tag_set may have been freed.
In theory, we don't need tagset for freeing scheduler tags which is
per-request-queue, not like driver tags.
However, the big trouble is that .exit_request() needs tagset, and this
one is a generic issue, not limited to ide.
Give me a little time, I will investigate and see if good solution can be
figured out. Otherwise, we may have to revert that commit.
Patch has been posted out:
https://lore.kernel.org/linux-block/20190604130802.17076-1-ming.lei@redha...
Thanks,
Ming
1202
days inactive
1202
days old
4 comments
3 participants
participants (3)
-
kernel test robot -
Ming Lei -
Rong Chen