FYI, we noticed the following commit: https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git linux-4.8.y-rt-rebase commit 43251cb83c44422010d61bc499ec98556a56b286 ("crypto: Convert crypto notifier chain to SRCU") in testcase: hackbench with following parameters: nr_threads: 1600% mode: process ipc: socket Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. on test machine: qemu-system-x86_64 -enable-kvm -cpu qemu64,+ssse3 -smp 4 -m 4G caused below changes: +-------------------------------------------------------------------------------+------------+------------+ | | 9cfed60b42 | 43251cb83c | +-------------------------------------------------------------------------------+------------+------------+ | boot_successes | 0 | 0 | | boot_failures | 8 | 11 | | BUG:kernel_hang_in_boot_stage | 8 | | | BUG:sleeping_function_called_from_invalid_context_at_kernel/locking/rtmutex.c | 0 | 9 | | calltrace:smpboot_thread_fn | 0 | 9 | | BUG:kernel_reboot-without-warning_in_test_stage | 0 | 2 | +-------------------------------------------------------------------------------+------------+------------+ Mounting Configuration File System... Starting udev Kernel Device Manager... Starting Preprocess NFS configuration... [ 18.441839] BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:931 [ 18.441841] in_atomic(): 1, irqs_disabled(): 0, pid: 27, name: ksoftirqd/2 [ 18.441842] 4 locks held by ksoftirqd/2/27: [ 18.441852] #0: (&per_cpu(local_softirq_locks[i], __cpu).lock){......}, at: [<ffffffff8107d9a3>] do_current_softirqs+0x99/0x267 [ 18.441858] #1: (rcu_callback){......}, at: [<ffffffff810c3bf4>] rcu_process_callbacks+0x27f/0x539 [ 18.441864] #2: (css_set_lock){......}, at: [<ffffffff810dd1c8>] put_css_set+0x2c/0x43 [ 18.441867] #3: (rcu_read_lock_sched){......}, at: [<ffffffff810db36f>] put_css_set_locked+0x1b3/0x259 [ 18.441870] CPU: 2 PID: 27 Comm: ksoftirqd/2 Not tainted 4.8.0-00263-g43251cb #1 [ 18.441871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 18.441875] 0000000000000000 ffff8801381b3c10 ffffffff813ff358 ffff880138196000 [ 18.441877] 0000000000000008 ffff8801381b3c28 ffffffff8109cfcb ffff88013b20e480 [ 18.441879] ffff8801381b3c40 ffffffff81cb0f36 000000000000e480 ffff8801381b3c78 [ 18.441880] Call Trace: [ 18.441884] [<ffffffff813ff358>] dump_stack+0x61/0x7e [ 18.441886] [<ffffffff8109cfcb>] ___might_sleep+0x113/0x116 [ 18.441891] [<ffffffff81cb0f36>] rt_spin_lock+0x1b/0x52 [ 18.441893] [<ffffffff8108cc22>] queue_work_on+0x54/0x148 [ 18.441895] [<ffffffff810dc25a>] css_release+0x7c/0x81 [ 18.441896] [<ffffffff810db404>] put_css_set_locked+0x248/0x259 [ 18.441898] [<ffffffff810dd1d0>] put_css_set+0x34/0x43 [ 18.441901] [<ffffffff810e1f78>] cgroup_free+0x63/0x6b [ 18.441905] [<ffffffff81077396>] __put_task_struct_cb+0x6d/0xe9 [ 18.441907] [<ffffffff810c3d83>] rcu_process_callbacks+0x40e/0x539 [ 18.441909] [<ffffffff81077329>] ? free_task+0xad/0xad [ 18.441911] [<ffffffff8107da59>] do_current_softirqs+0x14f/0x267 [ 18.441913] [<ffffffff8107dc18>] run_ksoftirqd+0x20/0x51 [ 18.441916] [<ffffffff81095882>] smpboot_thread_fn+0x1c6/0x1dd [ 18.441918] [<ffffffff810956bc>] ? sort_range+0x1d/0x1d [ 18.441920] [<ffffffff810929f5>] kthread+0xf6/0xfe [ 18.441923] [<ffffffff81caf271>] ? wait_for_common+0xf8/0x12c [ 18.441925] [<ffffffff81cb18df>] ret_from_fork+0x1f/0x40 [ 18.441927] [<ffffffff810928ff>] ? __kthread_parkme+0x7e/0x7e Starting Flush Journal to Persistent Storage... [ 19.549959] BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:931 [ 19.549961] in_atomic(): 1, irqs_disabled(): 0, pid: 19, name: ksoftirqd/1 [ 19.550031] 4 locks held by ksoftirqd/1/19: [ 19.550041] #0: (&per_cpu(local_softirq_locks[i], __cpu).lock){......}, at: [<ffffffff8107d9a3>] do_current_softirqs+0x99/0x267 [ 19.550046] #1: (rcu_callback){......}, at: [<ffffffff810c3bf4>] rcu_process_callbacks+0x27f/0x539 [ 19.550052] #2: (css_set_lock){......}, at: [<ffffffff810dd1c8>] put_css_set+0x2c/0x43 [ 19.550055] #3: (rcu_read_lock_sched){......}, at: [<ffffffff810db36f>] put_css_set_locked+0x1b3/0x259 [ 19.550058] CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 4.8.0-00263-g43251cb #1 [ 19.550059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 19.550063] 0000000000000000 ffff88013817bc10 ffffffff813ff358 ffff880138169800 [ 19.550065] 0000000000000008 ffff88013817bc28 ffffffff8109cfcb ffff88013b00e480 [ 19.550067] ffff88013817bc40 ffffffff81cb0f36 000000000000e480 ffff88013817bc78 [ 19.550068] Call Trace: [ 19.550072] [<ffffffff813ff358>] dump_stack+0x61/0x7e [ 19.550075] [<ffffffff8109cfcb>] ___might_sleep+0x113/0x116 [ 19.550080] [<ffffffff81cb0f36>] rt_spin_lock+0x1b/0x52 [ 19.550081] [<ffffffff8108cc22>] queue_work_on+0x54/0x148 [ 19.550083] [<ffffffff810dc25a>] css_release+0x7c/0x81 [ 19.550085] [<ffffffff810db404>] put_css_set_locked+0x248/0x259 [ 19.550087] [<ffffffff810dd1d0>] put_css_set+0x34/0x43 [ 19.550089] [<ffffffff810e1f78>] cgroup_free+0x63/0x6b [ 19.550093] [<ffffffff81077396>] __put_task_struct_cb+0x6d/0xe9 [ 19.550095] [<ffffffff810c3d83>] rcu_process_callbacks+0x40e/0x539 [ 19.550097] [<ffffffff81077329>] ? free_task+0xad/0xad [ 19.550099] [<ffffffff8107da59>] do_current_softirqs+0x14f/0x267 [ 19.550101] [<ffffffff8107dc18>] run_ksoftirqd+0x20/0x51 [ 19.550104] [<ffffffff81095882>] smpboot_thread_fn+0x1c6/0x1dd [ 19.550106] [<ffffffff810956bc>] ? sort_range+0x1d/0x1d [ 19.550109] [<ffffffff810929f5>] kthread+0xf6/0xfe [ 19.550111] [<ffffffff81caf271>] ? wait_for_common+0xf8/0x12c [ 19.550114] [<ffffffff81cb18df>] ret_from_fork+0x1f/0x40 [ 19.550116] [<ffffffff810928ff>] ? __kthread_parkme+0x7e/0x7e Starting Create Volatile Files and Directories... Starting Update UTMP about System Boot/Shutdown... Starting RPC bind portmap service... Starting Network Time Synchronization... [ 21.167101] BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:931 [ 21.167102] in_atomic(): 1, irqs_disabled(): 0, pid: 27, name: ksoftirqd/2 [ 21.167103] 4 locks held by ksoftirqd/2/27: [ 21.167112] #0: (&per_cpu(local_softirq_locks[i], __cpu).lock){......}, at: [<ffffffff8107d9a3>] do_current_softirqs+0x99/0x267 [ 21.167118] #1: (rcu_callback){......}, at: [<ffffffff810c3bf4>] rcu_process_callbacks+0x27f/0x539 [ 21.167122] #2: (css_set_lock){......}, at: [<ffffffff810dd1c8>] put_css_set+0x2c/0x43 [ 21.167125] #3: (rcu_read_lock_sched){......}, at: [<ffffffff810db36f>] put_css_set_locked+0x1b3/0x259 [ 21.167128] CPU: 2 PID: 27 Comm: ksoftirqd/2 Not tainted 4.8.0-00263-g43251cb #1 [ 21.167129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 21.167132] 0000000000000000 ffff8801381b3c10 ffffffff813ff358 ffff880138196000 [ 21.167134] 0000000000000008 ffff8801381b3c28 ffffffff8109cfcb ffff88013b20e480 [ 21.167136] ffff8801381b3c40 ffffffff81cb0f36 000000000000e480 ffff8801381b3c78 [ 21.167136] Call Trace: [ 21.167141] [<ffffffff813ff358>] dump_stack+0x61/0x7e [ 21.167143] [<ffffffff8109cfcb>] ___might_sleep+0x113/0x116 [ 21.167147] [<ffffffff81cb0f36>] rt_spin_lock+0x1b/0x52 [ 21.167149] [<ffffffff8108cc22>] queue_work_on+0x54/0x148 [ 21.167150] [<ffffffff810dc25a>] css_release+0x7c/0x81 [ 21.167152] [<ffffffff810db404>] put_css_set_locked+0x248/0x259 [ 21.167154] [<ffffffff810dd1d0>] put_css_set+0x34/0x43 [ 21.167155] [<ffffffff810e1f78>] cgroup_free+0x63/0x6b [ 21.167159] [<ffffffff81077396>] __put_task_struct_cb+0x6d/0xe9 [ 21.167161] [<ffffffff810c3d83>] rcu_process_callbacks+0x40e/0x539 [ 21.167162] [<ffffffff81077329>] ? free_task+0xad/0xad [ 21.167164] [<ffffffff8107da59>] do_current_softirqs+0x14f/0x267 [ 21.167165] [<ffffffff8107dc18>] run_ksoftirqd+0x20/0x51 [ 21.167168] [<ffffffff81095882>] smpboot_thread_fn+0x1c6/0x1dd [ 21.167170] [<ffffffff810956bc>] ? sort_range+0x1d/0x1d [ 21.167172] [<ffffffff810929f5>] kthread+0xf6/0xfe [ 21.167174] [<ffffffff81caf271>] ? wait_for_common+0xf8/0x12c [ 21.167176] [<ffffffff81cb18df>] ret_from_fork+0x1f/0x40 [ 21.167177] [<ffffffff810928ff>] ? __kthread_parkme+0x7e/0x7e Elapsed time: 40 To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Thanks, Xiaolong