FYI, we noticed the following commit: commit: 867786e506615777d96589532760026ee995e830 ("rcu: Use for_each_leaf_node_cpu() in force_qs_rnp()") https://github.com/fbq/linux.git leaf-node in testcase: boot on test machine: qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -m 2G caused below changes: +------------------------------------------------+------------+------------+ | | b024ae3068 | 867786e506 | +------------------------------------------------+------------+------------+ | boot_successes | 30 | 11 | | boot_failures | 0 | 21 | | BUG:kernel_hang_in_test_stage | 0 | 2 | | INFO:rcu_preempt_detected_stalls_on_CPUs/tasks | 0 | 19 | | calltrace:populate_rootfs | 0 | 19 | | calltrace:SyS_write | 0 | 5 | | calltrace:SyS_open | 0 | 2 | | calltrace:SyS_mkdir | 0 | 1 | | calltrace:SyS_newlstat | 0 | 1 | +------------------------------------------------+------------+------------+ [ 13.248262] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] [ 13.268999] PCI: CLS 0 bytes, default 64 [ 13.280821] Unpacking initramfs... [ 117.229392] INFO: rcu_preempt detected stalls on CPUs/tasks: [ 117.289959] All QSes seen, last rcu_preempt kthread activity 2 (4294948278-4294948276), jiffies_till_next_fqs=1, root ->qsmask 0x2 [ 117.312450] swapper/0 R running task 0 1 0 0x00080000 [ 117.331767] ffff880077403ee0 ffffffffa42f6494 ffffffffa5052540 ffff880077417940 [ 117.352989] ffff880077403f60 ffffffffa43214aa 0000000000000002 ffff880077403f18 [ 117.369357] ffff88007481ab40 ffffffffa437b7e6 ffffffffffffff66 0000000077403f28 [ 117.388704] Call Trace: [ 117.396733] <IRQ> [ 117.401563] [<ffffffffa42f6494>] sched_show_task+0xc9/0xce [ 117.417968] [<ffffffffa43214aa>] rcu_check_callbacks+0x633/0x7db [ 117.432872] [<ffffffffa437b7e6>] ? irq_trace+0x32/0x37 [ 117.445762] [<ffffffffa43271d1>] update_process_times+0x35/0x64 [ 117.460312] [<ffffffffa4338d32>] tick_nohz_handler+0xf6/0x15c [ 117.474251] [<ffffffffa423b2f0>] local_apic_timer_interrupt+0x4b/0x4e [ 117.487864] [<ffffffffa4ab0a64>] smp_apic_timer_interrupt+0x2e/0x3e [ 117.501506] [<ffffffffa4ab0001>] apic_timer_interrupt+0x91/0xa0 [ 117.510415] <EOI> [ 117.513338] [<ffffffffa471cbda>] ? simple_strtoull+0x61/0x61 [ 117.527140] [<ffffffffa51f33d4>] ? parse_header+0x37/0xe9 [ 117.540004] [<ffffffffa4402b99>] ? ___cache_free+0x245/0x2a9 [ 117.553745] [<ffffffffa4401d79>] ? compound_head+0x2b/0x3c [ 117.565148] [<ffffffffa437b7e6>] ? irq_trace+0x32/0x37 [ 117.578048] [<ffffffffa51f3593>] do_header+0x89/0x1cc [ 117.588672] [<ffffffffa51f2f82>] write_buffer+0x32/0x4a [ 117.597037] [<ffffffffa51f2fd4>] flush_buffer+0x3a/0xcc [ 117.606495] [<ffffffffa51f2e94>] ? initrd_load+0x4e/0x4e [ 117.616870] [<ffffffffa523290f>] __gunzip+0x38d/0x44c [ 117.627537] [<ffffffffa5232573>] ? bunzip2+0x4df/0x4df [ 117.635832] [<ffffffffa51f2f9a>] ? write_buffer+0x4a/0x4a [ 117.646073] [<ffffffffa52329ce>] ? __gunzip+0x44c/0x44c [ 117.655664] [<ffffffffa5232a18>] gunzip+0x4a/0x4c [ 117.666128] [<ffffffffa51f2e94>] ? initrd_load+0x4e/0x4e [ 117.678963] [<ffffffffa51f2f9a>] ? write_buffer+0x4a/0x4a [ 117.692345] [<ffffffffa51f3910>] unpack_to_rootfs+0x1da/0x339 [ 117.706340] [<ffffffffa51f2e94>] ? initrd_load+0x4e/0x4e [ 117.719276] [<ffffffffa51f3bc7>] ? maybe_link+0x158/0x158 [ 117.733437] [<ffffffffa51f18fd>] ? set_debug_rodata+0x20/0x20 [ 117.747078] [<ffffffffa51f3c3a>] populate_rootfs+0x73/0xdf [ 117.758969] [<ffffffffa51f3bc7>] ? maybe_link+0x158/0x158 [ 117.772448] [<ffffffffa420050a>] do_one_initcall+0xa7/0x18d [ 117.782438] [<ffffffffa51f18fd>] ? set_debug_rodata+0x20/0x20 [ 117.795624] [<ffffffffa51f2425>] kernel_init_freeable+0x1c5/0x270 [ 117.808457] [<ffffffffa4aa8065>] ? rest_init+0x8c/0x8c [ 117.820931] [<ffffffffa4aa8079>] kernel_init+0x14/0x14c [ 117.832116] [<ffffffffa4aa8065>] ? rest_init+0x8c/0x8c [ 117.844525] [<ffffffffa4aae92a>] ret_from_fork+0x2a/0x40 [ 140.516967] Freeing initrd memory: 140196K (ffff8800776e7000 - ffff88007ffd0000) [ 140.572471] Scanning for low memory corruption every 60 seconds [ 140.645128] AVX or AES-NI instructions are not detected. To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Ying Huang