Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue.git master commit f4a3e90ba5739cfd761b6befadae9728bd3641ed Author: Phil Sutter <phil(a)nwl.cc&gt; AuthorDate: Sat Aug 15 00:37:15 2015 +0200 Commit: David S. Miller <davem(a)davemloft.net&gt; CommitDate: Mon Aug 17 14:33:47 2015 -0700 rhashtable-test: extend to test concurrency After having tested insertion, lookup, table walk and removal, spawn a number of threads running operations on the same rhashtable. Each of them will: 1) insert it's own set of objects, 2) lookup every successfully inserted object and finally 3) remove objects in several rounds until all of them have been removed, making sure the remaining ones are still found after each round. This should put a good amount of load onto the system and due to synchronising thread startup via two semaphores also extensive concurrent table access. The default number of ten threads returned within half a second on my local VM with two cores. Running 200 threads took about four seconds. If slow systems suffer too much from this though, the default could be lowered or even set to zero so this extended test does not run at all by default. Signed-off-by: Phil Sutter <phil(a)nwl.cc&gt; Acked-by: Thomas Graf <tgraf(a)suug.ch&gt; Signed-off-by: David S. Miller <davem(a)davemloft.net&gt; +----------------------------------------------------+------------+------------+------------+ | | c1f066d4ee | f4a3e90ba5 | 6967aa466b | +----------------------------------------------------+------------+------------+------------+ | boot_successes | 1060 | 808 | 106 | | boot_failures | 1 | 102 | 21 | | INFO:possible_circular_locking_dependency_detected | 1 | | | | backtrace:vfs_readv | 1 | | | | backtrace:SyS_readv | 1 | | | | backtrace:blk_mq_sysfs_unregister | 1 | | | | backtrace:blk_mq_queue_reinit_notify | 1 | | | | backtrace:debug_hotplug_cpu | 1 | | | | backtrace:kernel_init_freeable | 1 | | | | EIP_is_at_lock_is_held | 0 | 63 | 16 | | Kernel_panic-not_syncing:softlockup:hung_tasks | 0 | 102 | 19 | | backtrace:threadfunc | 0 | 101 | 19 | | EIP_is_at_rcu_read_lock_held | 0 | 10 | 5 | | EIP_is_at_rcu_lockdep_current_cpu_online | 0 | 9 | | | EIP_is_at_thread_lookup_test | 0 | 11 | 1 | | EIP_is_at_lock_release | 0 | 3 | | | EIP_is_at_lockdep_rht_bucket_is_held | 0 | 3 | | | EIP_is_at_rcu_is_watching | 0 | 5 | | | backtrace:apic_timer_interrupt | 0 | 3 | | | EIP_is_at_lock_acquire | 0 | 12 | 2 | | EIP_is_at_jhash | 0 | 11 | | | EIP_is_at_raw_spin_lock_bh | 0 | 1 | | | EIP_is_at_debug_lockdep_rcu_enabled | 0 | 13 | | | EIP_is_at_lockdep_rht_mutex_is_held | 0 | 3 | 2 | | EIP_is_at_raw_spin_unlock_bh | 0 | 1 | | | EIP_is_at_do_raw_spin_lock | 0 | 1 | | | EIP_is_at__local_bh_enable_ip | 0 | 1 | | | EIP_is_at_threadfunc | 0 | 1 | | | IP-Config:Auto-configuration_of_network_failed | 0 | 0 | 2 | +----------------------------------------------------+------------+------------+------------+ [ 6.673473] Testing concurrent rhashtable access from 10 threads [ 32.066733] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [rhashtable_thra:103] [ 32.068432] irq event stamp: 1362768 [ 32.069045] hardirqs last enabled at (1362767): [<c2df1f47>] restore_all_notrace+0x0/0x18 [ 32.070037] hardirqs last disabled at (1362768): [<c2df29ab>] apic_timer_interrupt+0x2b/0x40 [ 32.070037] softirqs last enabled at (1362766): [<c11086ff>] __do_softirq+0x2af/0x380 [ 32.070037] softirqs last disabled at (1362761): [<c1072d5b>] do_softirq_own_stack+0x2b/0x40 [ 32.070037] CPU: 0 PID: 103 Comm: rhashtable_thra Not tainted 4.2.0-rc6-00979-gf4a3e90 #1 [ 32.070037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 32.070037] task: cb1acf80 ti: c0054000 task.ti: c0054000 [ 32.070037] EIP: 0060:[<c1188b82>] EFLAGS: 00000246 CPU: 0 [ 32.070037] EIP is at lock_is_held+0xb2/0xe0 [ 32.070037] EAX: 00000246 EBX: 00000246 ECX: 00000001 EDX: c385ee50 [ 32.070037] ESI: 00000001 EDI: 00000000 EBP: c0055ea8 ESP: c0055ea0 [ 32.070037] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 32.070037] CR0: 8005003b CR2: ffffffff CR3: 03cfa000 CR4: 000006b0 [ 32.070037] Stack: [ 32.083360] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [rhashtable_thra:109] [ 32.083362] irq event stamp: 1377938 [ 32.083372] hardirqs last enabled at (1377937): [<c2df1f47>] restore_all_notrace+0x0/0x18 [ 32.083376] hardirqs last disabled at (1377938): [<c2df29ab>] apic_timer_interrupt+0x2b/0x40 [ 32.083380] softirqs last enabled at (1377936): [<c11086ff>] __do_softirq+0x2af/0x380 [ 32.083384] softirqs last disabled at (1377933): [<c1072d5b>] do_softirq_own_stack+0x2b/0x40 [ 32.083388] CPU: 1 PID: 109 Comm: rhashtable_thra Not tainted 4.2.0-rc6-00979-gf4a3e90 #1 [ 32.083389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 32.083391] task: cb1a9a80 ti: ca914000 task.ti: ca914000 [ 32.083394] EIP: 0060:[<c173ff3e>] EFLAGS: 00000246 CPU: 1 [ 32.083399] EIP is at lockdep_rht_mutex_is_held+0xe/0x50 [ 32.083401] EAX: c46d3880 EBX: cb1f4000 ECX: 00000001 EDX: 00000001 [ 32.083402] ESI: 300003f1 EDI: 00000000 EBP: ca915eec ESP: ca915eb4 [ 32.083403] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 32.083405] CR0: 8005003b CR2: 00000000 CR3: 03cfa000 CR4: 000006b0 [ 32.083410] Stack: [ 32.083415] c17470a3 00000000 d35ec050 d35ec048 000001f8 300003f1 0004c780 000098f0 [ 32.083420] 000698f0 c46d3880 ca915ed4 00000000 d3899974 c0100000 ca915f34 c1748215 [ 32.083424] d35ec050 00000030 00000006 00000000 d35ec048 00061aa0 0000c354 d3899970 [ 32.083424] Call Trace: [ 32.083429] [<c17470a3>] ? thread_lookup_test+0x683/0x7f0 [ 32.083432] [<c1748215>] threadfunc+0x1005/0x1550 [ 32.083434] [<c1747210>] ? thread_lookup_test+0x7f0/0x7f0 [ 32.083436] [<c1747210>] ? thread_lookup_test+0x7f0/0x7f0 [ 32.083440] [<c113cd37>] kthread+0x127/0x170 [ 32.083443] [<c2df1d81>] ret_from_kernel_thread+0x21/0x30 [ 32.083445] [<c113cc10>] ? __kthread_unpark+0xd0/0xd0 [ 32.083479] Code: 0d 6b 01 83 05 d8 fb 6c c4 01 83 15 dc fb 6c c4 00 b8 ef ff ff ff 5b 5e 5d c3 8d 76 00 83 05 68 e0 6c c4 01 83 15 6c e0 6c c4 00 <8b> 15 d4 a5 8d c3 85 d2 75 08 b8 01 00 00 00 c3 66 90 55 89 e5 [ 32.083536] Kernel panic - not syncing: softlockup: hung tasks [ 32.083540] CPU: 1 PID: 109 Comm: rhashtable_thra Tainted: G L 4.2.0-rc6-00979-gf4a3e90 #1 [ 32.083541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 32.083547] 00000000 ca915db4 c2dc56f2 00000016 ca915dcc c2dc082b ee6b2800 00000016 [ 32.083553] ee6b2800 00000000 ca915df0 c1217583 c359d4c6 ca915e78 00000000 00000007 [ 32.083559] d1381800 d13813a0 c12171d0 ca915e24 c11cf4b3 00000000 7850e875 00000007 [ 32.083559] Call Trace: [ 32.083563] [<c2dc56f2>] dump_stack+0xb8/0x108 [ 32.083565] [<c2dc082b>] panic+0x12b/0x3e6 [ 32.083570] [<c1217583>] watchdog_timer_fn+0x3b3/0x430 [ 32.083572] [<c12171d0>] ? watchdog+0x70/0x70 [ 32.083578] [<c11cf4b3>] __hrtimer_run_queues+0x123/0x2e0 [ 32.083581] [<c11d07a4>] hrtimer_interrupt+0xc4/0x2c0 [ 32.083584] [<c1186ef7>] ? trace_hardirqs_off+0x27/0x40 [ 32.083587] [<c11c7a3a>] ? rcu_irq_enter+0x14a/0x160 [ 32.083590] [<c10c8117>] local_apic_timer_interrupt+0x37/0x90 [ 32.083593] [<c10c8d28>] smp_apic_timer_interrupt+0x58/0x90 [ 32.083596] [<c2df29b2>] apic_timer_interrupt+0x32/0x40 [ 32.083599] [<c173ff3e>] ? lockdep_rht_mutex_is_held+0xe/0x50 [ 32.083601] [<c17470a3>] ? thread_lookup_test+0x683/0x7f0 [ 32.083604] [<c1748215>] threadfunc+0x1005/0x1550 [ 32.083606] [<c1747210>] ? thread_lookup_test+0x7f0/0x7f0 [ 32.083608] [<c1747210>] ? thread_lookup_test+0x7f0/0x7f0 [ 32.083611] [<c113cd37>] kthread+0x127/0x170 [ 32.083614] [<c2df1d81>] ret_from_kernel_thread+0x21/0x30 [ 32.083616] [<c113cc10>] ? __kthread_unpark+0xd0/0xd0 [ 32.070037] d0984000 300000bd c0055eb0 c11bef6d c0055eec c1746eb1 00000000 d35ec008 [ 32.070037] d35ec000 0000005e 300000bd 0000d030 00001a06 00001a06 c46d3880 c0055ed4 [ 32.070037] 00000000 d364fa74 c0100000 c0055f34 c1748215 d35ec008 00000030 00000006 [ 32.070037] Call Trace: [ 32.070037] [<c11bef6d>] rcu_read_lock_held+0xdd/0xf0 [ 32.070037] [<c1746eb1>] thread_lookup_test+0x491/0x7f0 [ 32.070037] [<c1748215>] threadfunc+0x1005/0x1550 [ 32.070037] [<c1747210>] ? thread_lookup_test+0x7f0/0x7f0 [ 32.070037] [<c1747210>] ? thread_lookup_test+0x7f0/0x7f0 [ 32.070037] [<c113cd37>] kthread+0x127/0x170 [ 32.070037] [<c2df1d81>] ret_from_kernel_thread+0x21/0x30 [ 32.070037] [<c113cc10>] ? __kthread_unpark+0xd0/0xd0 [ 32.070037] Code: 89 f0 e8 d2 fe ff ff 89 c6 83 05 28 2e ec c3 01 83 15 2c 2e ec c3 00 64 a1 28 38 ce c3 c7 80 10 04 00 00 00 00 00 00 89 d8 50 9d <8d> 74 26 00 83 05 30 2e ec c3 01 83 15 34 2e ec c3 00 89 f0 5b [ 32.086676] Shutting down cpus with NMI [ 32.086676] Kernel Offset: disabled git bisect start 6967aa466b743f215b75a55d046ab4062d716ad5 2c6625cd545bdd66acff14f3394865d43920a5c7 -- git bisect good cbf6b6653e9a289d010d9a1b66a2397849821191 # 04:21 910+ 2 Merge 'kees/format-security' into devel-spot-201508210128 git bisect good 9ecca26e13e2d7ff423ae1d092413c629412035b # 04:56 908+ 0 Merge 'xen-tip/for-linus-4.3' into devel-spot-201508210128 git bisect good b5263785bfb0e5434cdb51bd21a82de2ebe1c676 # 05:49 910+ 3 Merge 'tip/perf/urgent' into devel-spot-201508210128 git bisect good b7a0925e47031d51041ce9a357edc02fb67d0b1d # 06:13 910+ 0 Merge tag 'wireless-drivers-next-for-davem-2015-07-31' of git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next git bisect good 90eb7fa51ccb9af58c437581af73c4a1d5fd2b9f # 06:44 910+ 0 Merge branch 'bpf_fanout' git bisect bad 4afbc0db720555f1bf0dd9f3e112819041231818 # 06:44 0- 53 net: 6lowpan: convert to using IFF_NO_QUEUE git bisect good c1f066d4ee0bde4bb0ff4ac295218b631729e0de # 07:32 910+ 0 Merge tag 'batman-adv-for-davem' of git://git.open-mesh.org/linux-merge git bisect good c0015bf3a34961342a27b021672049e535ab36a1 # 08:13 910+ 0 ieee802154: 6lowpan: fix non-lowpan wpan interfaces git bisect bad 2f52bdcf6ba1bd81597b1505a222430676548b3a # 08:22 107- 56 net: Updates to netif_index_is_vrf git bisect bad 0aa65cc0c2ca7e3908b1e4ae7946d909a4882249 # 08:34 94- 23 Merge branch 'for-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next git bisect bad 8b89f3a19de05992a70f727072b65f3ea1397075 # 08:51 118- 24 enic: move struct definition from .c to .h file git bisect bad ec120da6f0fe59f175c2a8faa0a7700280c39644 # 09:26 414- 73 ipv6: trivial whitespace fix git bisect bad f4a3e90ba5739cfd761b6befadae9728bd3641ed # 09:42 172- 22 rhashtable-test: extend to test concurrency # first bad commit: [f4a3e90ba5739cfd761b6befadae9728bd3641ed] rhashtable-test: extend to test concurrency git bisect good c1f066d4ee0bde4bb0ff4ac295218b631729e0de # 09:48 1003+ 0 Merge tag 'batman-adv-for-davem' of git://git.open-mesh.org/linux-merge # extra tests with DEBUG_INFO git bisect bad f4a3e90ba5739cfd761b6befadae9728bd3641ed # 09:57 21- 5 rhashtable-test: extend to test concurrency # extra tests on HEAD of linux-devel/devel-spot-201508210128 git bisect bad abe5543dfd124681d7c508588c300abb84972558 # 10:40 166- 70 0day head guard for 'devel-spot-201508210128' # extra tests on tree/branch jkirsher-next-queue/master # extra tests with first bad commit reverted # extra tests on tree/branch linus/master git bisect good e525293d97b93a04038e7dc15cee3e9075c32dbd # 12:17 1010+ 2 Add hch to .get_maintainer.ignore # extra tests on tree/branch linux-next/master git bisect bad 1ef981bcd18de26dc78bc79f092d6f4bb25e0e8f # 12:28 8- 4 Add linux-next specific files for 20150821 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=quantal-core-i386.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu kvm64 -kernel $kernel -initrd $initrd -m 300 -smp 2 -device e1000,netdev=net0 -netdev user,id=net0 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation