Hi Matthew, We find the softlockup and blocked_tasks issues related to your dax/wlist tree. The bisect to merge commit is not ideal, however the errors do look relevant. The attached reproduce-* script may help you debug the problem. git://internal_merge_and_test_tree devel-catchup-201702222304 commit a37748add0fa56e04c78ef2860f76eb894444164 Merge: fa1fcf7 947a7ea Author: 0day robot <fengguang.wu(a)intel.com&gt; AuthorDate: Wed Feb 22 23:04:13 2017 +0800 Commit: 0day robot <fengguang.wu(a)intel.com&gt; CommitDate: Wed Feb 22 23:04:13 2017 +0800 Merge 'dax/wlist' into devel-catchup-201702222304 fa1fcf710e Merge 'linux-review/Bartosz-Golaszewski/ARM-davinci-vpif-capture-display-support/20170222-221850' into devel-catchup-201702222304 947a7ea93f Convert semaphores to use waitlists a37748add0 Merge 'dax/wlist' into devel-catchup-201702222304 +------------------------------------------------------------------+------------+------------+------------+ | | fa1fcf710e | 947a7ea93f | a37748add0 | +------------------------------------------------------------------+------------+------------+------------+ | boot_successes | 135 | 0 | 4 | | boot_failures | 1 | 133 | 33 | | invoked_oom-killer:gfp_mask=0x | 1 | | | | Mem-Info | 1 | | | | page_allocation_failure:order:#,mode:#(GFP_KERNEL|__GFP_NORETRY) | 1 | | | | Kernel_panic-not_syncing:softlockup:hung_tasks | 0 | 133 | | | INFO:task_blocked_for_more_than#seconds | 0 | 0 | 33 | | calltrace:SyS_munmap | 0 | 0 | 25 | | Kernel_panic-not_syncing:hung_task:blocked_tasks | 0 | 0 | 32 | | calltrace:SyS_brk | 0 | 0 | 14 | | calltrace:SyS_mprotect | 0 | 0 | 1 | | BUG:kernel_hang_in_test_stage | 0 | 0 | 1 | +------------------------------------------------------------------+------------+------------+------------+ [ 275.146571] init: Temporary process spawn error: No such file or directory [ 275.146571] init: Temporary process spawn error: No such file or directory [ 275.274924] init: plymouth-upstart-bridge main process (345) terminated with status 1 [ 275.274924] init: plymouth-upstart-bridge main process (345) terminated with status 1 [ 326.624181] Writes: Total: 4 Max/Min: 0/0 Fail: 0 [ 367.584246] INFO: task udevd:304 blocked for more than 120 seconds. [ 367.584246] INFO: task udevd:304 blocked for more than 120 seconds. [ 367.587631] Not tainted 4.10.0-02694-ga37748a #1 [ 367.587631] Not tainted 4.10.0-02694-ga37748a #1 [ 367.589434] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 367.589434] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 367.591486] udevd D29176 304 1 0x00000004 [ 367.591486] udevd D29176 304 1 0x00000004 [ 367.593057] Call Trace: [ 367.593057] Call Trace: [ 367.593738] __schedule+0xa0d/0x1450 [ 367.593738] __schedule+0xa0d/0x1450 [ 367.594696] schedule+0x5c/0x120 [ 367.594696] schedule+0x5c/0x120 [ 367.595565] rwsem_down_read_failed+0x17e/0x270 [ 367.595565] rwsem_down_read_failed+0x17e/0x270 [ 367.596864] call_rwsem_down_read_failed+0x18/0x30 [ 367.596864] call_rwsem_down_read_failed+0x18/0x30 [ 367.598145] ? lock_contended+0x4e0/0x7a0 [ 367.598145] ? lock_contended+0x4e0/0x7a0 [ 367.599212] down_read+0x10c/0x140 [ 367.599212] down_read+0x10c/0x140 [ 367.604226] ? validate_mm+0xbd/0x660 [ 367.604226] ? validate_mm+0xbd/0x660 [ 367.605214] validate_mm+0xbd/0x660 [ 367.605214] validate_mm+0xbd/0x660 [ 367.606201] ? __asan_loadN+0xf/0x20 [ 367.606201] ? __asan_loadN+0xf/0x20 [ 367.607168] ? __vma_link_file+0x50/0x130 [ 367.607168] ? __vma_link_file+0x50/0x130 [ 367.612334] vma_link+0xb3/0x160 [ 367.612334] vma_link+0xb3/0x160 [ 367.613212] mmap_region+0xb06/0xf30 [ 367.613212] mmap_region+0xb06/0xf30 [ 367.614173] ? __asan_loadN+0xf/0x20 [ 367.614173] ? __asan_loadN+0xf/0x20 [ 367.615127] ? get_unmapped_area+0x1c0/0x380 [ 367.615127] ? get_unmapped_area+0x1c0/0x380 [ 367.620385] do_mmap+0x656/0xb70 [ 367.620385] do_mmap+0x656/0xb70 [ 367.621266] vm_mmap_pgoff+0xcb/0x130 [ 367.621266] vm_mmap_pgoff+0xcb/0x130 [ 367.622249] SyS_mmap_pgoff+0xd3/0x490 [ 367.622249] SyS_mmap_pgoff+0xd3/0x490 [ 367.623258] SyS_mmap+0x85/0xa0 [ 367.623258] SyS_mmap+0x85/0xa0 [ 367.628205] entry_SYSCALL_64_fastpath+0x1e/0xb2 [ 367.628205] entry_SYSCALL_64_fastpath+0x1e/0xb2 [ 367.629441] RIP: 0033:0x7ff85bea986a [ 367.629441] RIP: 0033:0x7ff85bea986a [ 367.630393] RSP: 002b:00007ffdf5773198 EFLAGS: 00000246 ORIG_RAX: 0000000000000009 [ 367.630393] RSP: 002b:00007ffdf5773198 EFLAGS: 00000246 ORIG_RAX: 0000000000000009 [ 367.636458] RAX: ffffffffffffffda RBX: 000055be8ff7e2d0 RCX: 00007ff85bea986a [ 367.636458] RAX: ffffffffffffffda RBX: 000055be8ff7e2d0 RCX: 00007ff85bea986a [ 367.638333] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 0000000000000000 [ 367.638333] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 0000000000000000 [ 367.644280] RBP: 000000000000011a R08: 00000000ffffffff R09: 0000000000000000 [ 367.644280] RBP: 000000000000011a R08: 00000000ffffffff R09: 0000000000000000 [ 367.648287] R10: 0000000000000022 R11: 0000000000000246 R12: 000055be8ff845f0 [ 367.648287] R10: 0000000000000022 R11: 0000000000000246 R12: 000055be8ff845f0 [ 367.650172] R13: 0000000000000000 R14: 0000000000000000 R15: 000055be8ffa8520 [ 367.650172] R13: 0000000000000000 R14: 0000000000000000 R15: 000055be8ffa8520 [ 367.656136] [ 367.656136] Showing all locks held in the system: [ 367.656136] [ 367.656136] Showing all locks held in the system: [ 367.657778] 2 locks held by khungtaskd/20: [ 367.657778] 2 locks held by khungtaskd/20: [ 367.658867] #0: (rcu_read_lock){......}, at: [<ffffffff812921b0>] watchdog+0x130/0xab0 [ 367.658867] #0: (rcu_read_lock){......}, at: [<ffffffff812921b0>] watchdog+0x130/0xab0 [ 367.661103] #1: (tasklist_lock){......}, at: [<ffffffff811c3d4b>] debug_show_all_locks+0x4b/0x1f0 [ 367.661103] #1: (tasklist_lock){......}, at: [<ffffffff811c3d4b>] debug_show_all_locks+0x4b/0x1f0 [ 367.663503] 2 locks held by udevd/304: [ 367.663503] 2 locks held by udevd/304: [ 367.664584] #0: (&mm->mmap_sem){......}, at: [<ffffffff8137f1c7>] vm_mmap_pgoff+0x57/0x130 [ 367.664584] #0: (&mm->mmap_sem){......}, at: [<ffffffff8137f1c7>] vm_mmap_pgoff+0x57/0x130 [ 367.666859] #1: (&anon_vma->rwsem){......}, at: [<ffffffff813bce1d>] validate_mm+0xbd/0x660 [ 367.666859] #1: (&anon_vma->rwsem){......}, at: [<ffffffff813bce1d>] validate_mm+0xbd/0x660 [ 367.669220] 2 locks held by udevd/316: [ 367.669220] 2 locks held by udevd/316: [ 367.670215] #0: (&mm->mmap_sem){......}, at: [<ffffffff813c6138>] SyS_munmap+0x48/0xb0 [ 367.670215] #0: (&mm->mmap_sem){......}, at: [<ffffffff813c6138>] SyS_munmap+0x48/0xb0 [ 367.674015] #1: (&anon_vma->rwsem){......}, at: [<ffffffff813bce1d>] validate_mm+0xbd/0x660 [ 367.674015] #1: (&anon_vma->rwsem){......}, at: [<ffffffff813bce1d>] validate_mm+0xbd/0x660 [ 367.676462] 2 locks held by udevd/331: [ 367.676462] 2 locks held by udevd/331: [ 367.677460] #0: (&mm->mmap_sem){......}, at: [<ffffffff8137f1c7>] vm_mmap_pgoff+0x57/0x130 [ 367.677460] #0: (&mm->mmap_sem){......}, at: [<ffffffff8137f1c7>] vm_mmap_pgoff+0x57/0x130 [ 367.679681] #1: (&anon_vma->rwsem){......}, at: [<ffffffff813bce1d>] validate_mm+0xbd/0x660 [ 367.679681] #1: (&anon_vma->rwsem){......}, at: [<ffffffff813bce1d>] validate_mm+0xbd/0x660 [ 367.682054] 2 locks held by trinity-main/340: [ 367.682054] 2 locks held by trinity-main/340: [ 367.683204] #0: (&sb->s_type->i_mutex_key){......}, at: [<ffffffff81476874>] iterate_dir+0x164/0x470 [ 367.683204] #0: (&sb->s_type->i_mutex_key){......}, at: [<ffffffff81476874>] iterate_dir+0x164/0x470 [ 367.685731] #1: (&mm->mmap_sem){......}, at: [<ffffffff81564511>] proc_map_files_readdir+0x391/0x7f0 [ 367.685731] #1: (&mm->mmap_sem){......}, at: [<ffffffff81564511>] proc_map_files_readdir+0x391/0x7f0 [ 367.688349] 1 lock held by killall5/362: [ 367.688349] 1 lock held by killall5/362: [ 367.689396] #0: (&mm->mmap_sem){......}, at: [<ffffffff81561f25>] proc_pid_cmdline_read+0x225/0xc60 [ 367.689396] #0: (&mm->mmap_sem){......}, at: [<ffffffff81561f25>] proc_pid_cmdline_read+0x225/0xc60 [ 367.691826] [ 367.691826] [ 367.692335] ============================================= [ 367.692335] [ 367.692335] ============================================= [ 367.692335] [ 367.694138] NMI backtrace for cpu 1 [ 367.694138] NMI backtrace for cpu 1 [ 367.695068] CPU: 1 PID: 20 Comm: khungtaskd Not tainted 4.10.0-02694-ga37748a #1 [ 367.695068] CPU: 1 PID: 20 Comm: khungtaskd Not tainted 4.10.0-02694-ga37748a #1 [ 367.696317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014 [ 367.696317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014 [ 367.696317] Call Trace: [ 367.696317] Call Trace: [ 367.696317] dump_stack+0xd5/0x113 [ 367.696317] dump_stack+0xd5/0x113 [ 367.696317] nmi_cpu_backtrace+0x242/0x270 [ 367.696317] nmi_cpu_backtrace+0x242/0x270 [ 367.696317] ? irq_force_complete_move+0x3f0/0x3f0 [ 367.696317] ? irq_force_complete_move+0x3f0/0x3f0 [ 367.696317] nmi_trigger_cpumask_backtrace+0x31e/0x390 [ 367.696317] nmi_trigger_cpumask_backtrace+0x31e/0x390 [ 367.696317] arch_trigger_cpumask_backtrace+0x19/0x20 [ 367.696317] arch_trigger_cpumask_backtrace+0x19/0x20 [ 367.696317] watchdog+0x948/0xab0 [ 367.696317] watchdog+0x948/0xab0 [ 367.696317] ? watchdog+0x130/0xab0 [ 367.696317] ? watchdog+0x130/0xab0 [ 367.696317] kthread+0x27b/0x280 [ 367.696317] kthread+0x27b/0x280 [ 367.696317] ? hung_task_panic+0x20/0x20 [ 367.696317] ? hung_task_panic+0x20/0x20 [ 367.696317] ? __kthread_create_on_node+0x3a0/0x3a0 [ 367.696317] ? __kthread_create_on_node+0x3a0/0x3a0 [ 367.696317] ret_from_fork+0x2c/0x40 [ 367.696317] ret_from_fork+0x2c/0x40 [ 367.712765] Sending NMI from CPU 1 to CPUs 0: [ 367.712765] Sending NMI from CPU 1 to CPUs 0: [ 367.713956] NMI backtrace for cpu 0 skipped: idling at pc 0xffffffff82f46c66 [ 367.713956] NMI backtrace for cpu 0 skipped: idling at pc 0xffffffff82f46c66 [ 367.715823] Kernel panic - not syncing: hung_task: blocked tasks [ 367.715823] Kernel panic - not syncing: hung_task: blocked tasks [ 367.716714] CPU: 1 PID: 20 Comm: khungtaskd Not tainted 4.10.0-02694-ga37748a #1 [ 367.716714] CPU: 1 PID: 20 Comm: khungtaskd Not tainted 4.10.0-02694-ga37748a #1 [ 367.716714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014 [ 367.716714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014 [ 367.716714] Call Trace: [ 367.716714] Call Trace: [ 367.716714] dump_stack+0xd5/0x113 [ 367.716714] dump_stack+0xd5/0x113 [ 367.716714] panic+0x18b/0x413 [ 367.716714] panic+0x18b/0x413 [ 367.716714] ? printk_nmi_flush+0x54/0x160 [ 367.716714] ? printk_nmi_flush+0x54/0x160 [ 367.716714] watchdog+0x95b/0xab0 [ 367.716714] watchdog+0x95b/0xab0 [ 367.716714] ? watchdog+0x130/0xab0 [ 367.716714] ? watchdog+0x130/0xab0 [ 367.716714] kthread+0x27b/0x280 [ 367.716714] kthread+0x27b/0x280 [ 367.716714] ? hung_task_panic+0x20/0x20 [ 367.716714] ? hung_task_panic+0x20/0x20 [ 367.716714] ? __kthread_create_on_node+0x3a0/0x3a0 [ 367.716714] ? __kthread_create_on_node+0x3a0/0x3a0 [ 367.716714] ret_from_fork+0x2c/0x40 [ 367.716714] ret_from_fork+0x2c/0x40 [ 367.716714] Kernel Offset: disabled [ 367.716714] Kernel Offset: disabled git bisect start 831b1e44a7e9e209c65506a86a120ea2156aee5f c470abd4fde40ea6a0846a2beab642a578c0b8cd -- git bisect bad a37748add0fa56e04c78ef2860f76eb894444164 # 17:58 0- 2 Merge 'dax/wlist' into devel-catchup-201702222304 git bisect good 953facbe4bd24d3d18d87a2ba4e3341042e9a5a2 # 19:22 30+ 0 0day base guard for 'devel-catchup-201702222304' git bisect good fa1fcf710e24af3bb27aab81a5ae3584554e0dba # 20:06 31+ 1 Merge 'linux-review/Bartosz-Golaszewski/ARM-davinci-vpif-capture-display-support/20170222-221850' into devel-catchup-201702222304 git bisect good 04fe36bd995189aacf44890ae85a59a04afb3114 # 12:31 30+ 34 Convert mutexes to use waitlists git bisect good 947a7ea93f3d85ddc885b02550ce263f15921544 # 00:10 30+ 34 Convert semaphores to use waitlists # first bad commit: [a37748add0fa56e04c78ef2860f76eb894444164] Merge 'dax/wlist' into devel-catchup-201702222304 git bisect good fa1fcf710e24af3bb27aab81a5ae3584554e0dba # 00:36 94+ 1 Merge 'linux-review/Bartosz-Golaszewski/ARM-davinci-vpif-capture-display-support/20170222-221850' into devel-catchup-201702222304 git bisect good 947a7ea93f3d85ddc885b02550ce263f15921544 # 13:11 93+ 129 Convert semaphores to use waitlists # extra tests with CONFIG_DEBUG_INFO_REDUCED git bisect bad a37748add0fa56e04c78ef2860f76eb894444164 # 13:33 2- 5 Merge 'dax/wlist' into devel-catchup-201702222304 # extra tests on HEAD of linux-devel/devel-catchup-201702222304 git bisect bad 831b1e44a7e9e209c65506a86a120ea2156aee5f # 13:33 0- 14 0day head guard for 'devel-catchup-201702222304' # extra tests on tree/branch linux-devel/devel-catchup-201702222304 git bisect bad 831b1e44a7e9e209c65506a86a120ea2156aee5f # 13:39 0- 14 0day head guard for 'devel-catchup-201702222304' --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation