Hi Peter, FYI, here is another bisected warning to this commit. Attached a reproduce script for your convenience. https://github.com/0day-ci/linux peter-enderborg-sonymobile-com/android-Collect-statistics-from-lowmemorykiller/20170215-004327 commit c3263ba15e872666094c18a1d882597bfe1ce1c1 Author: Peter Enderborg <peter.enderborg(a)sonymobile.com&gt; AuthorDate: Tue Feb 14 17:09:32 2017 +0100 Commit: 0day robot <fengguang.wu(a)intel.com&gt; CommitDate: Wed Feb 15 00:43:32 2017 +0800 mm: Remove RCU and tasklocks from lmk Fundamental changes: 1 Does NOT take any RCU lock in shrinker functions. 2 It returns same result for scan and counts, so we dont need to do shinker will know when it is pointless to call scan. 3 It does not lock any other process than the one that is going to be killed. Background. The low memory killer scans for process that can be killed to free memory. This can be cpu consuming when there is a high demand for memory. This can be seen by analysing the kswapd0 task work. The stats function added in earler patch adds a counter for waste work. How it works. This patch create a structure within the lowmemory killer that caches the user spaces processes that it might kill. It is done with a sorted rbtree so we can very easy find the candidate to be killed, and knows its properies as memory usage and sorted by oom_score_adj to look up the task with highest oom_score_adj. To be able to achive this it uses oom_score_notify events. This patch also as a other effect, we are now free to do other lowmemorykiller configurations. Without the patch there is a need for a tradeoff between freed memory and task and rcu locks. This is no longer a concern for tuning lmk. This patch is not intended to do any calculation changes other than we do use the cache for calculate the count values and that makes kswapd0 to shrink other areas. Signed-off-by: Peter Enderborg <peter.enderborg(a)sonymobile.com&gt; +-------------------------------------------------+------------+------------+------------------+ | | 0174c40bf1 | c3263ba15e | v4.10-rc8_021805 | +-------------------------------------------------+------------+------------+------------------+ | boot_successes | 261 | 0 | 70 | | boot_failures | 3 | 71 | 25 | | BUG:kernel_hang_in_test_stage | 3 | 1 | 4 | | inconsistent{IN-SOFTIRQ-W}->{SOFTIRQ-ON-W}usage | 0 | 6 | 10 | | calltrace:lmk_oom_score_notifier | 0 | 70 | | | calltrace:smpboot_thread_fn | 0 | 3 | | | calltrace:__set_oom_adj | 0 | 2 | | | inconsistent{SOFTIRQ-ON-W}->{IN-SOFTIRQ-W}usage | 0 | 64 | 10 | | calltrace:SyS_write | 0 | 63 | | | calltrace:rcu_process_callbacks | 0 | 64 | 10 | | calltrace:SyS_munmap | 0 | 1 | | | calltrace:irq_exit | 0 | 1 | 4 | | calltrace:SyS_mmap_pgoff | 0 | 1 | | | calltrace:copy_process | 0 | 2 | 3 | | calltrace:n_tty_write | 0 | 1 | | | BUG:kernel_in_stage | 0 | 0 | 1 | | calltrace:vprintk_emit | 0 | 0 | 2 | | calltrace:univ8250_release_irq | 0 | 0 | 1 | +-------------------------------------------------+------------+------------+------------------+ [ 9.180183] [ 9.180610] ================================= [ 9.181713] [ INFO: inconsistent lock state ] [ 9.182811] 4.10.0-rc7-00639-gc3263ba #2 Not tainted [ 9.184067] --------------------------------- [ 9.185260] inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage. [ 9.186780] ksoftirqd/0/6 [HC0[0]:SC1[1]:HE1:SE0] takes: [ 9.188173] (lmk_task_lock){+.?...}, at: [<ffffffff81441d54>] lmk_oom_score_notifier+0x64/0x150 [ 9.192310] {SOFTIRQ-ON-W} state was registered at: [ 9.193508] [ 9.193515] [<ffffffff81095509>] __lock_acquire+0x5d9/0x1760 [ 9.197638] [ 9.197644] [<ffffffff81096acf>] lock_acquire+0x8f/0xe0 [ 9.200487] [ 9.200494] [<ffffffff8151084c>] _raw_spin_lock+0x3c/0x80 [ 9.203921] [ 9.203930] [<ffffffff81441bf0>] lmk_task_watch+0x40/0x140 [ 9.207799] [ 9.207807] [<ffffffff81441d12>] lmk_oom_score_notifier+0x22/0x150 [ 9.211497] [ 9.211504] [<ffffffff8107dec4>] notifier_call_chain+0x34/0xa0 [ 9.216504] [ 9.216508] [<ffffffff8107df97>] atomic_notifier_call_chain+0x67/0x100 [ 9.218637] [ 9.218644] [<ffffffff810e3014>] oom_score_notify_update+0x24/0x40 [ 9.224726] [ 9.224734] [<ffffffff811a7c3d>] __set_oom_adj+0x15d/0x4a0 [ 9.226560] [ 9.226563] [<ffffffff811a803d>] oom_score_adj_write+0xbd/0xd0 [ 9.228466] [ 9.228471] [<ffffffff81149d33>] __vfs_write+0x23/0x120 [ 9.231499] [ 9.231503] [<ffffffff8114b8a4>] vfs_write+0xc4/0x1c0 [ 9.237215] [ 9.237221] [<ffffffff8114ba84>] SyS_write+0x44/0xa0 [ 9.238895] [ 9.238900] [<ffffffff815116fd>] entry_SYSCALL_64_fastpath+0x1f/0xc1 [ 9.241805] irq event stamp: 41246 [ 9.244701] hardirqs last enabled at (41246): [<ffffffff81112a4b>] free_hot_cold_page+0x1cb/0x2e0 [ 9.247554] hardirqs last disabled at (41245): [<ffffffff811129ae>] free_hot_cold_page+0x12e/0x2e0 [ 9.251752] softirqs last enabled at (41088): [<ffffffff81060c6a>] __do_softirq+0x20a/0x270 [ 9.256594] softirqs last disabled at (41093): [<ffffffff81060cef>] run_ksoftirqd+0x1f/0x70 [ 9.258397] [ 9.258397] other info that might help us debug this: [ 9.261476] Possible unsafe locking scenario: [ 9.261476] [ 9.264234] CPU0 [ 9.267364] ---- [ 9.267992] lock(lmk_task_lock); [ 9.268899] <Interrupt> [ 9.269565] lock(lmk_task_lock); [ 9.272459] [ 9.272459] *** DEADLOCK *** [ 9.272459] [ 9.273949] 2 locks held by ksoftirqd/0/6: [ 9.276583] #0: (rcu_callback){......}, at: [<ffffffff810abfe9>] rcu_process_callbacks+0x269/0x620 [ 9.280664] #1: (rcu_read_lock){......}, at: [<ffffffff8107df30>] atomic_notifier_call_chain+0x0/0x100 [ 9.283754] [ 9.283754] stack backtrace: [ 9.287175] CPU: 0 PID: 6 Comm: ksoftirqd/0 Not tainted 4.10.0-rc7-00639-gc3263ba #2 [ 9.289626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014 [ 9.294629] Call Trace: [ 9.296338] dump_stack+0x19/0x26 [ 9.297172] print_usage_bug+0x1df/0x1f0 [ 9.300938] mark_lock+0x5be/0x650 [ 9.301788] ? print_shortest_lock_dependencies+0x1a0/0x1a0 [ 9.303269] __lock_acquire+0x517/0x1760 [ 9.307267] ? kvm_clock_read+0x22/0x40 [ 9.308326] ? kvm_sched_clock_read+0x9/0x20 [ 9.309521] ? kvm_clock_read+0x22/0x40 [ 9.312740] ? kvm_sched_clock_read+0x9/0x20 [ 9.313959] lock_acquire+0x8f/0xe0 [ 9.315280] ? lmk_oom_score_notifier+0x64/0x150 [ 9.318345] _raw_spin_lock+0x3c/0x80 [ 9.319269] ? lmk_oom_score_notifier+0x64/0x150 [ 9.322357] lmk_oom_score_notifier+0x64/0x150 [ 9.323474] notifier_call_chain+0x34/0xa0 [ 9.327487] ? will_become_orphaned_pgrp+0x90/0x90 [ 9.328755] atomic_notifier_call_chain+0x67/0x100 [ 9.330050] ? notifier_call_chain+0xa0/0xa0 [ 9.333014] oom_score_notify_free+0x21/0x40 [ 9.334183] ? kmem_cache_free+0x18/0x40 [ 9.335978] __put_task_struct+0x57/0x170 [ 9.341088] delayed_put_task_struct+0x29/0x30 [ 9.342256] rcu_process_callbacks+0x2d0/0x620 [ 9.343371] ? rcu_process_callbacks+0x269/0x620 [ 9.344565] __do_softirq+0x12a/0x270 [ 9.346733] ? smpboot_thread_fn+0x23/0x250 [ 9.348353] run_ksoftirqd+0x1f/0x70 [ 9.351651] smpboot_thread_fn+0x16a/0x250 [ 9.352724] kthread+0x102/0x140 [ 9.353541] ? sort_range+0x20/0x20 [ 9.356468] ? kthread_create_on_node+0x40/0x40 [ 9.357650] ret_from_fork+0x31/0x40 [ 9.419370] random: trinity: uninitialized urandom read (4 bytes read) mountall: Event failed [ 9.462308] random: mountall: uninitialized urandom read (12 bytes read) [ 9.519336] init: Failed to create pty - disabling logging for job [ 9.529057] init: Temporary process spawn error: No such file or directory git bisect start 804340e99508771d2c3709a1605ca0d7aa7ef761 7089db84e356562f8ba737c29e472cc42d530dbc -- git bisect bad 0d2799238644a776f5bfcdb060e361ed56a36cd6 # 23:06 0- 16 Merge 'renesas/devel' into devel-hourly-2017021805 git bisect good 157f37653a4382f7eeddb3ea6d460af714fb50a8 # 23:18 65+ 1 Merge 'linux-review/Geert-Uytterhoeven/ARM-PM-Drop-useless-checks-for-PM_SUSPEND_STANDBY/20170218-025420' into devel-hourly-2017021805 git bisect good 1d4f149c684d839094ccadab501860440200d98a # 23:46 63+ 0 Merge 'linux-review/Olav-Haugan/staging-speakup-coding-style-Simplify-comparisons-to-NULL/20170217-131253' into devel-hourly-2017021805 git bisect bad cf14a82cebeec14fb05b5fa844f10e22964d658a # 23:58 0- 2 Merge 'linux-review/thor-thayer-linux-intel-com/mfd-altr-a10sr-Add-Arria10-SR-sysfs-attributes/20170215-022501' into devel-hourly-2017021805 git bisect bad 644e51d644f6c446930b4a3f6dbc664dd27776c6 # 00:11 0- 6 Merge 'linux-review/Colin-King/usb-misc-usbtest-remove-redundant-check-on-retval-0/20170213-024141' into devel-hourly-2017021805 git bisect good 54a136f5b3b9a36eb5c717644d3e97fd69a030b8 # 00:25 64+ 1 Merge 'linux-review/Bjorn-Andersson/arm64-dts-qcom-pm8994-Add-rtc-node/20170218-020240' into devel-hourly-2017021805 git bisect good e79a5f53ec94ec08e7b7997150ef85d4b516c41b # 00:45 63+ 0 Merge 'linux-review/Paolo-Bonzini/KVM-change-gfn-hva-cache-to-use-per-VCPU-memslots/20170216-065425' into devel-hourly-2017021805 git bisect good 9382124d4146803e057e7dbe8d92a73eabb0cf14 # 01:09 63+ 0 Merge 'vfio/next' into devel-hourly-2017021805 git bisect bad 108f1e75acb82a913bec494f26eec68d497098a3 # 01:38 0- 5 Merge 'linux-review/Alban/MIPS-Allow-compressed-images-to-be-loaded-at-any-address/20170214-053759' into devel-hourly-2017021805 git bisect bad 11bc53d63ef26e59ce1c7c8ca88a4b7f959b878d # 01:54 0- 1 Merge 'linux-review/peter-enderborg-sonymobile-com/android-Collect-statistics-from-lowmemorykiller/20170215-004327' into devel-hourly-2017021805 git bisect good 0174c40bf153def3ac7b287f000a885b15048a38 # 02:16 61+ 0 oom: Add notification for oom_score_adj git bisect bad c3263ba15e872666094c18a1d882597bfe1ce1c1 # 02:31 0- 15 mm: Remove RCU and tasklocks from lmk # first bad commit: [c3263ba15e872666094c18a1d882597bfe1ce1c1] mm: Remove RCU and tasklocks from lmk git bisect good 0174c40bf153def3ac7b287f000a885b15048a38 # 02:43 189+ 0 oom: Add notification for oom_score_adj # extra tests with CONFIG_DEBUG_INFO_REDUCED git bisect bad c3263ba15e872666094c18a1d882597bfe1ce1c1 # 03:16 0- 7 mm: Remove RCU and tasklocks from lmk # extra tests on HEAD of linux-devel/devel-hourly-2017021805 git bisect bad 804340e99508771d2c3709a1605ca0d7aa7ef761 # 03:16 0- 23 0day head guard for 'devel-hourly-2017021805' # extra tests on tree/branch linux-review/peter-enderborg-sonymobile-com/android-Collect-statistics-from-lowmemorykiller/20170215-004327 git bisect bad c3263ba15e872666094c18a1d882597bfe1ce1c1 # 03:23 0- 68 mm: Remove RCU and tasklocks from lmk # extra tests with first bad commit reverted git bisect good 1f3fd50274732605285e1de35c03b552107df194 # 16:21 187+ 1 Revert "mm: Remove RCU and tasklocks from lmk" # extra tests on tree/branch linus/master git bisect good 00ea1ceebe0d9f2dc1cc2b7bd575a00100c27869 # 05:14 183+ 4 ipv6: release dst on error in ip6_dst_lookup_tail # extra tests on tree/branch linux-next/master git bisect good 4ce4a759a3e221b5265ebd03c2f8888b69a7cf3e # 17:24 183+ 17 Add linux-next specific files for 20170217 --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation