Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/mel/linux mm-vmscan-node-lru-v3r10 commit 970df71e432f080bee5548dd6ee6606943ef3236 Author: Mel Gorman <mgorman(a)techsingularity.net&gt; AuthorDate: Thu Apr 16 17:30:55 2015 +0100 Commit: Mel Gorman <mgorman(a)techsingularity.net&gt; CommitDate: Wed Mar 30 11:33:22 2016 +0100 mm, vmscan: Move LRU lists to node This moves the LRU lists from the zone to the node and all related data such as counters, tracing, congestion tracking and writeback tracking. This is mostly a mechanical patch but note that it introduces a number of anomalies. For example, the scans are per-zone but using per-node counters. We also mark a node as congested when a zone is congested. This causes weird problems that are fixed later but is easier to review. Signed-off-by: Mel Gorman <mgorman(a)techsingularity.net&gt; Acked-by: Johannes Weiner <hannes(a)cmpxchg.org&gt; +------------------------------------------+------------+------------+------------+ | | 7db93e3a25 | 970df71e43 | 8405105053 | +------------------------------------------+------------+------------+------------+ | boot_successes | 190 | 16 | 0 | | boot_failures | 0 | 52 | 13 | | BUG:unable_to_handle_kernel | 0 | 52 | 13 | | Oops | 0 | 52 | 13 | | RIP:cpu_vm_stats_fold | 0 | 52 | 13 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 52 | 13 | | backtrace:debug_hotplug_cpu | 0 | 52 | 13 | | backtrace:kernel_init_freeable | 0 | 52 | 13 | +------------------------------------------+------------+------------+------------+ [ 2.795299] Key type trusted registered [ 2.797960] Key type encrypted registered [ 2.800424] Unregister pv shared memory for cpu 0 [ 2.847203] BUG: unable to handle kernel paging request at 0000000000016f29 [ 2.849384] IP: [<ffffffff812189da>] cpu_vm_stats_fold+0x10a/0x180 [ 2.851271] PGD 0 [ 2.852450] Oops: 0000 [#1] SMP [ 2.853953] Modules linked in: [ 2.855216] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.6.0-rc1-00004-g970df71 #1 [ 2.857632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 2.860605] task: ffff8800101c4b40 ti: ffff8800101c8000 task.ti: ffff8800101c8000 [ 2.863090] RIP: 0010:[<ffffffff812189da>] [<ffffffff812189da>] cpu_vm_stats_fold+0x10a/0x180 [ 2.865892] RSP: 0000:ffff8800101cbbf0 EFLAGS: 00010246 [ 2.867414] RAX: ffff880011529000 RBX: 0000000000000000 RCX: 0000000000000000 [ 2.869252] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffffffff8206d368 [ 2.871096] RBP: ffff8800101cbca0 R08: ffff88001152cc88 R09: 0000000000016f28 [ 2.872898] R10: ffffea00003ca2a0 R11: 0000000000000002 R12: ffff8800101cbc14 [ 2.874682] R13: 0000000000000000 R14: 0000000000000000 R15: 00000000fffffff7 [ 2.876604] FS: 0000000000000000(0000) GS:ffff880011300000(0000) knlGS:0000000000000000 [ 2.879359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2.881003] CR2: 0000000000016f29 CR3: 0000000001c06000 CR4: 00000000000006e0 [ 2.882868] Stack: [ 2.883886] 0000000011216f58 0000000000000000 0000000000000000 0000000000000000 [ 2.887027] 0000000800000000 00000000fffffff3 0000000000000000 0000000000000000 [ 2.890299] 0000000100000000 0000000000000008 0000000000000001 0000000000000000 [ 2.893633] Call Trace: [ 2.894898] [<ffffffff811f4e9f>] page_alloc_cpu_notify+0x4f/0x60 [ 2.896784] [<ffffffff810f011d>] notifier_call_chain+0x5d/0xb0 [ 2.898731] [<ffffffff810f022e>] __raw_notifier_call_chain+0xe/0x10 [ 2.900662] [<ffffffff810c3486>] __cpu_notify+0x36/0x50 [ 2.902482] [<ffffffff810c35aa>] cpu_notify_nofail+0x1a/0x50 [ 2.906200] [<ffffffff810c35e0>] ? cpu_notify_nofail+0x50/0x50 [ 2.908022] [<ffffffff810c3606>] notify_dead+0x26/0x110 [ 2.909880] [<ffffffff810c35e0>] ? cpu_notify_nofail+0x50/0x50 [ 2.913145] [<ffffffff810c3f29>] cpuhp_invoke_callback+0x79/0x1b0 [ 2.914975] [<ffffffff810c4358>] cpuhp_down_callbacks+0x58/0xe0 [ 2.916724] [<ffffffff8179a08f>] _cpu_down+0x19f/0x1f0 [ 2.918560] [<ffffffff810c522e>] do_cpu_down+0x4e/0x70 [ 2.920232] [<ffffffff810c5260>] cpu_down+0x10/0x20 [ 2.921866] [<ffffffff81030340>] _debug_hotplug_cpu+0xa0/0xf0 [ 2.923656] [<ffffffff82090cdb>] ? topology_init+0x9f/0x9f [ 2.926458] [<ffffffff82090ce8>] debug_hotplug_cpu+0xd/0x11 [ 2.928153] [<ffffffff8100041e>] do_one_initcall+0xde/0x2b0 [ 2.930113] [<ffffffff8208645f>] kernel_init_freeable+0x113/0x1d6 [ 2.932531] [<ffffffff8179945e>] kernel_init+0xe/0x180 [ 2.938010] [<ffffffff817a9de2>] ret_from_fork+0x22/0x40 [ 2.939919] [<ffffffff81799450>] ? rest_init+0x90/0x90 [ 2.941686] Code: 48 89 c7 e8 f9 dc ff ff 48 85 c0 0f 85 7a ff ff ff e8 4b dc ff ff 48 85 c0 74 5f 4c 8b 88 80 3c 00 00 4c 8d 80 88 3c 00 00 31 d2 <41> 0f be 4c 11 01 31 f6 84 c9 40 0f 95 c6 48 83 c6 02 48 83 04 [ 2.954461] RIP [<ffffffff812189da>] cpu_vm_stats_fold+0x10a/0x180 [ 2.956475] RSP <ffff8800101cbbf0> [ 2.957900] CR2: 0000000000016f29 [ 2.959336] ---[ end trace ba236c6a755d1712 ]--- [ 2.960801] Kernel panic - not syncing: Fatal exception git bisect start 84051050531602bc1d427bfc27d39fb8e2a6e23f f55532a0c0b8bb6148f4e07853b876ef73bc69ca -- git bisect bad 58766773d8b58762bea392f9f179e2ea5e602e2a # 22:29 0- 11 Merge 'linux-review/Lars-Peter-Clausen/usb-gadget-f_fs-Fix-EFAULT-generation-for-async-read-operations/20160330-201655' into devel-catchup-201603302038 git bisect bad 57cb0975edee7a62820dbc2dcbefe7a159687e71 # 22:29 0- 39 Merge 'linux-review/Shubhangi-Shrivastava/drm-i915-Splitting-intel_dp_detect/20160330-203510' into devel-catchup-201603302038 git bisect good b4fe8cd840d422f83211b03cd9b97c482e159ebf # 22:36 66+ 0 0day base guard for 'devel-catchup-201603302038' git bisect bad 46edea8c09c5dada2def7b4558b5d38781c9211a # 22:42 0- 47 Merge 'mel/mm-vmscan-node-lru-v3r10' into devel-catchup-201603302038 git bisect bad 404ff05311b002244746cfbed691f0d4c17a76f3 # 22:48 0- 25 mm, workingset: Make working set detection node-aware git bisect bad a6013470b90d9a2c8d3da20b9ee20ec32b42c30b # 22:52 0- 36 mm, vmscan: Make kswapd reclaim in terms of nodes git bisect good 7db93e3a253b53eefedfd27fcf391bc2cb02380b # 22:59 66+ 0 mm, vmscan: Move lru_lock to the node git bisect bad ac8fbf7357e93ccb5887ff9fb123d5b16bd7504f # 23:04 0- 17 mm, vmscan: Begin reclaiming pages on a per-node basis git bisect bad 970df71e432f080bee5548dd6ee6606943ef3236 # 23:09 0- 52 mm, vmscan: Move LRU lists to node # first bad commit: [970df71e432f080bee5548dd6ee6606943ef3236] mm, vmscan: Move LRU lists to node git bisect good 7db93e3a253b53eefedfd27fcf391bc2cb02380b # 23:12 190+ 0 mm, vmscan: Move lru_lock to the node # extra tests with DEBUG_INFO git bisect bad 970df71e432f080bee5548dd6ee6606943ef3236 # 23:16 0- 7 mm, vmscan: Move LRU lists to node # extra tests on HEAD of linux-devel/devel-catchup-201603302038 git bisect bad 84051050531602bc1d427bfc27d39fb8e2a6e23f # 23:16 0- 13 0day head guard for 'devel-catchup-201603302038' # extra tests on tree/branch mel/mm-vmscan-node-lru-v3r10 git bisect bad ddb4e8c2992639305e2ecadacb53ba9fe80718bc # 23:37 0- 7 mm: vmstat: Account per-zone stalls and pages skipped during reclaim # extra tests on tree/branch linus/master git bisect good 1993b176a8224e371e0732ffada7ab9eb3b0912b # 23:45 190+ 0 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/ide # extra tests on tree/branch linux-next/master git bisect good 67130d02fee17fd2176fff7d49d4f91ccbdede07 # 23:49 188+ 0 Add linux-next specific files for 20160330 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=quantal-core-x86_64.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu kvm64 -kernel $kernel -initrd $initrd -m 300 -smp 2 -device e1000,netdev=net0 -netdev user,id=net0 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation