Greetings,
0day kernel testing robot got the below dmesg and the first bad commit is
https://git.kernel.org/pub/scm/linux/kernel/git/mel/linux mm-vmscan-node-lru-v3r10
commit 970df71e432f080bee5548dd6ee6606943ef3236
Author: Mel Gorman <mgorman(a)techsingularity.net>
AuthorDate: Thu Apr 16 17:30:55 2015 +0100
Commit: Mel Gorman <mgorman(a)techsingularity.net>
CommitDate: Wed Mar 30 11:33:22 2016 +0100
mm, vmscan: Move LRU lists to node
This moves the LRU lists from the zone to the node and all related data
such as counters, tracing, congestion tracking and writeback tracking.
This is mostly a mechanical patch but note that it introduces a number
of anomalies. For example, the scans are per-zone but using per-node
counters. We also mark a node as congested when a zone is congested. This
causes weird problems that are fixed later but is easier to review.
Signed-off-by: Mel Gorman <mgorman(a)techsingularity.net>
Acked-by: Johannes Weiner <hannes(a)cmpxchg.org>
+------------------------------------------+------------+------------+------------+
| | 7db93e3a25 | 970df71e43 | 8405105053 |
+------------------------------------------+------------+------------+------------+
| boot_successes | 190 | 16 | 0 |
| boot_failures | 0 | 52 | 13 |
| BUG:unable_to_handle_kernel | 0 | 52 | 13 |
| Oops | 0 | 52 | 13 |
| RIP:cpu_vm_stats_fold | 0 | 52 | 13 |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 52 | 13 |
| backtrace:debug_hotplug_cpu | 0 | 52 | 13 |
| backtrace:kernel_init_freeable | 0 | 52 | 13 |
+------------------------------------------+------------+------------+------------+
[ 2.795299] Key type trusted registered
[ 2.797960] Key type encrypted registered
[ 2.800424] Unregister pv shared memory for cpu 0
[ 2.847203] BUG: unable to handle kernel paging request at 0000000000016f29
[ 2.849384] IP: [<ffffffff812189da>] cpu_vm_stats_fold+0x10a/0x180
[ 2.851271] PGD 0
[ 2.852450] Oops: 0000 [#1] SMP
[ 2.853953] Modules linked in:
[ 2.855216] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.6.0-rc1-00004-g970df71 #1
[ 2.857632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1
04/01/2014
[ 2.860605] task: ffff8800101c4b40 ti: ffff8800101c8000 task.ti: ffff8800101c8000
[ 2.863090] RIP: 0010:[<ffffffff812189da>] [<ffffffff812189da>]
cpu_vm_stats_fold+0x10a/0x180
[ 2.865892] RSP: 0000:ffff8800101cbbf0 EFLAGS: 00010246
[ 2.867414] RAX: ffff880011529000 RBX: 0000000000000000 RCX: 0000000000000000
[ 2.869252] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffffffff8206d368
[ 2.871096] RBP: ffff8800101cbca0 R08: ffff88001152cc88 R09: 0000000000016f28
[ 2.872898] R10: ffffea00003ca2a0 R11: 0000000000000002 R12: ffff8800101cbc14
[ 2.874682] R13: 0000000000000000 R14: 0000000000000000 R15: 00000000fffffff7
[ 2.876604] FS: 0000000000000000(0000) GS:ffff880011300000(0000)
knlGS:0000000000000000
[ 2.879359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2.881003] CR2: 0000000000016f29 CR3: 0000000001c06000 CR4: 00000000000006e0
[ 2.882868] Stack:
[ 2.883886] 0000000011216f58 0000000000000000 0000000000000000 0000000000000000
[ 2.887027] 0000000800000000 00000000fffffff3 0000000000000000 0000000000000000
[ 2.890299] 0000000100000000 0000000000000008 0000000000000001 0000000000000000
[ 2.893633] Call Trace:
[ 2.894898] [<ffffffff811f4e9f>] page_alloc_cpu_notify+0x4f/0x60
[ 2.896784] [<ffffffff810f011d>] notifier_call_chain+0x5d/0xb0
[ 2.898731] [<ffffffff810f022e>] __raw_notifier_call_chain+0xe/0x10
[ 2.900662] [<ffffffff810c3486>] __cpu_notify+0x36/0x50
[ 2.902482] [<ffffffff810c35aa>] cpu_notify_nofail+0x1a/0x50
[ 2.906200] [<ffffffff810c35e0>] ? cpu_notify_nofail+0x50/0x50
[ 2.908022] [<ffffffff810c3606>] notify_dead+0x26/0x110
[ 2.909880] [<ffffffff810c35e0>] ? cpu_notify_nofail+0x50/0x50
[ 2.913145] [<ffffffff810c3f29>] cpuhp_invoke_callback+0x79/0x1b0
[ 2.914975] [<ffffffff810c4358>] cpuhp_down_callbacks+0x58/0xe0
[ 2.916724] [<ffffffff8179a08f>] _cpu_down+0x19f/0x1f0
[ 2.918560] [<ffffffff810c522e>] do_cpu_down+0x4e/0x70
[ 2.920232] [<ffffffff810c5260>] cpu_down+0x10/0x20
[ 2.921866] [<ffffffff81030340>] _debug_hotplug_cpu+0xa0/0xf0
[ 2.923656] [<ffffffff82090cdb>] ? topology_init+0x9f/0x9f
[ 2.926458] [<ffffffff82090ce8>] debug_hotplug_cpu+0xd/0x11
[ 2.928153] [<ffffffff8100041e>] do_one_initcall+0xde/0x2b0
[ 2.930113] [<ffffffff8208645f>] kernel_init_freeable+0x113/0x1d6
[ 2.932531] [<ffffffff8179945e>] kernel_init+0xe/0x180
[ 2.938010] [<ffffffff817a9de2>] ret_from_fork+0x22/0x40
[ 2.939919] [<ffffffff81799450>] ? rest_init+0x90/0x90
[ 2.941686] Code: 48 89 c7 e8 f9 dc ff ff 48 85 c0 0f 85 7a ff ff ff e8 4b dc ff ff 48
85 c0 74 5f 4c 8b 88 80 3c 00 00 4c 8d 80 88 3c 00 00 31 d2 <41> 0f be 4c 11 01 31
f6 84 c9 40 0f 95 c6 48 83 c6 02 48 83 04
[ 2.954461] RIP [<ffffffff812189da>] cpu_vm_stats_fold+0x10a/0x180
[ 2.956475] RSP <ffff8800101cbbf0>
[ 2.957900] CR2: 0000000000016f29
[ 2.959336] ---[ end trace ba236c6a755d1712 ]---
[ 2.960801] Kernel panic - not syncing: Fatal exception
git bisect start 84051050531602bc1d427bfc27d39fb8e2a6e23f
f55532a0c0b8bb6148f4e07853b876ef73bc69ca --
git bisect bad 58766773d8b58762bea392f9f179e2ea5e602e2a # 22:29 0- 11 Merge
'linux-review/Lars-Peter-Clausen/usb-gadget-f_fs-Fix-EFAULT-generation-for-async-read-operations/20160330-201655'
into devel-catchup-201603302038
git bisect bad 57cb0975edee7a62820dbc2dcbefe7a159687e71 # 22:29 0- 39 Merge
'linux-review/Shubhangi-Shrivastava/drm-i915-Splitting-intel_dp_detect/20160330-203510'
into devel-catchup-201603302038
git bisect good b4fe8cd840d422f83211b03cd9b97c482e159ebf # 22:36 66+ 0 0day
base guard for 'devel-catchup-201603302038'
git bisect bad 46edea8c09c5dada2def7b4558b5d38781c9211a # 22:42 0- 47 Merge
'mel/mm-vmscan-node-lru-v3r10' into devel-catchup-201603302038
git bisect bad 404ff05311b002244746cfbed691f0d4c17a76f3 # 22:48 0- 25 mm,
workingset: Make working set detection node-aware
git bisect bad a6013470b90d9a2c8d3da20b9ee20ec32b42c30b # 22:52 0- 36 mm,
vmscan: Make kswapd reclaim in terms of nodes
git bisect good 7db93e3a253b53eefedfd27fcf391bc2cb02380b # 22:59 66+ 0 mm,
vmscan: Move lru_lock to the node
git bisect bad ac8fbf7357e93ccb5887ff9fb123d5b16bd7504f # 23:04 0- 17 mm,
vmscan: Begin reclaiming pages on a per-node basis
git bisect bad 970df71e432f080bee5548dd6ee6606943ef3236 # 23:09 0- 52 mm,
vmscan: Move LRU lists to node
# first bad commit: [970df71e432f080bee5548dd6ee6606943ef3236] mm, vmscan: Move LRU lists
to node
git bisect good 7db93e3a253b53eefedfd27fcf391bc2cb02380b # 23:12 190+ 0 mm,
vmscan: Move lru_lock to the node
# extra tests with DEBUG_INFO
git bisect bad 970df71e432f080bee5548dd6ee6606943ef3236 # 23:16 0- 7 mm,
vmscan: Move LRU lists to node
# extra tests on HEAD of linux-devel/devel-catchup-201603302038
git bisect bad 84051050531602bc1d427bfc27d39fb8e2a6e23f # 23:16 0- 13 0day
head guard for 'devel-catchup-201603302038'
# extra tests on tree/branch mel/mm-vmscan-node-lru-v3r10
git bisect bad ddb4e8c2992639305e2ecadacb53ba9fe80718bc # 23:37 0- 7 mm:
vmstat: Account per-zone stalls and pages skipped during reclaim
# extra tests on tree/branch linus/master
git bisect good 1993b176a8224e371e0732ffada7ab9eb3b0912b # 23:45 190+ 0 Merge
git://git.kernel.org/pub/scm/linux/kernel/git/davem/ide
# extra tests on tree/branch linux-next/master
git bisect good 67130d02fee17fd2176fff7d49d4f91ccbdede07 # 23:49 188+ 0 Add
linux-next specific files for 20160330
This script may reproduce the error.
----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
initrd=quantal-core-x86_64.cgz
wget --no-clobber
https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd
kvm=(
qemu-system-x86_64
-enable-kvm
-cpu kvm64
-kernel $kernel
-initrd $initrd
-m 300
-smp 2
-device e1000,netdev=net0
-netdev user,id=net0
-boot order=nc
-no-reboot
-watchdog i6300esb
-rtc base=localtime
-serial stdio
-display none
-monitor null
)
append=(
hung_task_panic=1
earlyprintk=ttyS0,115200
systemd.log_level=err
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
drbd.minor_count=8
)
"${kvm[@]}" --append "${append[*]}"
----------------------------------------------------------------------------
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation