[clockevents] WARNING: CPU: 0 PID: 0 at kernel/time/tick-oneshot.c:32 tick_stop_event()
by Jet Chen
Hi Viresh,
0day kernel testing robot got the below dmesg and the first bad commit is
git://git.linaro.org/people/vireshk/linux tick/ONESHOT-STOPPED
commit d5a085970b0b88b8c82303d2c45f09c9427fd02a
Author: Viresh Kumar <viresh.kumar(a)linaro.org>
AuthorDate: Tue Jun 10 13:52:25 2014 +0530
Commit: Viresh Kumar <viresh.kumar(a)linaro.org>
CommitDate: Thu Jun 19 10:35:12 2014 +0530
clockevents: x86: add support for ONESHOT_STOPPED mode
Clockevent devices can be stopped from core (hrtimers/tick) when they aren't
used anymore. This will mostly happen on a idle or tickless cpu. This is done to
avoid getting spurious interrupts when core doesn't want to handle them anymore.
For stopping clockevent devices, core requests drivers to switch to a special
optional mode: CLOCK_EVT_MODE_ONESHOT_STOPPED. At this time platforms can also
save some power by disabling these devices.
This patch adds support for handling CLOCK_EVT_MODE_ONESHOT_STOPPED mode for x86
drivers.
Signed-off-by: Viresh Kumar <viresh.kumar(a)linaro.org>
+-----------------------------------------------------------------+------------+------------+
| | f4f5df6f14 | d5a085970b |
+-----------------------------------------------------------------+------------+------------+
| boot_successes | 120 | 9 |
| boot_failures | 0 | 31 |
| WARNING:CPU:PID:at_kernel/time/tick-oneshot.c:tick_stop_event() | 0 | 31 |
| backtrace:cpu_startup_entry | 0 | 19 |
+-----------------------------------------------------------------+------------+------------+
[ 0.247455] WARNING: CPU: 0 PID: 0 at kernel/time/tick-oneshot.c:32 tick_stop_event+0x43/0x50()
[ 0.247455] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-rc1-00016-gd5a0859 #1
[ 0.247455] 0000000000000009 ffff880012203eb0 ffffffff81658a72 0000000000000000
[ 0.247455] ffff880012203ee8 ffffffff8108c128 ffff88001220ca80 7fffffffffffffff
[ 0.247455] 000000000e82ddd2 7fffffffffffffff ffff8800123cd560 ffff880012203ef8
[ 0.247455] Call Trace:
[ 0.247455] <IRQ> [<ffffffff81658a72>] dump_stack+0x4d/0x66
[ 0.247455] [<ffffffff8108c128>] warn_slowpath_common+0x78/0xa0
[ 0.247455] [<ffffffff8108c23a>] warn_slowpath_null+0x1a/0x20
[ 0.247455] [<ffffffff81101983>] tick_stop_event+0x43/0x50
[ 0.247455] [<ffffffff81102275>] __tick_nohz_idle_enter+0x515/0x580
[ 0.247455] [<ffffffff81102649>] ? tick_nohz_irq_exit+0x29/0x30
[ 0.247455] [<ffffffff81102649>] tick_nohz_irq_exit+0x29/0x30
[ 0.247455] [<ffffffff810913c5>] irq_exit+0xb5/0xd0
[ 0.247455] [<ffffffff81028095>] smp_call_function_single_interrupt+0x35/0x40
[ 0.247455] [<ffffffff81662892>] call_function_single_interrupt+0x72/0x80
[ 0.247455] <EOI> [<ffffffff81032b96>] ? native_safe_halt+0x6/0x10
[ 0.247455] [<ffffffff8100c964>] default_idle+0x24/0xe0
[ 0.247455] [<ffffffff8100d3cf>] arch_cpu_idle+0xf/0x20
[ 0.247455] [<ffffffff810d4f16>] cpu_startup_entry+0x386/0x4f0
[ 0.247455] [<ffffffff81652da6>] rest_init+0xc6/0xd0
[ 0.247455] [<ffffffff81652ce5>] ? rest_init+0x5/0xd0
[ 0.247455] [<ffffffff81f23f96>] start_kernel+0x41e/0x429
[ 0.247455] [<ffffffff81f23981>] ? set_init_arg+0x53/0x53
[ 0.247455] [<ffffffff81f23120>] ? early_idt_handlers+0x120/0x120
[ 0.247455] [<ffffffff81f23549>] x86_64_start_reservations+0x2a/0x2c
[ 0.247455] [<ffffffff81f23679>] x86_64_start_kernel+0x12e/0x13d
git bisect start 1ded43568e7bee60c6548e5fb3b4bd6d1d0783b1 a497c3ba1d97fc69c1e78e7b96435ba8c2cb42ee --
git bisect good 5f069b44108f255ab9a881ff366c793bc1cda5e9 # 14:27 21+ 0 Merge 'ipvs-next/master' into devel-hourly-2014062313
git bisect good 70eab242a357abfe990c30656a396f1b0a0ba465 # 14:33 21+ 0 Merge 'cifs/for-next' into devel-hourly-2014062313
git bisect good 64550da9a486e3766b85928f442756f29c39437a # 14:40 21+ 0 Merge 'rcu/rcu_cond_resched.2014.06.20c' into devel-hourly-2014062313
git bisect bad bf774a68a67dd0c0f938258f56346763d619bdbf # 14:44 0- 6 Merge 'vireshk/tick/lowres-go-tickless' into devel-hourly-2014062313
git bisect bad b1efa076fdd9532ba3fdf0bddb475ed968f33c13 # 14:50 2- 1 Merge 'vireshk/tick/ONESHOT-STOPPED' into devel-hourly-2014062313
git bisect good f935474b892829a400f31ec076385b792034a32e # 15:02 40+ 0 clockevents: Introduce CLOCK_EVT_MODE_ONESHOT_STOPPED mode
git bisect good f4f5df6f142f6797c9aae44d8b570843544dcca9 # 15:12 40+ 0 clockevents: mips: add support for ONESHOT_STOPPED mode
git bisect bad 2d74fb379fa8c9d388fcce15e84772713ac7331d # 15:16 5- 8 clockevents: misc: add support for ONESHOT_STOPPED mode
git bisect bad 6e9ed902730ca7f5cde9fcf47c6b1ca2a143babf # 15:33 1- 14 clockevents: drivers: add support for ONESHOT_STOPPED mode
git bisect bad d5a085970b0b88b8c82303d2c45f09c9427fd02a # 15:43 1- 2 clockevents: x86: add support for ONESHOT_STOPPED mode
# first bad commit: [d5a085970b0b88b8c82303d2c45f09c9427fd02a] clockevents: x86: add support for ONESHOT_STOPPED mode
git bisect good f4f5df6f142f6797c9aae44d8b570843544dcca9 # 15:46 120+ 0 clockevents: mips: add support for ONESHOT_STOPPED mode
git bisect bad 1ded43568e7bee60c6548e5fb3b4bd6d1d0783b1 # 15:46 0- 11 0day head guard for 'devel-hourly-2014062313'
git bisect good a497c3ba1d97fc69c1e78e7b96435ba8c2cb42ee # 15:49 120+ 0 Linux 3.16-rc2
git bisect good 58ae500a03a6bf68eee323c342431bfdd3f460b6 # 15:56 120+ 0 Add linux-next specific files for 20140623
This script may reproduce the error.
-----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
initrd=yocto-minimal-i386.cgz
wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/blob/master/initrd/$initrd
kvm=(
qemu-system-x86_64 -cpu kvm64 -enable-kvm
-kernel $kernel
-initrd $initrd
-smp 2
-m 256M
-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
-net user,vlan=0
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-serial stdio
-display none
-monitor null
)
append=(
debug
sched_debug
apic=debug
ignore_loglevel
sysrq_always_enabled
panic=10
prompt_ramdisk=0
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
)
"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------
Thanks,
Jet
6 years, 7 months
[Device/core] BUG: unable to handle kernel NULL pointer dereference at (null)
by Jet Chen
Hi Tianyu,
0day kernel testing robot got the below dmesg and the first bad commit is
git://bee.sh.intel.com/git/tianyu/usb.git dep_support
commit 6ce060a8738b36c60ad4b2567d393132b2620121
Author: Lan Tianyu <tianyu.lan(a)intel.com>
AuthorDate: Thu Apr 10 13:18:43 2014 +0800
Commit: Lan Tianyu <tianyu.lan(a)intel.com>
CommitDate: Wed Jun 18 10:40:29 2014 +0800
Device/core: Add platform_dev_attach/detach_notify() callback
For ACPI platform, ACPI table provides device probe dependency information
and ACPI subsystem needs to check the driver probing status of dependent
devices. This patch is to add two callbacks to notify platform when device
is attached and detached from driver.
Signed-off-by: Lan Tianyu <tianyu.lan(a)intel.com>
+------------------------------------------------------------+------------+------------+
| | 96e8c1a7c3 | 6ce060a873 |
+------------------------------------------------------------+------------+------------+
| boot_successes | 60 | 0 |
| boot_failures | 0 | 20 |
| BUG:unable_to_handle_kernel_NULL_pointer_dereference | 0 | 20 |
| Oops | 0 | 20 |
| WARNING:CPU:PID:at_kernel/smp.c:smp_call_function_single() | 0 | 20 |
| Kernel_panic-not_syncing:Attempted_to_kill_init_exitcode= | 0 | 20 |
| backtrace:__platform_driver_register | 0 | 20 |
| backtrace:regulator_dummy_init | 0 | 20 |
| backtrace:regulator_init | 0 | 20 |
| backtrace:kernel_init_freeable | 0 | 20 |
+------------------------------------------------------------+------------+------------+
[ 0.180030] prefetch64-sse: 137.000 MB/sec
[ 0.220055] generic_sse: 132.000 MB/sec
[ 0.220691] xor: using function: prefetch64-sse (137.000 MB/sec)
[ 0.222565] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 0.223565] IP: [< (null)>] (null)
[ 0.224000] PGD 0 [ 0.224000] Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 0.224000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.16.0-rc1-00011-g6ce060a #1
[ 0.224000] task: ffff8800128c8000 ti: ffff8800128c4000 task.ti: ffff8800128c4000
[ 0.224000] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 0.224000] RSP: 0000:ffff8800128c7d68 EFLAGS: 00010246
[ 0.224000] RAX: ffff8800128abbe8 RBX: ffff8800128ab7c0 RCX: 000000000000008f
[ 0.224000] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8800128ab7c0
[ 0.224000] RBP: ffff8800128c7d90 R08: 0000000000000002 R09: 000000000011e08f
[ 0.224000] R10: 0000000000000000 R11: 00000000001d2f90 R12: ffffffff827fda98
[ 0.224000] R13: ffffffff827fda98 R14: 0000000000000000 R15: 0000000000000000
[ 0.224000] FS: 0000000000000000(0000) GS:ffff880013600000(0000) knlGS:0000000000000000
[ 0.224000] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 0.224000] CR2: 0000000000000000 CR3: 000000000260d000 CR4: 00000000000006f0
[ 0.224000] Stack:
[ 0.224000] ffffffff816ee1a7 ffff8800128ab7c0 ffffffff827fda98 ffff8800128ab820
[ 0.224000] 0000000000000000 ffff8800128c7db8 ffffffff816ee4c6 0000000000000000
[ 0.224000] ffffffff827fda98 ffffffff816ee431 ffff8800128c7df0 ffffffff816ec084
[ 0.224000] Call Trace:
[ 0.224000] [<ffffffff816ee1a7>] ? driver_probe_device+0x8c/0x316
[ 0.224000] [<ffffffff816ee4c6>] __driver_attach+0x95/0xd3
[ 0.224000] [<ffffffff816ee431>] ? driver_probe_device+0x316/0x316
[ 0.224000] [<ffffffff816ec084>] bus_for_each_dev+0xbe/0xc9
[ 0.224000] [<ffffffff816ee65b>] driver_attach+0x20/0x29
[ 0.224000] [<ffffffff816ecc5d>] bus_add_driver+0x112/0x2ab
[ 0.224000] [<ffffffff816eefe3>] driver_register+0xec/0x152
[ 0.224000] [<ffffffff816f096f>] __platform_driver_register+0x61/0x6a
[ 0.224000] [<ffffffff82c81ca7>] regulator_dummy_init+0xa0/0xde
[ 0.224000] [<ffffffff82c81a2f>] regulator_init+0x90/0x9c
[ 0.224000] [<ffffffff82c8199f>] ? virtio_balloon_driver_init+0x20/0x20
[ 0.224000] [<ffffffff82c486e9>] do_one_initcall+0x171/0x248
[ 0.224000] [<ffffffff810e2f00>] ? parameq+0x6/0x2d
[ 0.224000] [<ffffffff810e31bd>] ? parse_args+0x296/0x3e8
[ 0.224000] [<ffffffff82c48aa9>] kernel_init_freeable+0x2e9/0x3e8
[ 0.224000] [<ffffffff82c47911>] ? do_early_param+0xd7/0xd7
[ 0.224000] [<ffffffff81caf6a1>] ? rest_init+0x115/0x115
[ 0.224000] [<ffffffff81caf6b1>] kernel_init+0x10/0x175
[ 0.224000] [<ffffffff81cd533c>] ret_from_fork+0x7c/0xb0
[ 0.224000] [<ffffffff81caf6a1>] ? rest_init+0x115/0x115
[ 0.224000] Code: Bad RIP value.
[ 0.224000] RIP [< (null)>] (null)
[ 0.224000] RSP <ffff8800128c7d68>
[ 0.224000] CR2: 0000000000000000
[ 0.224000] ---[ end trace d3295e197e7ac925 ]---
[ 0.224000] ------------[ cut here ]------------
git bisect start aafca41bf8b2c093befa74e04c59580a4401b8e9 a497c3ba1d97fc69c1e78e7b96435ba8c2cb42ee --
git bisect good 5bb41e68bb216e78a850319a64b549c0ceaeddf8 # 15:15 20+ 0 Merge 'ipvs-next/master' into devel-hourly-2014062314
git bisect good 8c804d4bb30cfef4a4af0f561d6ef31e84014ad5 # 15:23 20+ 0 Merge 'asoc/topic/intel' into devel-hourly-2014062314
git bisect good b3b5fba3e162ab85c6b32b5425074cce21fbdfbc # 15:28 20+ 0 Merge 'block/for-linus' into devel-hourly-2014062314
git bisect bad 07e8581d455a832a85e7917a9601aa1aa4b30965 # 15:29 0- 20 Merge 'tianyu/dep_support' into devel-hourly-2014062314
git bisect good 4033d5f4a151180ad86e6b0b45760f872b07f3e3 # 15:34 20+ 0 Merge 'net-next/master' into devel-hourly-2014062314
git bisect good ca2bb6349593d783b7524e550d468d3ba259d62b # 15:40 20+ 0 I2C/ACPI: Add i2c ACPI operation region support
git bisect bad f93d3e49ea1609fe7c7828a94b16619ff68c4f27 # 15:43 0- 20 ACPI: Resolve battery probe issue on the T100TA
git bisect bad 6ce060a8738b36c60ad4b2567d393132b2620121 # 15:46 0- 20 Device/core: Add platform_dev_attach/detach_notify() callback
git bisect good 96e8c1a7c36ba7e14644aba544283ff4dfa2bff8 # 15:50 20+ 0 I2C/ACPI: Clean up I2C ACPI code and Add CONFIG_I2C_ACPI config
# first bad commit: [6ce060a8738b36c60ad4b2567d393132b2620121] Device/core: Add platform_dev_attach/detach_notify() callback
git bisect good 96e8c1a7c36ba7e14644aba544283ff4dfa2bff8 # 15:52 60+ 0 I2C/ACPI: Clean up I2C ACPI code and Add CONFIG_I2C_ACPI config
git bisect bad aafca41bf8b2c093befa74e04c59580a4401b8e9 # 15:52 0- 13 0day head guard for 'devel-hourly-2014062314'
git bisect good a497c3ba1d97fc69c1e78e7b96435ba8c2cb42ee # 15:56 60+ 0 Linux 3.16-rc2
git bisect good 58ae500a03a6bf68eee323c342431bfdd3f460b6 # 16:03 60+ 0 Add linux-next specific files for 20140623
This script may reproduce the error.
-----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
kvm=(
qemu-system-x86_64 -cpu kvm64 -enable-kvm
-kernel $kernel
-smp 2
-m 256M
-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
-net user,vlan=0
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-serial stdio
-display none
-monitor null
)
append=(
debug
sched_debug
apic=debug
ignore_loglevel
sysrq_always_enabled
panic=10
prompt_ramdisk=0
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
)
"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------
Thanks,
Jet
6 years, 7 months
[tracing] 939c7a4f04f: -46.4% cpuidle.C3-IVT.time
by Jet Chen
Hi Yoshihiro,
FYI, we noticed the below changes on
commit 939c7a4f04fcd2162109744e8bf88194948a6e65 ("tracing: Introduce saved_cmdlines_size file")
test case: brickland3/aim7/3000-brk_test
beba4bb096201ce 939c7a4f04fcd2162109744e8
--------------- -------------------------
352643 ~40% -46.4% 189136 ~15% TOTAL cpuidle.C3-IVT.time
750 ~ 2% -13.3% 650 ~ 2% TOTAL vmstat.procs.b
362016 ~ 3% -6.0% 340296 ~ 3% TOTAL proc-vmstat.pgmigrate_success
362016 ~ 3% -6.0% 340296 ~ 3% TOTAL proc-vmstat.numa_pages_migrated
1642328 ~ 3% +13.3% 1860721 ~ 2% TOTAL softirqs.RCU
2278829 ~ 2% -14.0% 1959647 ~ 2% TOTAL time.voluntary_context_switches
207 ~ 0% -10.4% 185 ~ 0% TOTAL time.user_time
~ 0% +8.5% ~ 0% TOTAL turbostat.RAM_W
4418246 ~ 3% -6.4% 4136447 ~ 1% TOTAL time.minor_page_faults
~ 0% +3.4% ~ 0% TOTAL turbostat.Cor_W
~ 0% +3.0% ~ 0% TOTAL turbostat.Pkg_W
11354889 ~ 0% +2.0% 11579897 ~ 0% TOTAL time.involuntary_context_switches
Legend:
~XX% - stddev percent
[+-]XX% - change percent
time.voluntary_context_switches
2.4e+06 ++---------------------------------------------------*------------+
*.. *.. .. .. |
2.3e+06 ++ .*.. *.. .. * |
| *..*. . .*.. .. * *.. + *.. *..*
| *. * . .*..*.. + .. |
2.2e+06 ++ *..*. * * |
| |
2.1e+06 ++ |
| |
2e+06 O+ O O O |
| O O O O |
| O O O O O |
1.9e+06 ++ O O O O |
| O O O |
1.8e+06 ++----------------------------------------------------------------+
turbostat.Pkg_W
++--------------------------------------------------------------------+
O+ O O O O O |
| O O O O O O O O O O O O |
++ O O |
++ |
++ |
++ |
| |
++ |
++ |
++ *.. .*.. |
*+.*... .. *...*. *...*..*..*...*..*.. ..*.. *...*..*..*...*..*
| * *. .. |
++ * |
++--------------------------------------------------------------------+
turbostat.Cor_W
++--------------------------------------------------------------------+
O+ O O O O O O O O O |
| O O O O O O O O O O |
++ |
++ |
| |
++ |
++ |
++ |
| |
++ *.. |
*+.*... .. *...*..*..*...*..*..*...*..*.. ..*.. *... .*..*...*..*
| * *. .. *. |
++ * |
++--------------------------------------------------------------------+
turbostat.RAM_W
++---------------------------------------------------------------------+
| O O O O O O O O |
O+ O O O O O O O O O O |
| O |
++ |
| |
++ |
| |
++ |
| * |
++ .. + *.. |
| .*..* + *.. *.. + ..*..*.. .*... |
++ .. + .. .. . + *. *...*. *..*
*..*...*..* * *...* * |
++---------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Jet
6 years, 7 months
[net] b58537a1f56: +89.2% netperf.Throughput_Mbps
by Jet Chen
Hi Daniel,
FYI, we noticed the below changes on
commit b58537a1f5629bdc98a8b9dc2051ce0e952f6b4b ("net: sctp: fix permissions for rto_alpha and rto_beta knobs")
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
950 ~ 1% +93.7% 1841 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
750 ~ 1% +83.4% 1375 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1700 ~ 1% +89.2% 3217 ~ 0% TOTAL netperf.Throughput_Mbps
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
7841 ~ 0% +50.4% 11792 ~ 0% lkp-nex04/netperf/300s-200%-SCTP_RR
7603 ~ 0% +37.6% 10463 ~ 0% lkp-nex05/netperf/300s-200%-SCTP_RR
15445 ~ 0% +44.1% 22256 ~ 0% TOTAL netperf.Throughput_tps
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
5.93 ~ 1% -99.0% 0.06 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
6.53 ~ 1% -99.1% 0.06 ~12% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
12.46 ~ 1% -99.1% 0.12 ~ 6% TOTAL perf-profile.cpu-cycles.sctp_get_af_specific.sctp_cmp_addr_exact.sctp_assoc_lookup_paddr.sctp_endpoint_lookup_assoc.sctp_sendmsg
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
6.35 ~ 0% -98.3% 0.11 ~ 3% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.95 ~ 5% -100.0% 0.00 ~ 0% lkp-nex04/netperf/300s-200%-SCTP_RR
2.26 ~ 6% -100.0% 0.00 ~ 0% lkp-nex05/netperf/300s-200%-SCTP_RR
6.68 ~ 0% -98.5% 0.10 ~10% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
18.23 ~ 1% -98.9% 0.21 ~ 7% TOTAL perf-profile.cpu-cycles.sctp_get_af_specific.sctp_chunk_iif.sctp_ulpevent_make_rcvmsg.sctp_ulpq_tail_data.sctp_cmd_interpreter
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
3.07 ~ 0% -99.0% 0.03 ~12% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
4.84 ~ 0% -100.0% 0.00 ~ 0% lkp-nex04/netperf/300s-200%-SCTP_RR
3.92 ~ 5% -100.0% 0.00 ~ 0% lkp-nex05/netperf/300s-200%-SCTP_RR
3.17 ~ 0% -99.0% 0.03 ~12% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
15.01 ~ 1% -99.6% 0.06 ~12% TOTAL perf-profile.cpu-cycles.sctp_get_af_specific.sctp_cmp_addr_exact.sctp_assoc_lookup_paddr.sctp_assoc_is_match.__sctp_lookup_association
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.92 ~ 1% -97.9% 0.04 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
3.78 ~ 5% -99.3% 0.03 ~18% lkp-nex04/netperf/300s-200%-SCTP_RR
2.91 ~ 5% -99.0% 0.03 ~14% lkp-nex05/netperf/300s-200%-SCTP_RR
2.00 ~ 1% -98.2% 0.04 ~13% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
10.61 ~ 3% -98.8% 0.13 ~10% TOTAL perf-profile.cpu-cycles.sctp_get_af_specific.sctp_eat_data.sctp_sf_eat_data_6_2.sctp_do_sm.sctp_assoc_bh_rcv
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
3.36 ~ 0% -97.1% 0.10 ~ 4% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
6.16 ~ 0% -99.3% 0.04 ~23% lkp-nex04/netperf/300s-200%-SCTP_RR
4.91 ~ 5% -98.9% 0.05 ~22% lkp-nex05/netperf/300s-200%-SCTP_RR
3.47 ~ 1% -97.4% 0.09 ~ 7% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
17.90 ~ 2% -98.4% 0.28 ~11% TOTAL perf-profile.cpu-cycles.sctp_get_af_specific.sctp_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
6.94 ~ 0% -97.3% 0.18 ~ 4% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
7.54 ~ 0% -97.6% 0.18 ~10% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
14.48 ~ 0% -97.5% 0.36 ~ 7% TOTAL perf-profile.cpu-cycles.sctp_get_af_specific.sctp_sockaddr_af.sctp_sendmsg.inet_sendmsg.sock_sendmsg
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
6207 ~ 0% +273.8% 23202 ~36% lkp-nex05/netperf/300s-200%-SCTP_RR
6207 ~ 0% +273.8% 23202 ~36% TOTAL numa-vmstat.node3.nr_active_file
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
24831 ~ 0% +273.8% 92810 ~36% lkp-nex05/netperf/300s-200%-SCTP_RR
24831 ~ 0% +273.8% 92810 ~36% TOTAL numa-meminfo.node3.Active(file)
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
27053 ~ 3% +237.7% 91368 ~10% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
606 ~26% -68.2% 192 ~11% lkp-nex05/netperf/300s-200%-SCTP_RR
83164 ~14% +322.9% 351725 ~26% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
110824 ~12% +300.0% 443286 ~23% TOTAL cpuidle.C1E-NHM.usage
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
16056 ~ 8% +278.8% 60825 ~ 6% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
56085 ~13% +177.5% 155640 ~12% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
72141 ~12% +200.1% 216465 ~10% TOTAL cpuidle.C1-NHM.usage
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
2.96 ~ 0% +28.7% 3.81 ~ 4% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
4.55 ~ 2% -86.3% 0.62 ~ 7% lkp-nex04/netperf/300s-200%-SCTP_RR
3.73 ~ 4% -88.3% 0.44 ~ 2% lkp-nex05/netperf/300s-200%-SCTP_RR
3.04 ~ 0% +65.8% 5.04 ~ 1% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
14.29 ~ 2% -30.6% 9.91 ~ 2% TOTAL perf-profile.cpu-cycles.lock_timer_base.isra.34.mod_timer.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
20838 ~ 3% -42.8% 11916 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
62995 ~10% -78.7% 13392 ~10% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
83834 ~ 8% -69.8% 25309 ~ 6% TOTAL cpuidle.C3-NHM.usage
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
2.21 ~ 1% +22.3% 2.70 ~ 6% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.57 ~ 4% -86.5% 0.21 ~11% lkp-nex04/netperf/300s-200%-SCTP_RR
1.29 ~ 6% -89.2% 0.14 ~ 4% lkp-nex05/netperf/300s-200%-SCTP_RR
2.30 ~ 1% +64.5% 3.79 ~ 2% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
7.38 ~ 2% -7.3% 6.84 ~ 4% TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.mod_timer.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
6.74 ~ 0% +17.9% 7.94 ~ 5% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
5.93 ~ 0% -79.4% 1.22 ~ 4% lkp-nex04/netperf/300s-200%-SCTP_RR
4.78 ~ 6% -80.4% 0.94 ~ 2% lkp-nex05/netperf/300s-200%-SCTP_RR
6.84 ~ 0% +55.5% 10.63 ~ 1% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
24.28 ~ 1% -14.6% 20.73 ~ 3% TOTAL perf-profile.cpu-cycles.sctp_transport_timeout.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.86 ~ 4% +172.3% 2.34 ~ 4% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.09 ~ 9% +207.7% 3.34 ~ 5% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.94 ~ 7% +192.1% 5.68 ~ 5% TOTAL perf-profile.cpu-cycles._raw_spin_lock.free_one_page.__free_pages_ok.__free_pages.__free_kmem_pages
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
43260 ~ 2% +158.2% 111705 ~28% lkp-nex05/netperf/300s-200%-SCTP_RR
43260 ~ 2% +158.2% 111705 ~28% TOTAL numa-meminfo.node3.Active
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.439e+08 ~ 1% +94.1% 2.793e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.434e+08 ~ 2% +81.3% 2.599e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.872e+08 ~ 1% +87.7% 5.392e+08 ~ 0% TOTAL numa-numastat.node3.local_node
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.439e+08 ~ 1% +94.1% 2.793e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.434e+08 ~ 2% +81.3% 2.599e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.873e+08 ~ 1% +87.7% 5.392e+08 ~ 0% TOTAL numa-numastat.node3.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
52800 ~ 7% -53.7% 24464 ~ 3% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
52800 ~ 7% -53.7% 24464 ~ 3% TOTAL cpuidle.C6-NHM.usage
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.18 ~ 4% +93.1% 2.28 ~ 5% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.38 ~ 6% +141.2% 3.32 ~ 6% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.56 ~ 5% +119.0% 5.61 ~ 5% TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.alloc_kmem_pages_node.kmalloc_large_node
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
33854097 ~ 2% +111.8% 71718667 ~ 1% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
56631220 ~ 2% +99.7% 1.131e+08 ~ 1% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
90485318 ~ 2% +104.2% 1.848e+08 ~ 1% TOTAL proc-vmstat.pgalloc_dma32
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71085188 ~ 1% +93.5% 1.375e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
165875 ~31% +68.2% 279079 ~33% lkp-nex05/netperf/300s-200%-SCTP_RR
70852917 ~ 2% +80.5% 1.279e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.421e+08 ~ 1% +87.0% 2.657e+08 ~ 0% TOTAL numa-vmstat.node3.numa_local
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
568 ~16% +42.3% 809 ~29% lkp-nex04/netperf/300s-200%-SCTP_RR
568 ~16% +42.3% 809 ~29% TOTAL numa-meminfo.node1.Mlocked
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.452e+08 ~ 1% +93.1% 2.804e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.426e+08 ~ 1% +85.0% 2.638e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.878e+08 ~ 1% +89.1% 5.442e+08 ~ 0% TOTAL numa-numastat.node0.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.452e+08 ~ 1% +93.1% 2.804e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.426e+08 ~ 1% +85.0% 2.638e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.878e+08 ~ 1% +89.1% 5.442e+08 ~ 0% TOTAL numa-numastat.node0.local_node
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
5351090 ~19% +66.0% 8883481 ~14% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
7254502 ~10% +103.9% 14794044 ~16% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
12605592 ~14% +87.8% 23677525 ~15% TOTAL cpuidle.C1-NHM.time
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.446e+08 ~ 1% +93.2% 2.794e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.414e+08 ~ 0% +84.4% 2.608e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.86e+08 ~ 1% +88.9% 5.402e+08 ~ 0% TOTAL numa-numastat.node1.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.446e+08 ~ 1% +93.2% 2.794e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.414e+08 ~ 0% +84.4% 2.608e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.86e+08 ~ 1% +88.9% 5.402e+08 ~ 0% TOTAL numa-numastat.node1.local_node
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
3.043e+09 ~ 1% +93.6% 5.892e+09 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
3.001e+09 ~ 1% +83.4% 5.504e+09 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
6.044e+09 ~ 1% +88.5% 1.14e+10 ~ 0% TOTAL proc-vmstat.pgfree
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
5.777e+08 ~ 1% +93.5% 1.118e+09 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
5.698e+08 ~ 1% +83.3% 1.044e+09 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.147e+09 ~ 1% +88.4% 2.162e+09 ~ 0% TOTAL proc-vmstat.numa_local
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
5.777e+08 ~ 1% +93.5% 1.118e+09 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
5.698e+08 ~ 1% +83.3% 1.044e+09 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.148e+09 ~ 1% +88.4% 2.162e+09 ~ 0% TOTAL proc-vmstat.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71917522 ~ 1% +92.1% 1.382e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
70494566 ~ 0% +84.2% 1.298e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.424e+08 ~ 1% +88.2% 2.68e+08 ~ 0% TOTAL numa-vmstat.node0.numa_local
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71921559 ~ 1% +92.1% 1.382e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
70498641 ~ 0% +84.2% 1.298e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.424e+08 ~ 1% +88.2% 2.68e+08 ~ 0% TOTAL numa-vmstat.node0.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
3.009e+09 ~ 1% +93.4% 5.82e+09 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.945e+09 ~ 1% +83.1% 5.391e+09 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
5.954e+09 ~ 1% +88.3% 1.121e+10 ~ 0% TOTAL proc-vmstat.pgalloc_normal
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71389128 ~ 1% +92.7% 1.376e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
69896469 ~ 0% +83.3% 1.281e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.413e+08 ~ 1% +88.1% 2.657e+08 ~ 0% TOTAL numa-vmstat.node1.numa_local
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71445543 ~ 1% +92.7% 1.376e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
69943098 ~ 0% +83.2% 1.282e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.414e+08 ~ 1% +88.0% 2.658e+08 ~ 0% TOTAL numa-vmstat.node1.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71144941 ~ 1% +93.4% 1.376e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
244384 ~21% +45.4% 355361 ~26% lkp-nex05/netperf/300s-200%-SCTP_RR
70900937 ~ 2% +80.4% 1.279e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.423e+08 ~ 1% +86.8% 2.659e+08 ~ 0% TOTAL numa-vmstat.node3.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.44e+08 ~ 1% +93.7% 2.789e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.424e+08 ~ 3% +82.4% 2.597e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.864e+08 ~ 2% +88.1% 5.386e+08 ~ 0% TOTAL numa-numastat.node2.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.44e+08 ~ 1% +93.7% 2.789e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.424e+08 ~ 3% +82.4% 2.597e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.864e+08 ~ 2% +88.1% 5.386e+08 ~ 0% TOTAL numa-numastat.node2.local_node
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71218513 ~ 1% +92.8% 1.373e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
70566629 ~ 3% +81.1% 1.278e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.418e+08 ~ 2% +87.0% 2.652e+08 ~ 0% TOTAL numa-vmstat.node2.numa_local
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
71277689 ~ 1% +92.8% 1.374e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
70612117 ~ 3% +81.1% 1.279e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.419e+08 ~ 2% +86.9% 2.653e+08 ~ 0% TOTAL numa-vmstat.node2.numa_hit
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.049e+08 ~ 1% +93.7% 2.031e+08 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
3.011e+08 ~ 0% +50.4% 4.528e+08 ~ 0% lkp-nex04/netperf/300s-200%-SCTP_RR
2.92e+08 ~ 0% +37.6% 4.018e+08 ~ 0% lkp-nex05/netperf/300s-200%-SCTP_RR
1.034e+08 ~ 1% +83.4% 1.897e+08 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
8.014e+08 ~ 0% +55.7% 1.247e+09 ~ 0% TOTAL softirqs.NET_RX
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
144 ~16% +40.9% 203 ~29% lkp-nex04/netperf/300s-200%-SCTP_RR
234 ~15% -16.4% 196 ~14% lkp-nex05/netperf/300s-200%-SCTP_RR
379 ~15% +5.5% 400 ~21% TOTAL numa-vmstat.node1.nr_unevictable
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
578 ~16% +41.0% 815 ~29% lkp-nex04/netperf/300s-200%-SCTP_RR
938 ~15% -16.4% 784 ~14% lkp-nex05/netperf/300s-200%-SCTP_RR
1516 ~15% +5.5% 1600 ~21% TOTAL numa-meminfo.node1.Unevictable
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
4.99 ~ 0% +58.4% 7.90 ~ 1% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
4.43 ~ 1% +55.3% 6.87 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
9.41 ~ 1% +56.9% 14.77 ~ 1% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg.inet_sendmsg
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
156328 ~ 1% +45.6% 227560 ~15% lkp-nex05/netperf/300s-200%-SCTP_RR
156328 ~ 1% +45.6% 227560 ~15% TOTAL numa-meminfo.node3.FilePages
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
39081 ~ 1% +45.6% 56889 ~15% lkp-nex05/netperf/300s-200%-SCTP_RR
39081 ~ 1% +45.6% 56889 ~15% TOTAL numa-vmstat.node3.nr_file_pages
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.09 ~12% +42.2% 1.55 ~ 2% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.05 ~16% +43.8% 1.51 ~ 6% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.14 ~14% +43.0% 3.06 ~ 4% TOTAL perf-profile.cpu-cycles.get_page_from_freelist.__alloc_pages_nodemask.alloc_kmem_pages_node.kmalloc_large_node.__kmalloc_node_track_caller
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
11745167 ~15% +66.1% 19514308 ~22% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
11745167 ~15% +66.1% 19514308 ~22% TOTAL cpuidle.C1E-NHM.time
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
3.95 ~ 1% +47.3% 5.81 ~ 1% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
3.44 ~ 2% +47.2% 5.06 ~ 1% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
7.39 ~ 1% +47.3% 10.88 ~ 1% TOTAL perf-profile.cpu-cycles.memcpy.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter.sctp_do_sm
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
2.14 ~ 1% +44.7% 3.09 ~ 1% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.84 ~ 3% +42.7% 2.63 ~ 2% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
3.98 ~ 2% +43.7% 5.72 ~ 1% TOTAL perf-profile.cpu-cycles.memcpy.sctp_packet_transmit_chunk.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
5.27 ~ 0% +44.6% 7.62 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
4.71 ~ 1% +39.0% 6.55 ~ 1% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
9.98 ~ 1% +42.0% 14.17 ~ 1% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.skb_copy_datagram_iovec.sctp_recvmsg.sock_common_recvmsg.sock_recvmsg
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.92 ~ 3% +44.9% 1.34 ~ 3% lkp-nex04/netperf/300s-200%-SCTP_RR
0.94 ~ 3% +34.3% 1.26 ~ 4% lkp-nex05/netperf/300s-200%-SCTP_RR
1.86 ~ 3% +39.5% 2.60 ~ 3% TOTAL perf-profile.cpu-cycles.sctp_recvmsg
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.64 ~ 3% +40.8% 0.90 ~ 8% lkp-nex04/netperf/300s-200%-SCTP_RR
0.64 ~ 3% +40.8% 0.90 ~ 8% TOTAL perf-profile.cpu-cycles.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.10 ~ 2% +24.8% 1.37 ~ 3% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1.10 ~ 3% +55.9% 1.72 ~ 2% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
2.20 ~ 2% +40.4% 3.09 ~ 2% TOTAL perf-profile.cpu-cycles.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.81 ~12% +26.8% 1.03 ~ 2% lkp-nex04/netperf/300s-200%-SCTP_RR
0.81 ~12% +26.8% 1.03 ~ 2% TOTAL perf-profile.cpu-cycles.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.94 ~ 3% +40.4% 1.31 ~ 1% lkp-nex04/netperf/300s-200%-SCTP_RR
1.03 ~ 5% +35.5% 1.39 ~ 4% lkp-nex05/netperf/300s-200%-SCTP_RR
1.96 ~ 4% +37.8% 2.70 ~ 2% TOTAL perf-profile.cpu-cycles.__schedule.schedule.schedule_timeout.sctp_recvmsg.sock_common_recvmsg
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.72 ~11% +33.1% 0.96 ~ 2% lkp-nex05/netperf/300s-200%-SCTP_RR
0.72 ~11% +33.1% 0.96 ~ 2% TOTAL perf-profile.cpu-cycles.idle_cpu.select_task_rq_fair.try_to_wake_up.default_wake_function.autoremove_wake_function
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1.14 ~ 7% +36.1% 1.55 ~ 7% lkp-nex04/netperf/300s-200%-SCTP_RR
1.21 ~ 6% +37.5% 1.66 ~ 4% lkp-nex05/netperf/300s-200%-SCTP_RR
2.34 ~ 7% +36.8% 3.20 ~ 6% TOTAL perf-profile.cpu-cycles.sctp_sendmsg.inet_sendmsg.sock_sendmsg.___sys_sendmsg.__sys_sendmsg
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.73 ~ 7% +35.4% 0.99 ~ 2% lkp-nex04/netperf/300s-200%-SCTP_RR
0.75 ~ 4% +32.3% 0.99 ~ 2% lkp-nex05/netperf/300s-200%-SCTP_RR
1.48 ~ 6% +33.8% 1.98 ~ 2% TOTAL perf-profile.cpu-cycles.select_task_rq_fair.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.76 ~ 9% +38.9% 1.06 ~11% lkp-nex04/netperf/300s-200%-SCTP_RR
0.85 ~ 4% +33.4% 1.13 ~ 5% lkp-nex05/netperf/300s-200%-SCTP_RR
1.61 ~ 6% +36.0% 2.19 ~ 8% TOTAL perf-profile.cpu-cycles.sctp_check_transmitted.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1897 ~18% +27.9% 2427 ~11% lkp-nex04/netperf/300s-200%-SCTP_RR
1803 ~ 7% +16.4% 2099 ~ 4% lkp-nex05/netperf/300s-200%-SCTP_RR
3700 ~12% +22.3% 4527 ~ 8% TOTAL proc-vmstat.pgmigrate_success
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1897 ~18% +27.9% 2427 ~11% lkp-nex04/netperf/300s-200%-SCTP_RR
1803 ~ 7% +16.4% 2099 ~ 4% lkp-nex05/netperf/300s-200%-SCTP_RR
3700 ~12% +22.3% 4527 ~ 8% TOTAL proc-vmstat.numa_pages_migrated
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
0.63 ~ 9% -14.2% 0.54 ~14% lkp-nex04/netperf/300s-200%-SCTP_RR
0.63 ~ 9% -14.2% 0.54 ~14% TOTAL turbostat.%c1
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
3097 ~ 3% +19.3% 3695 ~ 8% lkp-nex05/netperf/300s-200%-SCTP_RR
3097 ~ 3% +19.3% 3695 ~ 8% TOTAL numa-vmstat.node3.nr_slab_reclaimable
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
12389 ~ 3% +19.3% 14780 ~ 8% lkp-nex05/netperf/300s-200%-SCTP_RR
12389 ~ 3% +19.3% 14780 ~ 8% TOTAL numa-meminfo.node3.SReclaimable
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
10788 ~ 0% +19.0% 12843 ~ 9% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
10788 ~ 0% +19.0% 12843 ~ 9% TOTAL numa-meminfo.node0.SReclaimable
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
2696 ~ 0% +19.0% 3210 ~ 9% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
2696 ~ 0% +19.0% 3210 ~ 9% TOTAL numa-vmstat.node0.nr_slab_reclaimable
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
634073 ~ 0% +12.3% 712155 ~ 5% lkp-nex05/netperf/300s-200%-SCTP_RR
634073 ~ 0% +12.3% 712155 ~ 5% TOTAL numa-meminfo.node3.MemUsed
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
727 ~ 4% +12.1% 816 ~ 4% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
727 ~ 4% +12.1% 816 ~ 4% TOTAL slabinfo.dnotify_mark.num_objs
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
727 ~ 4% +12.1% 816 ~ 4% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
727 ~ 4% +12.1% 816 ~ 4% TOTAL slabinfo.dnotify_mark.active_objs
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
121677 ~ 5% -6.7% 113473 ~ 5% lkp-nex04/netperf/300s-200%-SCTP_RR
121677 ~ 5% -6.7% 113473 ~ 5% TOTAL meminfo.DirectMap4k
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
1541914 ~ 3% -10.5% 1380345 ~ 2% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
1541914 ~ 3% -10.5% 1380345 ~ 2% TOTAL softirqs.RCU
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
37167 ~ 3% +10.3% 41005 ~ 2% lkp-nex05/netperf/300s-200%-SCTP_RR
37167 ~ 3% +10.3% 41005 ~ 2% TOTAL numa-meminfo.node3.Slab
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
93363 ~ 0% -9.2% 84779 ~ 1% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
93363 ~ 0% -9.2% 84779 ~ 1% TOTAL proc-vmstat.pgactivate
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
7017 ~ 3% +15.3% 8091 ~14% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
7017 ~ 3% +15.3% 8091 ~14% TOTAL numa-meminfo.node0.KernelStack
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
174505 ~ 1% +92.8% 336502 ~ 0% lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY
1942254 ~ 1% +51.6% 2944825 ~ 0% lkp-nex04/netperf/300s-200%-SCTP_RR
1904295 ~ 0% +37.7% 2621695 ~ 0% lkp-nex05/netperf/300s-200%-SCTP_RR
173470 ~ 1% +82.5% 316560 ~ 0% lkp-wsx02/netperf/300s-200%-10K-SCTP_STREAM_MANY
4194525 ~ 0% +48.3% 6219584 ~ 0% TOTAL vmstat.system.cs
e4f7ae930afafd4 b58537a1f5629bdc98a8b9dc2
--------------- -------------------------
66725 ~ 0% +3.8% 69259 ~ 0% lkp-nex04/netperf/300s-200%-SCTP_RR
67344 ~ 0% +2.5% 69040 ~ 0% lkp-nex05/netperf/300s-200%-SCTP_RR
134069 ~ 0% +3.2% 138300 ~ 0% TOTAL vmstat.system.in
Legend:
~XX% - stddev percent
[+-]XX% - change percent
netperf.Throughput_Mbps
1900 O+------O------------------------------------------------------------+
1800 ++ O O O O O O O O O O O O O O O O O |
| |
1700 ++ |
1600 ++ |
| |
1500 ++ |
1400 ++ |
1300 ++ |
| |
1200 ++ |
1100 ++ |
| |
1000 *+. .*..*..*.*..*..*. .*..*.*..*..*.*..*. .*..*. .*..*.*..*..*. .*
900 ++-*-----------------*--------------------*-------*---------------*--+
vmstat.system.cs
360000 ++-----------------------------------------------------------------+
340000 O+ O O O O |
| O O O O O O O O O O O O O O |
320000 ++ |
300000 ++ |
| |
280000 ++ |
260000 ++ |
240000 ++ |
| |
220000 ++ |
200000 ++ |
| |
180000 *+.*.*..*.*..*.*..*..*.*..*.*..*..*.*..*.*..*.*..*..*.*..*.*..*.*..*
160000 ++-----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Jet
6 years, 7 months
[HZ] INFO: task kworker/0:1:16 blocked for more than 120 seconds.
by Jet Chen
Hi Maarten,
0day kernel testing robot got the below dmesg and the first bad commit is
git://people.freedesktop.org/~mlankhorst/linux master
commit 52774d9273d6b8aff39822ccfe10e5a0d7c3a7f8
Author: Maarten Lankhorst <m.b.lankhorst(a)gmail.com>
AuthorDate: Thu Jan 16 16:24:04 2014 +0100
Commit: Maarten Lankhorst <maarten.lankhorst(a)canonical.com>
CommitDate: Wed Jun 18 14:19:52 2014 +0200
There's some really badly broken software out there that is entirely
dependant on HZ for its maximum performance. Raise the maximum HZ value
to some higher and slightly unreasonable values up to some higher and
completely obscene values.
-ck
+--------------------------------------------------+------------+------------+
| | c03339ffa3 | 52774d9273 |
+--------------------------------------------------+------------+------------+
| boot_successes | 60 | 0 |
| boot_failures | 0 | 20 |
| INFO:task_blocked_for_more_than_seconds | 0 | 20 |
| RIP:flat_send_IPI_mask | 0 | 20 |
| Kernel_panic-not_syncing:hung_task:blocked_tasks | 0 | 20 |
| backtrace:serio_handle_event | 0 | 20 |
| backtrace:watchdog | 0 | 20 |
+--------------------------------------------------+------------+------------+
[ 69.176969] torture thread lock_torture_writer parking due to system shutdown
[ 120.553588] spin_lock-torture: Writes: Total: 2 Max/Min: 0/0 Fail: 0 [ 120.554375] torture thread lock_torture_stats parking due to system shutdown
[ 240.031059] INFO: task kworker/0:1:16 blocked for more than 120 seconds.
[ 240.031837] Not tainted 3.16.0-rc1-00045-g52774d9 #1
[ 240.032443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 240.033280] kworker/0:1 D ffff880013612580 0 16 2 0x00000000
git bisect start 91283063c4f74fb202ac7809a8df3cc00f22203e 7171511eaec5bf23fb06078f59784a3a0626b38f --
git bisect good bc106bf69fdfba649ad98b3c7cfd2b63e84431a8 # 01:57 20+ 0 Merge 'dynticks/timers/missing-kick-v2' into devel-hourly-2014061900
git bisect bad 25f92e043e7e4ca029592e03a2d77a09f226f395 # 02:15 0- 1 Merge 'at91/abe/at91_3.16-rc1_CCF' into devel-hourly-2014061900
git bisect good 6c815b36fd13fd40f3545c9e6e33eb7d4b396d8d # 02:18 20+ 0 Merge 'efi/next' into devel-hourly-2014061900
git bisect good a235b89e0b1332b44baff6277b5dbc711df34b3d # 02:25 20+ 0 Merge 'kvm-ppc/kvm-ppc-queue' into devel-hourly-2014061900
git bisect bad 8b4870846e71960a223e729e3c18942edfe81f16 # 02:44 0- 4 Merge 'spi/for-next' into devel-hourly-2014061900
git bisect bad 2115cd10081ea65213b45d9970aeebe88dabcfb3 # 02:55 0- 5 Merge 'mlankhorst/master' into devel-hourly-2014061900
git bisect bad 52774d9273d6b8aff39822ccfe10e5a0d7c3a7f8 # 03:00 0- 4 There's some really badly broken software out there that is entirely
git bisect good 0da2b800ec8dfad78c888b33be2ad2aea3411796 # 03:06 20+ 0 Merge branch 'for-nouveau'
git bisect good c03339ffa3d9d2330c8c14b8d73e9a4da38101ea # 03:09 20+ 0 drm/nouveau: add guard pages to vm
# first bad commit: [52774d9273d6b8aff39822ccfe10e5a0d7c3a7f8] There's some really badly broken software out there that is entirely
git bisect good c03339ffa3d9d2330c8c14b8d73e9a4da38101ea # 03:12 60+ 0 drm/nouveau: add guard pages to vm
git bisect bad 91283063c4f74fb202ac7809a8df3cc00f22203e # 03:12 0- 13 0day head guard for 'devel-hourly-2014061900'
git bisect good e99cfa2d0634881b8a41d56c48b5956b9a3ba162 # 03:50 60+ 0 Merge tag 'dt-fixes-for-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux
git bisect good 4ca1fbc122fcb09efa2bb1448bff8b71b46434c8 # 04:05 60+ 0 Add linux-next specific files for 20140618
This script may reproduce the error.
-----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
initrd=yocto-minimal-x86_64.cgz
wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/blob/master/initrd/$initrd
kvm=(
qemu-system-x86_64 -cpu kvm64 -enable-kvm
-kernel $kernel
-initrd $initrd
-smp 2
-m 256M
-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
-net user,vlan=0
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-serial stdio
-display none
-monitor null
)
append=(
debug
sched_debug
apic=debug
ignore_loglevel
sysrq_always_enabled
panic=10
prompt_ramdisk=0
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
)
"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------
Thanks,
Jet
6 years, 7 months
[x86_64, vsyscall] 21d4ab4881a: -11.1% will-it-scale.per_process_ops
by Jet Chen
Hi Andy,
we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git x86/vsyscall
commit 21d4ab4881ad9b257bec75d04480105dad4336e1 ("x86_64,vsyscall: Move all of the gate_area code to vsyscall_64.c")
test case: lkp-wsx01/will-it-scale/signal1
a7781f1035319a7 21d4ab4881ad9b257bec75d04
--------------- -------------------------
259032 ~ 1% -11.1% 230288 ~ 0% TOTAL will-it-scale.per_process_ops
0.04 ~ 9% +4276.2% 1.84 ~ 1% TOTAL perf-profile.cpu-cycles.map_id_up.do_tkill.sys_tgkill.system_call_fastpath.raise
2.36 ~ 0% -63.8% 0.85 ~ 2% TOTAL perf-profile.cpu-cycles._atomic_dec_and_lock.free_uid.__sigqueue_free.__dequeue_signal.dequeue_signal
2.25 ~14% -55.2% 1.01 ~ 1% TOTAL perf-profile.cpu-cycles.raise
42.41 ~ 0% +34.5% 57.04 ~ 0% TOTAL perf-profile.cpu-cycles.__sigqueue_alloc.__send_signal.send_signal.do_send_sig_info.do_send_specific
40.70 ~ 0% -23.9% 30.96 ~ 0% TOTAL perf-profile.cpu-cycles.__sigqueue_free.part.11.__dequeue_signal.dequeue_signal.get_signal_to_deliver.do_signal
252 ~11% -18.8% 204 ~ 9% TOTAL numa-vmstat.node1.nr_page_table_pages
1012 ~11% -18.3% 827 ~ 9% TOTAL numa-meminfo.node1.PageTables
520 ~ 7% -17.1% 431 ~ 5% TOTAL cpuidle.C1-NHM.usage
Legend:
~XX% - stddev percent
[+-]XX% - change percent
will-it-scale.per_process_ops
270000 ++-----------------------------------------------------------------+
| |
260000 *+.*..*.. .*.. .*.. .*..*... .*.. *.. .*.. *.. .*..*
| *..*. *. *. *. .. *. .. *..*. |
250000 ++ * * |
| |
240000 ++ |
| |
230000 ++ O O O O O O O O O O O |
O O O |
220000 ++ O O O |
| O O O O |
210000 ++ O |
| |
200000 ++-----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Jet
6 years, 7 months
[bisect done] [ 0.013865] WARNING: CPU: 0 PID: 0 at fs/kernfs/dir.c:947 kernfs_activate()
by Jet Chen
Hi David,
0day kernel testing robot got the below dmesg and the first bad commit is
git://people.freedesktop.org/~dvdhrm/linux fops
commit 6a616be3329e4ae59ae33e8de7be9b62e9535baa
Author: David Herrmann <dh.herrmann(a)gmail.com>
AuthorDate: Tue Jun 17 22:02:20 2014 +0200
Commit: David Herrmann <dh.herrmann(a)gmail.com>
CommitDate: Tue Jun 17 22:13:23 2014 +0200
kactive: extract active-ref implementation of kernfs
This introduces a new reference-type "struct kactive". Unlike kref, this
type manages "active references". That means, references can only be
acquired if the object is active. At any time the object can be
deactivated, causing any new attempt to acquire an active reference to
fail.
This patch extracts the kactive code from kernfs and turns it into an
unlocked variant. kernfs is updated to use the new code.
Signed-off-by: David Herrmann <dh.herrmann(a)gmail.com>
+------------------------------------------------------+------------+------------+
| | 809f893c60 | 6a616be332 |
+------------------------------------------------------+------------+------------+
| boot_successes | 60 | 0 |
| boot_failures | 0 | 20 |
| WARNING:CPU:PID:at_fs/kernfs/dir.c:kernfs_activate() | 0 | 20 |
| backtrace:kernfs_create_root | 0 | 20 |
| backtrace:sysfs_init | 0 | 20 |
| backtrace:mnt_init | 0 | 20 |
| backtrace:vfs_caches_init | 0 | 20 |
| Out_of_memory:Kill_process | 0 | 1 |
+------------------------------------------------------+------------+------------+
[ 0.011522] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[ 0.012007] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[ 0.013069] ------------[ cut here ]------------
[ 0.013865] WARNING: CPU: 0 PID: 0 at fs/kernfs/dir.c:947 kernfs_activate+0x90/0xd0()
[ 0.016004] Modules linked in:
[ 0.016765] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-rc1-00002-g6a616be #18
[ 0.018102] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 0.019012] 0000000000000000 ffffffff82003e18 ffffffff819c2b6e 0000000000000000
[ 0.020393] ffffffff82003e50 ffffffff810e0e90 ffffffff81249048 ffff88001167f000
[ 0.021966] ffff88001167f000 0000000000000002 0000000000000000 ffffffff82003e60
[ 0.023587] Call Trace:
[ 0.024007] [<ffffffff819c2b6e>] dump_stack+0x4d/0x66
[ 0.024847] [<ffffffff810e0e90>] warn_slowpath_common+0x7f/0x98
[ 0.025744] [<ffffffff81249048>] ? kernfs_activate+0x90/0xd0
[ 0.026620] [<ffffffff810e0f57>] warn_slowpath_null+0x1a/0x1c
[ 0.028005] [<ffffffff81249048>] kernfs_activate+0x90/0xd0
[ 0.028887] [<ffffffff812492f7>] kernfs_create_root+0xd8/0xe7
[ 0.029779] [<ffffffff82182212>] sysfs_init+0x13/0x51
[ 0.030593] [<ffffffff8217fa9d>] mnt_init+0x116/0x215
[ 0.031415] [<ffffffff8217f6cb>] vfs_caches_init+0x99/0x110
[ 0.032007] [<ffffffff82153eb1>] start_kernel+0x3d2/0x424
[ 0.032864] [<ffffffff821538ed>] ? set_init_arg+0x53/0x53
[ 0.033713] [<ffffffff82153120>] ? early_idt_handlers+0x120/0x120
[ 0.034706] [<ffffffff821534a2>] x86_64_start_reservations+0x2a/0x2c
[ 0.036006] [<ffffffff821535df>] x86_64_start_kernel+0x13b/0x148
[ 0.036945] ---[ end trace a625764830fea92a ]---
[ 0.037981] Initializing cgroup subsys memory
git bisect start 9cafa0eb8a6d3009f961c7b7fa14a2c5f149f9e2 7171511eaec5bf23fb06078f59784a3a0626b38f --
git bisect good 809f893c604209db19b38927ca4b927358eed321 # 04:43 20+ 0 fs: add f_op invokation guards
git bisect bad 6a616be3329e4ae59ae33e8de7be9b62e9535baa # 04:44 0- 1 kactive: extract active-ref implementation of kernfs
# first bad commit: [6a616be3329e4ae59ae33e8de7be9b62e9535baa] kactive: extract active-ref implementation of kernfs
git bisect good 809f893c604209db19b38927ca4b927358eed321 # 04:46 60+ 0 fs: add f_op invokation guards
git bisect bad 9cafa0eb8a6d3009f961c7b7fa14a2c5f149f9e2 # 04:46 0- 13 fs: add f_op->xy() active-protection
git bisect good ebe06187bf2aec10d537ce4595e416035367d703 # 04:49 60+ 0 epoll: fix use-after-free in eventpoll_release_file
git bisect good 5f295cdf5c5dbbb0c40f10f2ddae02ff46bbf773 # 04:52 60+ 0 Add linux-next specific files for 20140617
This script may reproduce the error.
-----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
kvm=(
qemu-system-x86_64 -cpu kvm64 -enable-kvm
-kernel $kernel
-smp 2
-m 256M
-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
-net user,vlan=0
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-serial stdio
-display none
-monitor null
)
append=(
debug
sched_debug
apic=debug
ignore_loglevel
sysrq_always_enabled
panic=10
prompt_ramdisk=0
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
)
"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------
Thanks,
Jet
6 years, 7 months
[idr] WARNING: at lib/idr.c:669 idr_find_slowpath()
by Jet Chen
Hi Tejun,
0day kernel testing robot got the below dmesg and the first bad commit is
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 0ffc2a9c8072969253a20821c2c733a2cbb4c7c7
Author: Tejun Heo <tj(a)kernel.org>
AuthorDate: Wed Feb 27 17:05:08 2013 -0800
Commit: Linus Torvalds <torvalds(a)linux-foundation.org>
CommitDate: Wed Feb 27 19:10:21 2013 -0800
idr: implement lookup hint
While idr lookup isn't a particularly heavy operation, it still is too
substantial to use in hot paths without worrying about the performance
implications. With recent changes, each idr_layer covers 256 slots
which should be enough to cover most use cases with single idr_layer
making lookup hint very attractive.
This patch adds idr->hint which points to the idr_layer which
allocated an ID most recently and the fast path lookup becomes
if (look up target's prefix matches that of the hinted layer)
return hint->ary[ID's offset in the leaf layer];
which can be inlined.
idr->hint is set to the leaf node on idr_fill_slot() and cleared from
free_layer().
[andriy.shevchenko(a)linux.intel.com: always do slow path when hint is uninitialized]
Signed-off-by: Tejun Heo <tj(a)kernel.org>
Cc: Kirill A. Shutemov <kirill.shutemov(a)linux.intel.com>
Cc: Sasha Levin <sasha.levin(a)oracle.com>
Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds(a)linux-foundation.org>
+--------------------------------------------------------------------------+------------+------------+
| | 54616283c2 | 0ffc2a9c80 |
+--------------------------------------------------------------------------+------------+------------+
| boot_successes | 140 | 31 |
| boot_failures | 320 | 84 |
| BUG:soft_lockup-CPU_stuck_for_s | 128 | 40 |
| EIP_is_at_stop_machine_cpu_stop | 128 | 40 |
| Kernel_panic-not_syncing:softlockup:hung_tasks | 128 | 40 |
| backtrace:smpboot_thread_fn | 128 | 40 |
| BUG:unable_to_handle_kernel_paging_request | 67 | 18 |
| Oops | 40 | 12 |
| EIP_is_at_static_key_slow_inc | 35 | 10 |
| Kernel_panic-not_syncing:Fatal_exception | 39 | 12 |
| backtrace:perf_swevent_init | 37 | 10 |
| backtrace:perf_init_event | 144 | 35 |
| BUG:unable_to_handle_kernel_NULL_pointer_dereference | 2 | |
| EIP_is_at_jump_label_update | 2 | |
| backtrace:static_key_slow_inc | 2 | |
| WARNING:at_lib/idr.c:idr_find() | 119 | |
| EIP_is_at_atomic_dec_and_mutex_lock | 2 | 2 |
| backtrace:idr_find | 111 | |
| WARNING:at_kernel/jump_label.c:__static_key_slow_dec() | 3 | 1 |
| BUG:kernel_test_crashed | 18 | 4 |
| WARNING:at_kernel/lockdep.c:__bfs() | 1 | |
| EIP_is_at__bfs | 1 | |
| WARNING:at_arch/x86/kernel/hw_breakpoint.c:arch_install_hw_breakpoint() | 1 | |
| BUG:sleeping_function_called_from_invalid_context_at_arch/x86/mm/fault.c | 1 | |
| WARNING:at_lib/idr.c:idr_find_slowpath() | 0 | 27 |
| backtrace:idr_find_slowpath | 0 | 26 |
| WARNING:at_drivers/tty/tty_buffer.c:flush_to_ldisc() | 0 | 1 |
| backtrace:flush_to_ldisc | 0 | 1 |
+--------------------------------------------------------------------------+------------+------------+
Kernel tests: Boot OK!
[ 17.603698] sock: process `trinity-main' is using obsolete setsockopt SO_BSDCOMPAT
[ 17.798944] ------------[ cut here ]------------
[ 17.798972] WARNING: at lib/idr.c:669 idr_find_slowpath+0x28/0xca()
[ 17.798973] Hardware name: Bochs
[ 17.798976] Pid: 655, comm: trinity-main Not tainted 3.8.0-06482-g0ffc2a9 #68
[ 17.798976] Call Trace:
[ 17.798989] [<c022ace6>] warn_slowpath_common+0x72/0x89
[ 17.798992] [<c051f001>] ? idr_find_slowpath+0x28/0xca
[ 17.798994] [<c051f001>] ? idr_find_slowpath+0x28/0xca
[ 17.798997] [<c022ad9b>] warn_slowpath_null+0x1d/0x1f
[ 17.798999] [<c051f001>] idr_find_slowpath+0x28/0xca
[ 17.799007] [<c027fab7>] perf_init_event+0x82/0x124
[ 17.799013] [<c024321f>] ? __mutex_init+0x3f/0x44
[ 17.799015] [<c027fda9>] perf_event_alloc+0x250/0x360
[ 17.799018] [<c028040e>] sys_perf_event_open+0x336/0x664
[ 17.799036] [<c0944280>] syscall_call+0x7/0xb
[ 17.799042] [<c0940000>] ? SetAgcIf.isra.6+0x156/0x290
[ 17.799043] ---[ end trace 8e00e88be2b81021 ]---
[ 76.161086] Unregister pv shared memory for cpu 0
Attached dmesg for the parent commit, too, to help confirm whether it is a noise error.
git bisect start 6dbe51c251a327e012439c4772097a13df43c5b8 v3.8 --
git bisect good d414c104e26fd3b597f855cc29473a8b1527fb4c # 02:41 35+ 35 Merge tag 'please-pull-vm_unwrapped' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect bad ee89f81252179dcbf6cd65bd48299f5e52292d88 # 02:46 10- 14 Merge branch 'for-3.9/core' of git://git.kernel.dk/linux-block
git bisect good a6590b9f01dca2d45a38b8387f59812c44f68a2f # 03:05 40+ 40 Merge tag 'upstream-3.9-rc1' of git://git.infradead.org/linux-ubifs
git bisect good 09884964335e85e897876d17783c2ad33cf8a2e0 # 03:13 40+ 40 mm: do not grow the stack vma just because of an overrun on preceding vma
git bisect good ad60a9333035f2323840b71ab8ad07bbab728faf # 03:29 40+ 17 Merge branch 'misc' of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
git bisect good e3c4877de8b9d93bd47b6ee88eb594b1c1e10da5 # 04:16 40+ 15 Merge branch 'x86-efi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 2a7d2b96d5cba7568139d9ab157a0e97ab32440f # 04:29 1- 3 Merge branch 'akpm' (final batch from Andrew)
git bisect good 6fa780095f058f0960f220754217701a53a9e403 # 05:44 40+ 19 IB/cxgb3: convert to idr_alloc()
git bisect good 050a6b47d98e2bcea909c1129111e721668aaa2c # 06:06 40+ 36 idr: make idr_layer larger
git bisect bad 459a86d83dce6641f66311d37fbfe1419c477a5a # 06:28 19- 21 mtd: mtd_nandecctest: use prandom_bytes instead of get_random_bytes()
git bisect bad 06004e6eebdaf101a0f9b533fac77b5768d2c09c # 06:40 5- 6 block/partitions/mac.c: obey the state->limit constraint
git bisect bad d941aeae42942c2ef61ed5e05cc1ad0428085704 # 06:52 0- 1 ipmi: add new kernel options to prevent automatic ipmi init
git bisect bad 0ffc2a9c8072969253a20821c2c733a2cbb4c7c7 # 07:00 9- 10 idr: implement lookup hint
git bisect good 54616283c2948812a44240858ced610e7cacbde1 # 07:06 115+ 73 idr: add idr_layer->prefix
# first bad commit: [0ffc2a9c8072969253a20821c2c733a2cbb4c7c7] idr: implement lookup hint
git bisect good 54616283c2948812a44240858ced610e7cacbde1 # 07:15 345+ 320 idr: add idr_layer->prefix
git bisect good d67a0e110187abd560a1de63fa172894a52839d5 # 07:30 345+ 11 morphologies: Enable CONFIG_FHANDLE
git bisect good 0e04c641b199435f3779454055f6a7de258ecdfc # 07:45 345+ 22 Merge tag 'dm-3.16-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
git bisect good b1cce620fd4b6864c92e7307be7839789f9c8be0 # 08:03 345+ 18 Add linux-next specific files for 20140612
This script may reproduce the error.
-----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
initrd=yocto-minimal-i386.cgz
wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/blob/master/initrd/$initrd
kvm=(
qemu-system-x86_64 -cpu kvm64 -enable-kvm
-kernel $kernel
-initrd $initrd
-smp 2
-m 256M
-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
-net user,vlan=0
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-serial stdio
-display none
-monitor null
)
append=(
debug
sched_debug
apic=debug
ignore_loglevel
sysrq_always_enabled
panic=10
prompt_ramdisk=0
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
)
"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------
Thanks,
Jet
6 years, 7 months
[random] BUG: spinlock trylock failure on UP on CPU#0, swapper/1
by Jet Chen
Hi Greg,
0day kernel testing robot got the below dmesg and the first bad commit is
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 0fb7a01af5b0cbe5bf365891fc4d186f2caa23f7
Author: Greg Price <price(a)MIT.EDU>
AuthorDate: Thu Dec 5 19:32:19 2013 -0500
Commit: Theodore Ts'o <tytso(a)mit.edu>
CommitDate: Wed Mar 19 22:18:51 2014 -0400
random: simplify accounting code
With this we handle "reserved" in just one place. As a bonus the
code becomes less nested, and the "wakeup_write" flag variable
becomes unnecessary. The variable "flags" was already unused.
This code behaves identically to the previous version except in
two pathological cases that don't occur. If the argument "nbytes"
is already less than "min", then we didn't previously enforce
"min". If r->limit is false while "reserved" is nonzero, then we
previously applied "reserved" in checking whether we had enough
bits, even though we don't apply it to actually limit how many we
take. The callers of account() never exercise either of these cases.
Before the previous commit, it was possible for "nbytes" to be less
than "min" if userspace chose a pathological configuration, but no
longer.
Cc: Jiri Kosina <jkosina(a)suse.cz>
Cc: "H. Peter Anvin" <hpa(a)zytor.com>
Signed-off-by: Greg Price <price(a)mit.edu>
Signed-off-by: "Theodore Ts'o" <tytso(a)mit.edu>
+-------------------------------------------+------------+------------+
| | 8c2aa3390e | 0fb7a01af5 |
+-------------------------------------------+------------+------------+
| boot_successes | 66 | 2 |
| boot_failures | 0 | 20 |
| BUG:spinlock_trylock_failure_on_UP_on_CPU | 0 | 20 |
| backtrace:prandom_reseed | 0 | 11 |
| backtrace:kernel_init_freeable | 0 | 11 |
+-------------------------------------------+------------+------------+
[ 374.075477] raid6test: test_disks(62, 63): faila= 62(P) failb= 63(Q) OK
[ 374.192518] raid6test: [ 374.312137] raid6test: complete (2429 tests, 0 failures)
[ 374.434548] BUG: spinlock trylock failure on UP on CPU#0, swapper/1
[ 374.444114] lock: lock.13830+0x0/0x50, .magic: dead4ead, .owner: swapper/1, .owner_cpu: 0
[ 374.444114] CPU: 0 PID: 1 Comm: swapper Not tainted 3.14.0-rc2-00009-g0fb7a01 #1
[ 374.444114] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 374.444114] 0000000000000000 ffff88001386fc28 ffffffff81ec8af8 ffff88001386fc48
[ 374.444114] ffffffff81ec26d9 0000000000000000 ffffffff827db7a0 ffff88001386fc68
[ 374.444114] ffffffff81ec2704 ffffffff827db7a0 ffffffff823b74f3 ffff88001386fc80
[ 374.444114] Call Trace:
[ 374.444114] [<ffffffff81ec8af8>] dump_stack+0x19/0x1b
[ 374.444114] [<ffffffff81ec26d9>] spin_dump+0x85/0x8a
[ 374.444114] [<ffffffff81ec2704>] spin_bug+0x26/0x28
[ 374.444114] [<ffffffff810ae690>] do_raw_spin_trylock+0x37/0x3c
[ 374.444114] [<ffffffff81ed707b>] _raw_spin_lock_irqsave+0x4b/0x7a
[ 374.444114] [<ffffffff814f5f69>] ? __prandom_reseed+0x1a/0xad
[ 374.444114] [<ffffffff814f5f69>] __prandom_reseed+0x1a/0xad
[ 374.444114] [<ffffffff814f6183>] prandom_reseed_late+0xe/0x10
[ 374.444114] [<ffffffff8162f31c>] credit_entropy_bits+0x116/0x21e
[ 374.444114] [<ffffffff8162f8d9>] _xfer_secondary_pool+0x86/0x8f
[ 374.444114] [<ffffffff810abe66>] ? __lock_acquire.isra.20+0x4cf/0x5cd
[ 374.444114] [<ffffffff8162f963>] xfer_secondary_pool+0x5d/0x5f
[ 374.444114] [<ffffffff8162fa0a>] extract_entropy+0xa5/0x16b
[ 374.444114] [<ffffffff816301cc>] get_random_bytes+0x1b/0x1d
[ 374.444114] [<ffffffff814f5f8f>] __prandom_reseed+0x40/0xad
[ 374.444114] [<ffffffff810e7b32>] ? put_page+0x20/0x22
[ 374.444114] [<ffffffff8295f922>] ? radix_tree_init+0xce/0xce
[ 374.444114] [<ffffffff8295f92e>] prandom_reseed+0xc/0x49
[ 374.444114] [<ffffffff8295f922>] ? radix_tree_init+0xce/0xce
[ 374.444114] [<ffffffff8293ae11>] do_one_initcall+0x7d/0x108
[ 374.444114] [<ffffffff81098f9c>] ? parse_args+0x180/0x246
[ 374.444114] [<ffffffff8293af92>] kernel_init_freeable+0xf6/0x187
[ 374.444114] [<ffffffff8293a70e>] ? do_early_param+0x8a/0x8a
[ 374.444114] [<ffffffff81ebe658>] ? rest_init+0xbc/0xbc
[ 374.444114] [<ffffffff81ebe661>] kernel_init+0x9/0xd0
[ 374.444114] [<ffffffff81ed850a>] ret_from_fork+0x7a/0xb0
[ 374.444114] [<ffffffff81ebe658>] ? rest_init+0xbc/0xbc
Elapsed time: 380
qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel /kernel/x86_64-randconfig-hsxa1-06150953/0fb7a01af5b0cbe5bf365891fc4d186f2caa23f7/vmlinuz-3.14.0-rc2-00009-g0fb7a01 -append 'hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=10 softlockup_panic=1 nmi_watchdog=panic prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw link=/kbuild-tests/run-queue/kvm/x86_64-randconfig-hsxa1-06150953/linux-devel:devel-hourly-2014061509:0fb7a01af5b0cbe5bf365891fc4d186f2caa23f7:bisect-linux3/.vmlinuz-0fb7a01af5b0cbe5bf365891fc4d186f2caa23f7-20140615135429-17-ivb41 branch=linux-devel/devel-hourly-2014061509 BOOT_IMAGE=/kernel/x86_64-randconfig-hsxa1-06150953/0fb7a01af5b0cbe5bf365891fc4d186f2caa23f7/vmlinuz-3.14.0-rc2-00009-g0fb7a01 drbd.minor_count=8' -initrd /kernel-tests/initrd/yocto-minimal-x86_64.cgz -m 320 -smp 2 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb
-rtc base=localtime -pidfile /dev/shm/kboot/pid-yocto-ivb41-47 -serial file:/dev/shm/kboot/serial-yocto-ivb41-47 -daemonize -display none -monitor null
git bisect start v3.15 v3.14 --
git bisect bad 674366e90e5a2430e3ddef40ce51512d681944e4 # 10:49 0- 3 Merge tag 'pci-v3.15-fixes-1' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
git bisect bad 9f800363bb0ea459e15bef0928a72c88d374e489 # 11:28 2- 14 Merge tag 'fixes-non-critical-3.15' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect good 1aabc5990d205cdb0789a1c26143c53601e9bb07 # 11:38 21+ 0 x86: Fix dumpstack_64 to keep state of "used" variable in loop
git bisect good d0cb5f71c5cde8e1ef6d03983641366800ceabdb # 11:51 21+ 0 Merge tag 'vfio-v3.15-rc1' of git://github.com/awilliam/linux-vfio
git bisect good d15fee814d8d30bbb4859ef8fef7a1f96327635b # 12:07 21+ 0 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
git bisect bad d1d9cfc330e2622f4b4e0b82d3b41a34619614d4 # 12:16 1- 8 Merge tag 'random_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random
git bisect good 24e7ea3bea94fe05eae5019f5f12bcdc98fc5157 # 12:27 21+ 0 Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
git bisect good d15e03104eb9a4f8e244ab6ed3ca5a107e46db13 # 12:40 21+ 0 Merge tag 'xfs-for-linus-3.15-rc1' of git://oss.sgi.com/xfs/xfs
git bisect good cda540ace6a194850e23c79955cc2e46fd91c19a # 13:15 21+ 0 mm: get_user_pages(write,force) refuse to COW in shared areas
git bisect good 8c2aa3390ebb59cba4495a56557b70ad0575eef5 # 13:34 21+ 0 random: tighten bound on random_read_wakeup_thresh
git bisect bad 46884442fc5bb81a896f7245bd850fde9b435509 # 13:44 0- 2 random: use the architectural HWRNG for the SHA's IV in extract_buf()
git bisect bad 7d1b08c40c4f02c119476b29eca9bbc8d98d2a83 # 13:52 2- 9 random: entropy_bytes is actually bits
git bisect bad 0fb7a01af5b0cbe5bf365891fc4d186f2caa23f7 # 14:01 0- 5 random: simplify accounting code
# first bad commit: [0fb7a01af5b0cbe5bf365891fc4d186f2caa23f7] random: simplify accounting code
git bisect good 8c2aa3390ebb59cba4495a56557b70ad0575eef5 # 14:11 66+ 0 random: tighten bound on random_read_wakeup_thresh
git bisect bad 1d08ab37636d6a169c67e879cba37675cd82e6ac # 14:11 0- 11 0day head guard for 'devel-hourly-2014061509'
git bisect bad abf04af74a9f27a65172a43b43ccabcbc2bbdc39 # 14:27 3- 2 Merge tag 'scsi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
git bisect bad f9801c532e045c1ab89801d0597353c5e2a55671 # 14:41 5- 8 Add linux-next specific files for 20140613
This script may reproduce the error.
-----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
initrd=yocto-minimal-x86_64.cgz
wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/blob/master/initrd/$initrd
kvm=(
qemu-system-x86_64 -cpu kvm64 -enable-kvm
-kernel $kernel
-initrd $initrd
-smp 2
-m 256M
-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
-net user,vlan=0
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-serial stdio
-display none
-monitor null
)
append=(
debug
sched_debug
apic=debug
ignore_loglevel
sysrq_always_enabled
panic=10
prompt_ramdisk=0
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
)
"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------
Thanks,
Jet
6 years, 7 months
[CPU hotplug, smp] WARNING: CPU: 1 PID: 0 at kernel/smp.c:209 generic_smp_call_function_single_interrupt()
by Jet Chen
Hi Srivatsa,
0day kernel testing robot got the below dmesg and the first bad commit is
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit ab7a42783d939cdbe729c18ab32dbf0d25746ea2
Author: Srivatsa S. Bhat <srivatsa.bhat(a)linux.vnet.ibm.com>
AuthorDate: Thu May 22 10:44:06 2014 +1000
Commit: Stephen Rothwell <sfr(a)canb.auug.org.au>
CommitDate: Thu May 22 10:44:06 2014 +1000
CPU hotplug, smp: Flush any pending IPI callbacks before CPU offline
During CPU offline, in the stop-machine loop, we use 2 separate stages to
disable interrupts, to ensure that the CPU going offline doesn't get any
new IPIs from the other CPUs after it has gone offline.
However, an IPI sent much earlier might arrive late on the target CPU
(possibly _after_ the CPU has gone offline) due to hardware latencies, and
due to this, the smp-call-function callbacks queued on the outgoing CPU
might not get noticed (and hence not executed) at all.
This is somewhat theoretical, but in any case, it makes sense to
explicitly loop through the call_single_queue and flush any pending
callbacks before the CPU goes completely offline. So, flush the queued
smp-call-function callbacks in the MULTI_STOP_DISABLE_IRQ_ACTIVE stage,
after disabling interrupts on the active CPU. This can be trivially
achieved by invoking the generic_smp_call_function_single_interrupt()
function itself (and since the outgoing CPU is still online at this point,
we won't trigger the "IPI to offline CPU" warning in this function; so we
are safe to call it here).
This way, we would have handled all the queued callbacks before going
offline, and also, no new IPIs can be sent by the other CPUs to the
outgoing CPU at that point, because they will all be executing the
stop-machine code with interrupts disabled.
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat(a)linux.vnet.ibm.com>
Suggested-by: Frederic Weisbecker <fweisbec(a)gmail.com>
Reviewed-by: Tejun Heo <tj(a)kernel.org>
Cc: Peter Zijlstra <peterz(a)infradead.org>
Cc: Oleg Nesterov <oleg(a)redhat.com>
Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org>
+------------------------------------------------------------------------------+------------+------------+
| | c80d40e1f2 | ab7a42783d |
+------------------------------------------------------------------------------+------------+------------+
| boot_successes | 1194 | 261 |
| boot_failures | 6 | 39 |
| BUG:kernel_test_crashed | 6 | |
| WARNING:CPU:PID:at_kernel/smp.c:generic_smp_call_function_single_interrupt() | 0 | 39 |
| backtrace:stop_machine_from_inactive_cpu | 0 | 39 |
| backtrace:mtrr_ap_init | 0 | 39 |
| general_protection_fault | 0 | 0 |
| RIP:__lock_acquire | 0 | 0 |
| BUG:sleeping_function_called_from_invalid_context_at_kernel/locking/rwsem.c | 0 | 0 |
| INFO:lockdep_is_turned_off | 0 | 0 |
| backtrace:do_mount | 0 | 0 |
| backtrace:SyS_mount | 0 | 0 |
+------------------------------------------------------------------------------+------------+------------+
[ 62.119017] masked ExtINT on CPU#1
[ 62.119017] numa_add_cpu cpu 1 node 0: mask now 0-1
[ 62.261606] ------------[ cut here ]------------
[ 62.261606] WARNING: CPU: 1 PID: 0 at kernel/smp.c:209 generic_smp_call_function_single_interrupt+0xc5/0x155()
[ 62.261606] IPI on offline CPU 1
[ 62.261606] Modules linked in:
[ 62.261606] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.15.0-rc5-01473-gab7a427 #4510
[ 62.261606] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 62.261606] 0000000000000000 ffff88001e7e1d28 ffffffff81b01705 ffff88001e7e1d70
[ 62.261606] ffff88001e7e1d60 ffffffff810754b3 ffffffff810e282c 0000000000000000
[ 62.261606] 0000000000000000 ffffffff82568b01 0000000000000082 ffff88001e7e1dc0
[ 62.261606] Call Trace:
[ 62.261606] [<ffffffff81b01705>] dump_stack+0x4d/0x66
[ 62.261606] [<ffffffff810754b3>] warn_slowpath_common+0x7f/0x98
[ 62.261606] [<ffffffff810e282c>] ? generic_smp_call_function_single_interrupt+0xc5/0x155
[ 62.261606] [<ffffffff81075518>] warn_slowpath_fmt+0x4c/0x4e
[ 62.261606] [<ffffffff810e282c>] generic_smp_call_function_single_interrupt+0xc5/0x155
[ 62.261606] [<ffffffff810f4792>] multi_cpu_stop+0xcf/0x136
[ 62.261606] [<ffffffff810f5003>] stop_machine_from_inactive_cpu+0xf7/0x132
[ 62.261606] [<ffffffff81056cfe>] ? mtrr_restore+0x45/0x45
[ 62.261606] [<ffffffff810573ba>] mtrr_ap_init+0x5a/0x5c
[ 62.261606] [<ffffffff8104689e>] identify_secondary_cpu+0x19/0x1b
[ 62.261606] [<ffffffff8105beda>] smp_store_cpu_info+0x3d/0x3f
[ 62.261606] [<ffffffff8105c6cc>] start_secondary+0x16e/0x254
[ 62.261606] ---[ end trace 1752731e5a47c773 ]---
[ 62.312240] KVM setup async PF for cpu 1
git bisect start 27a4e439fe5cd92b70137ae237c7aa6888c07b5a 56b54984d49285d515dc6421c93648ccdc0a06bc --
git bisect bad 473496a03d9dbf8654f2fd789383d71df4dc7fde # 15:58 88- 3 Merge branch 'akpm-current/current'
git bisect good b41b3b6c0ff37f89caf1dd5ec8f203f00d5c9080 # 16:23 300+ 0 mm: page_alloc: convert hot/cold parameter and immediate callers to bool
git bisect good a0a7e7f4b2fbf31729ea5cd5d3d5cb7ee5c0b494 # 16:44 300+ 1 rtc: rtc-palmas: make of_device_id array const
git bisect bad ab7a42783d939cdbe729c18ab32dbf0d25746ea2 # 16:49 45- 3 CPU hotplug, smp: Flush any pending IPI callbacks before CPU offline
git bisect good faa7c21459067de9686cdd62cfb47685bc5c60fc # 16:55 300+ 1 fs/hfsplus/wrapper.c: replace min/casting by min_t
git bisect good 6a2ecc87dcbae7fa62ce1f156c4ac4a52d96b3fd # 17:01 300+ 1 signals: kill rm_from_queue(), change prepare_signal() to use for_each_thread()
git bisect good ebe499d62a3a08e94fb1067f3486e00eca49edca # 17:09 300+ 1 signals: introduce kernel_sigaction()
git bisect good ddb7fec09b32ec61e428e7e882b6821a10d3df3f # 17:43 300+ 1 smp-print-more-useful-debug-info-upon-receiving-ipi-on-an-offline-cpu-v5
git bisect good e840b4a6701321414618aa1fa623925368d66141 # 17:47 300+ 2 cpu-hotplug-stop-machine-plug-race-window-that-leads-to-ipi-to-offline-cpu-v3
git bisect good c80d40e1f29243cf7d932b8e35126bcae0e875b4 # 18:01 300+ 2 cpu-hotplug-stop-machine-plug-race-window-that-leads-to-ipi-to-offline-cpu-v5
# first bad commit: [ab7a42783d939cdbe729c18ab32dbf0d25746ea2] CPU hotplug, smp: Flush any pending IPI callbacks before CPU offline
git bisect good c80d40e1f29243cf7d932b8e35126bcae0e875b4 # 18:10 900+ 6 cpu-hotplug-stop-machine-plug-race-window-that-leads-to-ipi-to-offline-cpu-v5
git bisect bad 27a4e439fe5cd92b70137ae237c7aa6888c07b5a # 18:10 0- 7 Add linux-next specific files for 20140610
git bisect good 64b2d1fbbfda07765dae3f601862796a61b2c451 # 18:19 900+ 99 Merge tag 'for-f2fs-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
git bisect bad 27a4e439fe5cd92b70137ae237c7aa6888c07b5a # 18:19 0- 7 Add linux-next specific files for 20140610
Thanks,
Jet
6 years, 7 months