January 2016 - LKP - Ml01.01.Org

Re: [LKP] [rcu] kernel BUG at include/linux/pagemap.h:149!

by Frederic Weisbecker

On Fri, Sep 11, 2015 at 10:19:47AM +0800, Boqun Feng wrote: > Subject: [PATCH 01/27] rcu: Don't disable preemption for Tiny and Tree RCU > readers > > Because preempt_disable() maps to barrier() for non-debug builds, > it forces the compiler to spill and reload registers. Because Tree > RCU and Tiny RCU now only appear in CONFIG_PREEMPT=n builds, these > barrier() instances generate needless extra code for each instance of > rcu_read_lock() and rcu_read_unlock(). This extra code slows down Tree > RCU and bloats Tiny RCU. > > This commit therefore removes the preempt_disable() and preempt_enable() > from the non-preemptible implementations of __rcu_read_lock() and > __rcu_read_unlock(), respectively. > > For debug purposes, preempt_disable() and preempt_enable() are still > kept if CONFIG_PREEMPT_COUNT=y, which makes the detection of sleeping > inside atomic sections still work in non-preemptible kernels. > > Signed-off-by: Boqun Feng <boqun.feng(a)gmail.com> > Signed-off-by: Paul E. McKenney <paulmck(a)linux.vnet.ibm.com> > --- > include/linux/rcupdate.h | 6 ++++-- > include/linux/rcutiny.h | 1 + > kernel/rcu/tree.c | 9 +++++++++ > 3 files changed, 14 insertions(+), 2 deletions(-) > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h > index d63bb77..6c3cece 100644 > --- a/include/linux/rcupdate.h > +++ b/include/linux/rcupdate.h > @@ -297,12 +297,14 @@ void synchronize_rcu(void); > > static inline void __rcu_read_lock(void) > { > - preempt_disable(); > + if (IS_ENABLED(CONFIG_PREEMPT_COUNT)) > + preempt_disable(); preempt_disable() is a no-op when !CONFIG_PREEMPT_COUNT, right? Or rather it's a barrier(), which is anyway implied by rcu_read_lock(). So perhaps we can get rid of the IS_ENABLED() check?

1 year, 7 months

4
4
0 / 0

Extending the 0-day system with syzkaller?

by David Drysdale

Hi Fengguang / LKP-folk, Quick question -- how easy is it to add extra builds/tests/checks to your marvellous 0-day kbuild system? The reason I ask is that I've recently been exploring syzkaller [1], which is a system call fuzzer written by some of my colleagues here at Google (cc'ed). Although it's fairly new, it has uncovered a bunch of kernel bugs already [2] so I wondered if it might be a good candidate for inclusion in the 0-day checks at some point. (As an aside, I'm in the process of writing an article about syzkaller for LWN, which might also expose it to more folk.) What do you think? Thanks, David [1] https://github.com/google/syzkaller [2] https://github.com/google/syzkaller/wiki/Found-Bugs

4 years, 7 months

7
14
0 / 0

[slab] a1fd55538c: WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:2601 trace_hardirqs_on_caller()

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit a1fd55538cae9f411059c9b067a3d48c41aa876b Author: Jesper Dangaard Brouer <brouer(a)redhat.com> AuthorDate: Thu Jan 28 09:47:16 2016 +1100 Commit: Stephen Rothwell <sfr(a)canb.auug.org.au> CommitDate: Thu Jan 28 09:47:16 2016 +1100 slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB Deduplicate code in SLAB allocator functions slab_alloc() and slab_alloc_node() by using the slab_pre_alloc_hook() call, which is now shared between SLUB and SLAB. Signed-off-by: Jesper Dangaard Brouer <brouer(a)redhat.com> Cc: Christoph Lameter <cl(a)linux.com> Cc: Pekka Enberg <penberg(a)kernel.org> Cc: David Rientjes <rientjes(a)google.com> Cc: Joonsoo Kim <iamjoonsoo.kim(a)lge.com> Cc: Vladimir Davydov <vdavydov(a)virtuozzo.com> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> +-----------------------------------------------------------------+------------+------------+---------------+ | | 074b6f53c3 | a1fd55538c | next-20160128 | +-----------------------------------------------------------------+------------+------------+---------------+ | boot_successes | 40 | 0 | 0 | | boot_failures | 52 | 26 | 19 | | Kernel_panic-not_syncing:Attempted_to_kill_init!exitcode= | 52 | 26 | 14 | | WARNING:at_kernel/locking/lockdep.c:#trace_hardirqs_on_caller() | 0 | 26 | 19 | | backtrace:pcpu_mem_zalloc | 0 | 26 | 19 | | backtrace:percpu_init_late | 0 | 26 | 19 | | IP-Config:Auto-configuration_of_network_failed | 0 | 0 | 2 | +-----------------------------------------------------------------+------------+------------+---------------+ [ 0.000000] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes) [ 0.000000] Memory: 194224K/261624K available (10816K kernel code, 5060K rwdata, 6628K rodata, 988K init, 33076K bss, 67400K reserved, 0K cma-reserved) [ 0.000000] ------------[ cut here ]------------ [ 0.000000] WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:2601 trace_hardirqs_on_caller+0x341/0x380() [ 0.000000] DEBUG_LOCKS_WARN_ON(unlikely(early_boot_irqs_disabled)) [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.5.0-rc1-00069-ga1fd555 #1 [ 0.000000] ffffffff82403dd8 ffffffff82403d90 ffffffff813b937d ffffffff82403dc8 [ 0.000000] ffffffff810eb4d3 ffffffff812617cc 0000000000000001 ffff88000fcc50a8 [ 0.000000] ffff8800000984c0 00000000024000c0 ffffffff82403e28 ffffffff810eb5c7 [ 0.000000] Call Trace: [ 0.000000] [<ffffffff813b937d>] dump_stack+0x27/0x3a [ 0.000000] [<ffffffff810eb4d3>] warn_slowpath_common+0xa3/0x100 [ 0.000000] [<ffffffff812617cc>] ? cache_alloc_refill+0x7ac/0x910 [ 0.000000] [<ffffffff810eb5c7>] warn_slowpath_fmt+0x57/0x70 [ 0.000000] [<ffffffff81143e61>] trace_hardirqs_on_caller+0x341/0x380 [ 0.000000] [<ffffffff81143ebd>] trace_hardirqs_on+0x1d/0x30 [ 0.000000] [<ffffffff812617cc>] cache_alloc_refill+0x7ac/0x910 [ 0.000000] [<ffffffff8121df6a>] ? pcpu_mem_zalloc+0x5a/0xc0 [ 0.000000] [<ffffffff81261fce>] __kmalloc+0x24e/0x440 [ 0.000000] [<ffffffff8121df6a>] pcpu_mem_zalloc+0x5a/0xc0 [ 0.000000] [<ffffffff829213aa>] percpu_init_late+0x4d/0xbb [ 0.000000] [<ffffffff828f41c9>] start_kernel+0x30b/0x6e1 [ 0.000000] [<ffffffff828f3120>] ? early_idt_handler_array+0x120/0x120 [ 0.000000] [<ffffffff828f332f>] x86_64_start_reservations+0x46/0x4f [ 0.000000] [<ffffffff828f34d4>] x86_64_start_kernel+0x19c/0x1b2 [ 0.000000] ---[ end trace cb88537fdc8fa200 ]--- [ 0.000000] Running RCU self tests git bisect start 888c8375131656144c1605071eab2eb6ac49abc3 92e963f50fc74041b5e9e744c330dca48e04f08d -- git bisect good f664e02a71d85691fc33f116bae3eb7f0debd194 # 17:19 17+ 13 Merge remote-tracking branch 'kbuild/for-next' git bisect good c7173552fb5efc15dd092d3a90b5d6ad0f3d9421 # 17:35 17+ 2 Merge remote-tracking branch 'audit/next' git bisect good bd605d2e3cc724606fa7c0fd3d5d90276f07e979 # 17:47 17+ 2 Merge remote-tracking branch 'extcon/extcon-next' git bisect good 108776431802ced1ca8ba38a9765ef81c48513de # 18:06 17+ 5 Merge remote-tracking branch 'llvmlinux/for-next' git bisect good 56f1389517d2470a8abdb661c97d6ef640ca8cf3 # 18:30 17+ 3 Merge remote-tracking branch 'coresight/next' git bisect bad 3cb196d8ee7f94b78c3d609bb91f5b175b3841d8 # 19:17 0- 8 Merge branch 'akpm-current/current' git bisect good 49d5623e2407b26b532ca24f49d778b5b6fedb22 # 19:48 22+ 0 Merge remote-tracking branch 'rtc/rtc-next' git bisect bad 8ccfb34d7450299714a9a590a764934397a818c6 # 20:06 0- 22 mm: filemap: avoid unnecessary calls to lock_page when waiting for IO to complete during a read git bisect good d9dc8f2de4f863bef9a303b2cbae0bbd1c9dfceb # 20:32 22+ 22 ocfs2: add feature document for online file check git bisect bad ebea6ceb9754b02bcab987af96c64782c665aa91 # 20:56 0- 18 mm/slab: remove object status buffer for DEBUG_SLAB_LEAK git bisect bad 24d88722c03b13ef63b3b631f81454a63ac26cc4 # 21:06 0- 22 mm: kmemcheck skip object if slab allocation failed git bisect good 1fc2d06fe0cfca10e571e2e444a4a37693495502 # 21:26 22+ 22 ocfs2/dlm: move lock to the tail of grant queue while doing in-place convert git bisect good 3355ee84b3d96c7c30923d0bba228b0b7aa380d2 # 21:33 21+ 8 slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk git bisect good 074b6f53c320a81e975c0b5dd79daa5e78a711ba # 21:39 22+ 24 mm: fault-inject take over bootstrap kmem_cache check git bisect bad a1fd55538cae9f411059c9b067a3d48c41aa876b # 21:49 0- 26 slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB # first bad commit: [a1fd55538cae9f411059c9b067a3d48c41aa876b] slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB git bisect good 074b6f53c320a81e975c0b5dd79daa5e78a711ba # 21:53 66+ 52 mm: fault-inject take over bootstrap kmem_cache check # extra tests with DEBUG_INFO git bisect bad a1fd55538cae9f411059c9b067a3d48c41aa876b # 22:00 0- 36 slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB # extra tests on HEAD of linux-next/master git bisect bad 888c8375131656144c1605071eab2eb6ac49abc3 # 22:00 0- 19 Add linux-next specific files for 20160128 # extra tests on tree/branch linux-next/master git bisect bad 888c8375131656144c1605071eab2eb6ac49abc3 # 22:00 0- 19 Add linux-next specific files for 20160128 # extra tests with first bad commit reverted git bisect good fea4cd9180f321dd12ec9a7932a9bfb32bfaf4c4 # 22:32 66+ 30 Revert "slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB" # extra tests on tree/branch linus/master git bisect good 03c21cb775a313f1ff19be59c5d02df3e3526471 # 22:52 65+ 67 Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost # extra tests on tree/branch linux-next/master git bisect bad 888c8375131656144c1605071eab2eb6ac49abc3 # 22:52 0- 19 Add linux-next specific files for 20160128 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=yocto-minimal-x86_64.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu Haswell,+smep,+smap -kernel $kernel -initrd $initrd -m 256 -smp 1 -device e1000,netdev=net0 -netdev user,id=net0 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation

4 years, 10 months

3
2
0 / 0

[i2c-mux] 3489746df0: WARNING: CPU: 0 PID: 1 at drivers/base/devres.c:888 devm_kfree()

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://github.com/0day-ci/linux Peter-Rosin/i2c-mux-cleanup-and-locking-update/20160106-000205 commit 3489746df00f2f18b1a6dc2ba4de263e980cfbb0 Author: Peter Rosin <peda(a)axentia.se> AuthorDate: Tue Jan 5 16:57:13 2016 +0100 Commit: 0day robot <fengguang.wu(a)intel.com> CommitDate: Wed Jan 6 00:02:09 2016 +0800 i2c-mux: move the slave side adapter management to i2c_mux_core All muxes have slave side adapters, many have some arbitrary number of them. Handle this in the mux core, so that drivers are simplified. Add i2c_mux_reserve_adapter that can be used when it is known in advance how many child adapters that is to be added. This avoids reallocating memory. Drop i2c_del_mux_adapter and replace it with i2c_del_mux_adapters, since no mux driver is dynamically deleting individual child adapters anyway. Signed-off-by: Peter Rosin <peda(a)axentia.se> +------------------------------------------------------------------+------------+------------+------------+ | | f3ce0531d6 | 3489746df0 | 328de02ee7 | +------------------------------------------------------------------+------------+------------+------------+ | boot_successes | 16 | 0 | 0 | | boot_failures | 76 | 22 | 43 | | Out_of_memory:Kill_process | 46 | | | | Kernel_panic-not_syncing:Out_of_memory_and_no_killable_processes | 2 | | | | Kernel_panic-not_syncing:Attempted_to_kill_init!exitcode= | 30 | 21 | 13 | | WARNING:at_drivers/base/devres.c:#devm_kfree() | 0 | 22 | 43 | | backtrace:of_unittest | 0 | 22 | 43 | | backtrace:kernel_init_freeable | 0 | 22 | 43 | +------------------------------------------------------------------+------------+------------+------------+ [ 17.872856] overlay_removal_is_ok: overlay #5 is not topmost [ 17.876661] of_overlay_destroy: removal check failed for overlay #5 [ 17.881163] ------------[ cut here ]------------ [ 17.882437] WARNING: CPU: 0 PID: 1 at drivers/base/devres.c:888 devm_kfree+0x61/0x76() [ 17.884828] CPU: 0 PID: 1 Comm: swapper Not tainted 4.4.0-rc3-00086-g3489746 #1 [ 17.886694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 17.889121] 0000000000000000 ffff88000fa3bb60 ffffffff8177aefa ffff88000fa3bb98 [ 17.891562] ffffffff81119965 ffffffff81d0a80c 0000000000000001 00000000fffffffe [ 17.893975] ffff88000c7e1698 ffff88000c757820 ffff88000fa3bba8 ffffffff81119a50 [ 17.899218] Call Trace: [ 17.900013] [<ffffffff8177aefa>] dump_stack+0x19/0x1b [ 17.901286] [<ffffffff81119965>] warn_slowpath_common+0xb6/0xcf [ 17.902684] [<ffffffff81d0a80c>] ? devm_kfree+0x61/0x76 [ 17.903953] [<ffffffff81119a50>] warn_slowpath_null+0x1a/0x1c [ 17.905343] [<ffffffff81d0a80c>] devm_kfree+0x61/0x76 [ 17.906657] [<ffffffff8252ad4c>] i2c_mux_reserve_adapters+0xae/0xbf [ 17.908324] [<ffffffff82842105>] unittest_i2c_mux_probe+0x1af/0x234 [ 17.911148] [<ffffffff82841f56>] ? unittest_i2c_mux_remove+0x90/0x90 [ 17.912678] [<ffffffff82526726>] i2c_device_probe+0x370/0x3b9 [ 17.914076] [<ffffffff81d05602>] driver_probe_device+0x2c5/0x6db [ 17.915554] [<ffffffff81d05ab7>] __driver_attach+0x9f/0xd5 [ 17.917142] [<ffffffff81d05a18>] ? driver_probe_device+0x6db/0x6db [ 17.918704] [<ffffffff81d02f3a>] bus_for_each_dev+0x91/0xa9 [ 17.920106] [<ffffffff81d05dc4>] driver_attach+0x1e/0x20 [ 17.921441] [<ffffffff81d03bd7>] bus_add_driver+0x1f7/0x36d [ 17.922866] [<ffffffff81d06a93>] driver_register+0x10d/0x17f [ 17.924278] [<ffffffff82528714>] i2c_register_driver+0xb1/0x147 [ 17.925762] [<ffffffff84f32561>] of_unittest_overlay+0xf05/0x1304 [ 17.950459] [<ffffffff84f339e8>] of_unittest+0x1088/0x10b6 [ 17.951884] [<ffffffff817b1043>] ? debug_check_no_obj_freed+0x26/0x28 [ 17.953463] [<ffffffff8125a594>] ? kfree+0x2c7/0x307 [ 17.954749] [<ffffffff84e87520>] ? do_one_initcall+0xf8/0x26c [ 17.956209] [<ffffffff84f32960>] ? of_unittest_overlay+0x1304/0x1304 [ 17.958778] [<ffffffff84e875a1>] do_one_initcall+0x179/0x26c [ 17.960190] [<ffffffff84e877f7>] kernel_init_freeable+0x163/0x22b [ 17.961705] [<ffffffff82e3cc03>] ? rest_init+0x7a/0x7a [ 17.963019] [<ffffffff82e3cc11>] kernel_init+0xe/0x13f [ 17.964374] [<ffffffff82e4bfcf>] ret_from_fork+0x3f/0x70 [ 17.965704] [<ffffffff82e3cc03>] ? rest_init+0x7a/0x7a [ 17.967146] ---[ end trace 3b2b863e959dd3c5 ]--- [ 17.968973] i2c i2c-0: Added multiplexed i2c bus 1 git bisect start 328de02ee7475df3b3e9849542e8d15adaaaa894 168309855a7d1e16db751e9c647119fe2d2dc878 -- git bisect bad 6115822e890d81471eda3c35bf203bf0e1cf2726 # 01:23 0- 18 Merge 'linux-review/Janusz-Dziedzic/mac80211-check-requested-flags-in-ieee80211_tx_prepare_skb/20160105-183905' into devel-spot-201601060011 git bisect bad 579e9f8621453bc381b04a6ad787e9c113b1eeaa # 01:34 0- 2 Merge 'peterz-queue/x86/mm' into devel-spot-201601060011 git bisect bad c13a4bf22a1ca649904deec37080e617fabc661f # 01:47 0- 17 Merge 'linux-review/Adam-Thomson/ASoC-da7219-Correct-BCLK-inversion-for-DSP-DAI-format-mode/20160105-230859' into devel-spot-201601060011 git bisect bad 8cac6f8fb02d6f55c25415e705410fc070a588ec # 01:55 0- 6 Merge 'cgroup/for-4.5' into devel-spot-201601060011 git bisect bad 4dd116948235f2da4e8d88f7df549b8311b16358 # 02:10 0- 4 Merge 'linux-review/Craig-Gallek/soreuseport-change-consume_skb-to-kfree_skb-in-error-case/20160105-235855' into devel-spot-201601060011 git bisect good bdca81bda03d112d0a2f7e972a6da677ef8bedf3 # 02:19 22+ 24 0day base guard for 'devel-spot-201601060011' git bisect bad b5f2c558c3f81a5ddb89b1b34089d3a4357d05a1 # 02:26 0- 7 Merge 'linux-review/Peter-Rosin/i2c-mux-cleanup-and-locking-update/20160106-000205' into devel-spot-201601060011 git bisect good 5398b7ef6cdbc88581457b26b924081fc53d1de8 # 02:34 22+ 22 i2c: xlr: add interrupt support for Sigma Designs chips git bisect good fcd0f469776ea6a2bef9c8ee535053a644c3d01a # 02:50 21+ 18 Merge branch 'i2c/for-4.5' into i2c/for-next git bisect good bdcadbaaf804f56ee4060aada6c0162c91a540ec # 02:59 22+ 15 Merge branch 'i2c/for-4.5' into i2c/for-next git bisect good 4222e8c152a36c482959d975d714cfbd0ccbd1e6 # 03:04 22+ 22 i2c-mux: add common core data for every mux instance git bisect bad 3489746df00f2f18b1a6dc2ba4de263e980cfbb0 # 03:14 0- 3 i2c-mux: move the slave side adapter management to i2c_mux_core git bisect good f3ce0531d6bf0bed5682a36cdf2e8bf044f3b67f # 03:46 22+ 11 i2c-mux: move select and deselect ops to i2c_mux_core # first bad commit: [3489746df00f2f18b1a6dc2ba4de263e980cfbb0] i2c-mux: move the slave side adapter management to i2c_mux_core git bisect good f3ce0531d6bf0bed5682a36cdf2e8bf044f3b67f # 03:49 66+ 76 i2c-mux: move select and deselect ops to i2c_mux_core # extra tests with DEBUG_INFO git bisect bad 3489746df00f2f18b1a6dc2ba4de263e980cfbb0 # 04:00 0- 17 i2c-mux: move the slave side adapter management to i2c_mux_core # extra tests on HEAD of linux-devel/devel-spot-201601060011 git bisect bad 328de02ee7475df3b3e9849542e8d15adaaaa894 # 04:00 0- 43 0day head guard for 'devel-spot-201601060011' # extra tests on tree/branch linux-review/Peter-Rosin/i2c-mux-cleanup-and-locking-update/20160106-000205 git bisect bad b1d2a9bab92639356ad5a2fda8ea046d1216c11c # 04:15 0- 14 i2c-mux: relax locking of the top i2c adapter during i2c controlled muxing # extra tests on tree/branch linus/master git bisect good 168309855a7d1e16db751e9c647119fe2d2dc878 # 04:19 66+ 165 Linux 4.4-rc8 # extra tests on tree/branch linux-next/master git bisect good 8ef79cd05e6894c01ab9b41aa918a402fa8022a7 # 04:34 66+ 58 Add linux-next specific files for 20160105 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=yocto-minimal-x86_64.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu Haswell,+smep,+smap -kernel $kernel -initrd $initrd -m 256 -smp 1 -device e1000,netdev=net0 -netdev user,id=net0 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation

4 years, 11 months

3
2
0 / 0

[lkp] [pipe] 759c01142a: -51.5% hackbench.throughput

by kernel test robot

FYI, we noticed the below changes on https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master commit 759c01142a5d0f364a462346168a56de28a80f52 ("pipe: limit the per-user amount of pages allocated in pipes") ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase: gcc-4.9/performance/pipe/x86_64-rhel/threads/1600%/debian-x86_64-2015-02-07.cgz/lkp-wsx02/hackbench commit: 558041d8d21b48287224dd0e32cf19563c77607c 759c01142a5d0f364a462346168a56de28a80f52 558041d8d21b4828 759c01142a5d0f364a46234616 ---------------- -------------------------- %stddev %change %stddev \ | \ 219967 ± 4% -51.5% 106583 ± 1% hackbench.throughput 5.319e+08 ± 11% -19.7% 4.273e+08 ± 0% hackbench.time.involuntary_context_switches 2422668 ± 9% -36.4% 1540731 ± 2% hackbench.time.minor_page_faults 7127 ± 11% +9.4% 7800 ± 0% hackbench.time.percent_of_cpu_this_job_got 42924 ± 9% +11.9% 48049 ± 1% hackbench.time.system_time 1665 ± 11% -49.3% 844.05 ± 1% hackbench.time.user_time 534.25 ± 55% +298.5% 2129 ± 67% numa-meminfo.node3.AnonHugePages 8581 ± 57% -48.4% 4427 ± 1% uptime.idle 5705345 ± 13% +58.8% 9061490 ± 5% softirqs.RCU 279868 ± 8% +20.4% 336916 ± 1% softirqs.SCHED 26.75 ± 23% +245.8% 92.50 ± 39% vmstat.procs.b 9809 ± 13% -64.7% 3463 ± 7% vmstat.procs.r 284320 ± 13% -45.9% 153686 ± 1% vmstat.system.in 114.00 ± 10% -50.9% 56.00 ± 0% time.file_system_outputs 5.319e+08 ± 11% -19.7% 4.273e+08 ± 0% time.involuntary_context_switches 2422668 ± 9% -36.4% 1540731 ± 2% time.minor_page_faults 1665 ± 11% -49.3% 844.05 ± 1% time.user_time 89.86 ± 10% +8.9% 97.84 ± 0% turbostat.%Busy 2270 ± 10% +8.9% 2471 ± 0% turbostat.Avg_MHz 1.45 ± 8% -52.6% 0.69 ± 9% turbostat.CPU%c1 0.17 ± 13% -52.2% 0.08 ± 34% turbostat.CPU%c3 8.53 ±116% -83.7% 1.39 ± 3% turbostat.CPU%c6 63583716 ± 16% -98.1% 1183913 ± 53% numa-numastat.node0.local_node 63587769 ± 16% -98.1% 1189061 ± 53% numa-numastat.node0.numa_hit 69989955 ± 6% -96.8% 2239323 ± 8% numa-numastat.node1.local_node 69995282 ± 6% -96.8% 2244474 ± 8% numa-numastat.node1.numa_hit 56620071 ± 22% -97.3% 1520012 ± 32% numa-numastat.node2.local_node 56625301 ± 22% -97.3% 1522594 ± 31% numa-numastat.node2.numa_hit 63753303 ± 22% -96.1% 2508634 ± 14% numa-numastat.node3.local_node 63754612 ± 22% -96.1% 2511213 ± 14% numa-numastat.node3.numa_hit 32233121 ± 15% -97.4% 830659 ± 60% numa-vmstat.node0.numa_hit 32193958 ± 15% -97.5% 790185 ± 63% numa-vmstat.node0.numa_local 35020615 ± 10% -96.4% 1253289 ± 8% numa-vmstat.node1.numa_hit 34968563 ± 10% -96.6% 1201078 ± 9% numa-vmstat.node1.numa_local 30394713 ± 14% -97.2% 843134 ± 31% numa-vmstat.node2.numa_hit 30373607 ± 14% -97.3% 824676 ± 31% numa-vmstat.node2.numa_local 32334081 ± 21% -95.5% 1469250 ± 16% numa-vmstat.node3.numa_hit 32286373 ± 21% -95.6% 1420300 ± 16% numa-vmstat.node3.numa_local 1.868e+08 ± 11% -51.0% 91569694 ± 13% cpuidle.C1-NHM.time 1743049 ± 10% -34.6% 1139878 ± 11% cpuidle.C1-NHM.usage 1.918e+08 ± 12% -49.7% 96433830 ± 11% cpuidle.C1E-NHM.time 1243978 ± 16% -40.7% 738168 ± 12% cpuidle.C1E-NHM.usage 70404853 ± 8% -47.4% 37040591 ± 19% cpuidle.C3-NHM.time 4.723e+09 ±108% -81.6% 8.672e+08 ± 3% cpuidle.C6-NHM.time 311666 ± 18% -47.9% 162252 ± 15% cpuidle.C6-NHM.usage 2.456e+08 ± 18% -53.2% 1.15e+08 ± 9% cpuidle.POLL.time 260576 ± 21% -58.6% 107765 ± 17% cpuidle.POLL.usage 906770 ± 8% -19.4% 730530 ± 5% proc-vmstat.numa_hint_faults 2.54e+08 ± 11% -97.1% 7464740 ± 4% proc-vmstat.numa_hit 2.539e+08 ± 11% -97.1% 7449281 ± 4% proc-vmstat.numa_local 662291 ± 10% -33.4% 441016 ± 5% proc-vmstat.numa_pages_migrated 1203460 ± 6% -16.7% 1002984 ± 5% proc-vmstat.numa_pte_updates 5716982 ± 17% -97.2% 158482 ± 40% proc-vmstat.pgalloc_dma32 2.521e+08 ± 11% -96.3% 9338972 ± 3% proc-vmstat.pgalloc_normal 3627983 ± 4% -26.1% 2682860 ± 1% proc-vmstat.pgfault 2.576e+08 ± 11% -96.4% 9352801 ± 3% proc-vmstat.pgfree 662291 ± 10% -33.4% 441016 ± 5% proc-vmstat.pgmigrate_success 27845 ± 11% -79.8% 5624 ± 1% slabinfo.kmalloc-1024.active_objs 879.00 ± 11% -79.2% 183.00 ± 1% slabinfo.kmalloc-1024.active_slabs 28142 ± 11% -79.1% 5879 ± 0% slabinfo.kmalloc-1024.num_objs 879.00 ± 11% -79.2% 183.00 ± 1% slabinfo.kmalloc-1024.num_slabs 20288 ± 0% +129.9% 46639 ± 0% slabinfo.kmalloc-64.active_objs 317.00 ± 0% +131.4% 733.50 ± 0% slabinfo.kmalloc-64.active_slabs 20288 ± 0% +131.6% 46980 ± 0% slabinfo.kmalloc-64.num_objs 317.00 ± 0% +131.4% 733.50 ± 0% slabinfo.kmalloc-64.num_slabs 335.50 ± 9% -44.8% 185.25 ± 4% slabinfo.taskstats.active_objs 335.50 ± 9% -44.8% 185.25 ± 4% slabinfo.taskstats.num_objs 38.97 ± 29% +51.7% 59.11 ± 7% perf-profile.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 3.89 ± 60% +334.7% 16.93 ± 13% perf-profile.cycles-pp.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath 0.70 ± 30% +1969.3% 14.49 ± 11% perf-profile.cycles-pp.__wake_up_common.__wake_up_sync_key.pipe_read.__vfs_read.vfs_read 0.71 ± 30% +1971.0% 14.65 ± 11% perf-profile.cycles-pp.__wake_up_sync_key.pipe_read.__vfs_read.vfs_read.sys_read 0.69 ± 84% -92.3% 0.05 ± 63% perf-profile.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.syscall_return_slowpath 34.58 ± 33% +57.1% 54.33 ± 7% perf-profile.cycles-pp._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task 39.61 ± 28% +53.7% 60.89 ± 7% perf-profile.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function 0.70 ± 30% +1967.9% 14.47 ± 11% perf-profile.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read.__vfs_read 0.70 ± 30% +1965.0% 14.46 ± 11% perf-profile.cycles-pp.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read 38.99 ± 28% +54.5% 60.25 ± 7% perf-profile.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 39.45 ± 28% +53.9% 60.72 ± 7% perf-profile.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function 57.97 ± 20% +40.4% 81.35 ± 8% perf-profile.cycles-pp.entry_SYSCALL_64_fastpath 1.31 ± 60% -92.0% 0.10 ± 67% perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop 2.54 ± 68% -81.8% 0.46 ± 60% perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.default_wake_function.autoremove_wake_function 34.38 ± 33% +55.7% 53.54 ± 6% perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 3.67 ± 65% +354.5% 16.69 ± 11% perf-profile.cycles-pp.pipe_read.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath 5.22 ± 42% +250.6% 18.30 ± 21% perf-profile.cycles-pp.sys_read.entry_SYSCALL_64_fastpath 39.95 ± 28% +53.0% 61.10 ± 7% perf-profile.cycles-pp.ttwu_do_activate.constprop.86.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common 4.70 ± 48% +282.5% 17.96 ± 19% perf-profile.cycles-pp.vfs_read.sys_read.entry_SYSCALL_64_fastpath 16323355 ± 13% -100.0% 0.00 ± -1% latency_stats.avg.nfs_wait_on_request.nfs_updatepage.nfs_write_end.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.nfs_file_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 1702613 ± 26% +288.1% 6607185 ± 61% latency_stats.avg.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath 31.67 ± 74% +63999.7% 20298 ±149% latency_stats.avg.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 3607008 ±163% -100.0% 384.25 ± 14% latency_stats.avg.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range.filemap_write_and_wait_range.nfs4_file_fsync.[nfsv4].vfs_fsync_range.vfs_fsync.nfs4_file_flush.[nfsv4].filp_close.do_dup2.SyS_dup2.entry_SYSCALL_64_fastpath 1042512 ± 15% +145.7% 2561649 ± 35% latency_stats.avg.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 384229 ± 2% -76.9% 88748 ± 24% latency_stats.hits.call_rwsem_down_read_failed.do_exit.SyS_exit.entry_SYSCALL_64_fastpath 22676827 ± 14% +762.3% 1.955e+08 ± 0% latency_stats.hits.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 108077 ± 11% -96.9% 3361 ± 8% latency_stats.hits.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 9791700 ± 80% +173.5% 26782186 ± 72% latency_stats.max.call_rwsem_down_read_failed.__do_page_fault.do_page_fault.page_fault.copy_page_from_iter.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 8696153 ± 89% +196.7% 25800599 ± 75% latency_stats.max.call_rwsem_down_read_failed.__do_page_fault.do_page_fault.page_fault.copy_page_to_iter.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath 13849 ±173% -64.1% 4975 ±102% latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.return_from_execve 42431494 ± 56% -100.0% 0.00 ± -1% latency_stats.max.nfs_wait_on_request.nfs_updatepage.nfs_write_end.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.nfs_file_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 11940618 ± 27% +124.4% 26793707 ± 72% latency_stats.max.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 31.67 ± 74% +73210.5% 23215 ±124% latency_stats.max.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 27276522 ±155% -100.0% 473.50 ± 18% latency_stats.max.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range.filemap_write_and_wait_range.nfs4_file_fsync.[nfsv4].vfs_fsync_range.vfs_fsync.nfs4_file_flush.[nfsv4].filp_close.do_dup2.SyS_dup2.entry_SYSCALL_64_fastpath 86431834 ±106% +708.2% 6.985e+08 ± 40% latency_stats.sum.call_rwsem_down_read_failed.__do_page_fault.do_page_fault.page_fault.copy_page_to_iter.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath 14619 ±173% -65.9% 4980 ±102% latency_stats.sum.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.return_from_execve 44595961 ± 58% -100.0% 0.00 ± -1% latency_stats.sum.nfs_wait_on_request.nfs_updatepage.nfs_write_end.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.nfs_file_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 4.608e+08 ± 37% +202.9% 1.396e+09 ± 59% latency_stats.sum.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath 2.164e+10 ± 13% +960.4% 2.294e+11 ± 5% latency_stats.sum.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 1.117e+11 ± 16% -97.9% 2.338e+09 ± 29% latency_stats.sum.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 35826 ± 54% -70.9% 10422 ± 6% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_execute.rpc_run_task.nfs4_call_sync_sequence.[nfsv4]._nfs4_proc_access.[nfsv4].nfs4_proc_access.[nfsv4].nfs_do_access.nfs_permission.__inode_permission.inode_permission.link_path_walk 31.67 ± 74% +1.3e+05% 39883 ± 89% latency_stats.sum.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 40133821 ±161% -100.0% 2693 ± 14% latency_stats.sum.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range.filemap_write_and_wait_range.nfs4_file_fsync.[nfsv4].vfs_fsync_range.vfs_fsync.nfs4_file_flush.[nfsv4].filp_close.do_dup2.SyS_dup2.entry_SYSCALL_64_fastpath 9.857e+09 ± 19% +462.0% 5.539e+10 ± 40% latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 7.50 ± 54% +73.3% 13.00 ± 9% sched_debug.cfs_rq:/.load.17 12.00 ± 17% -35.4% 7.75 ± 14% sched_debug.cfs_rq:/.load.22 5.50 ± 41% +140.9% 13.25 ± 33% sched_debug.cfs_rq:/.load.23 6.25 ± 42% +100.0% 12.50 ± 23% sched_debug.cfs_rq:/.load.25 6.00 ± 51% +145.8% 14.75 ± 28% sched_debug.cfs_rq:/.load.29 5.75 ± 56% +121.7% 12.75 ± 29% sched_debug.cfs_rq:/.load.35 7.50 ± 74% +86.7% 14.00 ± 44% sched_debug.cfs_rq:/.load.43 11.75 ± 20% -34.0% 7.75 ± 34% sched_debug.cfs_rq:/.load.46 5.75 ± 39% +113.0% 12.25 ± 18% sched_debug.cfs_rq:/.load.53 12.75 ± 12% -31.4% 8.75 ± 9% sched_debug.cfs_rq:/.load.54 13.75 ± 31% -40.0% 8.25 ± 30% sched_debug.cfs_rq:/.load.58 27.00 ± 49% -58.3% 11.25 ± 20% sched_debug.cfs_rq:/.load_avg.24 28.25 ± 58% -61.9% 10.75 ± 20% sched_debug.cfs_rq:/.load_avg.26 22.25 ± 51% -51.7% 10.75 ± 13% sched_debug.cfs_rq:/.load_avg.28 20.75 ± 48% -53.0% 9.75 ± 15% sched_debug.cfs_rq:/.load_avg.62 182.64 ± 15% -41.5% 106.77 ± 18% sched_debug.cfs_rq:/.load_avg.max 1.14 ± 30% +65.7% 1.90 ± 18% sched_debug.cfs_rq:/.load_avg.min 30.53 ± 7% -39.5% 18.46 ± 20% sched_debug.cfs_rq:/.load_avg.stddev 61751943 ± 3% -29.5% 43522729 ± 13% sched_debug.cfs_rq:/.min_vruntime.1 63695496 ± 3% -25.5% 47470305 ± 13% sched_debug.cfs_rq:/.min_vruntime.13 64969515 ± 3% -27.9% 46847510 ± 14% sched_debug.cfs_rq:/.min_vruntime.17 65957991 ± 1% -27.8% 47648514 ± 13% sched_debug.cfs_rq:/.min_vruntime.21 64485085 ± 3% -26.1% 47642410 ± 13% sched_debug.cfs_rq:/.min_vruntime.25 64721403 ± 3% -28.9% 45992467 ± 16% sched_debug.cfs_rq:/.min_vruntime.29 65138437 ± 4% -29.3% 46039876 ± 12% sched_debug.cfs_rq:/.min_vruntime.33 66399877 ± 4% -26.6% 48722623 ± 12% sched_debug.cfs_rq:/.min_vruntime.37 67958189 ± 4% -29.4% 47968815 ± 16% sched_debug.cfs_rq:/.min_vruntime.41 67595662 ± 6% -27.4% 49096370 ± 13% sched_debug.cfs_rq:/.min_vruntime.45 67690444 ± 3% -27.6% 49000109 ± 12% sched_debug.cfs_rq:/.min_vruntime.49 64098981 ± 3% -26.2% 47277630 ± 14% sched_debug.cfs_rq:/.min_vruntime.5 67783223 ± 3% -27.4% 49216714 ± 12% sched_debug.cfs_rq:/.min_vruntime.53 66220648 ± 3% -25.3% 49481358 ± 11% sched_debug.cfs_rq:/.min_vruntime.57 68140688 ± 4% -30.7% 47207825 ± 16% sched_debug.cfs_rq:/.min_vruntime.61 65498493 ± 5% -25.8% 48590023 ± 13% sched_debug.cfs_rq:/.min_vruntime.65 65049958 ± 4% -28.2% 46691564 ± 18% sched_debug.cfs_rq:/.min_vruntime.69 68046848 ± 3% -26.8% 49835973 ± 13% sched_debug.cfs_rq:/.min_vruntime.73 66382113 ± 4% -26.9% 48557976 ± 18% sched_debug.cfs_rq:/.min_vruntime.77 65055617 ± 3% -28.7% 46408287 ± 15% sched_debug.cfs_rq:/.min_vruntime.9 64257203 ± 11% -19.4% 51816609 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg 81981181 ± 6% -14.9% 69761399 ± 5% sched_debug.cfs_rq:/.min_vruntime.max 7.25 ± 39% +348.3% 32.50 ± 68% sched_debug.cfs_rq:/.nr_spread_over.0 6.75 ± 28% +622.2% 48.75 ± 58% sched_debug.cfs_rq:/.nr_spread_over.1 3.50 ± 51% +1342.9% 50.50 ± 45% sched_debug.cfs_rq:/.nr_spread_over.10 4.25 ±100% +4358.8% 189.50 ±143% sched_debug.cfs_rq:/.nr_spread_over.13 7.25 ± 86% +2000.0% 152.25 ±109% sched_debug.cfs_rq:/.nr_spread_over.14 9.00 ± 39% +933.3% 93.00 ±105% sched_debug.cfs_rq:/.nr_spread_over.17 6.50 ± 77% +826.9% 60.25 ± 67% sched_debug.cfs_rq:/.nr_spread_over.18 9.75 ± 67% +220.5% 31.25 ± 79% sched_debug.cfs_rq:/.nr_spread_over.20 9.25 ± 67% +281.1% 35.25 ± 65% sched_debug.cfs_rq:/.nr_spread_over.21 4.25 ± 45% +1517.6% 68.75 ± 53% sched_debug.cfs_rq:/.nr_spread_over.22 5.50 ± 68% +581.8% 37.50 ± 92% sched_debug.cfs_rq:/.nr_spread_over.24 5.50 ± 30% +754.5% 47.00 ± 73% sched_debug.cfs_rq:/.nr_spread_over.25 6.25 ± 23% +1660.0% 110.00 ± 39% sched_debug.cfs_rq:/.nr_spread_over.26 8.00 ± 12% +343.8% 35.50 ± 39% sched_debug.cfs_rq:/.nr_spread_over.29 3.00 ± 78% +600.0% 21.00 ± 25% sched_debug.cfs_rq:/.nr_spread_over.32 5.75 ± 93% +856.5% 55.00 ± 67% sched_debug.cfs_rq:/.nr_spread_over.36 4.25 ± 65% +670.6% 32.75 ± 30% sched_debug.cfs_rq:/.nr_spread_over.37 4.75 ± 47% +1300.0% 66.50 ± 63% sched_debug.cfs_rq:/.nr_spread_over.38 6.25 ± 81% +812.0% 57.00 ± 43% sched_debug.cfs_rq:/.nr_spread_over.4 8.25 ± 89% +169.7% 22.25 ± 69% sched_debug.cfs_rq:/.nr_spread_over.41 3.00 ±102% +5608.3% 171.25 ±140% sched_debug.cfs_rq:/.nr_spread_over.42 5.25 ± 47% +566.7% 35.00 ± 27% sched_debug.cfs_rq:/.nr_spread_over.44 4.00 ± 46% +662.5% 30.50 ± 28% sched_debug.cfs_rq:/.nr_spread_over.45 3.25 ±121% +2761.5% 93.00 ± 87% sched_debug.cfs_rq:/.nr_spread_over.46 4.25 ± 58% +758.8% 36.50 ±104% sched_debug.cfs_rq:/.nr_spread_over.47 6.50 ± 60% +323.1% 27.50 ± 52% sched_debug.cfs_rq:/.nr_spread_over.48 7.50 ± 38% +800.0% 67.50 ± 81% sched_debug.cfs_rq:/.nr_spread_over.49 6.75 ± 21% +303.7% 27.25 ± 43% sched_debug.cfs_rq:/.nr_spread_over.5 2.75 ± 30% +1736.4% 50.50 ± 56% sched_debug.cfs_rq:/.nr_spread_over.50 3.75 ± 66% +1220.0% 49.50 ±117% sched_debug.cfs_rq:/.nr_spread_over.52 2.75 ± 64% +854.5% 26.25 ± 79% sched_debug.cfs_rq:/.nr_spread_over.53 2.75 ± 97% +3218.2% 91.25 ± 57% sched_debug.cfs_rq:/.nr_spread_over.54 3.00 ± 40% +900.0% 30.00 ± 66% sched_debug.cfs_rq:/.nr_spread_over.56 2.25 ± 57% +1700.0% 40.50 ± 77% sched_debug.cfs_rq:/.nr_spread_over.57 3.50 ± 58% +2771.4% 100.50 ± 69% sched_debug.cfs_rq:/.nr_spread_over.58 4.00 ±107% +893.8% 39.75 ± 82% sched_debug.cfs_rq:/.nr_spread_over.59 6.00 ± 72% +1283.3% 83.00 ± 36% sched_debug.cfs_rq:/.nr_spread_over.6 6.50 ± 77% +384.6% 31.50 ± 78% sched_debug.cfs_rq:/.nr_spread_over.61 3.50 ± 51% +1571.4% 58.50 ± 73% sched_debug.cfs_rq:/.nr_spread_over.62 6.25 ± 52% +356.0% 28.50 ± 45% sched_debug.cfs_rq:/.nr_spread_over.64 7.75 ± 84% +106.5% 16.00 ± 41% sched_debug.cfs_rq:/.nr_spread_over.65 6.50 ± 72% +319.2% 27.25 ± 83% sched_debug.cfs_rq:/.nr_spread_over.68 8.50 ± 59% +329.4% 36.50 ± 57% sched_debug.cfs_rq:/.nr_spread_over.69 5.75 ± 56% +308.7% 23.50 ± 60% sched_debug.cfs_rq:/.nr_spread_over.72 3.50 ± 65% +400.0% 17.50 ± 52% sched_debug.cfs_rq:/.nr_spread_over.73 3.25 ± 25% +876.9% 31.75 ± 68% sched_debug.cfs_rq:/.nr_spread_over.75 4.00 ± 63% +23906.2% 960.25 ±169% sched_debug.cfs_rq:/.nr_spread_over.76 5.00 ± 70% +720.0% 41.00 ± 54% sched_debug.cfs_rq:/.nr_spread_over.77 3.50 ± 47% +600.0% 24.50 ± 73% sched_debug.cfs_rq:/.nr_spread_over.8 6.75 ± 48% +370.4% 31.75 ± 83% sched_debug.cfs_rq:/.nr_spread_over.9 19.08 ± 14% +232.7% 63.48 ± 32% sched_debug.cfs_rq:/.nr_spread_over.avg 13.50 ± 21% -38.9% 8.25 ± 21% sched_debug.cfs_rq:/.runnable_load_avg.62 17.34 ± 33% -48.7% 8.90 ± 22% sched_debug.cfs_rq:/.runnable_load_avg.stddev 1650786 ± 96% +215.7% 5211288 ± 32% sched_debug.cfs_rq:/.spread0.24 3735888 ± 25% -66.2% 1261722 ± 76% sched_debug.cfs_rq:/.spread0.32 6947361 ±181% -153.7% -3730893 ±-158% sched_debug.cfs_rq:/.spread0.41 822224 ± 6% +262.2% 2978378 ± 55% sched_debug.cpu.avg_idle.11 899425 ± 22% +115.1% 1934526 ± 63% sched_debug.cpu.avg_idle.27 902557 ± 11% +161.2% 2357181 ± 71% sched_debug.cpu.avg_idle.50 855811 ± 2% +90.8% 1632603 ± 74% sched_debug.cpu.avg_idle.51 968895 ± 7% +160.6% 2524849 ± 36% sched_debug.cpu.avg_idle.52 1175701 ± 48% +111.3% 2484589 ± 71% sched_debug.cpu.avg_idle.69 5.00 ± 44% +120.0% 11.00 ± 33% sched_debug.cpu.cpu_load[0].23 13.50 ± 21% -38.9% 8.25 ± 21% sched_debug.cpu.cpu_load[0].62 0.94 ± 25% +74.1% 1.64 ± 20% sched_debug.cpu.cpu_load[0].min 5.00 ± 44% +120.0% 11.00 ± 33% sched_debug.cpu.cpu_load[1].23 5.75 ± 31% +91.3% 11.00 ± 23% sched_debug.cpu.cpu_load[1].45 13.50 ± 21% -38.9% 8.25 ± 21% sched_debug.cpu.cpu_load[1].62 0.94 ± 25% +74.1% 1.64 ± 20% sched_debug.cpu.cpu_load[1].min 17.26 ± 33% -40.2% 10.33 ± 41% sched_debug.cpu.cpu_load[1].stddev 5.50 ± 32% +100.0% 11.00 ± 23% sched_debug.cpu.cpu_load[2].45 13.50 ± 21% -38.9% 8.25 ± 21% sched_debug.cpu.cpu_load[2].62 17.07 ± 33% -43.4% 9.66 ± 32% sched_debug.cpu.cpu_load[2].stddev 5.00 ± 42% +120.0% 11.00 ± 23% sched_debug.cpu.cpu_load[3].45 13.75 ± 18% -40.0% 8.25 ± 21% sched_debug.cpu.cpu_load[3].62 16.85 ± 31% -42.2% 9.75 ± 32% sched_debug.cpu.cpu_load[3].stddev 4.75 ± 50% +131.6% 11.00 ± 23% sched_debug.cpu.cpu_load[4].45 13.75 ± 18% -40.0% 8.25 ± 21% sched_debug.cpu.cpu_load[4].62 13.50 ± 19% -37.0% 8.50 ± 24% sched_debug.cpu.cpu_load[4].78 16.46 ± 29% -40.2% 9.85 ± 33% sched_debug.cpu.cpu_load[4].stddev 45774 ± 14% -26.5% 33656 ± 21% sched_debug.cpu.curr->pid.67 7.50 ± 52% +76.7% 13.25 ± 8% sched_debug.cpu.load.17 12.25 ± 25% -36.7% 7.75 ± 14% sched_debug.cpu.load.20 11.75 ± 18% -36.2% 7.50 ± 14% sched_debug.cpu.load.22 5.75 ± 37% +126.1% 13.00 ± 32% sched_debug.cpu.load.23 6.00 ± 42% +104.2% 12.25 ± 26% sched_debug.cpu.load.25 5.75 ± 56% +126.1% 13.00 ± 30% sched_debug.cpu.load.35 12.00 ± 17% -35.4% 7.75 ± 34% sched_debug.cpu.load.46 5.75 ± 39% +117.4% 12.50 ± 18% sched_debug.cpu.load.53 12.50 ± 12% -30.0% 8.75 ± 9% sched_debug.cpu.load.54 505539 ± 1% +644.1% 3761938 ± 74% sched_debug.cpu.max_idle_balance_cost.11 719417 ± 52% +395.3% 3563254 ± 71% sched_debug.cpu.max_idle_balance_cost.37 744737 ± 44% +202.3% 2251473 ± 75% sched_debug.cpu.max_idle_balance_cost.40 534942 ± 7% +266.8% 1962058 ± 46% sched_debug.cpu.max_idle_balance_cost.52 2113103 ± 65% -68.9% 656815 ± 41% sched_debug.cpu.max_idle_balance_cost.65 630785 ± 17% -19.5% 507892 ± 1% sched_debug.cpu.max_idle_balance_cost.7 148.50 ± 45% -83.5% 24.50 ± 19% sched_debug.cpu.nr_running.0 165.00 ± 45% -85.8% 23.50 ± 38% sched_debug.cpu.nr_running.10 75.75 ± 29% -71.6% 21.50 ± 25% sched_debug.cpu.nr_running.11 124.50 ± 85% -82.9% 21.25 ± 19% sched_debug.cpu.nr_running.12 141.75 ± 29% -84.0% 22.75 ± 26% sched_debug.cpu.nr_running.14 102.25 ± 78% -71.1% 29.50 ± 33% sched_debug.cpu.nr_running.15 125.25 ± 83% -82.6% 21.75 ± 21% sched_debug.cpu.nr_running.16 157.25 ± 39% -85.2% 23.25 ± 34% sched_debug.cpu.nr_running.18 159.00 ± 66% -83.6% 26.00 ± 27% sched_debug.cpu.nr_running.19 148.00 ± 24% -80.9% 28.25 ± 29% sched_debug.cpu.nr_running.2 117.75 ± 49% -81.7% 21.50 ± 28% sched_debug.cpu.nr_running.20 135.00 ± 32% -86.1% 18.75 ± 27% sched_debug.cpu.nr_running.22 71.25 ± 27% -55.1% 32.00 ± 27% sched_debug.cpu.nr_running.23 243.50 ± 71% -89.4% 25.75 ± 8% sched_debug.cpu.nr_running.24 186.00 ± 51% -89.1% 20.25 ± 24% sched_debug.cpu.nr_running.26 95.75 ± 59% -73.4% 25.50 ± 29% sched_debug.cpu.nr_running.27 239.25 ± 99% -92.1% 19.00 ± 15% sched_debug.cpu.nr_running.28 98.25 ± 54% -66.9% 32.50 ± 22% sched_debug.cpu.nr_running.3 172.50 ± 44% -87.5% 21.50 ± 43% sched_debug.cpu.nr_running.30 66.50 ± 25% -63.9% 24.00 ± 25% sched_debug.cpu.nr_running.31 133.25 ± 48% -82.2% 23.75 ± 18% sched_debug.cpu.nr_running.32 139.75 ± 37% -82.3% 24.75 ± 27% sched_debug.cpu.nr_running.34 86.25 ± 75% -75.4% 21.25 ± 23% sched_debug.cpu.nr_running.36 62.75 ± 31% -58.2% 26.25 ± 30% sched_debug.cpu.nr_running.39 100.00 ± 72% -79.0% 21.00 ± 21% sched_debug.cpu.nr_running.4 92.00 ± 64% -78.0% 20.25 ± 15% sched_debug.cpu.nr_running.40 155.50 ± 46% -83.8% 25.25 ± 45% sched_debug.cpu.nr_running.42 62.25 ± 39% -55.8% 27.50 ± 30% sched_debug.cpu.nr_running.43 113.50 ±102% -79.5% 23.25 ± 24% sched_debug.cpu.nr_running.44 140.00 ± 29% -86.4% 19.00 ± 31% sched_debug.cpu.nr_running.46 311.50 ±114% -91.3% 27.25 ± 20% sched_debug.cpu.nr_running.47 112.75 ± 91% -81.2% 21.25 ± 5% sched_debug.cpu.nr_running.48 138.50 ± 34% -84.7% 21.25 ± 31% sched_debug.cpu.nr_running.50 88.75 ± 67% -72.7% 24.25 ± 33% sched_debug.cpu.nr_running.51 105.50 ± 51% -79.9% 21.25 ± 14% sched_debug.cpu.nr_running.52 129.50 ± 29% -84.6% 20.00 ± 21% sched_debug.cpu.nr_running.54 122.25 ± 51% -80.2% 24.25 ± 12% sched_debug.cpu.nr_running.56 163.25 ± 45% -87.7% 20.00 ± 36% sched_debug.cpu.nr_running.58 92.25 ± 66% -71.5% 26.25 ± 13% sched_debug.cpu.nr_running.59 230.75 ±101% -91.9% 18.75 ± 23% sched_debug.cpu.nr_running.60 144.75 ± 42% -82.6% 25.25 ± 41% sched_debug.cpu.nr_running.62 58.25 ± 32% -55.4% 26.00 ± 31% sched_debug.cpu.nr_running.63 122.25 ± 47% -81.6% 22.50 ± 9% sched_debug.cpu.nr_running.64 150.00 ± 38% -74.5% 38.25 ± 76% sched_debug.cpu.nr_running.66 89.25 ± 62% -75.1% 22.25 ± 43% sched_debug.cpu.nr_running.67 116.00 ± 60% -78.7% 24.75 ± 10% sched_debug.cpu.nr_running.68 88.25 ± 22% -67.4% 28.75 ± 25% sched_debug.cpu.nr_running.7 134.25 ± 40% -81.8% 24.50 ± 40% sched_debug.cpu.nr_running.70 69.00 ± 38% -59.8% 27.75 ± 22% sched_debug.cpu.nr_running.71 119.75 ± 42% -80.4% 23.50 ± 16% sched_debug.cpu.nr_running.72 138.50 ± 38% -74.5% 35.25 ± 93% sched_debug.cpu.nr_running.74 86.50 ± 61% -73.7% 22.75 ± 32% sched_debug.cpu.nr_running.75 115.50 ± 62% -81.8% 21.00 ± 10% sched_debug.cpu.nr_running.76 127.00 ± 35% -83.5% 21.00 ± 37% sched_debug.cpu.nr_running.78 59.25 ± 40% -66.2% 20.00 ± 44% sched_debug.cpu.nr_running.79 97.50 ± 54% -74.1% 25.25 ± 10% sched_debug.cpu.nr_running.8 106.72 ± 14% -69.9% 32.15 ± 23% sched_debug.cpu.nr_running.avg 975.12 ± 47% -72.2% 271.24 ± 72% sched_debug.cpu.nr_running.max 0.83 ± 26% +65.9% 1.38 ± 26% sched_debug.cpu.nr_running.min 189.39 ± 34% -74.1% 49.14 ± 62% sched_debug.cpu.nr_running.stddev 8049365 ± 5% -9.5% 7281434 ± 5% sched_debug.cpu.nr_switches.21 8222072 ± 4% -11.3% 7293398 ± 2% sched_debug.cpu.nr_switches.25 8355303 ± 5% -13.3% 7241351 ± 4% sched_debug.cpu.nr_switches.29 8377252 ± 5% -12.2% 7358538 ± 3% sched_debug.cpu.nr_switches.33 8646522 ± 6% -12.8% 7543084 ± 3% sched_debug.cpu.nr_switches.37 8680679 ± 6% -13.5% 7512566 ± 3% sched_debug.cpu.nr_switches.41 8862414 ± 6% -15.0% 7533561 ± 4% sched_debug.cpu.nr_switches.45 8513418 ± 5% -12.4% 7453546 ± 3% sched_debug.cpu.nr_switches.49 8554847 ± 6% -12.6% 7480415 ± 4% sched_debug.cpu.nr_switches.53 8818690 ± 3% -15.0% 7492325 ± 2% sched_debug.cpu.nr_switches.57 8446431 ± 5% -12.2% 7416662 ± 3% sched_debug.cpu.nr_switches.61 8424264 ± 6% -11.2% 7483465 ± 3% sched_debug.cpu.nr_switches.65 8874420 ± 6% -14.6% 7581001 ± 4% sched_debug.cpu.nr_switches.73 8548326 ± 3% -12.7% 7462215 ± 3% sched_debug.cpu.nr_switches.77 9638903 ± 5% -17.4% 7963391 ± 4% sched_debug.cpu.nr_switches.max 896810 ± 21% -68.9% 279227 ± 18% sched_debug.cpu.nr_switches.stddev 116.00 ± 29% -115.7% -18.25 ±-446% sched_debug.cpu.nr_uninterruptible.10 333.00 ± 85% -104.9% -16.25 ±-454% sched_debug.cpu.nr_uninterruptible.14 132.00 ± 65% -76.7% 30.75 ±119% sched_debug.cpu.nr_uninterruptible.16 170.00 ± 57% -100.9% -1.50 ±-8373% sched_debug.cpu.nr_uninterruptible.20 201.00 ± 66% -81.1% 38.00 ±318% sched_debug.cpu.nr_uninterruptible.24 277.00 ± 44% -74.4% 71.00 ± 96% sched_debug.cpu.nr_uninterruptible.25 213.75 ± 60% -76.8% 49.50 ± 63% sched_debug.cpu.nr_uninterruptible.33 80.25 ± 41% -109.7% -7.75 ±-722% sched_debug.cpu.nr_uninterruptible.40 350.00 ±100% -107.5% -26.25 ±-523% sched_debug.cpu.nr_uninterruptible.46 177.00 ± 35% -60.7% 69.50 ± 65% sched_debug.cpu.nr_uninterruptible.56 223.00 ± 55% -58.6% 92.25 ± 84% sched_debug.cpu.nr_uninterruptible.58 260.50 ± 67% -313.2% -555.50 ±-124% sched_debug.cpu.nr_uninterruptible.59 290.25 ± 78% -98.7% 3.75 ±246% sched_debug.cpu.nr_uninterruptible.6 139.00 ± 40% -87.4% 17.50 ±226% sched_debug.cpu.nr_uninterruptible.66 94.25 ± 96% -136.1% -34.00 ±-331% sched_debug.cpu.nr_uninterruptible.68 134.50 ± 69% -128.6% -38.50 ±-288% sched_debug.cpu.nr_uninterruptible.69 160.25 ± 60% -84.9% 24.25 ±377% sched_debug.cpu.nr_uninterruptible.73 224.00 ± 30% -71.5% 63.75 ± 24% sched_debug.cpu.nr_uninterruptible.78 104.50 ±112% -128.7% -30.00 ±-179% sched_debug.cpu.nr_uninterruptible.8 93.00 ± 30% -128.2% -26.25 ±-149% sched_debug.cpu.nr_uninterruptible.9 68.65 ± 25% -77.4% 15.53 ±150% sched_debug.cpu.nr_uninterruptible.avg 8050079 ± 5% -9.5% 7282381 ± 5% sched_debug.cpu.sched_count.21 8223158 ± 4% -11.3% 7293920 ± 2% sched_debug.cpu.sched_count.25 8355745 ± 5% -13.3% 7241867 ± 4% sched_debug.cpu.sched_count.29 8377564 ± 5% -12.2% 7358616 ± 3% sched_debug.cpu.sched_count.33 8647489 ± 6% -12.8% 7543179 ± 3% sched_debug.cpu.sched_count.37 8680933 ± 6% -13.5% 7512654 ± 3% sched_debug.cpu.sched_count.41 8862638 ± 6% -15.0% 7533639 ± 4% sched_debug.cpu.sched_count.45 8513659 ± 5% -12.5% 7453634 ± 3% sched_debug.cpu.sched_count.49 8555109 ± 6% -12.6% 7480502 ± 4% sched_debug.cpu.sched_count.53 8818934 ± 3% -15.0% 7492403 ± 2% sched_debug.cpu.sched_count.57 8446695 ± 5% -12.2% 7416735 ± 3% sched_debug.cpu.sched_count.61 8424736 ± 6% -11.2% 7483931 ± 3% sched_debug.cpu.sched_count.65 8874653 ± 6% -14.6% 7581099 ± 4% sched_debug.cpu.sched_count.73 8548563 ± 3% -12.7% 7462306 ± 3% sched_debug.cpu.sched_count.77 1277740 ± 13% -28.3% 915765 ± 1% sched_debug.cpu.sched_count.stddev 23897 ± 33% -52.1% 11437 ± 45% sched_debug.cpu.sched_goidle.0 24028 ± 11% -55.3% 10738 ± 19% sched_debug.cpu.sched_goidle.10 23579 ± 16% -50.5% 11679 ± 45% sched_debug.cpu.sched_goidle.12 25676 ± 27% -55.2% 11511 ± 21% sched_debug.cpu.sched_goidle.13 29635 ± 45% -54.6% 13447 ± 30% sched_debug.cpu.sched_goidle.15 24179 ± 31% -45.1% 13266 ± 33% sched_debug.cpu.sched_goidle.16 19085 ± 16% -50.5% 9456 ± 29% sched_debug.cpu.sched_goidle.2 26332 ± 24% -51.3% 12826 ± 23% sched_debug.cpu.sched_goidle.20 19407 ± 23% -26.8% 14211 ± 18% sched_debug.cpu.sched_goidle.21 23725 ± 10% -60.5% 9362 ± 22% sched_debug.cpu.sched_goidle.22 20885 ± 8% -30.4% 14541 ± 33% sched_debug.cpu.sched_goidle.23 26971 ± 35% -61.1% 10498 ± 25% sched_debug.cpu.sched_goidle.24 21959 ± 16% -41.4% 12865 ± 22% sched_debug.cpu.sched_goidle.25 21776 ± 12% -46.1% 11742 ± 23% sched_debug.cpu.sched_goidle.27 21108 ± 27% -56.6% 9163 ± 29% sched_debug.cpu.sched_goidle.30 18235 ± 46% -57.1% 7826 ± 51% sched_debug.cpu.sched_goidle.32 18074 ± 14% -44.3% 10059 ± 40% sched_debug.cpu.sched_goidle.33 15392 ± 23% -37.3% 9657 ± 13% sched_debug.cpu.sched_goidle.34 21667 ± 26% -59.5% 8783 ± 13% sched_debug.cpu.sched_goidle.38 19652 ± 10% -42.0% 11404 ± 25% sched_debug.cpu.sched_goidle.4 16821 ± 13% -57.7% 7113 ± 14% sched_debug.cpu.sched_goidle.42 17884 ± 27% -62.7% 6664 ± 72% sched_debug.cpu.sched_goidle.44 22708 ± 46% -63.9% 8192 ± 58% sched_debug.cpu.sched_goidle.48 19401 ± 26% -42.1% 11228 ± 5% sched_debug.cpu.sched_goidle.49 16767 ± 19% -44.3% 9332 ± 20% sched_debug.cpu.sched_goidle.50 15964 ± 27% -48.2% 8269 ± 29% sched_debug.cpu.sched_goidle.52 14561 ± 17% -31.7% 9951 ± 8% sched_debug.cpu.sched_goidle.53 16277 ± 22% -67.9% 5232 ± 35% sched_debug.cpu.sched_goidle.54 23015 ± 35% -51.2% 11236 ± 24% sched_debug.cpu.sched_goidle.6 15897 ± 14% -48.0% 8264 ± 27% sched_debug.cpu.sched_goidle.62 22641 ± 16% -46.0% 12217 ± 20% sched_debug.cpu.sched_goidle.64 23159 ± 27% -47.9% 12060 ± 15% sched_debug.cpu.sched_goidle.68 20571 ± 10% -30.7% 14246 ± 29% sched_debug.cpu.sched_goidle.69 21750 ± 25% -54.6% 9880 ± 38% sched_debug.cpu.sched_goidle.70 13572 ± 12% -35.5% 8757 ± 29% sched_debug.cpu.sched_goidle.73 15307 ± 20% -36.8% 9677 ± 40% sched_debug.cpu.sched_goidle.78 17866 ± 7% -33.2% 11942 ± 29% sched_debug.cpu.sched_goidle.9 18857 ± 12% -40.4% 11240 ± 18% sched_debug.cpu.sched_goidle.avg 43849 ± 13% -36.0% 28085 ± 10% sched_debug.cpu.sched_goidle.max 6115 ± 23% -50.4% 3034 ± 27% sched_debug.cpu.sched_goidle.min 7313 ± 9% -32.8% 4915 ± 8% sched_debug.cpu.sched_goidle.stddev 4341077 ± 9% +14.6% 4976682 ± 4% sched_debug.cpu.ttwu_count.0 372672 ± 13% -70.4% 110439 ± 8% sched_debug.cpu.ttwu_count.stddev 2955563 ± 9% +53.1% 4523498 ± 4% sched_debug.cpu.ttwu_local.0 3131208 ± 7% +43.3% 4486241 ± 5% sched_debug.cpu.ttwu_local.1 3001199 ± 15% +44.8% 4346826 ± 4% sched_debug.cpu.ttwu_local.10 3166403 ± 7% +35.4% 4288123 ± 7% sched_debug.cpu.ttwu_local.11 3100602 ± 13% +38.3% 4288026 ± 4% sched_debug.cpu.ttwu_local.12 3391546 ± 5% +30.8% 4435978 ± 5% sched_debug.cpu.ttwu_local.13 2959588 ± 16% +44.0% 4261936 ± 4% sched_debug.cpu.ttwu_local.14 3267265 ± 9% +32.4% 4326510 ± 7% sched_debug.cpu.ttwu_local.15 3137496 ± 18% +38.5% 4345588 ± 5% sched_debug.cpu.ttwu_local.16 3356134 ± 7% +31.1% 4399395 ± 4% sched_debug.cpu.ttwu_local.17 3108355 ± 17% +36.7% 4248516 ± 4% sched_debug.cpu.ttwu_local.18 3303788 ± 10% +31.6% 4346891 ± 6% sched_debug.cpu.ttwu_local.19 2790792 ± 16% +60.4% 4476856 ± 4% sched_debug.cpu.ttwu_local.2 3208488 ± 16% +36.0% 4363319 ± 5% sched_debug.cpu.ttwu_local.20 3323090 ± 6% +33.4% 4432674 ± 8% sched_debug.cpu.ttwu_local.21 2920237 ± 17% +50.4% 4391940 ± 3% sched_debug.cpu.ttwu_local.22 3133540 ± 6% +38.6% 4342196 ± 10% sched_debug.cpu.ttwu_local.23 3082052 ± 15% +40.8% 4340369 ± 5% sched_debug.cpu.ttwu_local.24 3410451 ± 2% +30.2% 4440794 ± 5% sched_debug.cpu.ttwu_local.25 2965013 ± 15% +44.9% 4295580 ± 5% sched_debug.cpu.ttwu_local.26 3140505 ± 7% +40.7% 4420128 ± 7% sched_debug.cpu.ttwu_local.27 3107859 ± 16% +40.0% 4352409 ± 5% sched_debug.cpu.ttwu_local.28 3558229 ± 8% +23.2% 4383658 ± 5% sched_debug.cpu.ttwu_local.29 3056265 ± 8% +48.4% 4534490 ± 5% sched_debug.cpu.ttwu_local.3 3064582 ± 16% +41.9% 4348948 ± 2% sched_debug.cpu.ttwu_local.30 3209662 ± 6% +36.0% 4366452 ± 6% sched_debug.cpu.ttwu_local.31 3525265 ± 14% +29.9% 4578608 ± 2% sched_debug.cpu.ttwu_local.32 3649268 ± 5% +25.8% 4592214 ± 5% sched_debug.cpu.ttwu_local.33 3181985 ± 18% +44.3% 4590269 ± 4% sched_debug.cpu.ttwu_local.34 3503558 ± 6% +34.7% 4719199 ± 5% sched_debug.cpu.ttwu_local.35 3510966 ± 13% +24.9% 4384477 ± 4% sched_debug.cpu.ttwu_local.36 3801989 ± 7% +19.7% 4550261 ± 5% sched_debug.cpu.ttwu_local.37 3197749 ± 14% +34.7% 4306364 ± 0% sched_debug.cpu.ttwu_local.38 3452371 ± 6% +27.1% 4387164 ± 7% sched_debug.cpu.ttwu_local.39 3106534 ± 14% +38.5% 4301950 ± 3% sched_debug.cpu.ttwu_local.4 3410508 ± 14% +27.4% 4345186 ± 5% sched_debug.cpu.ttwu_local.40 3854125 ± 6% +18.4% 4562285 ± 6% sched_debug.cpu.ttwu_local.41 3210250 ± 14% +38.7% 4453936 ± 3% sched_debug.cpu.ttwu_local.42 3393670 ± 7% +30.2% 4419376 ± 6% sched_debug.cpu.ttwu_local.43 3504231 ± 13% +24.3% 4356255 ± 4% sched_debug.cpu.ttwu_local.44 3828003 ± 6% +18.7% 4542494 ± 6% sched_debug.cpu.ttwu_local.45 3285330 ± 14% +31.1% 4305974 ± 3% sched_debug.cpu.ttwu_local.46 3457275 ± 11% +28.7% 4449353 ± 6% sched_debug.cpu.ttwu_local.47 3352884 ± 17% +32.3% 4436488 ± 4% sched_debug.cpu.ttwu_local.48 3674474 ± 4% +22.3% 4495693 ± 5% sched_debug.cpu.ttwu_local.49 3448117 ± 5% +27.9% 4408576 ± 7% sched_debug.cpu.ttwu_local.5 3300328 ± 15% +32.4% 4368728 ± 4% sched_debug.cpu.ttwu_local.50 3359540 ± 6% +31.7% 4426052 ± 6% sched_debug.cpu.ttwu_local.51 3398675 ± 15% +32.1% 4488804 ± 5% sched_debug.cpu.ttwu_local.52 3744113 ± 7% +22.3% 4579796 ± 5% sched_debug.cpu.ttwu_local.53 3186989 ± 15% +40.5% 4477419 ± 2% sched_debug.cpu.ttwu_local.54 3578762 ± 8% +25.2% 4478923 ± 9% sched_debug.cpu.ttwu_local.55 3390671 ± 15% +30.9% 4437633 ± 4% sched_debug.cpu.ttwu_local.56 3860784 ± 2% +17.4% 4534386 ± 4% sched_debug.cpu.ttwu_local.57 3205789 ± 12% +37.7% 4414254 ± 5% sched_debug.cpu.ttwu_local.58 3395352 ± 7% +32.7% 4506999 ± 7% sched_debug.cpu.ttwu_local.59 2915811 ± 16% +45.4% 4240300 ± 1% sched_debug.cpu.ttwu_local.6 3562805 ± 14% +23.5% 4401745 ± 6% sched_debug.cpu.ttwu_local.60 3715936 ± 6% +21.2% 4503014 ± 5% sched_debug.cpu.ttwu_local.61 3204936 ± 14% +39.2% 4461506 ± 2% sched_debug.cpu.ttwu_local.62 3507592 ± 4% +28.1% 4492218 ± 6% sched_debug.cpu.ttwu_local.63 3202967 ± 13% +37.3% 4398303 ± 4% sched_debug.cpu.ttwu_local.64 3483280 ± 7% +30.7% 4554002 ± 5% sched_debug.cpu.ttwu_local.65 3138219 ± 15% +39.9% 4389905 ± 4% sched_debug.cpu.ttwu_local.66 3153951 ± 6% +41.6% 4466967 ± 5% sched_debug.cpu.ttwu_local.67 3229444 ± 12% +37.5% 4441388 ± 5% sched_debug.cpu.ttwu_local.68 3394709 ± 3% +32.3% 4489810 ± 5% sched_debug.cpu.ttwu_local.69 3230002 ± 7% +33.4% 4308068 ± 6% sched_debug.cpu.ttwu_local.7 3056815 ± 13% +43.4% 4382407 ± 3% sched_debug.cpu.ttwu_local.70 3223738 ± 6% +37.6% 4437307 ± 7% sched_debug.cpu.ttwu_local.71 3460274 ± 13% +29.3% 4473753 ± 4% sched_debug.cpu.ttwu_local.72 3860628 ± 6% +19.2% 4600629 ± 4% sched_debug.cpu.ttwu_local.73 3361754 ± 15% +31.7% 4427085 ± 4% sched_debug.cpu.ttwu_local.74 3375963 ± 9% +32.4% 4468587 ± 5% sched_debug.cpu.ttwu_local.75 3416899 ± 13% +30.6% 4462381 ± 4% sched_debug.cpu.ttwu_local.76 3725021 ± 2% +22.7% 4571798 ± 4% sched_debug.cpu.ttwu_local.77 3301968 ± 9% +35.0% 4458653 ± 2% sched_debug.cpu.ttwu_local.78 3438172 ± 10% +31.5% 4519809 ± 8% sched_debug.cpu.ttwu_local.79 3296672 ± 16% +29.4% 4267155 ± 6% sched_debug.cpu.ttwu_local.8 3480776 ± 7% +27.5% 4437332 ± 5% sched_debug.cpu.ttwu_local.9 3329473 ± 7% +33.0% 4429258 ± 4% sched_debug.cpu.ttwu_local.avg 2541283 ± 13% +56.8% 3983513 ± 5% sched_debug.cpu.ttwu_local.min 414648 ± 17% -57.3% 176872 ± 3% sched_debug.cpu.ttwu_local.stddev 0.00 ± 85% -79.4% 0.00 ± 10% sched_debug.rt_rq:/.rt_time.24 0.03 ±149% -96.4% 0.00 ± 27% sched_debug.rt_rq:/.rt_time.4 0.00 ± 58% -80.4% 0.00 ± 27% sched_debug.rt_rq:/.rt_time.9 lkp-wsx02: Westmere-EX Memory: 128G 5e+11 ++----------------------------------------------------------------+ 4.5e+11 ++ O | | | 4e+11 ++ | 3.5e+11 ++ | | | 3e+11 ++ | 2.5e+11 +O O O O O OO | 2e+11 O+ OO O O O O O | | | 1.5e+11 ++ | 1e+11 ++ | | | 5e+10 +*. *.* .***. *.**.**.** .* .* .** .* .* .** .* .**.**.***.**.* .** 0 *+-*---*-----*O--O------*--*--*---*--*--*---*--*---------------*--+ softirqs.RCU 1e+07 ++--------------------O---------------------------------------------+ 9e+06 OO O OO O O O O | | O OO O O | 8e+06 ++ O | 7e+06 ++ | |*.**.**.* *.* .* .* * **.***.* **.**.**.**.**.**.**. ** 6e+06 ++ *.* * * * : : : : : **.* : | 5e+06 ++ : : :: :: :: :: | 4e+06 ++ : : :: :: :: * | * : : * * * | 3e+06 ++ :: | 2e+06 ++ :: | | :: | 1e+06 ++ : | 0 ++-----------*-O--O-------------------------------------------------+ cpuidle.C6-NHM.time 3e+10 *+-------------------------*---------*----------------------------+ | : : | 2.5e+10 ++ : : | | : : | | * :: :: | 2e+10 ++ : : : :: | | ::: : :: | 1.5e+10 ++ ::: : : : | |: : : : : : * | 1e+10 ++ : * : : : : | |: : : : : :: | |: : : : : : : | 5e+09 ++ .* : : : : : :| |*.**.**.***. *.** *.** **.***.* **.***.**.**.**.***.**.* ** 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ turbostat.Avg_MHz 2500 O+OO-OO-OO-OO-O*-O*-OO-OO-O--------------------------**-**-**-**----** | **.**.** : : * : *.**.**.* **.**.**.** * :| | : : : : : : : : : | 2000 ++ : : : * : : : :: | |: : : : :: : :: * | |: : : : :: :: | 1500 ++ : : * : : | * : : * * | 1000 ++ :: | | :: | | :: | 500 ++ : | | : | | : | 0 ++------------*O--O--------------------------------------------------+ turbostat._Busy 100 O+OO-OO-OO-OO-O--O--OO-OO-O-------------------------------------------+ 90 ++**.**.**.** *.**.**.** *.**.*.** *.**.**.**.**.**.**.**.** *.* | : : : : : : : : : | 80 ++ : : : * : : : :: | 70 ++ : : : :: : : : * | |: : : : :: :: | 60 ++ : : * : : | 50 *+ : : * * | 40 ++ :: | | :: | 30 ++ :: | 20 ++ : | | : | 10 ++ : | 0 ++------------*O--O---------------------------------------------------+ turbostat.CPU_c6 60 ++---------------------------------------------------------------------+ | | 50 *+ * * | | : : | | * : : | 40 ++ : :: :: | |: :: :: :: | 30 ++ ::: : :: | |: ::: : : : * | 20 ++ : : : : : : | |: : * : : : :: | | : : : : : : : | 10 ++: *. : : : : : : | | **.**.**.**. *.*.* **.* **.*.**.* **.**.**.**.*.**.**.**.** *.* 0 O+OO-OO-OO-OO-OO-O-OO-OO-OO--------------------------------------------+ 18 ++---------------------------------------------------------------------+ | O | 16 O+ O O O O | 14 ++ O O O O | | O O | 12 ++O O | 10 ++ O O | | | 8 ++ | 6 ++ | | | 4 ++ | 2 ++ | | *.**. .**. .* *.*.* *.**. .* *.* .* .* 0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+ 18 ++---------------------------------------------------------------------+ | O | 16 O+ O O O O | 14 ++ O O O O | | O O | 12 ++ O | 10 ++O O O | | | 8 ++ | 6 ++ | | | 4 ++ | 2 ++ | | *.**. .**. .* *.*.* *.**. .* *.* .* .* 0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+ 18 ++---------------------------------------------------------------------+ | O | 16 O+ O O O O | 14 ++ O O O O | | O O | 12 ++ O | 10 ++O O O | | | 8 ++ | 6 ++ | | | 4 ++ | 2 ++ | | *.**. .**. .* *.*.* *.**. .* *.* .* .* 0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+ 18 ++---------------------------------------------------------------------+ | O | 16 O+ O O O O | 14 ++ O O O O | | O O | 12 ++ O | 10 ++O O O | | | 8 ++ | 6 ++ | | | 4 ++ | 2 ++ | | *.**. .**. .* *.*.* *.**. .* *.* .* .* 0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+ hackbench.throughput 300000 ++-----------------------------------------------------------------+ | | 250000 ++ *.* | |*.**.**. *.* .* *.**.** *.**.**. : : .**. .*| * * : ***.** *.**.* *.* * **. *.** ** | 200000 ++ : : * * | : : | 150000 ++ : : | | : : | 100000 OO OO OO OO OO: O OO OO O | | :: | | :: | 50000 ++ : | | : | 0 ++-----------*-O-O-------------------------------------------------+ hackbench.time.percent_of_cpu_this_job_got 8000 O+OO-OO-OO-OO-O--O--OO-OO-O------------------------------------------+ | **.**.**.** *.**. *.** *.**.**.* **.**.**.**.**.**.**.**.* ** 7000 ++: : : * : : : : : : | 6000 ++ : : : * : : : :: | |: : : : :: : :: * | 5000 ++ : : : :: :: | | : : * : : | 4000 *+ : : * * | | :: | 3000 ++ :: | 2000 ++ :: | | : | 1000 ++ : | | : | 0 ++------------*O--O--------------------------------------------------+ numa-numastat.node1.numa_hit 1.2e+08 ++----------------------------------------------------------------+ | | 1e+08 ++ * * * | | + :* +: + :.** * | |* ** *: * * * ** : * ** * : *. * : | 8e+07 +++ + : :: * + : * :: : : :: : : : ** + : * * |: ** : : :: * * :: : *.* :: *.* : * * + | 6e+07 ++ : * :: :: :: :: * *| | : : * * : : | 4e+07 *+ : : * * | | : : | | :: | 2e+07 ++ : | O : | 0 +O-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ numa-numastat.node1.local_node 1.2e+08 ++----------------------------------------------------------------+ | | 1e+08 ++ * * * | | + :* +: + :.** * | |* ** *: * * * ** : * ** * : *. * : | 8e+07 +++ + : :: * + : * :: : : :: : : : ** + : * * |: ** : : :: * * :: : *.* :: *.* : * * + | 6e+07 ++ : * :: :: :: :: * *| | : : * * : : | 4e+07 *+ : : * * | | : : | | :: | 2e+07 ++ : | O : | 0 +O-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ numa-numastat.node2.numa_hit 9e+07 ++------------------------------------------------------------------+ | * * * * | 8e+07 +*.* + : * **. :: **. :+ : | 7e+07 ++ * * *. : : * * : * * : : *.* .* | |: : : * * *.* :: : : : : : * : * : *| 6e+07 ++ :: : :: : : :: : : : : + *. *.* : :* 5e+07 ++ * : :*.* : : : *: * * * :: | * : : * : ** * :: | 4e+07 ++ : : :+ * | 3e+07 ++ :: * | | :: | 2e+07 ++ :: | 1e+07 ++ : | | : | 0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ numa-numastat.node2.local_node 9e+07 ++------------------------------------------------------------------+ | * * * * | 8e+07 +*.* + : * **. :: **. :+ : | 7e+07 ++ * * *. : : * * : * * : : *.* .* | |: : : * * *.* :: : : : : : * : * : *| 6e+07 ++ :: : :: : : :: : : : : + *. *.* : :* 5e+07 ++ * : :*.* : : : *: * * * :: | * : : * : ** * :: | 4e+07 ++ : : :+ * | 3e+07 ++ :: * | | :: | 2e+07 ++ :: | 1e+07 ++ : | | : | 0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ numa-numastat.node3.numa_hit 1e+08 ++------------------------------------------------*-----------------+ 9e+07 ++ :: | |* * * : : *| 8e+07 +++ * :: :: * : * * :| 7e+07 ++ * * :: * .* * : *. * : *. .* : ** : :+: : | |: *.* : :: * + * : : : ** : : ** : : + : * : : | 6e+07 ++ :: * :* *.* :: : : : :: ** *.** * 5e+07 ++ : : : :: :: : : : | 4e+07 ++ * : : * : *: * | * : : * * | 3e+07 ++ :: | 2e+07 ++ :: | | : | 1e+07 ++ : | 0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ numa-numastat.node3.local_node 1e+08 ++------------------------------------------------*-----------------+ 9e+07 ++ :: | |* * * : : *| 8e+07 +++ * :: :: * : * * :| 7e+07 ++ * * :: * .* * : *. * : *. .* : ** : :+: : | |: *.* : :: * + * : : : ** : : ** : : + : * : : | 6e+07 ++ :: * :* *.* :: : : : :: ** *.** * 5e+07 ++ : : : :: :: : : : | 4e+07 ++ * : : * : *: * | * : : * * | 3e+07 ++ :: | 2e+07 ++ :: | | : | 1e+07 ++ : | 0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ numa-vmstat.node0.numa_hit 5e+07 ++---------------*------------------------------------------------+ 4.5e+07 ++ : | | * : : .* .* | 4e+07 ++ * : * * : * * : * * : * * | 3.5e+07 +++ : +: : : :: * : : +: : : + * * * + : | |* ** :* : : : : : * ** : * ** + : * : * *. | 3e+07 ++ :: : ** * : :: :: **. : : : *| 2.5e+07 ++ *: * :: :: :: ** * * 2e+07 *+ : : * * * | | : : | 1.5e+07 ++ : : | 1e+07 ++ :: | | : | 5e+06 ++ : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ numa-vmstat.node0.numa_local 5e+07 ++---------------*------------------------------------------------+ 4.5e+07 ++ : | | * : : .* .* | 4e+07 ++ * : * * : * * : * * : * * | 3.5e+07 +++ : +: : : :: * : : +: : : + * * * + : | |* ** :* : : : : : * ** : * ** + : * : * *. | 3e+07 ++ :: : ** * : :: :: **. : : : *| 2.5e+07 ++ *: * :: :: :: ** * * 2e+07 *+ : : * * * | | : : | 1.5e+07 ++ : : | 1e+07 ++ :: | | : | 5e+06 ++ : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ numa-vmstat.node1.numa_hit 6e+07 ++------------------------------------------------------------------+ | .* .* * | 5e+07 ++ * : * : * : | | : : : : + : | |* .* : * * * * * : * * :*.* * | 4e+07 ++: * *.* : :: .* : :: * ::: * * :+ : *.** *. | | : : : * : :: * : : :: :+ :: :+ * *.* + : ** 3e+07 *+ * : : :: * :: * * * * * | | : : : : | 2e+07 ++ : : * * | | :: | | :: | 1e+07 ++ :: | O : | 0 +O-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ numa-vmstat.node1.numa_local 6e+07 ++------------------------------------------------------------------+ | .* .* * | 5e+07 ++ * : * : * : | | : : : : + : | |* .* : * * * * * : * * :*.* * | 4e+07 ++: * *.* : :: .* : :: * ::: * * :+ : *.** *. | | : : : * : :: * : : :: :+ :: :+ * *.* + : ** 3e+07 *+ * : : :: * :: * * * * * | | : : : : | 2e+07 ++ : : * * | | :: | | :: | 1e+07 ++ :: | O : | 0 +O-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ numa-vmstat.node2.numa_hit 5e+07 ++----------------------------------------------------------------+ 4.5e+07 ++ * * * * | |*. + : : * :: * :: * | 4e+07 ++ ** * * : + *.* * + *.* * :+ *. | 3.5e+07 *+ : *: * : * : * : : * .***.* * | | : :: * :* : : : : : :: : * : ** 3e+07 ++ * : :+ : +: : * * * : : :: | 2.5e+07 ++ : : * : * :: :.* :: | 2e+07 ++ : : * :: * * | | : : * | 1.5e+07 ++ :: | 1e+07 ++ :: | | : | 5e+06 ++ : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ numa-vmstat.node2.numa_local 5e+07 ++----------------------------------------------------------------+ 4.5e+07 ++ * * * * | |*. + : : * :: * :: * | 4e+07 ++ ** * : + *.* * + *.* * :+ | 3.5e+07 *+ : ** * : * : * : : * .***.**.* | | : :: * :* : : : : : :: : * : ** 3e+07 ++ * : :+ : +: : * * * : : :: | 2.5e+07 ++ : : * : * :: :.* :: | 2e+07 ++ : : * :: * * | | : : * | 1.5e+07 ++ :: | 1e+07 ++ :: | | : | 5e+06 ++ : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ numa-vmstat.node3.numa_hit 6e+07 ++------------------------------------------------------------------+ | | 5e+07 ++ * | |* * * :: | |:: * : : : : * *| 4e+07 ++: .** * :: * : : ** : : *.* * :* : * ::| | ** : :+ * : : ::: *.* : : *.* : : * + : :: : : | 3e+07 *+ : : * ::.** *.* : * : * : : * : :: *.** * | :: : :* :: : : :: * * | 2e+07 ++ : : : * : : | | * : : * * | | :: | 1e+07 ++ :: | | : | 0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ numa-vmstat.node3.numa_local 6e+07 ++------------------------------------------------------------------+ | | 5e+07 ++ * | |* * * : | |:: * : : : : * *| 4e+07 ++: .** * :: * : : ** : : *.* * : * : * :| | ** : :+ * : : ::: *.* : : *.* : : ::+ : :: : : | 3e+07 *+ : : * ::.** :.* : * : * : : * * : :: *.* : * | :: : :* * :: : : :: * * * | 2e+07 ++ : : : * : : | | * : : * * | | :: | 1e+07 ++ :: | | : | 0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+ proc-vmstat.numa_hit 3.5e+08 ++----------------------------------------------------------------+ | *. | 3e+08 +*.**.**. * **.***. **.***. : * .* *| |: * * .**. *.** : * : * *. .*** *.* :| 2.5e+08 ++ : * * : * : : : ** : : * |: : : : : : : : :: | 2e+08 ++ : : :: :: :: * | | : : :: : : | 1.5e+08 *+ : : * * * | | : : | 1e+08 ++ :: | | :: | 5e+07 ++ : | | : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ proc-vmstat.numa_local 3.5e+08 ++----------------------------------------------------------------+ | *. | 3e+08 +*.**.**. * **.***. **.***. : * .* *| |: * * .**. *.** : * : * *. .*** *.* :| 2.5e+08 ++ : * * : * : : : ** : : * |: : : : : : : : :: | 2e+08 ++ : : :: :: :: * | | : : :: : : | 1.5e+08 *+ : : * * * | | : : | 1e+08 ++ :: | | :: | 5e+07 ++ : | | : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ proc-vmstat.pgalloc_normal 3.5e+08 ++----------------------------------------------------------------+ | | 3e+08 +*. .**. * **. * **. *.* *| |: ** *** .* *.** :*.* * :*.* * *. .***.**.* :| 2.5e+08 ++ : * *.* : * : : : ** : : * |: : : : : : : : :: | 2e+08 ++ : : :: :: :: * | | : : :: : : | 1.5e+08 *+ : : * * * | | : : | 1e+08 ++ :: | | :: | 5e+07 ++ : | | : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ proc-vmstat.pgfree 3.5e+08 ++----------------------------------------------------------------+ |*. * * *.* | 3e+08 ++ **.**. * :*.***. :*.***. : : .* *| |: * * .**. *.** : * : * *.* .*** *.* : | 2.5e+08 ++ : * * : * : : : * :: * | : : : : : : : * | 2e+08 ++ : : :: :: :: | | : : :: : : | 1.5e+08 *+ : : * * * | | : : | 1e+08 ++ :: | | :: | 5e+07 ++ : | | : | 0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+ slabinfo.taskstats.active_objs 450 ++--------------------------------------------------------------------+ | .* | 400 ++* *.* .* *. .*.* *. .**. * : *. * | 350 ++ *.* * * .* .* ** : ** * : ** * :.* **. : **. ::| |: : * * :+ : *.: +: * :+ * * :| 300 *+ : : * : : * * * * 250 ++ : : : : | | : : : | 200 ++OO OO OO OO:O: O OO O O | 150 O+ : : O | | :: | 100 ++ :: | 50 ++ : | | : | 0 ++------------*O--O---------------------------------------------------+ slabinfo.taskstats.num_objs 450 ++--------------------------------------------------------------------+ | .* | 400 ++* *.* .* *. .*.* *. .**. * : *. * | 350 ++ *.* * * .* .* ** : ** * : ** * :.* **. : **. ::| |: : * * :+ : *.: +: * :+ * * :| 300 *+ : : * : : * * * * 250 ++ : : : : | | : : : | 200 ++OO OO OO OO:O: O OO O O | 150 O+ : : O | | :: | 100 ++ :: | 50 ++ : | | : | 0 ++------------*O--O---------------------------------------------------+ kmsg.DMAR-IR:Failed_to_copy_IR_table_for_dmar0_from_previous_kernel 1 *+-*-*------------------------------------------*-*--*-----------*--*-* | : : : : | | : : : : | 0.8 ++ : : : : | | : : : : | | : : : : | 0.6 ++ : : : : | | : : : : | 0.4 ++ : : : : | | : : : : | | : : : : | 0.2 ++ : : : : | | : : : : | | : : : : | 0 ++------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*---------*--*-*--*------+ kmsg.DMAR-IR:Failed_to_copy_IR_table_for_dmar1_from_previous_kernel 1 *+-*-*------------------------------------------*-*--*-----------*--*-* | : : : : | | : : : : | 0.8 ++ : : : : | | : : : : | | : : : : | 0.6 ++ : : : : | | : : : : | 0.4 ++ : : : : | | : : : : | | : : : : | 0.2 ++ : : : : | | : : : : | | : : : : | 0 ++------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*---------*--*-*--*------+ kmsg.DMAR-IR:IRQ_remapping_was_enabled_on_dmar0_but_we_are_not_in_kdump_mode 1 ++-*-*--------------------------------------------*--*--------------*-* | : : : : : | | : : : : : | 0.8 ++: : : : : | | : : : : : | | : : : : : | 0.6 ++: : : : : | | : : : : : | 0.4 ++ : : : : | |: : : : : | |: : : : : | 0.2 ++ : : : : | | : : : : | | : : : : | 0 *+------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*--*------*--*-*--*-*----+ kmsg.DMAR-IR:IRQ_remapping_was_enabled_on_dmar1_but_we_are_not_in_kdump_mode 1 ++-*-*--------------------------------------------*--*--------------*-* | : : : : : | | : : : : : | 0.8 ++: : : : : | | : : : : : | | : : : : : | 0.6 ++: : : : : | | : : : : : | 0.4 ++ : : : : | |: : : : : | |: : : : : | 0.2 ++ : : : : | | : : : : | | : : : : | 0 *+------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*--*------*--*-*--*-*----+ kmsg.DHCP_BOOTP:Reply_not_for_us_op___xid___ 1 ++--------------------------------------------------------------------* | | | :| 0.8 ++ :| | :| | : | 0.6 ++ : | | : | 0.4 ++ : | | : | | : | 0.2 ++ : | | : | | : | 0 *+---*---*----*----*---*----*----*---*----*----*---*----*----*---*----+ [*] bisect-good sample [O] bisect-bad sample To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Ying Huang

4 years, 11 months

2
1
0 / 0

[lkp] [kallsyms] bf2d2b07db: kasan: GPF could be caused by NULL-ptr deref or user memory accessgeneral protection fault: 0000 [#1] general protection fault: 0000 [#1] PREEMPT PREEMPT KASANKASAN

by kernel test robot

FYI, we noticed the below changes on https://git.linaro.org/people/ard.biesheuvel/linux-arm arm64-kaslr-v4a commit bf2d2b07db19001ae0bd55826025b0ba47fae0c2 ("kallsyms: add support for relative offsets in kallsyms address table") +-----------------------------------------------------------+------------+------------+ | | 2c4d21df0f | bf2d2b07db | +-----------------------------------------------------------+------------+------------+ | boot_successes | 10 | 0 | | boot_failures | 6 | 36 | | Kernel_panic-not_syncing:Attempted_to_kill_init!exitcode= | 2 | | | IP-Config:Auto-configuration_of_network_failed | 4 | | | general_protection_fault:#[##]PREEMPT_PREEMPT_KASANKASAN | 0 | 36 | | general_protection_fault:#[##] | 0 | 36 | | BUG:kernel_boot_hang | 0 | 36 | +-----------------------------------------------------------+------------+------------+ [ 0.281636] kasan: CONFIG_KASAN_INLINE enabled [ 0.282416] kasan: GPF could be caused by NULL-ptr deref or user memory access [ 0.282416] kasan: GPF could be caused by NULL-ptr deref or user memory accessgeneral protection fault: 0000 [#1] general protection fault: 0000 [#1] PREEMPT PREEMPT KASANKASAN [ 0.284561] Modules linked in: [ 0.284561] Modules linked in: [ 0.285136] CPU: 0 PID: 1 Comm: swapper Not tainted 4.5.0-rc1-00036-gbf2d2b0 #1 [ 0.285136] CPU: 0 PID: 1 Comm: swapper Not tainted 4.5.0-rc1-00036-gbf2d2b0 #1 [ 0.286438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 0.286438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 0.288000] task: ffff88000fcb0000 ti: ffff88000fcb8000 task.ti: ffff88000fcb8000 [ 0.288000] task: ffff88000fcb0000 ti: ffff88000fcb8000 task.ti: ffff88000fcb8000 [ 0.289287] RIP: 0010:[<ffffffff8120688c>] Thanks, Ying Huang

4 years, 11 months

4
3
0 / 0

Re: [LKP] [slab] a1fd55538c: WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:2601 trace_hardirqs_on_caller()

by Jesper Dangaard Brouer

On Sat, 30 Jan 2016 02:09:30 -0500 Valdis.Kletnieks(a)vt.edu wrote: > On Thu, 28 Jan 2016 18:47:49 +0100, Jesper Dangaard Brouer said: > > I cannot reproduce below problem... have enabled all kind of debugging > > and also lockdep. > > > > Can I get a version of the .config file used? > > I'm not the 0day bot, but my laptop hits the same issue at boot. Thank you! I'm now able to reproduce, and I've found the issue. It only happens for SLAB, and with FAILSLAB disabled. The problem were introduced in the patch before: http://ozlabs.org/~akpm/mmots/broken-out/mm-fault-inject-take-over-bootst... which moved the check function: static bool slab_should_failslab(struct kmem_cache *cachep, gfp_t flags) { if (unlikely(cachep == kmem_cache)) return false; return should_failslab(cachep->object_size, flags, cachep->flags); } into the fault injection framework, call of should_failslab(). That change was wrong, as some very early boot code depend on SLAB failing, when still allocating from the bootstrap kmem_cache. SLUB seem to handle this better. In this case the percpu system, have a workqueue function, calling pcpu_extend_area_map() which sort-of probe the slab-allocator, and depending on it fails, until it is fully ready. I will fix up my patches, reverting this change... and let them go through Andrews quilt process. Let me know, if the linux-next tree need's an explicit fix? -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer

4 years, 11 months

2
3
0 / 0

[lkp] [locks] 7f3697e24d: +35.1% will-it-scale.per_thread_ops

by kernel test robot

FYI, we noticed the below changes on https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master commit 7f3697e24dc3820b10f445a4a7d914fc356012d1 ("locks: fix unlock when fcntl_setlk races with a close") ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase: gcc-4.9/performance/x86_64-rhel/debian-x86_64-2015-02-07.cgz/lkp-snb01/lock1/will-it-scale commit: 9189922675ecca0fab38931d86b676e9d79602dc 7f3697e24dc3820b10f445a4a7d914fc356012d1 9189922675ecca0f 7f3697e24dc3820b10f445a4a7 ---------------- -------------------------- %stddev %change %stddev \ | \ 2376432 ± 0% +2.1% 2427484 ± 0% will-it-scale.per_process_ops 807889 ± 0% +35.1% 1091496 ± 4% will-it-scale.per_thread_ops 22.08 ± 2% +89.1% 41.75 ± 5% will-it-scale.time.user_time 1238371 ± 14% +100.4% 2481345 ± 39% cpuidle.C1E-SNB.time 3098 ± 57% -66.6% 1035 ±171% numa-numastat.node1.other_node 379.25 ± 8% -21.4% 298.00 ± 12% numa-vmstat.node0.nr_alloc_batch 22.08 ± 2% +89.1% 41.75 ± 5% time.user_time 1795 ± 4% +7.5% 1930 ± 2% vmstat.system.cs 0.54 ± 5% +136.9% 1.28 ± 10% perf-profile.cycles.___might_sleep.__might_sleep.kmem_cache_alloc.locks_alloc_lock.__posix_lock_file 1.65 ± 57% +245.2% 5.70 ± 29% perf-profile.cycles.__fdget_raw.sys_fcntl.entry_SYSCALL_64_fastpath 1.58 ± 59% +248.3% 5.50 ± 31% perf-profile.cycles.__fget.__fget_light.__fdget_raw.sys_fcntl.entry_SYSCALL_64_fastpath 1.62 ± 58% +246.3% 5.63 ± 30% perf-profile.cycles.__fget_light.__fdget_raw.sys_fcntl.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 5.88 ± 11% perf-profile.cycles.__memset.locks_alloc_lock.__posix_lock_file.vfs_lock_file.do_lock_file_wait 2.50 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles.__memset.locks_alloc_lock.__posix_lock_file.vfs_lock_file.fcntl_setlk 1.29 ± 4% +138.8% 3.09 ± 11% perf-profile.cycles.__memset.locks_alloc_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.47 ± 9% +144.4% 1.16 ± 11% perf-profile.cycles.__might_fault.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.37 ± 12% +140.3% 0.90 ± 9% perf-profile.cycles.__might_sleep.__might_fault.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.86 ± 6% +137.7% 2.05 ± 10% perf-profile.cycles.__might_sleep.kmem_cache_alloc.locks_alloc_lock.__posix_lock_file.vfs_lock_file 0.61 ± 14% +56.8% 0.95 ± 14% perf-profile.cycles.__might_sleep.kmem_cache_alloc.locks_alloc_lock.fcntl_setlk.sys_fcntl 0.00 ± -1% +Inf% 39.84 ± 12% perf-profile.cycles.__posix_lock_file.vfs_lock_file.do_lock_file_wait.fcntl_setlk.sys_fcntl 16.44 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles.__posix_lock_file.vfs_lock_file.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 1.77 ± 11% perf-profile.cycles._raw_spin_lock.__posix_lock_file.vfs_lock_file.do_lock_file_wait.fcntl_setlk 59.34 ± 1% -72.4% 16.36 ± 33% perf-profile.cycles._raw_spin_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.46 ± 11% +144.9% 1.13 ± 19% perf-profile.cycles.avc_has_perm.inode_has_perm.file_has_perm.selinux_file_fcntl.security_file_fcntl 0.87 ± 6% +103.2% 1.77 ± 12% perf-profile.cycles.avc_has_perm.inode_has_perm.file_has_perm.selinux_file_lock.security_file_lock 0.81 ± 4% +135.7% 1.90 ± 10% perf-profile.cycles.copy_user_generic_string.sys_fcntl.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 41.86 ± 12% perf-profile.cycles.do_lock_file_wait.part.29.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.88 ± 6% +127.8% 2.00 ± 9% perf-profile.cycles.entry_SYSCALL_64 0.86 ± 4% +122.6% 1.92 ± 12% perf-profile.cycles.entry_SYSCALL_64_after_swapgs 84.98 ± 0% -9.1% 77.20 ± 2% perf-profile.cycles.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.76 ± 10% +142.1% 1.84 ± 14% perf-profile.cycles.file_has_perm.selinux_file_fcntl.security_file_fcntl.sys_fcntl.entry_SYSCALL_64_fastpath 1.35 ± 4% +106.3% 2.78 ± 11% perf-profile.cycles.file_has_perm.selinux_file_lock.security_file_lock.fcntl_setlk.sys_fcntl 0.00 ± -1% +Inf% 0.89 ± 12% perf-profile.cycles.flock_to_posix_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 6.90 ± 4% -48.6% 3.55 ± 27% perf-profile.cycles.fput.entry_SYSCALL_64_fastpath 0.51 ± 10% +140.5% 1.23 ± 16% perf-profile.cycles.inode_has_perm.isra.31.file_has_perm.selinux_file_fcntl.security_file_fcntl.sys_fcntl 0.98 ± 4% +97.7% 1.93 ± 11% perf-profile.cycles.inode_has_perm.isra.31.file_has_perm.selinux_file_lock.security_file_lock.fcntl_setlk 0.00 ± -1% +Inf% 6.56 ± 10% perf-profile.cycles.kmem_cache_alloc.locks_alloc_lock.__posix_lock_file.vfs_lock_file.do_lock_file_wait 2.75 ± 4% -100.0% 0.00 ± -1% perf-profile.cycles.kmem_cache_alloc.locks_alloc_lock.__posix_lock_file.vfs_lock_file.fcntl_setlk 1.53 ± 7% +119.7% 3.37 ± 13% perf-profile.cycles.kmem_cache_alloc.locks_alloc_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 1.79 ± 11% perf-profile.cycles.kmem_cache_free.locks_free_lock.__posix_lock_file.vfs_lock_file.do_lock_file_wait 0.46 ± 14% +257.0% 1.66 ± 11% perf-profile.cycles.kmem_cache_free.locks_free_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.40 ± 7% +158.6% 1.05 ± 17% perf-profile.cycles.kmem_cache_free.locks_free_lock.locks_dispose_list.__posix_lock_file.vfs_lock_file 0.00 ± -1% +Inf% 0.96 ± 10% perf-profile.cycles.lg_local_lock.locks_insert_lock_ctx.__posix_lock_file.vfs_lock_file.do_lock_file_wait 0.00 ± -1% +Inf% 14.69 ± 10% perf-profile.cycles.locks_alloc_lock.__posix_lock_file.vfs_lock_file.do_lock_file_wait.fcntl_setlk 6.38 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles.locks_alloc_lock.__posix_lock_file.vfs_lock_file.fcntl_setlk.sys_fcntl 3.28 ± 6% +127.1% 7.45 ± 12% perf-profile.cycles.locks_alloc_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 9.75 ± 13% perf-profile.cycles.locks_delete_lock_ctx.__posix_lock_file.vfs_lock_file.do_lock_file_wait.fcntl_setlk 3.61 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles.locks_delete_lock_ctx.__posix_lock_file.vfs_lock_file.fcntl_setlk.sys_fcntl 0.00 ± -1% +Inf% 1.84 ± 11% perf-profile.cycles.locks_dispose_list.__posix_lock_file.vfs_lock_file.do_lock_file_wait.fcntl_setlk 0.00 ± -1% +Inf% 2.42 ± 10% perf-profile.cycles.locks_free_lock.__posix_lock_file.vfs_lock_file.do_lock_file_wait.fcntl_setlk 1.00 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles.locks_free_lock.__posix_lock_file.vfs_lock_file.fcntl_setlk.sys_fcntl 0.63 ± 11% +224.1% 2.05 ± 10% perf-profile.cycles.locks_free_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 1.22 ± 14% perf-profile.cycles.locks_free_lock.locks_dispose_list.__posix_lock_file.vfs_lock_file.do_lock_file_wait 0.00 ± -1% +Inf% 6.17 ± 15% perf-profile.cycles.locks_insert_lock_ctx.__posix_lock_file.vfs_lock_file.do_lock_file_wait.fcntl_setlk 2.31 ± 6% -100.0% 0.00 ± -1% perf-profile.cycles.locks_insert_lock_ctx.__posix_lock_file.vfs_lock_file.fcntl_setlk.sys_fcntl 0.00 ± -1% +Inf% 8.96 ± 13% perf-profile.cycles.locks_unlink_lock_ctx.locks_delete_lock_ctx.__posix_lock_file.vfs_lock_file.do_lock_file_wait 3.27 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles.locks_unlink_lock_ctx.locks_delete_lock_ctx.__posix_lock_file.vfs_lock_file.fcntl_setlk 53.88 ± 1% -79.7% 10.92 ± 46% perf-profile.cycles.native_queued_spin_lock_slowpath._raw_spin_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 2.75 ± 0% +183.3% 7.79 ± 13% perf-profile.cycles.put_pid.locks_unlink_lock_ctx.locks_delete_lock_ctx.__posix_lock_file.vfs_lock_file 1.11 ± 9% +137.2% 2.63 ± 14% perf-profile.cycles.security_file_fcntl.sys_fcntl.entry_SYSCALL_64_fastpath 1.69 ± 4% +118.2% 3.69 ± 11% perf-profile.cycles.security_file_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.91 ± 9% +139.0% 2.17 ± 14% perf-profile.cycles.selinux_file_fcntl.security_file_fcntl.sys_fcntl.entry_SYSCALL_64_fastpath 1.39 ± 4% +114.6% 2.97 ± 10% perf-profile.cycles.selinux_file_lock.security_file_lock.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 41.12 ± 12% perf-profile.cycles.vfs_lock_file.do_lock_file_wait.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 17.04 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles.vfs_lock_file.fcntl_setlk.sys_fcntl.entry_SYSCALL_64_fastpath 34.75 ±148% +132.4% 80.75 ± 82% sched_debug.cfs_rq:/.load.8 15.00 ± 9% +198.3% 44.75 ± 72% sched_debug.cfs_rq:/.load_avg.21 25.00 ± 29% +574.0% 168.50 ± 78% sched_debug.cfs_rq:/.load_avg.9 38.47 ± 5% +29.1% 49.65 ± 26% sched_debug.cfs_rq:/.load_avg.avg 63.17 ± 10% +44.3% 91.16 ± 36% sched_debug.cfs_rq:/.load_avg.stddev 893865 ± 12% -12.5% 782455 ± 0% sched_debug.cfs_rq:/.min_vruntime.25 18.25 ± 26% +52.1% 27.75 ± 25% sched_debug.cfs_rq:/.runnable_load_avg.9 -57635 ±-68% -196.4% 55548 ±130% sched_debug.cfs_rq:/.spread0.1 -802264 ±-25% -29.5% -565458 ±-49% sched_debug.cfs_rq:/.spread0.8 -804662 ±-25% -29.4% -567811 ±-48% sched_debug.cfs_rq:/.spread0.min 1233 ± 5% +30.9% 1614 ± 28% sched_debug.cfs_rq:/.tg_load_avg.0 1233 ± 5% +30.9% 1614 ± 28% sched_debug.cfs_rq:/.tg_load_avg.1 1228 ± 5% +30.3% 1601 ± 27% sched_debug.cfs_rq:/.tg_load_avg.10 1228 ± 5% +30.4% 1601 ± 27% sched_debug.cfs_rq:/.tg_load_avg.11 1228 ± 5% +30.3% 1601 ± 27% sched_debug.cfs_rq:/.tg_load_avg.12 1229 ± 5% +30.0% 1598 ± 27% sched_debug.cfs_rq:/.tg_load_avg.13 1228 ± 5% +30.1% 1598 ± 27% sched_debug.cfs_rq:/.tg_load_avg.14 1229 ± 5% +30.0% 1598 ± 27% sched_debug.cfs_rq:/.tg_load_avg.15 1226 ± 5% +30.3% 1598 ± 27% sched_debug.cfs_rq:/.tg_load_avg.16 1226 ± 5% +30.2% 1597 ± 27% sched_debug.cfs_rq:/.tg_load_avg.17 1227 ± 5% +30.1% 1595 ± 27% sched_debug.cfs_rq:/.tg_load_avg.18 1227 ± 5% +29.4% 1588 ± 26% sched_debug.cfs_rq:/.tg_load_avg.19 1233 ± 5% +30.4% 1609 ± 27% sched_debug.cfs_rq:/.tg_load_avg.2 1222 ± 5% +29.9% 1587 ± 26% sched_debug.cfs_rq:/.tg_load_avg.20 1223 ± 5% +24.2% 1519 ± 20% sched_debug.cfs_rq:/.tg_load_avg.21 1223 ± 5% +23.8% 1515 ± 20% sched_debug.cfs_rq:/.tg_load_avg.22 1223 ± 5% +23.9% 1515 ± 20% sched_debug.cfs_rq:/.tg_load_avg.23 1223 ± 5% +23.9% 1515 ± 20% sched_debug.cfs_rq:/.tg_load_avg.24 1223 ± 5% +23.5% 1511 ± 19% sched_debug.cfs_rq:/.tg_load_avg.25 1224 ± 5% +23.5% 1512 ± 19% sched_debug.cfs_rq:/.tg_load_avg.26 1223 ± 5% +23.1% 1506 ± 19% sched_debug.cfs_rq:/.tg_load_avg.27 1223 ± 5% +22.5% 1499 ± 19% sched_debug.cfs_rq:/.tg_load_avg.28 1224 ± 5% +22.5% 1499 ± 19% sched_debug.cfs_rq:/.tg_load_avg.29 1233 ± 5% +30.3% 1607 ± 27% sched_debug.cfs_rq:/.tg_load_avg.3 1223 ± 5% +22.2% 1495 ± 18% sched_debug.cfs_rq:/.tg_load_avg.30 1224 ± 5% +22.0% 1493 ± 19% sched_debug.cfs_rq:/.tg_load_avg.31 1234 ± 5% +30.0% 1604 ± 28% sched_debug.cfs_rq:/.tg_load_avg.4 1233 ± 5% +30.0% 1604 ± 28% sched_debug.cfs_rq:/.tg_load_avg.5 1231 ± 5% +30.3% 1604 ± 28% sched_debug.cfs_rq:/.tg_load_avg.6 1233 ± 5% +30.0% 1603 ± 27% sched_debug.cfs_rq:/.tg_load_avg.7 1231 ± 5% +30.1% 1601 ± 27% sched_debug.cfs_rq:/.tg_load_avg.8 1228 ± 5% +30.3% 1601 ± 27% sched_debug.cfs_rq:/.tg_load_avg.9 1228 ± 5% +27.8% 1569 ± 24% sched_debug.cfs_rq:/.tg_load_avg.avg 1246 ± 5% +30.7% 1628 ± 27% sched_debug.cfs_rq:/.tg_load_avg.max 1212 ± 5% +22.2% 1481 ± 19% sched_debug.cfs_rq:/.tg_load_avg.min 15.00 ± 9% +198.3% 44.75 ± 72% sched_debug.cfs_rq:/.tg_load_avg_contrib.21 25.00 ± 29% +574.0% 168.50 ± 78% sched_debug.cfs_rq:/.tg_load_avg_contrib.9 38.53 ± 5% +29.0% 49.71 ± 26% sched_debug.cfs_rq:/.tg_load_avg_contrib.avg 63.34 ± 10% +44.1% 91.30 ± 36% sched_debug.cfs_rq:/.tg_load_avg_contrib.stddev 532.25 ± 2% +8.5% 577.50 ± 6% sched_debug.cfs_rq:/.util_avg.15 210.75 ± 14% -14.4% 180.50 ± 4% sched_debug.cfs_rq:/.util_avg.29 450.00 ± 22% +50.7% 678.00 ± 18% sched_debug.cfs_rq:/.util_avg.9 955572 ± 4% -10.2% 857813 ± 5% sched_debug.cpu.avg_idle.6 23.99 ± 60% -76.2% 5.71 ± 24% sched_debug.cpu.clock.stddev 23.99 ± 60% -76.2% 5.71 ± 24% sched_debug.cpu.clock_task.stddev 2840 ± 37% -47.4% 1492 ± 65% sched_debug.cpu.curr->pid.25 34.75 ±148% +132.4% 80.75 ± 82% sched_debug.cpu.load.8 61776 ± 7% -7.1% 57380 ± 0% sched_debug.cpu.nr_load_updates.25 6543 ± 2% +20.4% 7879 ± 9% sched_debug.cpu.nr_switches.0 5256 ± 23% +177.1% 14566 ± 52% sched_debug.cpu.nr_switches.27 7915 ± 3% +8.7% 8605 ± 3% sched_debug.cpu.nr_switches.avg -0.25 ±-519% +1900.0% -5.00 ±-24% sched_debug.cpu.nr_uninterruptible.12 2.00 ± 93% -125.0% -0.50 ±-300% sched_debug.cpu.nr_uninterruptible.24 17468 ± 14% +194.3% 51413 ± 75% sched_debug.cpu.sched_count.15 2112 ± 2% +20.8% 2552 ± 11% sched_debug.cpu.sched_goidle.0 2103 ± 34% +219.0% 6709 ± 55% sched_debug.cpu.sched_goidle.27 3159 ± 3% +8.2% 3418 ± 4% sched_debug.cpu.sched_goidle.avg 1323 ± 64% -72.7% 361.50 ± 15% sched_debug.cpu.ttwu_count.23 3264 ± 12% +94.4% 6347 ± 41% sched_debug.cpu.ttwu_count.27 3860 ± 3% +9.0% 4208 ± 3% sched_debug.cpu.ttwu_count.avg 2358 ± 3% +28.7% 3035 ± 9% sched_debug.cpu.ttwu_local.0 1110 ± 22% +54.6% 1716 ± 28% sched_debug.cpu.ttwu_local.27 1814 ± 8% +16.1% 2106 ± 5% sched_debug.cpu.ttwu_local.stddev lkp-snb01: Sandy Bridge-EP Memory: 32G will-it-scale.per_thread_ops 1.2e+06 ++---------------------------------------------------------------+ | O | 1.15e+06 O+O O O O O O O O | 1.1e+06 ++ | | O O O O O OO | 1.05e+06 ++ O O | 1e+06 ++ | | | 950000 ++ | 900000 ++ | | | 850000 ++ | 800000 *+*.*.*.*.*.*.*.*.*.*.*.*. .*.*. *.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.* | * * | 750000 ++---------------------------------------------------------------+ will-it-scale.time.user_time 50 ++---------------------------------------------------------------------+ | | 45 ++ O O O O O O | O O O O | | O O O O O | 40 ++ O O O O | | | 35 ++ | | | 30 ++ | | | | * | 25 ++ + + | *.*.*.*..*.* *.*.*..*.*.*.*.*.*.*..*.*.*.*.*.*.*..*.*.*.*.*.*..*.*.*.* 20 ++---------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Ying Huang

4 years, 12 months

3
3
0 / 0

[lkp] [wext] 6a9a24baba: BUG: spinlock bad magic on CPU#0, swapper/0/1

by kernel test robot

FYI, we noticed the below changes on https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit 6a9a24baba28599b0bb8453ae6a6a06ad79701a0 ("wext: fix message delay/ordering") [ 0.821194] e820: reserve RAM buffer [mem 0x167e0000-0x17ffffff] [ 0.823299] NET: Registered protocol family 23 [ 0.823299] NET: Registered protocol family 23 [ 0.824526] BUG: spinlock bad magic on CPU#0, swapper/0/1 [ 0.824526] BUG: spinlock bad magic on CPU#0, swapper/0/1 [ 0.826041] lock: init_net+0xe78/0xf00, .magic: 00000000, .owner: swapper/0/1, .owner_cpu: 0 [ 0.826041] lock: init_net+0xe78/0xf00, .magic: 00000000, .owner: swapper/0/1, .owner_cpu: 0 [ 0.828388] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.4.0-03435-g6a9a24b #1 [ 0.828388] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.4.0-03435-g6a9a24b #1 [ 0.830342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 0.830342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 0.832868] ffffffff886ed7f8 [ 0.832868] ffffffff886ed7f8 ffff8800131b7d88 ffff8800131b7d88 ffffffff87eb8f86 ffffffff87eb8f86 ffff8800131b2040 ffff8800131b2040 [ 0.835143] ffff8800131b7da8 [ 0.835143] ffff8800131b7da8 ffffffff87d25897 ffffffff87d25897 ffffffff886ed7f8 ffffffff886ed7f8 ffffffff886ed7f8 ffffffff886ed7f8 [ 0.837376] ffff8800131b7dc0 [ 0.837376] ffff8800131b7dc0 ffffffff87d25b10 ffffffff87d25b10 0000000000000286 0000000000000286 ffff8800131b7de0 ffff8800131b7de0 [ 0.842451] Call Trace: [ 0.842451] Call Trace: [ 0.843040] [<ffffffff87eb8f86>] dump_stack+0x4b/0x63 [ 0.843040] [<ffffffff87eb8f86>] dump_stack+0x4b/0x63 [ 0.844370] [<ffffffff87d25897>] spin_dump+0x78/0xbb [ 0.844370] [<ffffffff87d25897>] spin_dump+0x78/0xbb [ 0.845662] [<ffffffff87d25b10>] do_raw_spin_unlock+0x64/0xba [ 0.845662] [<ffffffff87d25b10>] do_raw_spin_unlock+0x64/0xba [ 0.847114] [<ffffffff88123874>] _raw_spin_unlock_irqrestore+0x2c/0x5d [ 0.847114] [<ffffffff88123874>] _raw_spin_unlock_irqrestore+0x2c/0x5d [ 0.848654] [<ffffffff8807c341>] skb_dequeue+0x59/0x67 [ 0.848654] [<ffffffff8807c341>] skb_dequeue+0x59/0x67 [ 0.849874] [<ffffffff880edfe4>] wireless_nlevent_flush+0x54/0x8d [ 0.849874] [<ffffffff880edfe4>] wireless_nlevent_flush+0x54/0x8d [ 0.851321] [<ffffffff880ee02b>] wext_netdev_notifier_call+0xe/0x15 [ 0.851321] [<ffffffff880ee02b>] wext_netdev_notifier_call+0xe/0x15 [ 0.853020] [<ffffffff8808aa37>] register_netdevice_notifier+0x8a/0x1b9 [ 0.853020] [<ffffffff8808aa37>] register_netdevice_notifier+0x8a/0x1b9 [ 0.854689] [<ffffffff8895e4f6>] ? wext_pernet_init+0x4a/0x4a [ 0.854689] [<ffffffff8895e4f6>] ? wext_pernet_init+0x4a/0x4a [ 0.856159] [<ffffffff8895e506>] wireless_nlevent_init+0x10/0x22 [ 0.856159] [<ffffffff8895e506>] wireless_nlevent_init+0x10/0x22 [ 0.857556] [<ffffffff87c003d9>] do_one_initcall+0xb3/0x1fa [ 0.857556] [<ffffffff87c003d9>] do_one_initcall+0xb3/0x1fa [ 0.858853] [<ffffffff87cf5024>] ? parse_args+0x26b/0x4c4 [ 0.858853] [<ffffffff87cf5024>] ? parse_args+0x26b/0x4c4 [ 0.860344] [<ffffffff889170c2>] kernel_init_freeable+0x10f/0x194 [ 0.860344] [<ffffffff889170c2>] kernel_init_freeable+0x10f/0x194 [ 0.861889] [<ffffffff881168ac>] ? rest_init+0xcc/0xcc [ 0.861889] [<ffffffff881168ac>] ? rest_init+0xcc/0xcc [ 0.863210] [<ffffffff881168ba>] kernel_init+0xe/0xde [ 0.863210] [<ffffffff881168ba>] kernel_init+0xe/0xde [ 0.864587] [<ffffffff8812462f>] ret_from_fork+0x3f/0x70 [ 0.864587] [<ffffffff8812462f>] ret_from_fork+0x3f/0x70 [ 0.866178] [<ffffffff881168ac>] ? rest_init+0xcc/0xcc [ 0.866178] [<ffffffff881168ac>] ? rest_init+0xcc/0xcc [ 0.867927] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [ 0.867927] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [ 0.870286] clocksource: Switched to clocksource kvm-clock Thanks, Kernel Test Robot

4 years, 12 months

1
0
0 / 0

[netfilter: nf_conntrack] b16c29191d: BUG: spinlock recursion on CPU#1, kworker/u4:2/89

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit b16c29191dc89bd877af99a7b04ce4866728a3e0 Author: Sasha Levin <sasha.levin(a)oracle.com> AuthorDate: Mon Jan 18 19:23:51 2016 -0500 Commit: Pablo Neira Ayuso <pablo(a)netfilter.org> CommitDate: Wed Jan 20 14:15:31 2016 +0100 netfilter: nf_conntrack: use safer way to lock all buckets When we need to lock all buckets in the connection hashtable we'd attempt to lock 1024 spinlocks, which is way more preemption levels than supported by the kernel. Furthermore, this behavior was hidden by checking if lockdep is enabled, and if it was - use only 8 buckets(!). Fix this by using a global lock and synchronize all buckets on it when we need to lock them all. This is pretty heavyweight, but is only done when we need to resize the hashtable, and that doesn't happen often enough (or at all). Signed-off-by: Sasha Levin <sasha.levin(a)oracle.com> Acked-by: Jesper Dangaard Brouer <brouer(a)redhat.com> Reviewed-by: Florian Westphal <fw(a)strlen.de> Signed-off-by: Pablo Neira Ayuso <pablo(a)netfilter.org> +-------------------------------+------------+------------+------------+ | | 35b815392a | b16c29191d | 9f59b59fdd | +-------------------------------+------------+------------+------------+ | boot_successes | 910 | 118 | 38 | | boot_failures | 0 | 2 | 8 | | BUG:spinlock_recursion_on_CPU | 0 | 2 | 8 | | backtrace:cleanup_net | 0 | 2 | 7 | +-------------------------------+------------+------------+------------+ [child0:1965] uid changed! Was: 0, now 65535 [child1:1964] child exiting. Bailing main loop. Exit reason: UID changed. [ 24.822451] BUG: spinlock recursion on CPU#1, kworker/u4:2/89 [ 24.823358] lock: nf_conntrack_locks+0x0/0x12780, .magic: dead4ead, .owner: kworker/u4:2/89, .owner_cpu: 1 [ 24.824802] CPU: 1 PID: 89 Comm: kworker/u4:2 Not tainted 4.4.0-03418-gb16c291 #1 [ 24.825913] Workqueue: netns cleanup_net [ 24.826552] ffffffff81e08880 ffff88001265fbe8 ffffffff813cf0c9 ffff8800125e6100 [ 24.827743] ffff88001265fc08 ffffffff810b37e8 ffffffff81e08880 ffffffff81e08880 [ 24.828930] ffff88001265fc38 ffffffff810b39aa ffffffff81e08898 ffffffff81e08880 [ 24.830114] Call Trace: [ 24.830510] [<ffffffff813cf0c9>] dump_stack+0x4b/0x72 [ 24.831277] [<ffffffff810b37e8>] spin_dump+0x78/0xc0 [ 24.832043] [<ffffffff810b39aa>] do_raw_spin_lock+0x11a/0x150 [ 24.832922] [<ffffffff8185022d>] _raw_spin_lock+0x5d/0x80 [ 24.833753] [<ffffffff81701922>] ? nf_conntrack_lock+0x12/0x60 [ 24.834648] [<ffffffff81701922>] nf_conntrack_lock+0x12/0x60 [ 24.835353] caif:caif_disconnect_client(): nothing to disconnect [ 24.849447] [<ffffffff8170bce2>] ctnl_untimeout+0x82/0xb0 [ 24.850371] [<ffffffff8170bd3b>] cttimeout_net_exit+0x2b/0x80 [ 24.851245] [<ffffffff816b4ff8>] ops_exit_list+0x38/0x60 [ 24.852153] [<ffffffff816b5dae>] cleanup_net+0x1ae/0x270 [ 24.852986] [<ffffffff81080b71>] process_one_work+0x1c1/0x500 [ 24.853879] [<ffffffff81080ae9>] ? process_one_work+0x139/0x500 [ 24.854805] [<ffffffff81080efe>] worker_thread+0x4e/0x490 [ 24.855649] [<ffffffff81080eb0>] ? process_one_work+0x500/0x500 [ 24.856571] [<ffffffff81080eb0>] ? process_one_work+0x500/0x500 [ 24.857507] [<ffffffff81087851>] kthread+0x101/0x120 [ 24.858285] [<ffffffff81087750>] ? kthread_stop+0x120/0x120 [ 24.859186] [<ffffffff8185146f>] ret_from_fork+0x3f/0x70 [ 24.860035] [<ffffffff81087750>] ? kthread_stop+0x120/0x120 [ 34.718759] init: tty4 main process (1966) terminated with status 1 [ 34.721009] init: tty4 main process ended, respawning [ 34.730863] init: tty5 main process (1967) terminated with status 1 git bisect start 9f59b59fddd4ad4a398007059b179cc804e9a85f 92e963f50fc74041b5e9e744c330dca48e04f08d -- git bisect bad d2907ebfd7892cc6919288e306782d28b1aec054 # 11:25 14- 3 Merge 'linux-review/Nicolas-Ferre/spi-atmel-fix-gpio-chip-select-in-case-of-non-DT-platform/20160128-005054' into devel-spot-201601280923 git bisect good 2f451ad637b99e6073555ee77b4861ba4c0b33d8 # 11:37 120+ 0 Merge 'asoc/for-next' into devel-spot-201601280923 git bisect bad 5bf7f3913eb09a58a29f8c49d35e6d564521efc3 # 11:43 0- 1 Merge 'asoc/topic/ssm4567' into devel-spot-201601280923 git bisect good c0cd80413f024d2d2b0306cd73ff905211b2dfdf # 12:00 120+ 0 Merge 'spi/for-next' into devel-spot-201601280923 git bisect good cec507cd4a1097ff1db46a2138b5a63a34e19a4e # 12:25 119+ 0 Merge 'linux-review/Ross-Zwisler/ext2-ext4-Fix-issue-with-missing-journal-entry/20160128-030742' into devel-spot-201601280923 git bisect bad fab357e5a0b27fe98447194f4ff3b57cf161353a # 12:48 5- 5 Merge 'bluetooth/master' into devel-spot-201601280923 git bisect bad 41a751e8480e093999fed3af75cfd17ad3822f77 # 13:02 4- 8 Merge 'linux-review/Eric-Dumazet/tcp-beware-of-alignments-in-tcp_get_info/20160128-025549' into devel-spot-201601280923 git bisect bad 0e03f563a04207cc8e5db6afe63309a585995de7 # 13:21 3- 5 net: mvneta: sort the headers in alphabetic order git bisect bad 8034e1efcb330d2aecef8cbf8a83f206270c1775 # 13:39 2- 3 Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf git bisect good 81e8f2e930fe76b9814c71b9d87c30760b5eb705 # 13:48 120+ 0 net: dp83640: Fix tx timestamp overflow handling. git bisect good d6b3347bf178266259af64b1f27b5cf54acf62c8 # 14:19 120+ 0 netfilter: xt_TCPMSS: handle CHECKSUM_COMPLETE in tcpmss_tg6() git bisect bad b16c29191dc89bd877af99a7b04ce4866728a3e0 # 16:32 75- 2 netfilter: nf_conntrack: use safer way to lock all buckets git bisect good 35b815392a6b6c268baf3b63d7f2ba350597024f # 17:14 310+ 0 netfilter: nf_tables_netdev: fix error path in module initialization # first bad commit: [b16c29191dc89bd877af99a7b04ce4866728a3e0] netfilter: nf_conntrack: use safer way to lock all buckets git bisect good 35b815392a6b6c268baf3b63d7f2ba350597024f # 17:27 905+ 0 netfilter: nf_tables_netdev: fix error path in module initialization # extra tests with DEBUG_INFO git bisect bad b16c29191dc89bd877af99a7b04ce4866728a3e0 # 17:51 104- 4 netfilter: nf_conntrack: use safer way to lock all buckets # extra tests on HEAD of linux-devel/devel-spot-201601280923 git bisect bad 9f59b59fddd4ad4a398007059b179cc804e9a85f # 17:51 0- 8 0day head guard for 'devel-spot-201601280923' # extra tests on tree/branch linux-next/master git bisect bad 888c8375131656144c1605071eab2eb6ac49abc3 # 18:04 16- 3 Add linux-next specific files for 20160128 # extra tests with first bad commit reverted git bisect good 8aef8bd67e3bcbe1894bf46992346fc0a5ef4647 # 18:32 905+ 0 Revert "netfilter: nf_conntrack: use safer way to lock all buckets" # extra tests on tree/branch linus/master git bisect good 03c21cb775a313f1ff19be59c5d02df3e3526471 # 19:20 910+ 0 Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost # extra tests on tree/branch linux-next/master git bisect bad 888c8375131656144c1605071eab2eb6ac49abc3 # 19:20 0- 64 Add linux-next specific files for 20160128 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=quantal-core-x86_64.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu kvm64 -kernel $kernel -initrd $initrd -m 300 -smp 2 -device e1000,netdev=net0 -netdev user,id=net0 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation

4 years, 12 months

2
1
0 / 0

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP January 2016