October 2014 - LKP - Ml01.01.Org

[fentry] WARNING: CPU: 0 PID: 1 at kernel/trace/trace_functions_graph.c:224 ftrace_return_to_handler+0xd2/0x1a8()

by Fengguang Wu

FYI, we noticed the below changes on git://flatbed.openfabrics.org/~amirv/linux.git for-netdev commit de89e9380c82b0da22a6609f973dd6e9d67656e3 ("regression: Disable FENTRY") +-----------------------------------------------------------------------------+------------+------------+ | | d068b02cfd | de89e9380c | +-----------------------------------------------------------------------------+------------+------------+ | boot_successes | 0 | 0 | | boot_failures | 10 | 10 | | BUG:sleeping_function_called_from_invalid_context_at_kernel/locking/mutex.c | 7 | | | backtrace:do_vfs_ioctl | 7 | | | backtrace:SyS_ioctl | 7 | | | Out_of_memory:Kill_process | 3 | | | Kernel_panic-not_syncing:Out_of_memory_and_no_killable_processes | 2 | | | backtrace:do_execve | 2 | | | backtrace:SyS_execve | 2 | | | backtrace:do_fork | 3 | | | backtrace:SyS_clone | 3 | | | backtrace:do_sys_open | 1 | | | backtrace:SyS_open | 1 | | | backtrace:pgd_alloc | 1 | | | backtrace:mm_init | 1 | | | backtrace:SYSC_reboot | 1 | | | backtrace:SyS_reboot | 1 | | | WARNING:at_kernel/trace/trace_functions_graph.c:ftrace_return_to_handler() | 0 | 10 | | backtrace:register_tracer | 0 | 10 | | backtrace:init_graph_trace | 0 | 10 | | backtrace:kernel_init_freeable | 0 | 10 | +-----------------------------------------------------------------------------+------------+------------+ [ 10.644181] PASSED [ 10.644938] Testing tracer function_graph: [ 11.136016] ------------[ cut here ]------------ [ 11.140000] WARNING: CPU: 0 PID: 1 at kernel/trace/trace_functions_graph.c:224 ftrace_return_to_handler+0xd2/0x1a8() [ 11.140000] Bad frame pointer: expected ffff880023603eb8, received ffff880023603ef0 [ 11.140000] from func __calc_delta return to ffffffff810e981c [ 11.140000] Modules linked in: [ 11.140000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc7-01516-g8585242 #920 [ 11.140000] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 11.140000] 0000000000000000 ffff880023603db8 ffffffff82d0d960 ffff880023603e00 [ 11.140000] ffff880023603df0 ffffffff810c3476 ffffffff81164abb ffffffff82d05448 [ 11.140000] ffff8800237d6238 ffff8800237d6180 0000000000000000 ffff880023603e58 [ 11.140000] Call Trace: [ 11.140000] <IRQ> [<ffffffff82d0d960>] dump_stack+0x4d/0x66 [ 11.140000] [<ffffffff810c3476>] warn_slowpath_common+0x7f/0x98 [ 11.140000] [<ffffffff81164abb>] ? ftrace_return_to_handler+0xd2/0x1a8 [ 11.140000] [<ffffffff82d05448>] ? set_tsk_thread_flag+0x13/0x13 [ 11.140000] [<ffffffff810c34d7>] warn_slowpath_fmt+0x48/0x50 [ 11.140000] [<ffffffff810e9461>] ? __calc_delta+0x15/0xae [ 11.140000] [<ffffffff81164abb>] ftrace_return_to_handler+0xd2/0x1a8 [ 11.140000] [<ffffffff810e981c>] ? sched_slice+0x74/0x87 [ 11.140000] [<ffffffff82d3485e>] ? ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff810e981c>] sched_slice+0x74/0x87 [ 11.140000] [<ffffffff810e981c>] ? sched_slice+0x74/0x87 [ 11.140000] [<ffffffff82d34873>] return_to_handler+0x15/0x32 [ 11.140000] [<ffffffff82d3485e>] ? ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff810ebe79>] task_tick_fair+0x88/0x109 [ 11.140000] [<ffffffff8107a7e3>] ? ftrace_write+0x23/0x4c [ 11.140000] [<ffffffff82d3485e>] ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff810e54be>] scheduler_tick+0x57/0x89 [ 11.140000] [<ffffffff82d3485e>] ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff811195c8>] update_process_times+0x56/0x65 [ 11.140000] [<ffffffff82d3485e>] ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff811240dd>] tick_periodic+0xa1/0xad [ 11.140000] [<ffffffff82d3485e>] ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff8112427d>] tick_handle_periodic+0x26/0x60 [ 11.140000] [<ffffffff81089300>] ? __virt_addr_valid+0x66/0x8d [ 11.140000] [<ffffffff82d3485e>] ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff8107556a>] local_apic_timer_interrupt+0x54/0x57 [ 11.140000] [<ffffffff82d3485e>] ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff82d3501d>] smp_apic_timer_interrupt+0x2b/0x3c [ 11.140000] [<ffffffff82d33082>] apic_timer_interrupt+0x72/0x80 [ 11.140000] <EOI> [<ffffffff81080ac3>] ? pvclock_clocksource_read+0x76/0xa0 [ 11.140000] [<ffffffff81164d97>] ? trace_graph_entry+0x175/0x18b [ 11.140000] [<ffffffff8108937b>] ? __phys_addr_symbol+0x4/0x24 [ 11.140000] [<ffffffff8107ff56>] kvm_clock_read+0x27/0x31 [ 11.140000] [<ffffffff81051ebc>] paravirt_sched_clock+0x9/0xd [ 11.140000] [<ffffffff81052506>] sched_clock+0x9/0xb [ 11.140000] [<ffffffff8114fda6>] trace_clock_local+0x11/0x1b [ 11.140000] [<ffffffff81164916>] ftrace_push_return_trace+0x76/0x149 [ 11.140000] [<ffffffff8107a7e3>] ? ftrace_write+0x23/0x4c [ 11.140000] [<ffffffff8108937b>] ? __phys_addr_symbol+0x4/0x24 [ 11.140000] [<ffffffff8107af30>] prepare_ftrace_return+0xa2/0xb5 [ 11.140000] [<ffffffff8108937b>] ? __phys_addr_symbol+0x4/0x24 [ 11.140000] [<ffffffff82d34833>] ftrace_graph_caller+0x53/0x7e [ 11.140000] [<ffffffff810f9873>] ? trace_hardirqs_on_caller+0x18f/0x1ab [ 11.140000] [<ffffffff81164f4a>] ? trace_graph_return+0x8e/0x97 [ 11.140000] [<ffffffff8107a7e3>] ? ftrace_write+0x23/0x4c [ 11.140000] [<ffffffff811420f6>] ? audit_find_rule+0x12/0x108 [ 11.140000] [<ffffffff811420f1>] ? audit_find_rule+0xd/0x108 [ 11.140000] [<ffffffff81089380>] ? __phys_addr_symbol+0x9/0x24 [ 11.140000] [<ffffffff81089380>] ? __phys_addr_symbol+0x9/0x24 [ 11.140000] [<ffffffff82d3485e>] ftrace_graph_caller+0x7e/0x7e [ 11.140000] [<ffffffff8107aca9>] ftrace_replace_code+0x1d6/0x331 [ 11.140000] [<ffffffff81164ebc>] ? trace_graph_function+0x78/0x78 [ 11.140000] [<ffffffff811525f1>] ftrace_modify_all_code+0x41/0xca [ 11.140000] [<ffffffff8107ae19>] arch_ftrace_update_code+0x15/0x1e [ 11.140000] [<ffffffff81152b2c>] ftrace_run_update_code+0x2b/0x1b2 [ 11.140000] [<ffffffff81164ebc>] ? trace_graph_function+0x78/0x78 [ 11.140000] [<ffffffff81152ce5>] ftrace_startup_enable+0x32/0x34 [ 11.140000] [<ffffffff81152e25>] ftrace_startup+0x13e/0x148 [ 11.140000] [<ffffffff8115443b>] register_ftrace_graph+0x235/0x252 [ 11.140000] [<ffffffff8115f982>] ? ftrace_dump+0x241/0x241 [ 11.140000] [<ffffffff8441060b>] trace_selftest_startup_function_graph+0x57/0xe3 [ 11.140000] [<ffffffff8115eeac>] register_tracer+0x165/0x288 [ 11.140000] [<ffffffff84410c87>] ? init_graph_debugfs+0x2f/0x2f [ 11.140000] [<ffffffff84410ced>] init_graph_trace+0x66/0x68 [ 11.140000] [<ffffffff8100216d>] do_one_initcall+0xee/0x17e [ 11.140000] [<ffffffff810db300>] ? parse_args+0xfc/0x2b8 [ 11.140000] [<ffffffff843e80bf>] kernel_init_freeable+0x1e2/0x26f [ 11.140000] [<ffffffff82cfcab5>] ? rest_init+0xc9/0xc9 [ 11.140000] [<ffffffff82cfcac3>] kernel_init+0xe/0xdf [ 11.140000] [<ffffffff82d3203c>] ret_from_fork+0x7c/0xb0 [ 11.140000] [<ffffffff82cfcab5>] ? rest_init+0xc9/0xc9 [ 11.140000] ---[ end trace 94c1e7868964d6b1 ]--- [ 11.140000] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 3.17.0-rc7-01516-g8585242 #920 The "BUG: sleeping function called" in the parent commit "disappeared" in commit de89e93 because it happens in much later time: [ 322.056221] smpboot: CPU 1 is now offline [ 322.081569] bnx2fc: CPU 1 offline: Remove Rx thread [ 322.088300] CPU 1 offline: Remove Rx thread mount.nfs: Connection timed out run-parts: /etc/kernel-tests/99-trinity exited with return code 32 [ 362.720433] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:583 [ 362.733241] in_atomic(): 1, irqs_disabled(): 0, pid: 6357, name: reboot [ 362.734867] 2 locks held by reboot/6357: [ 362.735995] #0: (rtnl_mutex){+.+.+.}, at: [<ffffffff829d5b5d>] rtnl_lock+0x17/0x19 [ 362.738773] #1: (rcu_read_lock){......}, at: [<ffffffff810dca0e>] rcu_lock_acquire+0x0/0x20 [ 362.741686] CPU: 0 PID: 6357 Comm: reboot Not tainted 3.17.0-rc6-01531-gd068b02 #1333 [ 362.750958] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 362.752463] 0000000000000000 ffff88001858fa30 ffffffff82d0d3bf 00000000000018d5 [ 362.754976] ffff88001858fa50 ffffffff810e1e35 ffffffff83fc43c0 0000000000000000 [ 362.757523] ffff88001858fad0 ffffffff82d2e0f7 000000000052001f 0000000000000001 [ 362.760103] Call Trace: [ 362.782376] [<ffffffff82d0d3bf>] dump_stack+0x4d/0x66 [ 362.793968] [<ffffffff810e1e35>] __might_sleep+0x125/0x12a [ 362.795422] [<ffffffff82d2e0f7>] mutex_lock_nested+0x3e/0x3ec [ 362.796958] [<ffffffff81d60f4e>] cxgbi_device_find_by_netdev+0x63/0x102 [ 362.798562] [<ffffffff81d60f4e>] ? cxgbi_device_find_by_netdev+0x63/0x102 [ 362.800309] [<ffffffff81d68648>] cxgbi_inet6addr_handler+0x3c/0x83 [ 362.801817] [<ffffffff810dc923>] notifier_call_chain+0x6d/0x93 [ 362.803274] [<ffffffff810dcdc7>] __atomic_notifier_call_chain+0x4c/0x79 [ 362.804991] [<ffffffff810dce08>] atomic_notifier_call_chain+0x14/0x16 [ 362.806572] [<ffffffff82b239b6>] inet6addr_notifier_call_chain+0x1b/0x1d [ 362.808348] [<ffffffff82aefb5c>] addrconf_ifdown+0x28d/0x321 [ 362.809759] [<ffffffff82af0d7e>] addrconf_notify+0x717/0x7d5 [ 362.811192] [<ffffffff810c6ce5>] ? __local_bh_enable_ip+0xaf/0xb4 [ 362.812797] [<ffffffff82d30fe6>] ? _raw_spin_unlock_bh+0x35/0x38 [ 362.815980] [<ffffffff82af80bb>] ? fib6_run_gc+0xc4/0xcb [ 362.817453] [<ffffffff810dc923>] notifier_call_chain+0x6d/0x93 [ 362.818944] [<ffffffff810dc923>] ? notifier_call_chain+0x6d/0x93 [ 362.820545] [<ffffffff810dc96d>] raw_notifier_call_chain+0x14/0x16 [ 362.822095] [<ffffffff829c52bb>] call_netdevice_notifiers_info+0x52/0x59 [ 362.823725] [<ffffffff829c52d5>] call_netdevice_notifiers+0x13/0x15 [ 362.825368] [<ffffffff829cc19e>] __dev_notify_flags+0x53/0x81 [ 362.826838] [<ffffffff829cc814>] dev_change_flags+0x4e/0x59 [ 362.828374] [<ffffffff82aa877a>] devinet_ioctl+0x267/0x566 [ 362.829731] [<ffffffff82aa91f7>] inet_ioctl+0x86/0xa2 [ 362.831085] [<ffffffff82aa91f7>] ? inet_ioctl+0x86/0xa2 [ 362.832482] [<ffffffff829b25cc>] sock_do_ioctl+0x25/0x42 [ 362.833941] [<ffffffff829b2ad8>] sock_ioctl+0x213/0x21f [ 362.835346] [<ffffffff811e532f>] vfs_ioctl+0x18/0x34 [ 362.836692] [<ffffffff811e5b9a>] do_vfs_ioctl+0x39a/0x442 [ 362.838237] [<ffffffff811e5c99>] SyS_ioctl+0x57/0x79 [ 362.839545] [<ffffffff82d31d50>] tracesys+0xdd/0xe2 [ 364.099570] Unregister pv shared memory for cpu 0 [ 364.137326] reboot: Restarting system [ 364.138391] reboot: machine restart Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com

7 years, 7 months

1
0
0 / 0

[x86, this_cpu_ptr] BUG: unable to handle kernel paging request at 000000000000d968

by Fengguang Wu

Hi Christoph, FYI, we noticed the below changes on git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit 89cbc76768c2fa4ed95545bf961f3a14ddfeed21 ("x86: Replace __get_cpu_var uses") 532d0d0690d1532 89cbc76768c2fa4ed95545bf9 --------------- ------------------------- 0 +Inf% 0 ±50% dmesg.BUG:unable_to_handle_kernel 0 +Inf% 1 ± 0% last_state.running 0 +Inf% 1 ± 0% last_state.is_incomplete_run run: /lkp/lkp/src/monitors/wrapper numa-vmstat {} run: /lkp/lkp/src/monitors/wrapper numa-meminfo {} run: /lkp/lkp/src/monitors/w [ 45.378447] BUG: unable to handle kernel paging request at 000000000000d968 [ 45.379430] IP:PROGRESS CODE: V3020003 I0 Loading PEIM at 0x000FFFD57C0 EntryPoint=0x000FFFD673C Thanks, Fengguang

7 years, 7 months

1
0
0 / 0

[x86, kaslr] PANIC: early exception 0e rip 10:ffffffff842122df error 0 cr2 ffffffff829c9798

by Fengguang Wu

Hi Kees, There are a number of oops that are bisect to commit 82fa9637a2ba285bcc7c5050c73010b2c1b3d803 ("x86, kaslr: Select random position from e820 maps"), unfortunately they are mostly hard to reproduce (often only 1 out of 100 boots is bad). However this one is very reproducible. I hope it can sever as a good case for debugging this issue. +------------------------------------------+------------+------------+---------------+ | | 5bfce5ef55 | 82fa9637a2 | next-20140919 | +------------------------------------------+------------+------------+---------------+ | boot_successes | 1000 | 6 | 49 | | boot_failures | 0 | 894 | 4 | | PANIC:early_exception | 0 | 894 | 2 | | BUG:kernel_boot_hang | 0 | 894 | 2 | | general_protection_fault | 0 | 0 | 1 | | RIP:__lock_acquire | 0 | 0 | 1 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 0 | 2 | | backtrace:free_reserved_area | 0 | 0 | 1 | | backtrace:free_init_pages | 0 | 0 | 1 | | backtrace:populate_rootfs | 0 | 0 | 1 | | backtrace:kernel_init_freeable | 0 | 0 | 1 | | backtrace:kvm_get_tsc_khz | 0 | 0 | 2 | | backtrace:kvmclock_init | 0 | 0 | 2 | | BUG:unable_to_handle_kernel | 0 | 0 | 1 | | Oops | 0 | 0 | 1 | | RIP:setup_real_mode | 0 | 0 | 1 | +------------------------------------------+------------+------------+---------------+ [ 0.000000] BRK [0x06bd4000, 0x06bd4fff] PGTABLE [ 0.000000] BRK [0x06bd5000, 0x06bd5fff] PGTABLE [ 0.000000] BRK [0x06bd6000, 0x06bd6fff] PGTABLE PANIC: early exception 0e rip 10:ffffffff842122df error 0 cr2 ffffffff829c9798 [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 3.12.0-rc4-00005-g82fa963 #1 PANIC: early exception 0e rip 10:ffffffff842126f6 error 0 cr2 ffffffff829c9798 BUG: kernel boot hang Elapsed time: 305 qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel /kernel/x86_64-randconfig-ib1-09191856/82fa9637a2ba285bcc7c5050c73010b2c1b3d803/vmlinuz-3.12.0-rc4-00005-g82fa963 -append 'hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw link=/kbuild-tests/run-queue/kvm/x86_64-randconfig-ib1-09191856/next:master:82fa9637a2ba285bcc7c5050c73010b2c1b3d803:bisect-linux-5/.vmlinuz-82fa9637a2ba285bcc7c5050c73010b2c1b3d803-20140920200809-480-ivb41 branch=next/master BOOT_IMAGE=/kernel/x86_64-randconfig-ib1-09191856/82fa9637a2ba285bcc7c5050c73010b2c1b3d803/vmlinuz-3.12.0-rc4-00005-g82fa963 drbd.minor_count=8' -initrd /kernel-tests/initrd/yocto-minimal-x86_64.cgz -m 320 -smp 1 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -pidfile /dev/shm/kboot/pid-yocto-ivb41-24 -serial file:/dev/shm/kboot/serial-yocto-ivb41-24 -daemonize -display none -monitor null git bisect start v3.14 v3.13 -- git bisect bad 494479038d97f1b9f76fc633a360a681acdf035c # 02:20 373- 19 Merge tag 'pinctrl-v3.14-2' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad 1b17366d695c8ab03f98d0155357e97a427e1dce # 02:33 310- 11 Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc git bisect bad 60eaa0190f6b39dce18eb1975d9773ed8bc9a534 # 02:44 153- 7 Merge tag 'trace-3.14' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace git bisect good 74e8ee8262c3f93bbc41804037b43f07b95897bb # 03:21 900+ 7 Merge branch 'x86-platform-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 02d0a752460ea5dab34ce36c9ddc9c682e846a0d # 03:33 273- 8 Merge branch 'i2c/for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux git bisect bad 82b51734b4f228c76b6064b6e899d9d3d4c17c1a # 03:47 378- 16 Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux git bisect bad f4bcd8ccddb02833340652e9f46f5127828eb79d # 03:57 158- 1 Merge branch 'x86-kaslr-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good fab5669d556200c4dd119af705bff14085845d1e # 04:21 900+ 0 Merge branch 'x86-ras-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 7fe67a1180db49d41a3f764c379a08f8e31580ec # 05:31 900+ 4 Merge branch 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 6145cfe394a7f138f6b64491c5663f97dba12450 # 05:50 3- 4 x86, kaslr: Raise the maximum virtual address to -1 GiB on x86_64 git bisect good 8ab3820fd5b2896d66da7bb2a906bc382e63e7bc # 20:01 900+ 0 x86, kaslr: Return location from decompress_kernel git bisect bad 82fa9637a2ba285bcc7c5050c73010b2c1b3d803 # 20:13 0- 32 x86, kaslr: Select random position from e820 maps git bisect good 5bfce5ef55cbe78ee2ee6e97f2e26a8a582008f3 # 22:04 900+ 0 x86, kaslr: Provide randomness functions # first bad commit: [82fa9637a2ba285bcc7c5050c73010b2c1b3d803] x86, kaslr: Select random position from e820 maps git bisect good 5bfce5ef55cbe78ee2ee6e97f2e26a8a582008f3 # 22:18 1000+ 0 x86, kaslr: Provide randomness functions git bisect bad 6a10bca9b608df445baa23c3bfafc510d93d425b # 22:31 0- 4 Add linux-next specific files for 20140919 git bisect bad 46be7b73e82453447cd97b3440d523159eab09f8 # 22:44 169- 13 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs git bisect bad 6a10bca9b608df445baa23c3bfafc510d93d425b # 22:44 0- 4 Add linux-next specific files for 20140919 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 kvm=( qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel $kernel -m 320 -smp 1 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com

7 years, 7 months

1
2
0 / 0

[x86] double fault: ffff [#1]

by Fengguang Wu

Hi Andi, Here is another bisect result with much earlier oops. commit b8a868e9ea876a1b40020397305533c095921d7a Author: Andi Kleen <ak(a)linux.intel.com> AuthorDate: Wed Apr 23 13:26:20 2014 -0700 Commit: Andi Kleen <ak(a)linux.intel.com> CommitDate: Fri Oct 3 15:19:56 2014 -0700 x86: Add support for rd/wr fs/gs base ... v2: Change to save/restore GS instead of using swapgs based on the value. Large scale changes. Signed-off-by: Andi Kleen <ak(a)linux.intel.com> Attached dmesg for the parent commit, too, to help confirm whether it is a noise error. +-------------------------------------------------------+------------+------------+------------------+ | | 598d570a05 | b8a868e9ea | v3.17-rc7_100409 | +-------------------------------------------------------+------------+------------+------------------+ | boot_successes | 207 | 24 | 17 | | boot_failures | 3 | 46 | 4 | | BUG:kernel_boot_crashed | 3 | | | | double_fault:ffff | 0 | 39 | 4 | | RIP:trace_hardirqs_off_thunk | 0 | 22 | 1 | | BUG:unable_to_handle_kernel | 0 | 17 | 1 | | Oops | 0 | 15 | 1 | | RIP:show_stack_log_lvl | 0 | 14 | 1 | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 0 | 20 | 2 | | RIP:trace_hardirqs_off_caller | 0 | 6 | 1 | | BUG:kernel_boot_hang | 0 | 23 | 2 | | backtrace:rescuer_thread | 0 | 1 | | +-------------------------------------------------------+------------+------------+------------------+ [ 0.212404] regulator-dummy: no parameters [ 0.215602] NET: Registered protocol family 16 [ 0.215602] NET: Registered protocol family 16 [ 0.232049] double fault: ffff [#1] [ 0.232049] double fault: ffff [#1] PREEMPT PREEMPT SMP SMP DEBUG_PAGEALLOCDEBUG_PAGEALLOC [ 0.233000] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.17.0-rc7-00004-gb8a868e #1 [ 0.233000] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.17.0-rc7-00004-gb8a868e #1 [ 0.233000] task: ffff8800121ad500 ti: ffff88001234c000 task.ti: ffff88001234c000 [ 0.233000] task: ffff8800121ad500 ti: ffff88001234c000 task.ti: ffff88001234c000 [ 0.233000] RIP: 0010:[<ffffffff818273f5>] [ 0.233000] RIP: 0010:[<ffffffff818273f5>] [<ffffffff818273f5>] trace_hardirqs_off_thunk+0x35/0x3c [<ffffffff818273f5>] trace_hardirqs_off_thunk+0x35/0x3c [ 0.233000] RSP: 0000:ffff880012500000 EFLAGS: 00010086 [ 0.233000] RSP: 0000:ffff880012500000 EFLAGS: 00010086 [ 0.233000] RAX: 0000000082767ff0 RBX: 0000000000000001 RCX: ffffffff82767ff0 [ 0.233000] RAX: 0000000082767ff0 RBX: 0000000000000001 RCX: ffffffff82767ff0 [ 0.233000] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff82769326 [ 0.233000] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff82769326 [ 0.233000] RBP: ffff880012500110 R08: 0000000000000000 R09: 0000000000000001 [ 0.233000] RBP: ffff880012500110 R08: 0000000000000000 R09: 0000000000000001 [ 0.233000] R10: ffff880012503e98 R11: ffffffff838c7fb0 R12: 0000000000013ec0 [ 0.233000] R10: ffff880012503e98 R11: ffffffff838c7fb0 R12: 0000000000013ec0 [ 0.233000] R13: 0000000000013ec0 R14: 0000000000000000 R15: 0000000000000000 [ 0.233000] R13: 0000000000013ec0 R14: 0000000000000000 R15: 0000000000000000 BUG: kernel boot hang Elapsed time: 305 qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel /kernel/x86_64-randconfig-hsb0-10041007/b8a868e9ea876a1b40020397305533c095921d7a/vmlinuz-3.17.0-rc7-00004-gb8a868e -append 'hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw link=/kbuild-tests/run-queue/kvm/x86_64-randconfig-hsb0-10041007/linux-devel:devel-hourly-2014100409:b8a868e9ea876a1b40020397305533c095921d7a:bisect-linux-3/.vmlinuz-b8a868e9ea876a1b40020397305533c095921d7a-20141004114524-27-lkp-nex04 branch=linux-devel/devel-hourly-2014100409 BOOT_IMAGE=/kernel/x86_64-randconfig-hsb0-10041007/b8a868e9ea876a1b40020397305533c095921d7a/vmlinuz-3.17.0-rc7-00004-gb8a868e drbd.minor_count=8' -initrd /kernel-tests/initrd/quantal-core-x86_64.cgz -m 320 -smp 2 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -pidfile /dev/shm/kboot/pid-quantal-lkp-nex04-129 -serial file:/dev/shm/kboot/serial-quantal-lkp-nex04-129 -daemonize -display none -monitor null git bisect start cd94c0185489f0cddcbfa80e86258bb4609b6e1d fe82dcec644244676d55a1384c958d5f67979adb -- git bisect good a3f8f887e2a36003026b5d3effdd418d507cf7c8 # 11:19 70+ 0 Merge 'spi/for-linus' into devel-hourly-2014100409 git bisect good 2e6b2ee6893da99e9a156087aee97f90b0c8429c # 11:26 70+ 0 Merge 'trace/for-next' into devel-hourly-2014100409 git bisect good 7ea38a3061ab82dcfb705c3f36f55ddd8c351c83 # 11:31 70+ 1 Merge 'usb/usb-next' into devel-hourly-2014100409 git bisect bad 6d3ac51e9688b5cc2d4a63ba5f2e9b057db75ac8 # 11:35 0- 24 Merge 'driver-core/driver-core-next' into devel-hourly-2014100409 git bisect bad a33db30bb2f4d53f325e1c465f594894f559843c # 11:39 0- 1 Merge 'ak/x86/fsgs-2' into devel-hourly-2014100409 git bisect bad b8a868e9ea876a1b40020397305533c095921d7a # 11:46 0- 23 x86: Add support for rd/wr fs/gs base git bisect good a0b0be64599f50dc2c9fa85734026701221f186a # 11:53 70+ 1 x86: Naturally align the debug IST stack git bisect good 598d570a05cd31500fb15a843a92f68ddb1b3618 # 11:59 70+ 0 x86: Add intrinsics/macros for new rd/wr fs/gs base instructions # first bad commit: [b8a868e9ea876a1b40020397305533c095921d7a] x86: Add support for rd/wr fs/gs base git bisect good 598d570a05cd31500fb15a843a92f68ddb1b3618 # 12:05 210+ 3 x86: Add intrinsics/macros for new rd/wr fs/gs base instructions git bisect bad cd94c0185489f0cddcbfa80e86258bb4609b6e1d # 12:05 0- 4 0day head guard for 'devel-hourly-2014100409' git bisect good 126d4576cb73c8a440adc37c129589cd66051bcc # 12:11 210+ 7 Merge branch 'i2c/for-current' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux git bisect good 2e1d004b9645628c64a2db55ef6b81fadf5e6e91 # 12:19 210+ 2 Add linux-next specific files for 20141003 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 kvm=( qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel $kernel -m 320 -smp 2 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com

7 years, 7 months

1
0
0 / 0

[perf/x86/intel] WARNING: CPU: 48 PID: 7540 at arch/x86/kernel/cpu/perf_event_intel_ds.c:316 reserve_ds_buffers+0x208/0x430()

by Fengguang Wu

Hi David, FYI, we noticed the below changes on commit 4485154138f6ffa5b252cb490aba3e8eb30124e4 ("perf/x86/intel: Avoid spamming kernel log for BTS buffer failure") +---------------------------------------------------------------------------+------------+------------+ | | 338b522ca4 | 4485154138 | +---------------------------------------------------------------------------+------------+------------+ | boot_successes | 3 | 5 | | boot_failures | 21 | 15 | | page_allocation_failure:order:,mode | 11 | | | backtrace:reserve_ds_buffers | 11 | 5 | | backtrace:x86_pmu_event_init | 11 | 5 | | backtrace:perf_init_event | 11 | 5 | | backtrace:SYSC_perf_event_open | 11 | 5 | | backtrace:SyS_perf_event_open | 11 | 5 | | BUG:kernel_test_crashed | 10 | 10 | | WARNING:at_arch/x86/kernel/cpu/perf_event_intel_ds.c:reserve_ds_buffers() | 0 | 5 | +---------------------------------------------------------------------------+------------+------------+ 2014-10-03 01:27:35 ./usemem --runtime 300 -t 64 -f /tmp/vm-scalability/sparse-mremap-xread-rand-mt-60 -E --prealloc --readonly --random 17179869184 2014-10-03 01:27:36 ./usemem --runtime 300 -t 64 -f /tmp/vm-scalability/sparse-mremap-xread-rand-mt-62 -E --prealloc --readonly --random 17179869184 [ 192.496567] ------------[ cut here ]------------ [ 192.505045] WARNING: CPU: 48 PID: 7540 at arch/x86/kernel/cpu/perf_event_intel_ds.c:316 reserve_ds_buffers+0x208/0x430() [ 192.524168] alloc_bts_buffer: BTS buffer allocation failure [ 192.533339] Modules linked in: loop ipmi_watchdog nfsd auth_rpcgss dm_mod fuse sg sd_mod crct10dif_generic sr_mod crc_t10dif cdrom crct10dif_common ata_generic pata_acpi mgag200 syscopyarea snd_pcm sysfillrect sysimgblt snd_timer ttm snd drm_kms_helper soundcore ata_piix pcspkr i7core_edac drm libata megaraid_sas i2c_i801 edac_core ipmi_si ipmi_msghandler acpi_cpufreq [ 192.581450] CPU: 48 PID: 7540 Comm: perf Not tainted 3.16.0-rc2-00011-g4485154 #1 [ 192.592393] Hardware name: Quanta MP Server/QSSC-S4R, BIOS QSSC-S4R.QCI.01.00.0028.090220101116 09/02/2010 [ 192.606455] 0000000000000009 ffff88005de53ce8 ffffffff81844e55 ffff88005de53d30 [ 192.618370] ffff88005de53d20 ffffffff8106de6d 0000000000000000 0000000000010528 [ 192.630786] 0000000000000000 0000000000000000 0000000000000000 ffff88005de53d80 [ 192.642465] Call Trace: [ 192.647913] [<ffffffff81844e55>] dump_stack+0x4d/0x66 [ 192.656643] [<ffffffff8106de6d>] warn_slowpath_common+0x7d/0xa0 [ 192.666961] [<ffffffff8106dedc>] warn_slowpath_fmt+0x4c/0x50 [ 192.677047] [<ffffffff81030863>] ? reserve_ds_buffers+0x2b3/0x430 [ 192.687420] [<ffffffff810307b8>] reserve_ds_buffers+0x208/0x430 [ 192.697838] [<ffffffff8118abec>] ? handle_mm_fault+0x8ac/0xf70 [ 192.707522] [<ffffffff8102a354>] x86_pmu_event_init+0x434/0x450 [ 192.717351] [<ffffffff8115241b>] perf_init_event+0xdb/0x140 [ 192.727292] [<ffffffff81152808>] perf_event_alloc+0x388/0x440 [ 192.738117] [<ffffffff81152c60>] SYSC_perf_event_open+0x3a0/0xd20 [ 192.750511] [<ffffffff81153979>] SyS_perf_event_open+0x9/0x10 [ 192.761657] [<ffffffff8184da29>] system_call_fastpath+0x16/0x1b [ 192.771904] ---[ end trace 6d601571820ebee0 ]--- 4331667464 bytes / 300293184 usecs = 14086 KB/s Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com

7 years, 7 months

2
1
0 / 0

[PCI/MSI] WARNING: CPU: 0 PID: 1 at drivers/pci/msi.c:1034 pci_enable_msi_range+0x2c1/0x2f0()

by Fengguang Wu

Hi Alexander, FYI, we noticed the below changes on git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git pci/msi commit 4ab2ee7c77875669230a797a3ed2228990f82545 ("PCI/MSI: Rename pci_msi_check_device() to pci_msi_supported()") +-----------------------------------------------------+------------+------------+ | | 888215f389 | 4ab2ee7c77 | +-----------------------------------------------------+------------+------------+ | boot_successes | 15 | 2 | | early-boot-hang | 1 | | | boot_failures | 0 | 13 | | WARNING:at_drivers/pci/msi.c:pci_enable_msi_range() | 0 | 5 | | WARNING:at_fs/sysfs/dir.c:sysfs_warn_dup() | 0 | 5 | | backtrace:ip_auto_config | 0 | 5 | | backtrace:kernel_init_freeable | 0 | 5 | | BUG:kernel_test_crashed | 0 | 8 | +-----------------------------------------------------+------------+------------+ [ 5.410055] evm: HMAC attrs: 0x1 [ 5.413893] rtc_cmos 00:01: setting system clock to 2014-09-25 00:13:24 UTC (1411604004) [ 5.447469] ------------[ cut here ]------------ [ 5.452212] WARNING: CPU: 0 PID: 1 at drivers/pci/msi.c:1034 pci_enable_msi_range+0x2c1/0x2f0() [ 5.462479] Modules linked in: [ 5.465712] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc2-00011-g4ab2ee7 #1 [ 5.473467] Hardware name: Supermicro X7DB8/X7DB8, BIOS 6.00 04/28/2006 [ 5.480202] 0000000000000009 ffff880139ab7c00 ffffffff8185b75a 0000000000000000 [ 5.488043] ffff880139ab7c38 ffffffff8107056d 0000000000000001 ffff88007f8ec000 [ 5.495879] ffff88013974e098 ffff88013974e000 ffff88007f8ecdb8 ffff880139ab7c48 [ 5.503722] Call Trace: [ 5.506293] [<ffffffff8185b75a>] dump_stack+0x4d/0x66 [ 5.511552] [<ffffffff8107056d>] warn_slowpath_common+0x7d/0xa0 [ 5.517675] [<ffffffff8107064a>] warn_slowpath_null+0x1a/0x20 [ 5.523631] [<ffffffff81449961>] pci_enable_msi_range+0x2c1/0x2f0 [ 5.529932] [<ffffffff815b490a>] e1000_open+0x22a/0x4b0 [ 5.535368] [<ffffffff8172d466>] __dev_open+0xb6/0x130 [ 5.540712] [<ffffffff8172d77d>] __dev_change_flags+0x9d/0x160 [ 5.546754] [<ffffffff8172d869>] dev_change_flags+0x29/0x70 [ 5.552534] [<ffffffff81e5707e>] ip_auto_config+0x1c4/0xe7f [ 5.558314] [<ffffffff81e56eba>] ? root_nfs_parse_addr+0xba/0xba [ 5.564529] [<ffffffff81405f1e>] ? kasprintf+0x3e/0x40 [ 5.569876] [<ffffffff81e56eba>] ? root_nfs_parse_addr+0xba/0xba [ 5.576091] [<ffffffff81002130>] do_one_initcall+0xc0/0x1f0 [ 5.581871] [<ffffffff81002130>] ? do_one_initcall+0xc0/0x1f0 [ 5.587826] [<ffffffff8108f000>] ? parse_args+0x180/0x4e0 [ 5.593436] [<ffffffff81df511b>] kernel_init_freeable+0x1b3/0x23b [ 5.599735] [<ffffffff8184ed70>] ? rest_init+0x90/0x90 [ 5.605079] [<ffffffff8184ed7e>] kernel_init+0xe/0xf0 [ 5.610341] [<ffffffff8186437c>] ret_from_fork+0x7c/0xb0 [ 5.615861] [<ffffffff8184ed70>] ? rest_init+0x90/0x90 [ 5.621207] ---[ end trace 3e09fff841f00690 ]--- [ 5.625958] e1000e 0000:06:00.0: irq 27 for MSI/MSI-X Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com

7 years, 7 months

3
2
0 / 0

[compiler/gcc4] a9f180345f5: -100.0% last_state.is_incomplete_run

by Fengguang Wu

Hi Steven, FYI, we noticed that your commit a9f180345f5378ac87d80ed0bea55ba421d83859 ("compiler/gcc4: Make quirk for asm_volatile_goto() unconditional") fixed a number of machine boot failures in our LKP test farm. This is really helpful! Our gcc version is 4.9.1 (Debian 4.9.1-11). 569d6557ab957d6 a9f180345f5378ac87d80ed0b --------------- ------------------------- %stddev %change %stddev \ | / 1 ± 0% -100.0% 0 ± 0% lkp-st02/dd-write/5m-11HDD-RAID5-cfq-btrfs-100dd 1 ± 0% -100.0% 0 ± 0% TOTAL dmesg.kernel_BUG_at_fs/nfs/pagelist.c 569d6557ab957d6 a9f180345f5378ac87d80ed0b --------------- ------------------------- 1 ± 0% -100.0% 0 ± 0% lkp-st02/dd-write/5m-11HDD-RAID5-cfq-btrfs-100dd 1 ± 0% -100.0% 0 ± 0% TOTAL dmesg.Kernel_panic-not_syncing:Fatal_exception 569d6557ab957d6 a9f180345f5378ac87d80ed0b --------------- ------------------------- 1 ± 0% -100.0% 0 ± 0% lkp-st02/dd-write/5m-11HDD-RAID5-cfq-btrfs-100dd 1 ± 0% -100.0% 0 ± 0% TOTAL dmesg.invalid_opcode 569d6557ab957d6 a9f180345f5378ac87d80ed0b --------------- ------------------------- 1 ± 0% -100.0% 0 ± 0% ivb42/will-it-scale/futex4 1 ± 0% -100.0% 0 ± 0% ivb44/fsmark/1x-1t-1HDD-xfs-4M-60G-NoSync 1 ± 0% -100.0% 0 ± 0% ivb44/fsmark/1x-64t-1BRD_48G-btrfs-4M-40G-fsyncBeforeClose 1 ± 0% -100.0% 0 ± 0% lkp-bdw01/blogbench/1SSD-btrfs 1 ± 0% -100.0% 0 ± 0% lkp-hsw01/vm-scalability/300s-anon-rx-rand-mt 1 ± 0% -100.0% 0 ± 0% lkp-sbx04/will-it-scale/futex3 1 ± 0% -100.0% 0 ± 0% lkp-sbx04/will-it-scale/page_fault3 1 ± 0% -100.0% 0 ± 0% lkp-st02/dd-write/5m-11HDD-RAID5-cfq-btrfs-100dd 8 ± 0% -100.0% 0 ± 0% TOTAL last_state.is_incomplete_run 569d6557ab957d6 a9f180345f5378ac87d80ed0b --------------- ------------------------- 1 ± 0% -100.0% 0 ± 0% ivb42/will-it-scale/futex4 1 ± 0% -100.0% 0 ± 0% ivb44/fsmark/1x-1t-1HDD-xfs-4M-60G-NoSync 1 ± 0% -100.0% 0 ± 0% ivb44/fsmark/1x-64t-1BRD_48G-btrfs-4M-40G-fsyncBeforeClose 1 ± 0% -100.0% 0 ± 0% lkp-bdw01/blogbench/1SSD-btrfs 1 ± 0% -100.0% 0 ± 0% lkp-hsw01/vm-scalability/300s-anon-rx-rand-mt 1 ± 0% -100.0% 0 ± 0% lkp-sbx04/will-it-scale/futex3 1 ± 0% -100.0% 0 ± 0% lkp-sbx04/will-it-scale/page_fault3 1 ± 0% -100.0% 0 ± 0% lkp-st02/dd-write/5m-11HDD-RAID5-cfq-btrfs-100dd 8 ± 0% -100.0% 0 ± 0% TOTAL last_state.booting Thanks, Fengguang

7 years, 7 months

2
1
0 / 0

[mm/slab] BUG: unable to handle kernel paging request at 00010023

by Fengguang Wu

Hi Joonsoo, 0day kernel testing robot got the below dmesg and the first bad commit is git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit 36fbfebe776eb5871d61e7a755c9feb1c96cc4aa Author: Joonsoo Kim <iamjoonsoo.kim(a)lge.com> AuthorDate: Tue Sep 23 11:52:35 2014 +1000 Commit: Stephen Rothwell <sfr(a)canb.auug.org.au> CommitDate: Tue Sep 23 11:52:35 2014 +1000 mm/slab: support slab merge Slab merge is good feature to reduce fragmentation. If new creating slab have similar size and property with exsitent slab, this feature reuse it rather than creating new one. As a result, objects are packed into fewer slabs so that fragmentation is reduced. Below is result of my testing. * After boot, sleep 20; cat /proc/meminfo | grep Slab <Before> Slab: 25136 kB <After> Slab: 24364 kB We can save 3% memory used by slab. For supporting this feature in SLAB, we need to implement SLAB specific kmem_cache_flag() and __kmem_cache_alias(), because SLUB implements some SLUB specific processing related to debug flag and object size change on these functions. Signed-off-by: Joonsoo Kim <iamjoonsoo.kim(a)lge.com> Cc: Christoph Lameter <cl(a)linux.com> Cc: Pekka Enberg <penberg(a)kernel.org> Cc: David Rientjes <rientjes(a)google.com> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> +----------------------------------------------+------------+------------+---------------+ | | 8c087489b8 | 36fbfebe77 | next-20140923 | +----------------------------------------------+------------+------------+---------------+ | boot_successes | 60 | 0 | 1 | | boot_failures | 0 | 20 | 314 | | BUG:unable_to_handle_kernel | 0 | 20 | 312 | | Oops | 0 | 20 | 312 | | EIP_is_at_kernfs_link_sibling | 0 | 4 | 14 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 20 | 312 | | backtrace:acpi_bus_scan | 0 | 4 | 14 | | backtrace:acpi_scan_init | 0 | 20 | 45 | | backtrace:acpi_init | 0 | 20 | 45 | | backtrace:kernel_init_freeable | 0 | 20 | 312 | | EIP_is_at_kernfs_add_one | 0 | 16 | 298 | | backtrace:kobject_add_internal | 0 | 16 | 31 | | backtrace:kobject_init_and_add | 0 | 16 | 31 | | backtrace:acpi_scan_add_handler_with_hotplug | 0 | 16 | 31 | | backtrace:acpi_pci_root_init | 0 | 16 | 31 | | backtrace:tty_register_driver | 0 | 0 | 106 | | backtrace:pty_init | 0 | 0 | 106 | | backtrace:acpi_bus_register_driver | 0 | 0 | 1 | | backtrace:acpi_button_driver_init | 0 | 0 | 1 | | BUG:kernel_boot_crashed | 0 | 0 | 1 | | BUG:kernel_test_crashed | 0 | 0 | 1 | | backtrace:subsys_system_register | 0 | 0 | 160 | | backtrace:container_dev_init | 0 | 0 | 160 | | backtrace:driver_init | 0 | 0 | 160 | +----------------------------------------------+------------+------------+---------------+ [ 0.463788] ACPI: (supports S0 S5) [ 0.464003] ACPI: Using IOAPIC for interrupt routing [ 0.464738] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.466034] BUG: unable to handle kernel paging request at 00010023 [ 0.466989] IP: [<c117dcf9>] kernfs_add_one+0x89/0x130 [ 0.467812] *pdpt = 0000000000000000 *pde = f000ff53f000ff53 [ 0.468000] Oops: 0002 [#1] SMP [ 0.468000] Modules linked in: [ 0.468000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc6-00089-g36fbfeb #1 [ 0.468000] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 0.468000] task: d303ec90 ti: d3040000 task.ti: d3040000 [ 0.468000] EIP: 0060:[<c117dcf9>] EFLAGS: 00010286 CPU: 0 [ 0.468000] EIP is at kernfs_add_one+0x89/0x130 [ 0.468000] EAX: 542572cb EBX: 00010003 ECX: 00000008 EDX: 2c8de598 [ 0.468000] ESI: d311de10 EDI: d311de70 EBP: d3041dd8 ESP: d3041db4 [ 0.468000] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 0.468000] CR0: 8005003b CR2: 00010023 CR3: 01a8a000 CR4: 000006f0 [ 0.468000] Stack: [ 0.468000] d3006f00 00000202 d311de70 d311de10 d3041dd8 c117dba0 d311de10 c159a5c0 [ 0.468000] c1862a00 d3041df0 c117f0f2 00000000 c18629f4 d311de70 00000000 d3041e2c [ 0.468000] c117f8b5 00001000 00000000 c159a5c0 c18629f4 00000000 00000001 c1862a00 [ 0.468000] Call Trace: [ 0.468000] [<c117dba0>] ? kernfs_new_node+0x30/0x40 [ 0.468000] [<c117f0f2>] __kernfs_create_file+0x92/0xc0 [ 0.468000] [<c117f8b5>] sysfs_add_file_mode_ns+0x95/0x190 [ 0.468000] [<c117f9d7>] sysfs_create_file_ns+0x27/0x40 [ 0.468000] [<c1252ef6>] kobject_add_internal+0x136/0x2c0 [ 0.468000] [<c125e360>] ? kvasprintf+0x40/0x50 [ 0.468000] [<c1252a92>] ? kobject_set_name_vargs+0x42/0x60 [ 0.468000] [<c12530b5>] kobject_init_and_add+0x35/0x50 [ 0.468000] [<c12ad04f>] acpi_sysfs_add_hotplug_profile+0x24/0x4a [ 0.468000] [<c12a7280>] acpi_scan_add_handler_with_hotplug+0x21/0x28 [ 0.468000] [<c18df524>] acpi_pci_root_init+0x20/0x22 [ 0.468000] [<c18df0e1>] acpi_scan_init+0x24/0x16d [ 0.468000] [<c18def73>] acpi_init+0x20c/0x224 [ 0.468000] [<c18ded67>] ? acpi_sleep_init+0xab/0xab [ 0.468000] [<c100041e>] do_one_initcall+0x7e/0x1b0 [ 0.468000] [<c18ded67>] ? acpi_sleep_init+0xab/0xab [ 0.468000] [<c18b24ba>] ? repair_env_string+0x12/0x54 [ 0.468000] [<c18b24a8>] ? initcall_blacklist+0x7c/0x7c [ 0.468000] [<c105e100>] ? parse_args+0x160/0x3f0 [ 0.468000] [<c18b2bd1>] kernel_init_freeable+0xfc/0x179 [ 0.468000] [<c156782b>] kernel_init+0xb/0xd0 [ 0.468000] [<c1574601>] ret_from_kernel_thread+0x21/0x30 [ 0.468000] [<c1567820>] ? rest_init+0xb0/0xb0 [ 0.468000] Code: 26 00 83 e1 10 75 5b 8b 46 24 e8 b3 ea ff ff 89 46 38 89 f0 e8 d9 f9 ff ff 85 c0 89 c3 75 ca 8b 5f 5c 85 db 74 11 e8 97 90 f1 ff <89> 43 20 89 53 24 89 43 28 89 53 2c b8 c0 2f 85 c1 e8 d1 36 3f [ 0.468000] EIP: [<c117dcf9>] kernfs_add_one+0x89/0x130 SS:ESP 0068:d3041db4 [ 0.468000] CR2: 0000000000010023 [ 0.468000] ---[ end trace 4fa173691404b63f ]--- [ 0.468000] Kernel panic - not syncing: Fatal exception git bisect start 55f21306900abf9f9d2a087a127ff49c6d388ad2 0f33be009b89d2268e94194dc4fd01a7851b6d51 -- git bisect good 18c13e2d9b75e2760e6520f2fde00401192956f3 # 17:56 20+ 0 Merge remote-tracking branch 'bluetooth/master' git bisect good abf79495f38ba66f750566b3f0a8da8dd94b4dc3 # 18:03 20+ 0 Merge remote-tracking branch 'ftrace/for-next' git bisect good 0bed22034e26a3c37ee4407fccffa8c095d5e144 # 18:09 20+ 0 Merge remote-tracking branch 'pinctrl/for-next' git bisect good 15c9281a15ed7718868d115d4d00619b0b7a2624 # 18:14 20+ 0 Merge remote-tracking branch 'clk/clk-next' git bisect good 50939531dea1b913b7fa29f9bbc69feafefd090c # 18:23 20+ 0 Merge branch 'rd-docs/master' git bisect good aa881e3c5e87c8aa23519f40554897d56f32b935 # 18:49 20+ 0 Merge remote-tracking branch 'powerpc-mpe/next' git bisect bad 81b63d14db32bd7706c955d1e04e65b152b2277a # 18:57 0- 2 Merge branch 'akpm-current/current' git bisect bad f313ca82d72066a3c44fd6c66cee57b25de43aa9 # 19:31 0- 1 introduce-dump_vma-fix git bisect good 2c5fe9213048c5640b8e46407f5614038c03ad93 # 20:16 20+ 0 mm: fix kmemcheck.c build errors git bisect bad 69454f8be7f621ac8c3c6c9763bb70e116988942 # 20:35 0- 18 block_dev: implement readpages() to optimize sequential read git bisect bad 66a31d528a1e3d483be2b1c993ec1268412f0074 # 20:40 0- 1 memory-hotplug-add-sysfs-zones_online_to-attribute-fix-2 git bisect good 5e8acb68610c077b08cb3f16305aa3cc22e5d2a8 # 21:07 20+ 0 kernel/kthread.c: partial revert of 81c98869faa5 ("kthread: ensure locality of task_struct allocations") git bisect bad 36fbfebe776eb5871d61e7a755c9feb1c96cc4aa # 22:06 0- 10 mm/slab: support slab merge git bisect good 11e57381eced875ef5a6fea4005fdf72b6f68eff # 22:19 20+ 0 mm/slab_common: commonize slab merge logic git bisect good 8c087489b8a32b9235f7f9417390c62d93aba522 # 22:24 20+ 0 mm/slab_common: fix build failure if CONFIG_SLUB # first bad commit: [36fbfebe776eb5871d61e7a755c9feb1c96cc4aa] mm/slab: support slab merge git bisect good 8c087489b8a32b9235f7f9417390c62d93aba522 # 22:26 60+ 0 mm/slab_common: fix build failure if CONFIG_SLUB git bisect bad 55f21306900abf9f9d2a087a127ff49c6d388ad2 # 22:26 0- 314 Add linux-next specific files for 20140923 git bisect good f4cb707e7ad9727a046b463232f2de166e327d3e # 22:32 60+ 0 Merge tag 'pm+acpi-3.17-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm git bisect bad 4d8426f9ac601db2a64fa7be64051d02b9c9fe01 # 22:36 0- 60 Add linux-next specific files for 20140926 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 kvm=( qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel $kernel -m 320 -smp 1 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com

7 years, 7 months

2
2
0 / 0

[net] 4ed2d765dfa: ltp.recv01.4.TFAIL

by Fengguang Wu

Hi Willem, FYI, we noticed the below LTP failures on commit 4ed2d765dfaccff5ebdac68e2064b59125033a3b ("net-timestamp: TCP timestamping") test case: nhm-white/ltp/syscalls nhm-white is a Nehalem box with 6G memory. e7fd2885385157d 4ed2d765dfaccff5ebdac68e2 --------------- ------------------------- 0 +Inf% 1 ± 0% TOTAL ltp.recv01.4.TFAIL 0 +Inf% 1 ± 0% TOTAL ltp.recvfrom01.4.TFAIL 0 +Inf% 1 ± 0% TOTAL ltp.recvfrom01.6.TFAIL 0 +Inf% 1 ± 0% TOTAL ltp.recvmsg01.9.TFAIL ltp.recvmsg01.9.TFAIL 1 O+-O-O--O-O--O--O-O--O-O--O----O--O-O--O-O--O----O--O-O--O--O-O--O-O--O | | | | 0.8 ++ | | | | | 0.6 ++ | | | 0.4 ++ | | | | | 0.2 ++ | | | | | 0 ++---------------------------O-----------------O----------------------+ [*] bisect-good sample [O] bisect-bad sample To reproduce: apt-get install ruby ruby-oj git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/setup-local job.yaml # the job file attached in this email bin/run-local job.yaml Thanks, Fengguang

7 years, 7 months

4
3
0 / 0

[loopback] b17c706987f: +175.7% netperf.Throughput_Mbps

by Fengguang Wu

Hi Daniel, FYI, we noticed nice performance improvement in commit b17c706987fa6f28bdc1771c8266e7a69e22adcb ("loopback: sctp: add NETIF_F_SCTP_CSUM to device features") test case: lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY 72f8e06f3ea022d b17c706987fa6f28bdc1771c8 --------------- ------------------------- %stddev %change %stddev \ | / 664 ± 0% +175.7% 1832 ± 0% TOTAL netperf.Throughput_Mbps 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[63]:/.nr_running 947669 ± 2% +681.8% 7408572 ± 1% TOTAL sched_debug.cfs_rq[63]:/.min_vruntime 19701 ± 3% +2814.0% 574098 ± 1% TOTAL sched_debug.cpu#63.ttwu_local 41754 ± 1% -99.5% 200 ±43% TOTAL softirqs.HRTIMER 5 ±20% +400.0% 29 ± 2% TOTAL sched_debug.cpu#63.cpu_load[4] 2.59 ± 1% -100.0% 0.00 ± 0% TOTAL perf-profile.cpu-cycles.intel_idle.cpuidle_enter_state.cpuidle_idle_call.arch_cpu_idle.cpu_startup_entry 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#63.nr_running 72 ±48% -95.8% 3 ±42% TOTAL sched_debug.cfs_rq[62]:/.blocked_load_avg 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[62]:/.nr_running 0.24 ± 7% +2565.6% 6.50 ± 4% TOTAL perf-profile.cpu-cycles.sctp_transport_timeout.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter 0.22 ± 3% +1442.3% 3.42 ± 2% TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller 0.27 ± 6% +1100.0% 3.29 ± 2% TOTAL perf-profile.cpu-cycles._raw_spin_lock.free_one_page.__free_pages_ok.__free_pages.__free_memcg_kmem_pages 0.04 ±10% +8463.2% 3.25 ± 2% TOTAL perf-profile.cpu-cycles.lock_timer_base.isra.35.mod_timer.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork 0.04 ±13% +6227.8% 2.28 ± 3% TOTAL perf-profile.cpu-cycles.memcpy.sctp_packet_transmit_chunk.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter 0.00 +Inf% 1.97 ± 8% TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.mod_timer.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork 11 ±44% +9151.8% 1036 ±31% TOTAL sched_debug.cfs_rq[62]:/.nr_spread_over 1.15 ± 7% -94.4% 0.06 ± 7% TOTAL perf-profile.cpu-cycles._raw_spin_lock_bh.lock_sock_nested.sctp_recvmsg.sock_common_recvmsg.sock_recvmsg 33688875 ± 2% -89.6% 3504734 ±18% TOTAL cpuidle.C1-NHM.time 281217 ± 2% -90.9% 25698 ±41% TOTAL cpuidle.C1-NHM.usage 45558795 ± 0% -99.9% 27898 ±39% TOTAL cpuidle.C3-NHM.usage 39.61 ± 0% -99.7% 0.11 ±30% TOTAL turbostat.%c1 5.60 ± 0% -94.0% 0.34 ± 1% TOTAL turbostat.%c3 876992 ± 3% +736.3% 7333987 ± 1% TOTAL sched_debug.cfs_rq[62]:/.min_vruntime 19686 ± 2% +2810.9% 573039 ± 1% TOTAL sched_debug.cpu#62.ttwu_local 5 ±25% +403.4% 29 ± 3% TOTAL sched_debug.cpu#62.cpu_load[4] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#62.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[61]:/.nr_running 8 ±22% +13481.4% 1168 ±36% TOTAL sched_debug.cfs_rq[61]:/.nr_spread_over 899343 ± 6% +715.7% 7335525 ± 2% TOTAL sched_debug.cfs_rq[61]:/.min_vruntime 19673 ± 4% +2790.5% 568655 ± 2% TOTAL sched_debug.cpu#61.ttwu_local 989853 ± 2% -99.2% 7480 ±48% TOTAL sched_debug.cpu#61.sched_goidle 4 ±40% +581.0% 28 ± 3% TOTAL sched_debug.cpu#61.cpu_load[4] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#61.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[60]:/.nr_running 893662 ± 5% +724.8% 7370847 ± 0% TOTAL sched_debug.cfs_rq[60]:/.min_vruntime 19607 ± 3% +2827.4% 573986 ± 1% TOTAL sched_debug.cpu#60.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#60.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[59]:/.nr_running 86308 ±18% +7142.5% 6250836 ± 8% TOTAL sched_debug.cfs_rq[59]:/.max_vruntime 899110 ± 4% +717.5% 7350607 ± 0% TOTAL sched_debug.cfs_rq[59]:/.min_vruntime 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#0.nr_running 86308 ±18% +7142.5% 6250836 ± 8% TOTAL sched_debug.cfs_rq[59]:/.MIN_vruntime 20221 ± 4% +2739.2% 574120 ± 1% TOTAL sched_debug.cpu#59.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#59.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[58]:/.nr_running 370050 ± 3% -98.8% 4378 ±24% TOTAL sched_debug.cpu#0.sched_goidle 32479 ± 4% +1698.6% 584171 ± 1% TOTAL sched_debug.cpu#0.ttwu_local 947194 ± 4% +676.5% 7354983 ± 1% TOTAL sched_debug.cfs_rq[58]:/.min_vruntime 19 ±12% +26074.0% 5025 ±32% TOTAL sched_debug.cfs_rq[0]:/.nr_spread_over 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[0]:/.nr_running 20721 ± 4% +2663.3% 572601 ± 1% TOTAL sched_debug.cpu#58.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#58.nr_running 0 ± 0% +Inf% 1 ±33% TOTAL sched_debug.cfs_rq[57]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#1.nr_running 964313 ± 4% +662.7% 7355115 ± 0% TOTAL sched_debug.cfs_rq[57]:/.min_vruntime 21834 ± 6% +2531.4% 574540 ± 1% TOTAL sched_debug.cpu#57.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#57.nr_running 354246 ± 1% -98.7% 4745 ±38% TOTAL sched_debug.cpu#1.sched_goidle 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[56]:/.nr_running 28829 ± 5% +1914.4% 580717 ± 0% TOTAL sched_debug.cpu#1.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[1]:/.nr_running 960131 ± 3% +667.5% 7368750 ± 1% TOTAL sched_debug.cfs_rq[56]:/.min_vruntime 21418 ± 2% +2584.0% 574857 ± 0% TOTAL sched_debug.cpu#56.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#2.nr_running 4 ±17% +534.8% 29 ± 3% TOTAL sched_debug.cpu#56.cpu_load[4] 5 ±18% +433.3% 28 ± 2% TOTAL sched_debug.cpu#56.cpu_load[3] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#56.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[55]:/.nr_running 71260 ±42% +8857.8% 6383312 ± 6% TOTAL sched_debug.cfs_rq[55]:/.max_vruntime 935340 ± 4% +686.3% 7354731 ± 1% TOTAL sched_debug.cfs_rq[55]:/.min_vruntime 363420 ± 1% -98.9% 3993 ±40% TOTAL sched_debug.cpu#2.sched_goidle 71260 ±42% +8857.8% 6383312 ± 6% TOTAL sched_debug.cfs_rq[55]:/.MIN_vruntime 19268 ± 3% +2867.0% 571688 ± 2% TOTAL sched_debug.cpu#55.ttwu_local 29222 ± 4% +1889.9% 581489 ± 0% TOTAL sched_debug.cpu#2.ttwu_local 5 ±33% +457.7% 29 ± 2% TOTAL sched_debug.cpu#55.cpu_load[4] 21 ±10% +5539.8% 1218 ±19% TOTAL sched_debug.cfs_rq[2]:/.nr_spread_over 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[2]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#55.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[54]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#3.nr_running 45064 ±18% +14562.1% 6607460 ± 5% TOTAL sched_debug.cfs_rq[54]:/.max_vruntime 934539 ± 6% +685.1% 7337176 ± 1% TOTAL sched_debug.cfs_rq[54]:/.min_vruntime 45064 ±18% +14562.1% 6607460 ± 5% TOTAL sched_debug.cfs_rq[54]:/.MIN_vruntime 19262 ± 6% +2872.1% 572494 ± 1% TOTAL sched_debug.cpu#54.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#54.nr_running 356613 ± 1% -99.2% 2934 ±29% TOTAL sched_debug.cpu#3.sched_goidle 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[53]:/.nr_running 27577 ± 5% +2003.9% 580222 ± 1% TOTAL sched_debug.cpu#3.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[3]:/.nr_running 933523 ± 4% +686.2% 7339367 ± 1% TOTAL sched_debug.cfs_rq[53]:/.min_vruntime 19832 ± 3% +2803.2% 575758 ± 1% TOTAL sched_debug.cpu#53.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#4.nr_running 5 ±46% +410.7% 28 ± 4% TOTAL sched_debug.cpu#53.cpu_load[3] 6 ±47% +354.8% 28 ± 5% TOTAL sched_debug.cpu#53.cpu_load[0] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#53.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[52]:/.nr_running 923091 ± 3% +698.0% 7366148 ± 2% TOTAL sched_debug.cfs_rq[52]:/.min_vruntime 348410 ± 2% -98.8% 4230 ±39% TOTAL sched_debug.cpu#4.sched_goidle 19701 ± 3% +2820.0% 575293 ± 1% TOTAL sched_debug.cpu#52.ttwu_local 26645 ± 4% +2076.8% 580019 ± 1% TOTAL sched_debug.cpu#4.ttwu_local 178678 ±25% +3627.5% 6660167 ± 3% TOTAL sched_debug.cfs_rq[4]:/.MIN_vruntime 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#52.nr_running 178678 ±25% +3627.5% 6660167 ± 3% TOTAL sched_debug.cfs_rq[4]:/.max_vruntime 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[4]:/.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[51]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#5.nr_running 929685 ± 4% +688.0% 7325719 ± 1% TOTAL sched_debug.cfs_rq[51]:/.min_vruntime 19761 ± 4% +2813.8% 575800 ± 1% TOTAL sched_debug.cpu#51.ttwu_local 5 ±37% +461.5% 29 ± 2% TOTAL sched_debug.cpu#51.cpu_load[4] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#51.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[50]:/.nr_running 26189 ± 3% +2110.9% 579032 ± 0% TOTAL sched_debug.cpu#5.ttwu_local 963932 ± 3% +657.4% 7301196 ± 1% TOTAL sched_debug.cfs_rq[50]:/.min_vruntime 23557 ± 2% +2326.4% 571608 ± 2% TOTAL sched_debug.cpu#23.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[5]:/.nr_running 20098 ± 5% +2752.9% 573394 ± 1% TOTAL sched_debug.cpu#50.ttwu_local 6 ±34% +380.0% 28 ± 1% TOTAL sched_debug.cpu#50.cpu_load[4] 6 ±35% +343.8% 28 ± 1% TOTAL sched_debug.cpu#50.cpu_load[3] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#6.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#50.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[49]:/.nr_running 16 ± 8% +4700.0% 768 ±49% TOTAL sched_debug.cfs_rq[49]:/.nr_spread_over 25690 ± 2% +2152.9% 578774 ± 1% TOTAL sched_debug.cpu#6.ttwu_local 988774 ± 1% +643.1% 7347606 ± 1% TOTAL sched_debug.cfs_rq[49]:/.min_vruntime 17 ±16% +6484.7% 1119 ±36% TOTAL sched_debug.cfs_rq[6]:/.nr_spread_over 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[6]:/.nr_running 21655 ± 3% +2554.9% 574933 ± 1% TOTAL sched_debug.cpu#49.ttwu_local 917931 ± 0% -99.2% 7364 ±47% TOTAL sched_debug.cpu#49.sched_goidle 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#49.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#7.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[48]:/.nr_running 963896 ± 5% +662.9% 7353616 ± 1% TOTAL sched_debug.cfs_rq[48]:/.min_vruntime 20962 ± 5% +2638.5% 574061 ± 1% TOTAL sched_debug.cpu#48.ttwu_local 342563 ± 2% -99.0% 3297 ±40% TOTAL sched_debug.cpu#7.sched_goidle 25834 ± 4% +2138.5% 578300 ± 0% TOTAL sched_debug.cpu#7.ttwu_local 4 ±48% +500.0% 28 ± 1% TOTAL sched_debug.cpu#48.cpu_load[3] 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[7]:/.nr_running 4 ±44% +513.0% 28 ± 1% TOTAL sched_debug.cpu#48.cpu_load[0] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#48.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#8.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[47]:/.nr_running 1026500 ± 4% +606.8% 7255532 ± 1% TOTAL sched_debug.cfs_rq[47]:/.min_vruntime 21776 ± 3% +2530.7% 572873 ± 1% TOTAL sched_debug.cpu#47.ttwu_local 940486 ± 1% -99.3% 6905 ±46% TOTAL sched_debug.cpu#47.sched_goidle 375887 ± 2% -98.8% 4502 ±40% TOTAL sched_debug.cpu#8.sched_goidle 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#47.nr_running 46 ±43% -94.0% 2 ±41% TOTAL sched_debug.cfs_rq[46]:/.blocked_load_avg 27667 ± 3% +1978.0% 574926 ± 1% TOTAL sched_debug.cpu#8.ttwu_local 6 ±34% +370.0% 28 ± 1% TOTAL sched_debug.cfs_rq[46]:/.runnable_load_avg 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[46]:/.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[8]:/.nr_running 973194 ± 3% +646.5% 7264433 ± 1% TOTAL sched_debug.cfs_rq[46]:/.min_vruntime 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#9.nr_running 22161 ± 2% +2482.1% 572230 ± 1% TOTAL sched_debug.cpu#46.ttwu_local 4 ±15% +525.0% 30 ± 2% TOTAL sched_debug.cpu#46.cpu_load[4] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#46.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[45]:/.nr_running 359971 ± 2% -99.1% 3085 ±41% TOTAL sched_debug.cpu#9.sched_goidle 1018595 ± 4% +611.1% 7243421 ± 1% TOTAL sched_debug.cfs_rq[45]:/.min_vruntime 27131 ± 2% +2017.8% 574580 ± 1% TOTAL sched_debug.cpu#9.ttwu_local 134404 ±31% +4758.2% 6529609 ± 8% TOTAL sched_debug.cfs_rq[9]:/.MIN_vruntime 22294 ± 4% +2467.1% 572326 ± 1% TOTAL sched_debug.cpu#45.ttwu_local 134404 ±31% +4758.2% 6529609 ± 8% TOTAL sched_debug.cfs_rq[9]:/.max_vruntime 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[9]:/.nr_running 5 ±22% +455.6% 30 ± 2% TOTAL sched_debug.cpu#45.cpu_load[4] 0 ± 0% +Inf% 2 ±18% TOTAL sched_debug.cpu#45.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[44]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#10.nr_running 360383 ± 1% -98.8% 4421 ±45% TOTAL sched_debug.cpu#10.sched_goidle 27338 ± 3% +2000.0% 574125 ± 1% TOTAL sched_debug.cpu#10.ttwu_local 1021430 ± 4% +609.5% 7247197 ± 1% TOTAL sched_debug.cfs_rq[44]:/.min_vruntime 231918 ±24% +2549.3% 6144271 ± 8% TOTAL sched_debug.cfs_rq[10]:/.MIN_vruntime 231918 ±24% +2549.3% 6144271 ± 8% TOTAL sched_debug.cfs_rq[10]:/.max_vruntime 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[10]:/.nr_running 22562 ± 3% +2442.1% 573549 ± 1% TOTAL sched_debug.cpu#44.ttwu_local 5 ±25% +492.0% 29 ± 1% TOTAL sched_debug.cpu#44.cpu_load[4] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#11.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#44.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[43]:/.nr_running 1019742 ± 2% +614.0% 7280699 ± 1% TOTAL sched_debug.cfs_rq[43]:/.min_vruntime 366743 ± 3% -98.8% 4544 ±48% TOTAL sched_debug.cpu#11.sched_goidle 22781 ± 3% +2411.2% 572090 ± 1% TOTAL sched_debug.cpu#43.ttwu_local 27283 ± 5% +1994.9% 571569 ± 1% TOTAL sched_debug.cpu#11.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#43.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[11]:/.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[42]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#12.nr_running 1013625 ± 3% +610.3% 7199592 ± 1% TOTAL sched_debug.cfs_rq[42]:/.min_vruntime 23285 ± 2% +2357.3% 572199 ± 1% TOTAL sched_debug.cpu#42.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#42.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[41]:/.nr_running 363201 ± 5% -99.2% 3010 ±32% TOTAL sched_debug.cpu#12.sched_goidle 1026049 ± 2% +605.1% 7234511 ± 2% TOTAL sched_debug.cfs_rq[41]:/.min_vruntime 26558 ± 1% +2059.8% 573625 ± 1% TOTAL sched_debug.cpu#12.ttwu_local 24712 ± 3% +2225.8% 574772 ± 1% TOTAL sched_debug.cpu#41.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[12]:/.nr_running 908047 ± 0% -99.3% 6771 ±46% TOTAL sched_debug.cpu#41.sched_goidle 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#41.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[40]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#13.nr_running 1045087 ± 1% +592.6% 7238603 ± 1% TOTAL sched_debug.cfs_rq[40]:/.min_vruntime 23905 ± 4% +2300.9% 573931 ± 1% TOTAL sched_debug.cpu#40.ttwu_local 362073 ± 4% -99.0% 3783 ±43% TOTAL sched_debug.cpu#13.sched_goidle 6 ±45% +383.3% 29 ± 0% TOTAL sched_debug.cpu#40.cpu_load[4] 27170 ± 6% +2006.6% 572362 ± 1% TOTAL sched_debug.cpu#13.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#40.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[13]:/.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[39]:/.nr_running 956859 ± 2% +664.8% 7318047 ± 1% TOTAL sched_debug.cfs_rq[39]:/.min_vruntime 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#14.nr_running 22156 ± 6% +2515.2% 579425 ± 1% TOTAL sched_debug.cpu#39.ttwu_local 5 ±20% +400.0% 29 ± 2% TOTAL sched_debug.cpu#39.cpu_load[4] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#39.nr_running 342979 ± 2% -99.1% 3153 ±32% TOTAL sched_debug.cpu#14.sched_goidle 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[38]:/.nr_running 25831 ± 2% +2108.8% 570574 ± 1% TOTAL sched_debug.cpu#14.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[14]:/.nr_running 998185 ± 4% +630.6% 7292251 ± 1% TOTAL sched_debug.cfs_rq[38]:/.min_vruntime 21969 ± 3% +2536.5% 579224 ± 1% TOTAL sched_debug.cpu#38.ttwu_local 0 ± 0% +Inf% 2 ±18% TOTAL sched_debug.cpu#15.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#38.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[37]:/.nr_running 8 ±22% +7077.3% 631 ±47% TOTAL sched_debug.cfs_rq[37]:/.nr_spread_over 976930 ± 2% +653.4% 7359797 ± 1% TOTAL sched_debug.cfs_rq[37]:/.min_vruntime 365758 ± 2% -98.9% 3877 ±47% TOTAL sched_debug.cpu#15.sched_goidle 22211 ± 5% +2503.0% 578152 ± 0% TOTAL sched_debug.cpu#37.ttwu_local 25988 ± 4% +2107.9% 573784 ± 1% TOTAL sched_debug.cpu#15.ttwu_local 4 ±23% +559.1% 29 ± 2% TOTAL sched_debug.cpu#37.cpu_load[4] 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[15]:/.nr_running 5 ±14% +442.3% 28 ± 2% TOTAL sched_debug.cpu#37.cpu_load[3] 5 ± 9% +414.8% 27 ± 2% TOTAL sched_debug.cpu#37.cpu_load[2] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#37.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[36]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#16.nr_running 10 ±17% +6420.8% 691 ±26% TOTAL sched_debug.cfs_rq[36]:/.nr_spread_over 951225 ± 5% +672.6% 7349084 ± 1% TOTAL sched_debug.cfs_rq[36]:/.min_vruntime 22668 ± 5% +2450.8% 578229 ± 1% TOTAL sched_debug.cpu#36.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#36.nr_running 25231 ± 3% +2177.7% 574692 ± 1% TOTAL sched_debug.cpu#16.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[35]:/.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[16]:/.nr_running 969215 ± 3% +658.8% 7354084 ± 2% TOTAL sched_debug.cfs_rq[35]:/.min_vruntime 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#17.nr_running 22948 ± 6% +2424.6% 579345 ± 1% TOTAL sched_debug.cpu#35.ttwu_local 4 ±44% +615.0% 28 ± 3% TOTAL sched_debug.cpu#35.cpu_load[4] 4 ±48% +518.2% 27 ± 4% TOTAL sched_debug.cpu#35.cpu_load[0] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#35.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[34]:/.nr_running 24619 ± 2% +2234.0% 574630 ± 1% TOTAL sched_debug.cpu#17.ttwu_local 985770 ± 4% +646.0% 7353877 ± 1% TOTAL sched_debug.cfs_rq[34]:/.min_vruntime 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[17]:/.nr_running 23297 ± 6% +2389.1% 579887 ± 1% TOTAL sched_debug.cpu#34.ttwu_local 4 ±42% +534.8% 29 ± 2% TOTAL sched_debug.cpu#34.cpu_load[4] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#18.nr_running 5 ±41% +429.6% 28 ± 3% TOTAL sched_debug.cpu#34.cpu_load[3] 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#34.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[33]:/.nr_running 11 ±39% +7127.3% 795 ±29% TOTAL sched_debug.cfs_rq[33]:/.nr_spread_over 952580 ± 1% +670.8% 7342099 ± 1% TOTAL sched_debug.cfs_rq[33]:/.min_vruntime 24246 ± 4% +2265.3% 573496 ± 1% TOTAL sched_debug.cpu#18.ttwu_local 24645 ± 6% +2249.7% 579112 ± 1% TOTAL sched_debug.cpu#33.ttwu_local 933626 ± 1% -99.3% 6613 ±47% TOTAL sched_debug.cpu#33.sched_goidle 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[18]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#33.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[32]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#19.nr_running 950196 ± 2% +670.3% 7319391 ± 2% TOTAL sched_debug.cfs_rq[32]:/.min_vruntime 25033 ± 5% +2211.6% 578682 ± 1% TOTAL sched_debug.cpu#32.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#32.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[31]:/.nr_running 23931 ± 3% +2304.2% 575367 ± 1% TOTAL sched_debug.cpu#19.ttwu_local 23792 ± 1% +2309.0% 573173 ± 1% TOTAL sched_debug.cpu#31.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[19]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#31.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[30]:/.nr_running 45.19 ± 0% -100.0% 0.00 ± 0% TOTAL perf-profile.cpu-cycles.__crc32c_le.chksum_update.crypto_shash_update.crc32c.sctp_csum_update 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#20.nr_running 23990 ± 2% +2299.0% 575544 ± 1% TOTAL sched_debug.cpu#30.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#30.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[29]:/.nr_running 333744 ± 2% -99.1% 3032 ±30% TOTAL sched_debug.cpu#20.sched_goidle 23769 ± 2% +2321.8% 575648 ± 1% TOTAL sched_debug.cpu#20.ttwu_local 23963 ± 2% +2289.1% 572507 ± 1% TOTAL sched_debug.cpu#29.ttwu_local 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[20]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#29.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[28]:/.nr_running 16 ±16% +14806.2% 2385 ±48% TOTAL sched_debug.cfs_rq[28]:/.nr_spread_over 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#21.nr_running 23955 ± 2% +2294.9% 573707 ± 1% TOTAL sched_debug.cpu#28.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#28.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[27]:/.nr_running 200672 ±41% +3130.2% 6482112 ±11% TOTAL sched_debug.cfs_rq[27]:/.max_vruntime 333344 ± 3% -99.4% 2160 ±44% TOTAL sched_debug.cpu#21.sched_goidle 200672 ±41% +3130.2% 6482112 ±11% TOTAL sched_debug.cfs_rq[27]:/.MIN_vruntime 24388 ± 2% +2254.3% 574189 ± 1% TOTAL sched_debug.cpu#27.ttwu_local 23766 ± 2% +2316.8% 574378 ± 1% TOTAL sched_debug.cpu#21.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#27.nr_running 16 ±23% +8230.9% 1349 ±31% TOTAL sched_debug.cfs_rq[21]:/.nr_spread_over 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[21]:/.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[26]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#22.nr_running 24877 ± 2% +2200.2% 572209 ± 1% TOTAL sched_debug.cpu#26.ttwu_local 333909 ± 4% -98.5% 4876 ±31% TOTAL sched_debug.cpu#26.sched_goidle 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#26.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[25]:/.nr_running 21 ±22% +8784.9% 1883 ±35% TOTAL sched_debug.cfs_rq[25]:/.nr_spread_over 24697 ± 4% +2214.3% 571582 ± 1% TOTAL sched_debug.cpu#25.ttwu_local 24175 ± 5% +2283.3% 576162 ± 1% TOTAL sched_debug.cpu#22.ttwu_local 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#25.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[22]:/.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[24]:/.nr_running 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#23.nr_running 25076 ± 2% +2194.8% 575457 ± 1% TOTAL sched_debug.cpu#24.ttwu_local 336969 ± 2% -99.1% 3193 ±49% TOTAL sched_debug.cpu#24.sched_goidle 0 ± 0% +Inf% 2 ± 0% TOTAL sched_debug.cpu#24.nr_running 0 ± 0% +Inf% 1 ± 0% TOTAL sched_debug.cfs_rq[23]:/.nr_running 420529 ±14% -80.8% 80883 ± 6% TOTAL sched_debug.cpu#24.avg_idle 35 ± 1% +394.3% 174 ± 0% TOTAL vmstat.procs.r 320932 ±15% -77.7% 71461 ± 8% TOTAL sched_debug.cpu#26.avg_idle 295750 ±21% -76.1% 70793 ±17% TOTAL sched_debug.cpu#2.avg_idle 623387 ± 0% -79.5% 127567 ± 1% TOTAL softirqs.SCHED 6 ±42% +320.6% 28 ± 1% TOTAL sched_debug.cpu#40.cpu_load[3] 6 ±33% +376.7% 28 ± 3% TOTAL sched_debug.cpu#51.cpu_load[3] 6 ±27% +358.1% 28 ± 4% TOTAL sched_debug.cpu#62.cpu_load[3] 6 ±27% +358.1% 28 ± 2% TOTAL sched_debug.cpu#59.cpu_load[4] 5 ±20% +403.4% 29 ± 2% TOTAL sched_debug.cpu#44.cpu_load[3] 5 ±25% +437.0% 29 ± 2% TOTAL sched_debug.cpu#38.cpu_load[4] 6 ±21% +361.3% 28 ± 3% TOTAL sched_debug.cpu#46.cpu_load[1] 6 ±10% +380.0% 28 ± 4% TOTAL sched_debug.cpu#46.cpu_load[2] 6 ±18% +364.5% 28 ± 2% TOTAL sched_debug.cpu#44.cpu_load[2] 6 ±32% +364.5% 28 ± 1% TOTAL sched_debug.cpu#36.cpu_load[4] 6 ±21% +358.1% 28 ± 2% TOTAL sched_debug.cpu#44.cpu_load[1] 6 ±29% +354.8% 28 ± 3% TOTAL sched_debug.cpu#44.cpu_load[0] 6 ±44% +361.3% 28 ± 1% TOTAL sched_debug.cpu#32.cpu_load[4] 6 ±33% +373.3% 28 ± 2% TOTAL sched_debug.cpu#55.cpu_load[3] 6 ±28% +323.5% 28 ± 4% TOTAL sched_debug.cpu#46.cpu_load[0] 5 ± 6% +406.9% 29 ± 3% TOTAL sched_debug.cpu#46.cpu_load[3] 340647 ±14% -77.1% 77839 ±14% TOTAL sched_debug.cpu#25.avg_idle 6 ±18% +376.7% 28 ± 2% TOTAL sched_debug.cpu#38.cpu_load[3] 4 ±37% +508.7% 28 ± 2% TOTAL sched_debug.cpu#35.cpu_load[3] 6 ±29% +314.7% 28 ± 4% TOTAL sched_debug.cpu#56.cpu_load[0] 6 ±15% +346.9% 28 ± 2% TOTAL sched_debug.cpu#63.cpu_load[3] 7 ±47% +305.7% 28 ± 1% TOTAL sched_debug.cpu#32.cpu_load[3] 6 ±17% +317.6% 28 ± 3% TOTAL sched_debug.cpu#52.cpu_load[4] 6 ±15% +343.8% 28 ± 3% TOTAL sched_debug.cpu#56.cpu_load[2] 6 ±37% +337.5% 28 ± 3% TOTAL sched_debug.cpu#55.cpu_load[2] 6 ±23% +343.8% 28 ± 1% TOTAL sched_debug.cpu#39.cpu_load[3] 6 ±29% +351.6% 28 ± 5% TOTAL sched_debug.cpu#34.cpu_load[1] 6 ±31% +354.8% 28 ± 4% TOTAL sched_debug.cpu#34.cpu_load[2] 6 ±42% +358.1% 28 ± 5% TOTAL sched_debug.cpu#53.cpu_load[1] 6 ±40% +373.3% 28 ± 3% TOTAL sched_debug.cpu#53.cpu_load[2] 6 ±28% +327.3% 28 ± 1% TOTAL sched_debug.cpu#36.cpu_load[3] 6 ±20% +327.3% 28 ± 2% TOTAL sched_debug.cpu#56.cpu_load[1] 7 ±29% +297.1% 27 ± 1% TOTAL sched_debug.cpu#50.cpu_load[2] 7 ±44% +265.8% 27 ± 1% TOTAL sched_debug.cpu#32.cpu_load[2] 317802 ± 9% -77.7% 70785 ± 9% TOTAL sched_debug.cpu#4.avg_idle 346844 ±10% -77.6% 77616 ± 8% TOTAL sched_debug.cpu#16.avg_idle 4 ±40% +475.0% 27 ± 2% TOTAL sched_debug.cpu#35.cpu_load[1] 5 ±35% +452.0% 27 ± 2% TOTAL sched_debug.cfs_rq[48]:/.runnable_load_avg 4 ±40% +470.8% 27 ± 2% TOTAL sched_debug.cpu#35.cpu_load[2] 5 ±18% +385.7% 27 ± 4% TOTAL sched_debug.cpu#37.cpu_load[0] 5 ± 6% +369.0% 27 ± 2% TOTAL sched_debug.cpu#37.cpu_load[1] 366939 ±13% -78.7% 78115 ± 9% TOTAL sched_debug.cpu#30.avg_idle 1023 ±37% +323.5% 4333 ± 2% TOTAL sched_debug.cpu#48.curr->pid 327585 ± 9% -77.2% 74734 ±11% TOTAL sched_debug.cpu#17.avg_idle 12595 ± 0% -76.7% 2929 ± 0% TOTAL uptime.idle 338343 ±14% -78.4% 73057 ± 5% TOTAL sched_debug.cpu#31.avg_idle 6 ±37% +353.1% 29 ± 4% TOTAL sched_debug.cpu#58.cpu_load[4] 7 ±23% +322.9% 29 ± 3% TOTAL sched_debug.cpu#47.cpu_load[4] 7 ±28% +316.7% 30 ± 2% TOTAL sched_debug.cpu#42.cpu_load[4] 316565 ±15% -74.7% 80204 ±11% TOTAL sched_debug.cpu#23.avg_idle 295009 ±22% -75.0% 73750 ± 9% TOTAL sched_debug.cpu#22.avg_idle 19299589 ± 1% +316.6% 80411460 ± 1% TOTAL proc-vmstat.pgalloc_dma32 6 ±26% +311.8% 28 ± 4% TOTAL sched_debug.cpu#62.cpu_load[2] 6 ±21% +393.3% 29 ± 2% TOTAL sched_debug.cpu#45.cpu_load[3] 7 ±18% +311.4% 28 ± 2% TOTAL sched_debug.cpu#45.cpu_load[2] 7 ±10% +286.8% 29 ± 1% TOTAL sched_debug.cfs_rq[45]:/.runnable_load_avg 8 ±27% +234.9% 28 ± 2% TOTAL sched_debug.cpu#42.cpu_load[1] 6 ±19% +323.5% 28 ± 4% TOTAL sched_debug.cpu#60.cpu_load[4] 8 ±26% +265.0% 29 ± 2% TOTAL sched_debug.cpu#42.cpu_load[3] 7 ±13% +297.2% 28 ± 5% TOTAL sched_debug.cfs_rq[38]:/.runnable_load_avg 8 ±27% +234.9% 28 ± 2% TOTAL sched_debug.cpu#42.cpu_load[2] 6 ±40% +364.5% 28 ± 4% TOTAL sched_debug.cpu#54.cpu_load[4] 306463 ±16% -72.5% 84417 ±11% TOTAL sched_debug.cpu#7.avg_idle 294085 ±13% -74.4% 75243 ±11% TOTAL sched_debug.cpu#58.avg_idle 295178 ± 8% -74.8% 74330 ±13% TOTAL sched_debug.cpu#19.avg_idle 310735 ±12% -74.4% 79628 ±23% TOTAL sched_debug.cpu#12.avg_idle 7 ±21% +275.7% 27 ± 3% TOTAL sched_debug.cpu#59.cpu_load[3] 7 ±30% +277.8% 27 ± 5% TOTAL sched_debug.cpu#62.cpu_load[1] 7 ±40% +300.0% 28 ± 0% TOTAL sched_debug.cpu#40.cpu_load[2] 6 ±34% +314.7% 28 ± 2% TOTAL sched_debug.cpu#51.cpu_load[2] 7 ±49% +280.6% 27 ± 4% TOTAL sched_debug.cfs_rq[40]:/.runnable_load_avg 7 ±34% +297.1% 27 ± 4% TOTAL sched_debug.cpu#55.cpu_load[1] 8 ±20% +245.0% 27 ± 1% TOTAL sched_debug.cpu#36.cpu_load[1] 7 ±23% +286.1% 27 ± 1% TOTAL sched_debug.cpu#36.cpu_load[2] 7 ±16% +288.9% 28 ± 3% TOTAL sched_debug.cpu#52.cpu_load[3] 7 ±13% +278.4% 28 ± 2% TOTAL sched_debug.cpu#63.cpu_load[2] 7 ±23% +288.9% 28 ± 2% TOTAL sched_debug.cpu#39.cpu_load[2] 7 ±28% +271.1% 28 ± 1% TOTAL sched_debug.cpu#39.cpu_load[1] 7 ±18% +288.9% 28 ± 5% TOTAL sched_debug.cfs_rq[63]:/.runnable_load_avg 6 ±28% +302.9% 27 ± 2% TOTAL sched_debug.cfs_rq[37]:/.runnable_load_avg 7 ±30% +255.3% 27 ± 6% TOTAL sched_debug.cpu#38.cpu_load[0] 7 ±25% +283.3% 27 ± 4% TOTAL sched_debug.cpu#38.cpu_load[1] 6 ±22% +324.2% 28 ± 3% TOTAL sched_debug.cpu#38.cpu_load[2] 6 ±25% +308.8% 27 ± 1% TOTAL sched_debug.cfs_rq[56]:/.runnable_load_avg 7 ±38% +267.6% 27 ± 5% TOTAL sched_debug.cpu#62.cpu_load[0] 317966 ±13% -74.2% 82042 ± 9% TOTAL sched_debug.cpu#27.avg_idle 296746 ±16% -74.7% 75078 ± 9% TOTAL sched_debug.cpu#14.avg_idle 298670 ±15% -73.3% 79831 ±22% TOTAL sched_debug.cpu#6.avg_idle 6 ±32% +302.9% 27 ± 5% TOTAL sched_debug.cpu#34.cpu_load[0] 7 ±10% +260.5% 27 ± 3% TOTAL sched_debug.cpu#52.cpu_load[2] 6 ±23% +297.1% 27 ± 0% TOTAL sched_debug.cfs_rq[61]:/.runnable_load_avg 8 ±39% +216.3% 27 ± 1% TOTAL sched_debug.cpu#32.cpu_load[1] 7 ±30% +252.6% 26 ± 2% TOTAL sched_debug.cpu#50.cpu_load[0] 6 ±38% +302.9% 27 ± 1% TOTAL sched_debug.cpu#40.cpu_load[1] 7 ±16% +283.3% 27 ± 2% TOTAL sched_debug.cpu#63.cpu_load[1] 7 ±27% +255.3% 27 ± 2% TOTAL sched_debug.cpu#50.cpu_load[1] 6 ±44% +335.5% 27 ± 3% TOTAL sched_debug.cfs_rq[35]:/.runnable_load_avg 6 ±39% +300.0% 27 ± 2% TOTAL sched_debug.cpu#40.cpu_load[0] 294207 ±16% -73.9% 76789 ±18% TOTAL sched_debug.cpu#20.avg_idle 1454 ±47% +193.6% 4270 ± 1% TOTAL sched_debug.cpu#32.curr->pid 274384 ±28% -64.9% 96227 ±33% TOTAL sched_debug.cpu#33.avg_idle 287321 ± 8% -73.1% 77157 ±10% TOTAL sched_debug.cpu#3.avg_idle 7 ±10% +255.3% 27 ± 4% TOTAL sched_debug.cpu#52.cpu_load[1] 1284 ±29% +228.6% 4222 ± 0% TOTAL sched_debug.cpu#43.curr->pid 349008 ±15% -74.2% 90084 ± 6% TOTAL sched_debug.cpu#21.avg_idle 9 ±11% +230.6% 32 ± 4% TOTAL sched_debug.cpu#49.cpu_load[4] 7 ±30% +281.6% 29 ± 2% TOTAL sched_debug.cpu#43.cpu_load[2] 7 ±29% +297.3% 29 ± 2% TOTAL sched_debug.cpu#43.cpu_load[3] 6 ±38% +323.5% 28 ± 4% TOTAL sched_debug.cpu#54.cpu_load[3] 8 ±29% +223.3% 27 ±10% TOTAL sched_debug.cfs_rq[58]:/.runnable_load_avg 7 ±18% +271.8% 29 ± 2% TOTAL sched_debug.cpu#47.cpu_load[3] 7 ±31% +322.9% 29 ± 2% TOTAL sched_debug.cpu#43.cpu_load[4] 7 ±32% +283.8% 28 ± 4% TOTAL sched_debug.cpu#58.cpu_load[3] 9 ±25% +222.2% 29 ± 3% TOTAL sched_debug.cpu#42.cpu_load[0] 291808 ±12% -72.2% 81207 ±12% TOTAL sched_debug.cpu#0.avg_idle 12900 ± 6% +248.9% 45016 ± 0% TOTAL sched_debug.cfs_rq[61]:/.avg->runnable_avg_sum 282 ± 6% +247.8% 982 ± 0% TOTAL sched_debug.cfs_rq[61]:/.tg_runnable_contrib 307360 ±20% -75.2% 76165 ± 7% TOTAL sched_debug.cpu#13.avg_idle 1213 ±36% +252.3% 4273 ± 0% TOTAL sched_debug.cpu#35.curr->pid 2108032 ± 5% -71.2% 607806 ± 1% TOTAL sched_debug.cpu#34.sched_count 2130952 ± 9% -71.4% 609735 ± 1% TOTAL sched_debug.cpu#33.sched_count 7 ±39% +268.4% 28 ± 3% TOTAL sched_debug.cpu#54.cpu_load[0] 7 ±39% +278.4% 28 ± 3% TOTAL sched_debug.cfs_rq[53]:/.runnable_load_avg 8 ±23% +223.3% 27 ± 2% TOTAL sched_debug.cpu#36.cpu_load[0] 7 ±20% +286.1% 27 ± 4% TOTAL sched_debug.cpu#60.cpu_load[3] 7 ±21% +265.8% 27 ± 4% TOTAL sched_debug.cfs_rq[50]:/.runnable_load_avg 7 ±31% +276.3% 28 ± 2% TOTAL sched_debug.cpu#45.cpu_load[0] 8 ±24% +226.2% 27 ± 2% TOTAL sched_debug.cfs_rq[55]:/.runnable_load_avg 6 ±36% +320.6% 28 ± 4% TOTAL sched_debug.cfs_rq[44]:/.runnable_load_avg 7 ±26% +281.1% 28 ± 2% TOTAL sched_debug.cpu#51.cpu_load[1] 7 ±27% +265.8% 27 ± 4% TOTAL sched_debug.cpu#51.cpu_load[0] 8 ±27% +233.3% 28 ± 2% TOTAL sched_debug.cpu#39.cpu_load[0] 7 ±21% +276.3% 28 ± 2% TOTAL sched_debug.cpu#45.cpu_load[1] 293 ± 8% +234.7% 980 ± 0% TOTAL sched_debug.cfs_rq[56]:/.tg_runnable_contrib 2074224 ± 3% -70.6% 609675 ± 2% TOTAL sched_debug.cpu#63.sched_count 2127419 ± 6% -71.7% 601604 ± 1% TOTAL sched_debug.cpu#46.sched_count 2115949 ± 3% -65.2% 737051 ±35% TOTAL sched_debug.cpu#36.sched_count 13414 ± 8% +235.7% 45027 ± 0% TOTAL sched_debug.cfs_rq[56]:/.avg->runnable_avg_sum 267808 ±13% -72.1% 74782 ±10% TOTAL sched_debug.cpu#5.avg_idle 2069493 ± 8% -68.4% 654987 ±18% TOTAL sched_debug.cpu#45.sched_count 14044 ±11% +220.7% 45039 ± 0% TOTAL sched_debug.cfs_rq[55]:/.avg->runnable_avg_sum 2053686 ± 4% -70.7% 602700 ± 2% TOTAL sched_debug.cpu#52.sched_count 289450 ±13% -68.8% 90401 ±15% TOTAL sched_debug.cpu#1.avg_idle 307 ±11% +219.5% 980 ± 0% TOTAL sched_debug.cfs_rq[55]:/.tg_runnable_contrib 2002864 ± 8% -69.9% 602844 ± 1% TOTAL sched_debug.cpu#40.sched_count 2035365 ± 2% -70.3% 603651 ± 1% TOTAL sched_debug.cpu#47.sched_count 9 ±24% +197.8% 26 ± 1% TOTAL sched_debug.cfs_rq[59]:/.runnable_load_avg 8 ±24% +226.8% 26 ± 6% TOTAL sched_debug.cpu#60.cpu_load[0] 8 ±23% +235.0% 26 ± 4% TOTAL sched_debug.cpu#59.cpu_load[1] 7 ±19% +252.6% 26 ± 4% TOTAL sched_debug.cpu#60.cpu_load[1] 8 ±23% +234.1% 27 ± 3% TOTAL sched_debug.cpu#59.cpu_load[2] 8 ±36% +209.1% 27 ± 2% TOTAL sched_debug.cfs_rq[32]:/.runnable_load_avg 7 ±20% +253.8% 27 ± 2% TOTAL sched_debug.cpu#63.cpu_load[0] 8 ±16% +206.8% 27 ± 4% TOTAL sched_debug.cfs_rq[52]:/.runnable_load_avg 7 ±19% +257.9% 27 ± 4% TOTAL sched_debug.cpu#60.cpu_load[2] 2008711 ± 1% -70.0% 602980 ± 1% TOTAL sched_debug.cpu#61.nr_switches 2100707 ± 7% -71.1% 606142 ± 2% TOTAL sched_debug.cpu#53.sched_count 2046800 ± 1% -65.3% 710307 ±28% TOTAL sched_debug.cpu#60.sched_count 2091856 ± 3% -70.6% 614687 ± 3% TOTAL sched_debug.cpu#38.sched_count 2062814 ± 3% -70.7% 604560 ± 2% TOTAL sched_debug.cpu#54.sched_count 13554 ±11% +232.3% 45048 ± 0% TOTAL sched_debug.cfs_rq[35]:/.avg->runnable_avg_sum 296 ±11% +231.2% 982 ± 0% TOTAL sched_debug.cfs_rq[35]:/.tg_runnable_contrib 1474 ±29% +188.2% 4249 ± 0% TOTAL sched_debug.cpu#62.curr->pid 2049101 ± 6% -70.4% 607303 ± 2% TOTAL sched_debug.cpu#41.sched_count 2167538 ± 9% -71.9% 609753 ± 2% TOTAL sched_debug.cpu#39.sched_count 2036680 ± 3% -70.3% 605379 ± 1% TOTAL sched_debug.cpu#51.sched_count 2017924 ± 1% -70.0% 604898 ± 0% TOTAL sched_debug.cpu#59.sched_count 14509 ±13% +210.5% 45046 ± 0% TOTAL sched_debug.cfs_rq[63]:/.avg->runnable_avg_sum 2076058 ± 5% -69.2% 640162 ±10% TOTAL sched_debug.cpu#37.sched_count 1225 ±36% +253.2% 4329 ± 3% TOTAL sched_debug.cpu#53.curr->pid 1148 ±27% +268.3% 4230 ± 1% TOTAL sched_debug.cpu#61.curr->pid 14570 ±13% +209.0% 45024 ± 0% TOTAL sched_debug.cfs_rq[34]:/.avg->runnable_avg_sum 2020274 ± 0% -69.8% 610215 ± 2% TOTAL sched_debug.cpu#62.nr_switches 319 ±13% +208.0% 982 ± 0% TOTAL sched_debug.cfs_rq[34]:/.tg_runnable_contrib 317 ±13% +208.1% 978 ± 0% TOTAL sched_debug.cfs_rq[63]:/.tg_runnable_contrib 2042951 ± 1% -69.7% 618701 ± 2% TOTAL sched_debug.cpu#62.sched_count 2058101 ± 7% -70.8% 600726 ± 1% TOTAL sched_debug.cpu#44.sched_count 1984924 ± 1% -69.5% 604880 ± 0% TOTAL sched_debug.cpu#59.nr_switches 1971315 ± 1% -69.3% 604542 ± 2% TOTAL sched_debug.cpu#54.nr_switches 2014162 ± 6% -65.2% 700760 ±28% TOTAL sched_debug.cpu#42.sched_count 1953776 ± 2% -68.8% 609656 ± 2% TOTAL sched_debug.cpu#63.nr_switches 2055362 ± 7% -65.1% 717767 ±33% TOTAL sched_debug.cpu#43.sched_count 1231 ±29% +249.4% 4303 ± 2% TOTAL sched_debug.cpu#50.curr->pid 313 ±10% +213.7% 982 ± 0% TOTAL sched_debug.cfs_rq[62]:/.tg_runnable_contrib 7 ±13% +255.3% 27 ± 6% TOTAL sched_debug.cpu#52.cpu_load[0] 8 ±31% +202.3% 26 ± 3% TOTAL sched_debug.cpu#32.cpu_load[0] 1991792 ± 1% -69.3% 611077 ± 1% TOTAL sched_debug.cpu#60.nr_switches 14342 ±10% +214.0% 45031 ± 0% TOTAL sched_debug.cfs_rq[62]:/.avg->runnable_avg_sum 1942697 ± 1% -69.0% 602682 ± 2% TOTAL sched_debug.cpu#52.nr_switches 1950644 ± 1% -69.2% 601592 ± 1% TOTAL sched_debug.cpu#46.nr_switches 1992472 ± 1% -61.6% 765257 ±25% TOTAL sched_debug.cpu#50.sched_count 8 ±20% +241.5% 28 ± 4% TOTAL sched_debug.cpu#58.cpu_load[1] 9 ±12% +215.6% 28 ± 2% TOTAL sched_debug.cfs_rq[39]:/.runnable_load_avg 7 ±31% +283.8% 28 ± 2% TOTAL sched_debug.cpu#54.cpu_load[2] 8 ±22% +238.1% 28 ± 2% TOTAL sched_debug.cpu#47.cpu_load[2] 8 ±26% +250.0% 28 ± 4% TOTAL sched_debug.cpu#58.cpu_load[2] 13572 ±12% +232.0% 45056 ± 0% TOTAL sched_debug.cfs_rq[37]:/.avg->runnable_avg_sum 2047201 ± 6% -67.2% 670954 ±18% TOTAL sched_debug.cpu#35.sched_count 297 ±12% +230.8% 983 ± 0% TOTAL sched_debug.cfs_rq[37]:/.tg_runnable_contrib 1955174 ± 2% -69.1% 603605 ± 1% TOTAL sched_debug.cpu#55.nr_switches 1971669 ± 5% -69.1% 608414 ± 1% TOTAL sched_debug.cpu#49.sched_count 1939844 ± 1% -68.8% 605845 ± 1% TOTAL sched_debug.cpu#36.nr_switches 1949899 ± 1% -68.9% 606124 ± 2% TOTAL sched_debug.cpu#53.nr_switches 315 ±12% +211.7% 984 ± 0% TOTAL sched_debug.cfs_rq[51]:/.tg_runnable_contrib 12 ±27% +166.7% 32 ± 6% TOTAL sched_debug.cpu#49.cpu_load[2] 11 ±21% +184.2% 32 ± 5% TOTAL sched_debug.cpu#49.cpu_load[3] 14438 ±12% +212.0% 45043 ± 0% TOTAL sched_debug.cfs_rq[51]:/.avg->runnable_avg_sum 2001341 ± 4% -66.4% 671981 ±17% TOTAL sched_debug.cpu#57.sched_count 303710 ±17% -70.0% 91263 ±13% TOTAL sched_debug.cpu#9.avg_idle 1900887 ± 1% -68.4% 600434 ± 1% TOTAL sched_debug.cpu#45.nr_switches 14272 ±12% +215.8% 45065 ± 0% TOTAL sched_debug.cfs_rq[59]:/.avg->runnable_avg_sum 1937164 ± 0% -68.5% 609385 ± 1% TOTAL sched_debug.cpu#37.nr_switches 1974368 ± 1% -69.1% 609730 ± 2% TOTAL sched_debug.cpu#39.nr_switches 1973848 ± 2% -62.6% 738151 ±35% TOTAL sched_debug.cpu#58.sched_count 1912735 ± 1% -68.5% 603190 ± 2% TOTAL sched_debug.cpu#50.nr_switches 1931643 ± 2% -68.6% 607250 ± 2% TOTAL sched_debug.cpu#58.nr_switches 1377 ±14% +216.7% 4363 ± 1% TOTAL sched_debug.cpu#49.curr->pid 312 ±12% +214.1% 981 ± 0% TOTAL sched_debug.cfs_rq[59]:/.tg_runnable_contrib 1944033 ± 1% -68.9% 605363 ± 1% TOTAL sched_debug.cpu#51.nr_switches 1915699 ± 1% -68.1% 610629 ± 1% TOTAL sched_debug.cpu#35.nr_switches 15047 ±12% +199.1% 45011 ± 0% TOTAL sched_debug.cfs_rq[32]:/.avg->runnable_avg_sum 46418 ± 2% +213.2% 145374 ± 0% TOTAL sched_debug.cfs_rq[62]:/.exec_clock 1924241 ± 2% -68.3% 609789 ± 2% TOTAL sched_debug.cpu#38.nr_switches 1912400 ± 1% -68.4% 603629 ± 1% TOTAL sched_debug.cpu#47.nr_switches 329 ±12% +197.8% 979 ± 0% TOTAL sched_debug.cfs_rq[32]:/.tg_runnable_contrib 1895051 ± 2% -68.3% 600708 ± 1% TOTAL sched_debug.cpu#44.nr_switches 13983 ±14% +222.1% 45044 ± 0% TOTAL sched_debug.cfs_rq[44]:/.avg->runnable_avg_sum 1901704 ± 1% -67.9% 609715 ± 1% TOTAL sched_debug.cpu#33.nr_switches 14514 ± 2% +214.1% 45583 ± 0% TOTAL sched_debug.cfs_rq[33]:/.avg->runnable_avg_sum 306 ±14% +221.0% 982 ± 0% TOTAL sched_debug.cfs_rq[44]:/.tg_runnable_contrib 1926663 ± 0% -68.2% 613403 ± 2% TOTAL sched_debug.cpu#48.sched_count 317 ± 2% +212.6% 993 ± 0% TOTAL sched_debug.cfs_rq[33]:/.tg_runnable_contrib 1203 ±34% +258.3% 4310 ± 1% TOTAL sched_debug.cpu#56.curr->pid 46837 ± 2% +210.4% 145375 ± 0% TOTAL sched_debug.cfs_rq[60]:/.exec_clock 1874381 ± 1% -67.9% 600905 ± 1% TOTAL sched_debug.cpu#42.nr_switches 8 ±16% +238.1% 28 ± 3% TOTAL sched_debug.cfs_rq[47]:/.runnable_load_avg 9 ±12% +200.0% 27 ± 4% TOTAL sched_debug.cpu#58.cpu_load[0] 9 ±24% +213.3% 28 ± 2% TOTAL sched_debug.cpu#47.cpu_load[1] 9 ±28% +200.0% 28 ± 1% TOTAL sched_debug.cpu#47.cpu_load[0] 8 ±34% +239.0% 27 ± 4% TOTAL sched_debug.cpu#55.cpu_load[0] 8 ±15% +222.7% 28 ± 3% TOTAL sched_debug.cfs_rq[36]:/.runnable_load_avg 8 ±23% +246.3% 28 ± 3% TOTAL sched_debug.cpu#43.cpu_load[1] 8 ±22% +250.0% 28 ± 2% TOTAL sched_debug.cfs_rq[43]:/.runnable_load_avg 9 ±37% +213.3% 28 ± 4% TOTAL sched_debug.cfs_rq[34]:/.runnable_load_avg 7 ±32% +268.4% 28 ± 3% TOTAL sched_debug.cpu#54.cpu_load[1] 9 ±37% +208.9% 27 ± 4% TOTAL sched_debug.cfs_rq[51]:/.runnable_load_avg 1916043 ± 2% -68.3% 607993 ± 2% TOTAL sched_debug.cpu#57.nr_switches 265392 ±17% -64.3% 94795 ±22% TOTAL sched_debug.cpu#43.avg_idle 1889707 ± 1% -68.3% 599744 ± 1% TOTAL sched_debug.cpu#43.nr_switches 9 ±29% +227.1% 31 ± 3% TOTAL sched_debug.cpu#33.cpu_load[4] 39 ±46% +117.4% 84 ±32% TOTAL sched_debug.cfs_rq[17]:/.tg_load_contrib 1867119 ± 0% -67.6% 605812 ± 1% TOTAL sched_debug.cpu#49.nr_switches 1889889 ± 1% -67.8% 607783 ± 1% TOTAL sched_debug.cpu#34.nr_switches 47035 ± 4% +209.1% 145403 ± 0% TOTAL sched_debug.cfs_rq[61]:/.exec_clock 1860154 ± 1% -67.6% 602684 ± 2% TOTAL sched_debug.cpu#48.nr_switches 325 ±10% +202.2% 982 ± 0% TOTAL sched_debug.cfs_rq[39]:/.tg_runnable_contrib 66798546 ± 0% +206.8% 2.049e+08 ± 0% TOTAL softirqs.NET_RX 13894 ±13% +224.1% 45029 ± 0% TOTAL sched_debug.cfs_rq[48]:/.avg->runnable_avg_sum 14866 ±10% +202.7% 44994 ± 0% TOTAL sched_debug.cfs_rq[39]:/.avg->runnable_avg_sum 14874 ±11% +207.3% 45713 ± 0% TOTAL sched_debug.cfs_rq[57]:/.avg->runnable_avg_sum 1893238 ± 1% -67.9% 608368 ± 1% TOTAL sched_debug.cpu#56.sched_count 304 ±13% +222.6% 981 ± 0% TOTAL sched_debug.cfs_rq[48]:/.tg_runnable_contrib 1850633 ± 0% -67.2% 606443 ± 2% TOTAL sched_debug.cpu#41.nr_switches 273840 ±22% -69.2% 84378 ± 8% TOTAL sched_debug.cpu#60.avg_idle 325 ±12% +206.1% 996 ± 0% TOTAL sched_debug.cfs_rq[57]:/.tg_runnable_contrib 14957 ±12% +201.5% 45098 ± 0% TOTAL sched_debug.cfs_rq[58]:/.avg->runnable_avg_sum 47628 ± 2% +205.5% 145491 ± 0% TOTAL sched_debug.cfs_rq[51]:/.exec_clock 271141 ±15% -64.6% 96084 ±33% TOTAL sched_debug.cpu#47.avg_idle 47097 ± 2% +208.9% 145500 ± 0% TOTAL sched_debug.cfs_rq[59]:/.exec_clock 1395 ±13% +210.6% 4334 ± 2% TOTAL sched_debug.cpu#55.curr->pid 15193 ±10% +196.8% 45096 ± 0% TOTAL sched_debug.cfs_rq[50]:/.avg->runnable_avg_sum 326 ±12% +200.6% 981 ± 0% TOTAL sched_debug.cfs_rq[58]:/.tg_runnable_contrib 315 ± 8% +211.4% 982 ± 0% TOTAL sched_debug.cfs_rq[38]:/.tg_runnable_contrib 14456 ± 8% +211.3% 45004 ± 0% TOTAL sched_debug.cfs_rq[38]:/.avg->runnable_avg_sum 47896 ± 3% +203.5% 145372 ± 0% TOTAL sched_debug.cfs_rq[54]:/.exec_clock 332 ±10% +195.7% 984 ± 0% TOTAL sched_debug.cfs_rq[50]:/.tg_runnable_contrib 1865184 ± 2% -67.4% 608353 ± 1% TOTAL sched_debug.cpu#56.nr_switches 47722 ± 2% +204.7% 145411 ± 0% TOTAL sched_debug.cfs_rq[55]:/.exec_clock 47334 ± 2% +207.3% 145475 ± 0% TOTAL sched_debug.cfs_rq[52]:/.exec_clock 15320 ± 5% +193.8% 45011 ± 0% TOTAL sched_debug.cfs_rq[46]:/.avg->runnable_avg_sum 48192 ± 1% +201.6% 145362 ± 0% TOTAL sched_debug.cfs_rq[63]:/.exec_clock 48037 ± 1% +202.7% 145411 ± 0% TOTAL sched_debug.cfs_rq[32]:/.exec_clock 1497 ±19% +193.5% 4395 ± 5% TOTAL sched_debug.cpu#52.curr->pid 1846875 ± 0% -67.1% 607481 ± 1% TOTAL sched_debug.cpu#32.nr_switches 287922 ±23% -70.6% 84722 ±16% TOTAL sched_debug.cpu#11.avg_idle 48162 ± 3% +202.0% 145448 ± 0% TOTAL sched_debug.cfs_rq[36]:/.exec_clock 335 ± 5% +192.7% 982 ± 0% TOTAL sched_debug.cfs_rq[46]:/.tg_runnable_contrib 47591 ± 2% +205.6% 145417 ± 0% TOTAL sched_debug.cfs_rq[53]:/.exec_clock 270817 ±22% -66.3% 91254 ±13% TOTAL sched_debug.cpu#63.avg_idle 15169 ±10% +196.9% 45031 ± 0% TOTAL sched_debug.cfs_rq[36]:/.avg->runnable_avg_sum 332 ±10% +196.3% 984 ± 0% TOTAL sched_debug.cfs_rq[36]:/.tg_runnable_contrib 9 ±31% +195.7% 27 ± 3% TOTAL sched_debug.cfs_rq[54]:/.runnable_load_avg 1552 ±27% +175.8% 4281 ± 2% TOTAL sched_debug.cpu#60.curr->pid 49056 ± 1% +199.2% 146799 ± 0% TOTAL sched_debug.cfs_rq[33]:/.exec_clock 1431 ±22% +201.8% 4321 ± 1% TOTAL sched_debug.cpu#34.curr->pid 48712 ± 2% +198.8% 145545 ± 0% TOTAL sched_debug.cfs_rq[50]:/.exec_clock 48311 ± 1% +201.0% 145402 ± 0% TOTAL sched_debug.cfs_rq[39]:/.exec_clock 332 ± 9% +201.0% 999 ± 0% TOTAL sched_debug.cfs_rq[49]:/.tg_runnable_contrib 226926 ±13% -62.6% 84981 ±24% TOTAL sched_debug.cpu#38.avg_idle 15180 ± 9% +201.5% 45763 ± 0% TOTAL sched_debug.cfs_rq[49]:/.avg->runnable_avg_sum 48719 ± 2% +198.5% 145438 ± 0% TOTAL sched_debug.cfs_rq[56]:/.exec_clock 1795031 ± 0% -66.4% 602823 ± 1% TOTAL sched_debug.cpu#40.nr_switches 48496 ± 2% +199.9% 145443 ± 0% TOTAL sched_debug.cfs_rq[58]:/.exec_clock 15260 ±11% +195.2% 45042 ± 0% TOTAL sched_debug.cfs_rq[40]:/.avg->runnable_avg_sum 285472 ±25% -68.5% 89860 ±28% TOTAL sched_debug.cpu#15.avg_idle 334 ±11% +194.3% 982 ± 0% TOTAL sched_debug.cfs_rq[40]:/.tg_runnable_contrib 49933 ± 2% +194.4% 147012 ± 0% TOTAL sched_debug.cfs_rq[57]:/.exec_clock 49101 ± 2% +196.1% 145384 ± 0% TOTAL sched_debug.cfs_rq[37]:/.exec_clock 49003 ± 1% +196.8% 145444 ± 0% TOTAL sched_debug.cfs_rq[35]:/.exec_clock 1390 ±22% +206.7% 4265 ± 0% TOTAL sched_debug.cpu#37.curr->pid 48581 ± 3% +199.4% 145429 ± 0% TOTAL sched_debug.cfs_rq[48]:/.exec_clock 1630 ±25% +160.5% 4247 ± 2% TOTAL sched_debug.cpu#47.curr->pid 49123 ± 2% +196.0% 145404 ± 0% TOTAL sched_debug.cfs_rq[46]:/.exec_clock 266050 ±20% -59.4% 107922 ±24% TOTAL sched_debug.cpu#52.avg_idle 14978 ±10% +200.9% 45075 ± 0% TOTAL sched_debug.cfs_rq[45]:/.avg->runnable_avg_sum 1417 ±21% +204.3% 4312 ± 2% TOTAL sched_debug.cpu#39.curr->pid 12 ±26% +160.7% 31 ± 9% TOTAL sched_debug.cpu#49.cpu_load[1] 327 ±10% +198.7% 978 ± 0% TOTAL sched_debug.cfs_rq[45]:/.tg_runnable_contrib 270121 ±17% -61.7% 103375 ±20% TOTAL sched_debug.cpu#32.avg_idle 50428 ± 2% +189.1% 145774 ± 0% TOTAL sched_debug.cfs_rq[42]:/.exec_clock 1574 ±34% +166.5% 4194 ± 1% TOTAL sched_debug.cpu#45.curr->pid 50663 ± 1% +189.6% 146717 ± 0% TOTAL sched_debug.cfs_rq[49]:/.exec_clock 49273 ± 2% +195.3% 145509 ± 0% TOTAL sched_debug.cfs_rq[34]:/.exec_clock 15310 ±13% +194.0% 45007 ± 0% TOTAL sched_debug.cfs_rq[60]:/.avg->runnable_avg_sum 236078 ±11% -65.9% 80473 ±14% TOTAL sched_debug.cpu#41.avg_idle 49673 ± 2% +192.7% 145415 ± 0% TOTAL sched_debug.cfs_rq[38]:/.exec_clock 8 ±26% +235.0% 26 ± 5% TOTAL sched_debug.cpu#59.cpu_load[0] 50433 ± 2% +188.4% 145447 ± 0% TOTAL sched_debug.cfs_rq[45]:/.exec_clock 15213 ± 5% +196.2% 45068 ± 0% TOTAL sched_debug.cfs_rq[52]:/.avg->runnable_avg_sum 334 ±13% +193.4% 980 ± 0% TOTAL sched_debug.cfs_rq[60]:/.tg_runnable_contrib 50577 ± 1% +187.9% 145609 ± 0% TOTAL sched_debug.cfs_rq[43]:/.exec_clock 249630 ±15% -65.3% 86741 ± 5% TOTAL sched_debug.cpu#8.avg_idle 332 ± 5% +194.9% 980 ± 0% TOTAL sched_debug.cfs_rq[52]:/.tg_runnable_contrib 235932 ± 9% -62.3% 88954 ±19% TOTAL sched_debug.cpu#34.avg_idle 50890 ± 2% +185.7% 145384 ± 0% TOTAL sched_debug.cfs_rq[47]:/.exec_clock 50930 ± 1% +185.7% 145517 ± 0% TOTAL sched_debug.cfs_rq[40]:/.exec_clock 14328 ±16% +214.3% 45037 ± 0% TOTAL sched_debug.cfs_rq[53]:/.avg->runnable_avg_sum 16062 ± 9% +180.4% 45031 ± 0% TOTAL sched_debug.cfs_rq[47]:/.avg->runnable_avg_sum 51840 ± 1% +183.3% 146858 ± 0% TOTAL sched_debug.cfs_rq[41]:/.exec_clock 1499 ±20% +187.0% 4301 ± 1% TOTAL sched_debug.cpu#51.curr->pid 50662 ± 2% +187.3% 145570 ± 0% TOTAL sched_debug.cfs_rq[44]:/.exec_clock 269032 ± 7% -66.8% 89230 ±17% TOTAL sched_debug.cpu#59.avg_idle 313 ±16% +212.5% 980 ± 0% TOTAL sched_debug.cfs_rq[53]:/.tg_runnable_contrib 16303 ±17% +176.2% 45035 ± 0% TOTAL sched_debug.cfs_rq[54]:/.avg->runnable_avg_sum 356 ±17% +174.9% 979 ± 0% TOTAL sched_debug.cfs_rq[54]:/.tg_runnable_contrib 351 ± 9% +178.0% 977 ± 0% TOTAL sched_debug.cfs_rq[47]:/.tg_runnable_contrib 15587 ±11% +189.2% 45070 ± 0% TOTAL sched_debug.cfs_rq[43]:/.avg->runnable_avg_sum 341 ±11% +187.8% 982 ± 0% TOTAL sched_debug.cfs_rq[43]:/.tg_runnable_contrib 99841369 ± 0% +180.1% 2.796e+08 ± 1% TOTAL numa-numastat.node2.local_node 99843858 ± 0% +180.1% 2.796e+08 ± 1% TOTAL numa-numastat.node2.numa_hit 10 ±17% +196.2% 30 ± 3% TOTAL sched_debug.cpu#57.cpu_load[4] 11 ±12% +189.1% 31 ± 8% TOTAL sched_debug.cpu#41.cpu_load[4] 1560 ±11% +175.1% 4292 ± 1% TOTAL sched_debug.cpu#59.curr->pid 49642548 ± 0% +177.8% 1.379e+08 ± 1% TOTAL numa-vmstat.node2.numa_local 8 ±28% +220.5% 28 ± 4% TOTAL sched_debug.cpu#43.cpu_load[0] 49698361 ± 0% +177.6% 1.38e+08 ± 1% TOTAL numa-vmstat.node2.numa_hit 219871 ±18% -57.9% 92485 ±15% TOTAL sched_debug.cpu#62.avg_idle 50123901 ± 0% +178.0% 1.393e+08 ± 1% TOTAL numa-vmstat.node0.numa_local 50127863 ± 0% +177.9% 1.393e+08 ± 1% TOTAL numa-vmstat.node0.numa_hit 16250 ± 5% +178.5% 45253 ± 0% TOTAL sched_debug.cfs_rq[42]:/.avg->runnable_avg_sum 49861680 ± 0% +176.2% 1.377e+08 ± 1% TOTAL numa-vmstat.node3.numa_local 355 ± 5% +177.0% 985 ± 0% TOTAL sched_debug.cfs_rq[42]:/.tg_runnable_contrib 49917445 ± 0% +176.0% 1.378e+08 ± 1% TOTAL numa-vmstat.node3.numa_hit 1564 ±12% +175.2% 4304 ± 2% TOTAL sched_debug.cpu#54.curr->pid 49911338 ± 0% +175.9% 1.377e+08 ± 1% TOTAL numa-vmstat.node1.numa_local 248524 ±15% -58.0% 104482 ±29% TOTAL sched_debug.cpu#46.avg_idle 49966794 ± 0% +175.7% 1.377e+08 ± 1% TOTAL numa-vmstat.node1.numa_hit 1429 ±20% +198.6% 4269 ± 0% TOTAL sched_debug.cpu#44.curr->pid 16484 ± 8% +175.8% 45465 ± 0% TOTAL sched_debug.cfs_rq[41]:/.avg->runnable_avg_sum 361 ± 8% +174.8% 992 ± 0% TOTAL sched_debug.cfs_rq[41]:/.tg_runnable_contrib 1453 ±23% +197.1% 4318 ± 1% TOTAL sched_debug.cpu#57.curr->pid 1424 ±28% +213.8% 4470 ± 5% TOTAL sched_debug.cpu#63.curr->pid 1420 ±23% +201.2% 4278 ± 3% TOTAL sched_debug.cpu#58.curr->pid 211265 ±21% -57.9% 88934 ±31% TOTAL sched_debug.cpu#39.avg_idle 37 ±32% +129.8% 86 ±10% TOTAL sched_debug.cfs_rq[3]:/.tg_load_contrib 2788857 ± 1% +163.2% 7340919 ± 1% TOTAL sched_debug.cfs_rq[16]:/.min_vruntime 224622 ±17% -59.3% 91521 ±15% TOTAL sched_debug.cpu#35.avg_idle 2764205 ± 1% +161.1% 7216910 ± 1% TOTAL sched_debug.cfs_rq[8]:/.min_vruntime 2819278 ± 1% +156.7% 7238266 ± 1% TOTAL sched_debug.cfs_rq[10]:/.min_vruntime 2837162 ± 1% +159.3% 7357411 ± 1% TOTAL sched_debug.cfs_rq[24]:/.min_vruntime 2817770 ± 1% +160.8% 7349390 ± 1% TOTAL sched_debug.cfs_rq[17]:/.min_vruntime 12 ±31% +160.0% 31 ± 4% TOTAL sched_debug.cpu#33.cpu_load[3] 2839736 ± 1% +154.5% 7225910 ± 1% TOTAL sched_debug.cfs_rq[13]:/.min_vruntime 37 ±38% +86.6% 69 ±45% TOTAL sched_debug.cfs_rq[27]:/.tg_load_contrib 1826 ±15% +133.4% 4262 ± 0% TOTAL sched_debug.cpu#36.curr->pid 2880692 ± 3% +154.0% 7316909 ± 1% TOTAL sched_debug.cfs_rq[0]:/.min_vruntime 1543 ±16% +175.8% 4257 ± 0% TOTAL sched_debug.cpu#46.curr->pid 2903723 ± 1% +153.0% 7346423 ± 1% TOTAL sched_debug.cfs_rq[20]:/.min_vruntime 2853451 ± 1% +156.8% 7328372 ± 1% TOTAL sched_debug.cfs_rq[18]:/.min_vruntime 2881811 ± 2% +154.8% 7342530 ± 0% TOTAL sched_debug.cfs_rq[25]:/.min_vruntime 3.22 ± 1% +149.8% 8.05 ± 3% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg.inet_sendmsg 2848635 ± 1% +153.7% 7225776 ± 1% TOTAL sched_debug.cfs_rq[12]:/.min_vruntime 2878875 ± 2% +155.2% 7346322 ± 1% TOTAL sched_debug.cfs_rq[2]:/.min_vruntime 222782 ±27% -61.7% 85268 ± 9% TOTAL sched_debug.cpu#56.avg_idle 2831503 ± 1% +154.5% 7205461 ± 1% TOTAL sched_debug.cfs_rq[11]:/.min_vruntime 2835663 ± 1% +154.6% 7218527 ± 1% TOTAL sched_debug.cfs_rq[9]:/.min_vruntime 1704 ±26% +152.9% 4309 ± 1% TOTAL sched_debug.cpu#41.curr->pid 12 ±35% +130.6% 28 ± 2% TOTAL sched_debug.cfs_rq[48]:/.load 2844115 ± 0% +154.4% 7235577 ± 1% TOTAL sched_debug.cfs_rq[15]:/.min_vruntime 1665 ±28% +159.1% 4315 ± 1% TOTAL sched_debug.cpu#33.curr->pid 2892891 ± 1% +154.2% 7353510 ± 1% TOTAL sched_debug.cfs_rq[19]:/.min_vruntime 2901579 ± 2% +152.7% 7332677 ± 1% TOTAL sched_debug.cfs_rq[22]:/.min_vruntime 2896475 ± 2% +154.3% 7366784 ± 0% TOTAL sched_debug.cfs_rq[23]:/.min_vruntime 2911824 ± 1% +151.7% 7327805 ± 1% TOTAL sched_debug.cfs_rq[1]:/.min_vruntime 2966470 ± 1% +147.9% 7354082 ± 1% TOTAL sched_debug.cfs_rq[29]:/.min_vruntime 2884101 ± 0% +150.3% 7219627 ± 1% TOTAL sched_debug.cfs_rq[14]:/.min_vruntime 2925842 ± 1% +151.9% 7369360 ± 1% TOTAL sched_debug.cfs_rq[31]:/.min_vruntime 2902721 ± 1% +152.7% 7334172 ± 1% TOTAL sched_debug.cfs_rq[21]:/.min_vruntime 2924791 ± 2% +150.8% 7336302 ± 1% TOTAL sched_debug.cfs_rq[26]:/.min_vruntime 11 ±26% +154.2% 30 ± 2% TOTAL sched_debug.cpu#57.cpu_load[3] 2910713 ± 1% +151.6% 7322791 ± 1% TOTAL sched_debug.cfs_rq[3]:/.min_vruntime 2952231 ± 2% +149.6% 7369935 ± 0% TOTAL sched_debug.cfs_rq[27]:/.min_vruntime 1327 ±42% +219.3% 4239 ± 0% TOTAL sched_debug.cpu#40.curr->pid 2975600 ± 0% +146.9% 7348061 ± 1% TOTAL sched_debug.cfs_rq[28]:/.min_vruntime 2927020 ± 2% +150.3% 7326407 ± 1% TOTAL sched_debug.cfs_rq[5]:/.min_vruntime 2937147 ± 0% +148.1% 7287431 ± 1% TOTAL sched_debug.cfs_rq[7]:/.min_vruntime 12 ±16% +151.6% 32 ±12% TOTAL sched_debug.cpu#49.cpu_load[0] 2972203 ± 1% +146.7% 7331240 ± 0% TOTAL sched_debug.cfs_rq[30]:/.min_vruntime 2917430 ± 1% +149.9% 7291729 ± 1% TOTAL sched_debug.cfs_rq[4]:/.min_vruntime 9 ±19% +177.6% 27 ± 4% TOTAL sched_debug.cfs_rq[60]:/.runnable_load_avg 2903992 ± 2% +150.8% 7282050 ± 1% TOTAL sched_debug.cfs_rq[6]:/.min_vruntime 37 ±31% +113.3% 80 ±23% TOTAL sched_debug.cfs_rq[29]:/.tg_load_contrib 230273 ±17% -51.7% 111201 ±32% TOTAL sched_debug.cpu#50.avg_idle 0.70 ± 1% +140.3% 1.68 ± 3% TOTAL perf-profile.cpu-cycles.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve 206691 ±15% -57.8% 87201 ± 8% TOTAL sched_debug.cpu#61.avg_idle 259396 ± 5% +122.7% 577766 ± 1% TOTAL sched_debug.cpu#62.ttwu_count 12 ±34% +134.9% 29 ± 3% TOTAL sched_debug.cpu#57.cpu_load[2] 269666 ±16% -57.7% 114191 ±17% TOTAL sched_debug.cpu#36.avg_idle 14 ±36% +119.7% 31 ± 5% TOTAL sched_debug.cpu#33.cpu_load[2] 1706 ±16% +149.8% 4262 ± 0% TOTAL sched_debug.cpu#38.curr->pid 242840 ± 4% -54.8% 109685 ±27% TOTAL sched_debug.cpu#45.avg_idle 262430 ± 5% +120.6% 578956 ± 1% TOTAL sched_debug.cpu#60.ttwu_count 139692 ± 4% -54.6% 63480 ± 2% TOTAL proc-vmstat.numa_hint_faults 12 ±19% +148.4% 31 ± 9% TOTAL sched_debug.cpu#41.cpu_load[3] 264847 ± 9% +116.3% 572756 ± 2% TOTAL sched_debug.cpu#61.ttwu_count 0.50 ± 7% +122.0% 1.11 ±12% TOTAL perf-profile.cpu-cycles.sctp_sendmsg.inet_sendmsg.sock_sendmsg.___sys_sendmsg.__sys_sendmsg 30 ±12% +115.3% 64 ±46% TOTAL sched_debug.cfs_rq[19]:/.tg_load_contrib 11 ±28% +139.0% 28 ± 5% TOTAL sched_debug.cfs_rq[42]:/.runnable_load_avg 263882 ± 5% +119.9% 580288 ± 1% TOTAL sched_debug.cpu#59.ttwu_count 15 ±41% +105.3% 30 ± 7% TOTAL sched_debug.cpu#33.cpu_load[1] 15 ±41% +97.5% 31 ±10% TOTAL sched_debug.cpu#33.cpu_load[0] 232649 ± 8% -53.9% 107165 ± 9% TOTAL sched_debug.cpu#37.avg_idle 1870 ±19% +126.4% 4234 ± 0% TOTAL sched_debug.cpu#42.curr->pid 276936 ± 2% +110.2% 582259 ± 1% TOTAL sched_debug.cpu#32.ttwu_count 32 ±13% +116.7% 70 ±31% TOTAL sched_debug.cfs_rq[24]:/.tg_load_contrib 3.48 ± 0% +111.1% 7.35 ± 1% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.skb_copy_datagram_iovec.sctp_recvmsg.sock_common_recvmsg.sock_recvmsg 276884 ± 2% +111.1% 584406 ± 1% TOTAL sched_debug.cpu#33.ttwu_count 278248 ± 4% +107.0% 575968 ± 2% TOTAL sched_debug.cpu#55.ttwu_count 276813 ± 5% +109.4% 579782 ± 1% TOTAL sched_debug.cpu#51.ttwu_count 274908 ± 2% +110.4% 578527 ± 1% TOTAL sched_debug.cpu#52.ttwu_count 276883 ± 5% +108.8% 578248 ± 1% TOTAL sched_debug.cpu#53.ttwu_count 13 ±48% +127.3% 30 ± 8% TOTAL sched_debug.cpu#57.cpu_load[0] 12 ±42% +130.2% 29 ± 3% TOTAL sched_debug.cpu#57.cpu_load[1] 12 ±47% +159.0% 31 ±13% TOTAL sched_debug.cfs_rq[41]:/.runnable_load_avg 96427 ± 3% -51.4% 46839 ± 0% TOTAL proc-vmstat.numa_hint_faults_local 14 ±23% +113.5% 31 ±11% TOTAL sched_debug.cpu#41.cpu_load[2] 281119 ± 3% +105.5% 577638 ± 1% TOTAL sched_debug.cpu#63.ttwu_count 279804 ± 7% +105.6% 575372 ± 1% TOTAL sched_debug.cpu#54.ttwu_count 285799 ± 3% +104.0% 583075 ± 1% TOTAL sched_debug.cpu#39.ttwu_count 291642 ±21% -55.2% 130660 ±18% TOTAL sched_debug.cpu#40.avg_idle 140 ±31% -57.1% 60 ±44% TOTAL sched_debug.cfs_rq[43]:/.tg_load_contrib 13879 ± 0% +104.3% 28362 ± 0% TOTAL proc-vmstat.pgactivate 251355 ±27% -49.9% 126023 ±33% TOTAL sched_debug.cpu#42.avg_idle 284829 ± 5% +104.2% 581719 ± 1% TOTAL sched_debug.cpu#36.ttwu_count 287754 ± 5% +101.5% 579892 ± 1% TOTAL sched_debug.cpu#57.ttwu_count 290676 ± 4% +98.4% 576782 ± 1% TOTAL sched_debug.cpu#50.ttwu_count 15 ±34% +92.4% 30 ±11% TOTAL sched_debug.cfs_rq[33]:/.runnable_load_avg 28 ±12% +100.7% 57 ±30% TOTAL sched_debug.cfs_rq[31]:/.tg_load_contrib 287909 ± 4% +101.5% 580220 ± 0% TOTAL sched_debug.cpu#56.ttwu_count 281438 ± 6% +105.0% 576839 ± 1% TOTAL sched_debug.cpu#58.ttwu_count 223246 ±22% -59.3% 90847 ±16% TOTAL sched_debug.cpu#57.avg_idle 292834 ± 2% +98.9% 582416 ± 1% TOTAL sched_debug.cpu#35.ttwu_count 299007 ± 1% +93.8% 579492 ± 1% TOTAL sched_debug.cpu#49.ttwu_count 296051 ± 3% +96.4% 581581 ± 0% TOTAL sched_debug.cpu#37.ttwu_count 155938 ± 3% -48.9% 79614 ± 2% TOTAL proc-vmstat.numa_pte_updates 292818 ± 4% +99.2% 583153 ± 1% TOTAL sched_debug.cpu#34.ttwu_count 291339 ± 3% +97.8% 576203 ± 1% TOTAL sched_debug.cpu#46.ttwu_count 1173228 ±19% -48.8% 601063 ± 1% TOTAL sched_debug.cpu#8.sched_count 291043 ± 6% +98.2% 576756 ± 1% TOTAL sched_debug.cpu#48.ttwu_count 32644 ± 0% +92.8% 62936 ± 0% TOTAL sched_debug.cfs_rq[0]:/.tg->runnable_avg 32648 ± 0% +92.8% 62937 ± 0% TOTAL sched_debug.cfs_rq[1]:/.tg->runnable_avg 32650 ± 0% +92.8% 62937 ± 0% TOTAL sched_debug.cfs_rq[2]:/.tg->runnable_avg 32654 ± 0% +92.7% 62937 ± 0% TOTAL sched_debug.cfs_rq[3]:/.tg->runnable_avg 32665 ± 0% +92.7% 62937 ± 0% TOTAL sched_debug.cfs_rq[4]:/.tg->runnable_avg 32674 ± 0% +92.6% 62938 ± 0% TOTAL sched_debug.cfs_rq[5]:/.tg->runnable_avg 32689 ± 0% +92.5% 62938 ± 0% TOTAL sched_debug.cfs_rq[8]:/.tg->runnable_avg 32678 ± 0% +92.6% 62937 ± 0% TOTAL sched_debug.cfs_rq[6]:/.tg->runnable_avg 32690 ± 0% +92.5% 62938 ± 0% TOTAL sched_debug.cfs_rq[10]:/.tg->runnable_avg 32691 ± 0% +92.5% 62938 ± 0% TOTAL sched_debug.cfs_rq[9]:/.tg->runnable_avg 32686 ± 0% +92.6% 62937 ± 0% TOTAL sched_debug.cfs_rq[7]:/.tg->runnable_avg 32692 ± 0% +92.5% 62938 ± 0% TOTAL sched_debug.cfs_rq[11]:/.tg->runnable_avg 32696 ± 0% +92.5% 62938 ± 0% TOTAL sched_debug.cfs_rq[12]:/.tg->runnable_avg 32701 ± 0% +92.5% 62938 ± 0% TOTAL sched_debug.cfs_rq[13]:/.tg->runnable_avg 32704 ± 0% +92.4% 62938 ± 0% TOTAL sched_debug.cfs_rq[14]:/.tg->runnable_avg 32702 ± 0% +92.5% 62938 ± 0% TOTAL sched_debug.cfs_rq[15]:/.tg->runnable_avg 32717 ± 0% +92.4% 62937 ± 0% TOTAL sched_debug.cfs_rq[18]:/.tg->runnable_avg 32707 ± 0% +92.4% 62938 ± 0% TOTAL sched_debug.cfs_rq[16]:/.tg->runnable_avg 32713 ± 0% +92.4% 62937 ± 0% TOTAL sched_debug.cfs_rq[17]:/.tg->runnable_avg 32722 ± 0% +92.3% 62938 ± 0% TOTAL sched_debug.cfs_rq[19]:/.tg->runnable_avg 32727 ± 0% +92.3% 62938 ± 0% TOTAL sched_debug.cfs_rq[20]:/.tg->runnable_avg 32732 ± 0% +92.3% 62938 ± 0% TOTAL sched_debug.cfs_rq[21]:/.tg->runnable_avg 32740 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[24]:/.tg->runnable_avg 32739 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[23]:/.tg->runnable_avg 32743 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[26]:/.tg->runnable_avg 32739 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[22]:/.tg->runnable_avg 32743 ± 0% +92.2% 62937 ± 0% TOTAL sched_debug.cfs_rq[27]:/.tg->runnable_avg 32746 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[25]:/.tg->runnable_avg 32751 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[29]:/.tg->runnable_avg 32751 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[30]:/.tg->runnable_avg 32747 ± 0% +92.2% 62938 ± 0% TOTAL sched_debug.cfs_rq[28]:/.tg->runnable_avg 32752 ± 0% +92.2% 62939 ± 0% TOTAL sched_debug.cfs_rq[31]:/.tg->runnable_avg 32752 ± 0% +92.2% 62939 ± 0% TOTAL sched_debug.cfs_rq[32]:/.tg->runnable_avg 32759 ± 0% +92.1% 62939 ± 0% TOTAL sched_debug.cfs_rq[33]:/.tg->runnable_avg 32770 ± 0% +92.1% 62939 ± 0% TOTAL sched_debug.cfs_rq[37]:/.tg->runnable_avg 32757 ± 0% +92.1% 62939 ± 0% TOTAL sched_debug.cfs_rq[34]:/.tg->runnable_avg 32766 ± 0% +92.1% 62939 ± 0% TOTAL sched_debug.cfs_rq[36]:/.tg->runnable_avg 32765 ± 0% +92.1% 62939 ± 0% TOTAL sched_debug.cfs_rq[35]:/.tg->runnable_avg 32767 ± 0% +92.1% 62939 ± 0% TOTAL sched_debug.cfs_rq[38]:/.tg->runnable_avg 32774 ± 0% +92.0% 62939 ± 0% TOTAL sched_debug.cfs_rq[41]:/.tg->runnable_avg 32775 ± 0% +92.0% 62939 ± 0% TOTAL sched_debug.cfs_rq[42]:/.tg->runnable_avg 32774 ± 0% +92.0% 62939 ± 0% TOTAL sched_debug.cfs_rq[39]:/.tg->runnable_avg 32779 ± 0% +92.0% 62939 ± 0% TOTAL sched_debug.cfs_rq[43]:/.tg->runnable_avg 32773 ± 0% +92.0% 62939 ± 0% TOTAL sched_debug.cfs_rq[40]:/.tg->runnable_avg 32822 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[63]:/.tg->runnable_avg 32820 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[62]:/.tg->runnable_avg 32793 ± 0% +91.9% 62939 ± 0% TOTAL sched_debug.cfs_rq[47]:/.tg->runnable_avg 32796 ± 0% +91.9% 62940 ± 0% TOTAL sched_debug.cfs_rq[48]:/.tg->runnable_avg 32818 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[61]:/.tg->runnable_avg 32821 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[60]:/.tg->runnable_avg 32790 ± 0% +91.9% 62939 ± 0% TOTAL sched_debug.cfs_rq[44]:/.tg->runnable_avg 32791 ± 0% +91.9% 62939 ± 0% TOTAL sched_debug.cfs_rq[46]:/.tg->runnable_avg 32819 ± 0% +91.8% 62941 ± 0% TOTAL sched_debug.cfs_rq[59]:/.tg->runnable_avg 32793 ± 0% +91.9% 62939 ± 0% TOTAL sched_debug.cfs_rq[45]:/.tg->runnable_avg 32812 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[56]:/.tg->runnable_avg 32817 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[57]:/.tg->runnable_avg 32809 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[52]:/.tg->runnable_avg 32811 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[55]:/.tg->runnable_avg 32801 ± 0% +91.9% 62940 ± 0% TOTAL sched_debug.cfs_rq[49]:/.tg->runnable_avg 32822 ± 0% +91.8% 62941 ± 0% TOTAL sched_debug.cfs_rq[58]:/.tg->runnable_avg 32808 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[54]:/.tg->runnable_avg 32812 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[53]:/.tg->runnable_avg 32809 ± 0% +91.8% 62940 ± 0% TOTAL sched_debug.cfs_rq[51]:/.tg->runnable_avg 32804 ± 0% +91.9% 62940 ± 0% TOTAL sched_debug.cfs_rq[50]:/.tg->runnable_avg 309179 ± 2% +87.8% 580793 ± 1% TOTAL sched_debug.cpu#41.ttwu_count 81 ±44% -62.2% 30 ± 4% TOTAL sched_debug.cfs_rq[62]:/.tg_load_contrib 5254994 ± 0% +89.6% 9963624 ± 0% TOTAL softirqs.TIMER 3.69 ± 1% +86.0% 6.87 ± 4% TOTAL perf-profile.cpu-cycles.memcpy.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter.sctp_do_sm 308199 ± 4% +86.7% 575358 ± 1% TOTAL sched_debug.cpu#42.ttwu_count 305544 ± 4% +90.6% 582390 ± 1% TOTAL sched_debug.cpu#38.ttwu_count 309614 ± 1% +86.5% 577365 ± 1% TOTAL sched_debug.cpu#43.ttwu_count 14 ±33% +98.6% 28 ± 2% TOTAL sched_debug.cpu#35.load 309719 ± 5% +85.8% 575506 ± 1% TOTAL sched_debug.cpu#45.ttwu_count 1081161 ± 1% -46.1% 582374 ± 1% TOTAL sched_debug.cpu#29.ttwu_count 316152 ± 3% +82.7% 577646 ± 1% TOTAL sched_debug.cpu#40.ttwu_count 27 ± 8% +107.9% 57 ±49% TOTAL sched_debug.cfs_rq[9]:/.tg_load_contrib 1083438 ± 0% -45.9% 586638 ± 0% TOTAL sched_debug.cpu#7.ttwu_count 315864 ± 5% +82.6% 576755 ± 1% TOTAL sched_debug.cpu#47.ttwu_count 31 ± 7% +69.2% 52 ±36% TOTAL sched_debug.cfs_rq[13]:/.tg_load_contrib 1084154 ± 1% -45.7% 588437 ± 1% TOTAL sched_debug.cpu#30.ttwu_count 311962 ± 5% +84.9% 576905 ± 1% TOTAL sched_debug.cpu#44.ttwu_count 1060523 ± 1% -45.1% 582533 ± 1% TOTAL sched_debug.cpu#21.ttwu_count 1054699 ± 1% -45.1% 579000 ± 1% TOTAL sched_debug.cpu#14.ttwu_count 17 ±22% +70.8% 30 ± 2% TOTAL sched_debug.cpu#8.cpu_load[4] 15 ±26% +103.9% 31 ±12% TOTAL sched_debug.cpu#41.cpu_load[1] 28 ± 4% +85.4% 53 ±38% TOTAL sched_debug.cfs_rq[30]:/.tg_load_contrib 1075268 ± 1% -45.5% 586405 ± 1% TOTAL sched_debug.cpu#28.ttwu_count 1059908 ± 2% -44.9% 584032 ± 1% TOTAL sched_debug.cpu#31.ttwu_count 1068780 ± 1% -45.5% 582915 ± 1% TOTAL sched_debug.cpu#27.ttwu_count 1068310 ± 0% -44.9% 588354 ± 1% TOTAL sched_debug.cpu#5.ttwu_count 1055705 ± 2% -45.0% 580123 ± 2% TOTAL sched_debug.cpu#23.ttwu_count 1054570 ± 1% -44.7% 583394 ± 1% TOTAL sched_debug.cpu#20.ttwu_count 254366 ±43% -59.3% 103437 ±13% TOTAL sched_debug.cpu#54.avg_idle 1063589 ± 2% -44.7% 587718 ± 1% TOTAL sched_debug.cpu#6.ttwu_count 1058139 ± 3% -44.6% 585929 ± 1% TOTAL sched_debug.cpu#22.ttwu_count 1067145 ± 1% -44.9% 588220 ± 1% TOTAL sched_debug.cpu#4.ttwu_count 1057669 ± 0% -44.3% 588982 ± 1% TOTAL sched_debug.cpu#1.ttwu_count 1059310 ± 1% -44.3% 589806 ± 1% TOTAL sched_debug.cpu#3.ttwu_count 1043843 ± 1% -44.2% 581948 ± 1% TOTAL sched_debug.cpu#18.ttwu_count 1031973 ± 1% -43.9% 579237 ± 1% TOTAL sched_debug.cpu#11.ttwu_count 18 ±24% +70.7% 31 ±13% TOTAL sched_debug.cpu#41.cpu_load[0] 1047788 ± 2% -44.3% 583196 ± 1% TOTAL sched_debug.cpu#19.ttwu_count 1044163 ± 2% -44.2% 583017 ± 1% TOTAL sched_debug.cpu#26.ttwu_count 1288766 ±38% -52.8% 608462 ± 1% TOTAL sched_debug.cpu#1.sched_count 1032898 ± 1% -43.8% 579984 ± 1% TOTAL sched_debug.cpu#13.ttwu_count 1049631 ± 1% -43.7% 590788 ± 0% TOTAL sched_debug.cpu#2.ttwu_count 1034514 ± 1% -43.7% 582699 ± 1% TOTAL sched_debug.cpu#15.ttwu_count 16 ±20% +75.0% 29 ± 4% TOTAL sched_debug.cpu#9.cpu_load[0] 1023516 ± 1% -43.1% 582541 ± 1% TOTAL sched_debug.cpu#17.ttwu_count 1039799 ± 3% -44.3% 579547 ± 2% TOTAL sched_debug.cpu#25.ttwu_count 1302779 ±36% -54.0% 598642 ± 1% TOTAL sched_debug.cpu#9.sched_count 1012139 ± 0% -42.5% 581867 ± 1% TOTAL sched_debug.cpu#9.ttwu_count 1032088 ± 2% -43.7% 581367 ± 1% TOTAL sched_debug.cpu#12.ttwu_count 1015497 ± 1% -42.6% 582939 ± 1% TOTAL sched_debug.cpu#16.ttwu_count 1023139 ± 1% -43.1% 582533 ± 1% TOTAL sched_debug.cpu#10.ttwu_count 1039034 ± 1% -42.7% 595606 ± 1% TOTAL sched_debug.cpu#0.ttwu_count 1125131 ±22% -46.7% 600109 ± 2% TOTAL sched_debug.cpu#12.sched_count 9019 ± 6% -42.0% 5232 ±10% TOTAL proc-vmstat.pgmigrate_success 9019 ± 6% -42.0% 5232 ±10% TOTAL proc-vmstat.numa_pages_migrated 1020221 ± 2% -42.6% 586085 ± 1% TOTAL sched_debug.cpu#24.ttwu_count 18 ± 9% +69.2% 30 ± 2% TOTAL sched_debug.cpu#9.cpu_load[4] 19 ±14% +84.8% 36 ±34% TOTAL sched_debug.cfs_rq[38]:/.load 30 ±12% +56.7% 47 ±18% TOTAL sched_debug.cfs_rq[25]:/.tg_load_contrib 17 ±13% +62.5% 28 ± 3% TOTAL sched_debug.cpu#1.cpu_load[0] 17 ±15% +64.4% 28 ± 1% TOTAL sched_debug.cpu#22.cpu_load[3] 19 ±16% +52.1% 29 ± 2% TOTAL sched_debug.cpu#25.cpu_load[3] 991690 ± 0% -41.2% 582981 ± 1% TOTAL sched_debug.cpu#8.ttwu_count 18 ±12% +60.2% 29 ± 4% TOTAL sched_debug.cpu#15.cpu_load[4] 18 ±22% +66.7% 30 ± 2% TOTAL sched_debug.cpu#8.cpu_load[3] 17 ±11% +72.4% 30 ± 2% TOTAL sched_debug.cpu#9.cpu_load[3] 21 ±26% +70.5% 35 ±38% TOTAL sched_debug.cpu#61.load 19 ±25% +56.8% 29 ± 2% TOTAL sched_debug.cpu#43.load 18 ±16% +52.7% 27 ± 1% TOTAL sched_debug.cpu#30.cpu_load[2] 18 ±18% +56.0% 28 ± 3% TOTAL sched_debug.cpu#26.cpu_load[4] 17 ±17% +59.1% 28 ± 3% TOTAL sched_debug.cpu#22.cpu_load[1] 17 ± 4% +58.0% 27 ± 4% TOTAL sched_debug.cpu#3.cpu_load[0] 17 ±16% +55.1% 27 ± 3% TOTAL sched_debug.cpu#22.cpu_load[0] 18 ±19% +56.7% 28 ± 4% TOTAL sched_debug.cpu#26.cpu_load[3] 17 ±17% +64.0% 28 ± 2% TOTAL sched_debug.cpu#22.cpu_load[2] 2664 ±15% +63.7% 4362 ± 1% TOTAL sched_debug.cpu#20.curr->pid 19 ±14% +56.6% 31 ± 3% TOTAL sched_debug.cfs_rq[45]:/.load 71 ±43% -52.1% 34 ±13% TOTAL sched_debug.cfs_rq[47]:/.tg_load_contrib 13306 ± 0% +61.5% 21495 ± 0% TOTAL proc-vmstat.nr_shmem 53249 ± 0% +61.5% 85996 ± 0% TOTAL meminfo.Shmem 18 ± 8% +58.7% 29 ± 1% TOTAL sched_debug.cpu#11.cpu_load[3] 18 ±12% +57.1% 28 ± 1% TOTAL sched_debug.cpu#30.cpu_load[3] 17 ±22% +65.2% 29 ± 3% TOTAL sched_debug.cfs_rq[9]:/.runnable_load_avg 19 ± 6% +54.7% 29 ± 1% TOTAL sched_debug.cpu#23.cpu_load[4] 17 ±15% +69.8% 29 ± 3% TOTAL sched_debug.cpu#9.cpu_load[2] 18 ± 7% +59.1% 29 ± 2% TOTAL sched_debug.cpu#11.cpu_load[4] 17 ±17% +70.6% 29 ± 4% TOTAL sched_debug.cpu#9.cpu_load[1] 18 ± 8% +55.4% 28 ± 3% TOTAL sched_debug.cpu#1.cpu_load[1] 18 ± 7% +55.3% 29 ± 3% TOTAL sched_debug.cpu#1.cpu_load[2] 19 ±14% +54.7% 29 ± 1% TOTAL sched_debug.cpu#25.cpu_load[4] 17 ±14% +65.5% 28 ± 1% TOTAL sched_debug.cpu#22.cpu_load[4] 18 ± 9% +53.8% 28 ± 4% TOTAL sched_debug.cfs_rq[13]:/.runnable_load_avg 19 ±11% +54.7% 29 ± 5% TOTAL sched_debug.cpu#15.cpu_load[3] 18 ±12% +55.4% 28 ± 1% TOTAL sched_debug.cfs_rq[23]:/.runnable_load_avg 18 ± 2% +57.1% 28 ± 3% TOTAL sched_debug.cpu#3.cpu_load[3] 18 ± 2% +60.4% 29 ± 2% TOTAL sched_debug.cpu#3.cpu_load[4] 17 ± 2% +60.7% 28 ± 3% TOTAL sched_debug.cpu#3.cpu_load[2] 2777 ±10% +60.7% 4463 ± 7% TOTAL sched_debug.cpu#27.curr->pid 2744 ±13% +58.3% 4342 ± 3% TOTAL sched_debug.cpu#22.curr->pid 988578 ±10% -38.9% 604501 ± 1% TOTAL sched_debug.cpu#24.sched_count 1083808 ±31% -42.7% 621044 ± 7% TOTAL sched_debug.cpu#6.sched_count 19 ±12% +53.1% 29 ± 4% TOTAL sched_debug.cpu#19.cpu_load[3] 19 ±13% +53.6% 29 ± 4% TOTAL sched_debug.cpu#19.cpu_load[4] 966177 ±24% -37.8% 600790 ± 2% TOTAL sched_debug.cpu#27.sched_count 1146030 ±46% -46.9% 608122 ± 1% TOTAL sched_debug.cpu#2.sched_count 18 ± 3% +55.6% 28 ± 3% TOTAL sched_debug.cpu#3.cpu_load[1] 19 ±19% +41.4% 28 ± 2% TOTAL sched_debug.cpu#5.cpu_load[0] 17 ± 9% +58.4% 28 ± 5% TOTAL sched_debug.cfs_rq[3]:/.runnable_load_avg 18 ±19% +52.2% 28 ± 5% TOTAL sched_debug.cpu#26.cpu_load[2] 18 ±17% +50.5% 28 ± 3% TOTAL sched_debug.cpu#30.cpu_load[0] 18 ±17% +50.5% 27 ± 2% TOTAL sched_debug.cpu#30.cpu_load[1] 19 ±11% +49.0% 28 ± 4% TOTAL sched_debug.cpu#15.cpu_load[2] 18 ±12% +56.7% 28 ± 1% TOTAL sched_debug.cpu#13.cpu_load[0] 20 ±13% +38.0% 27 ± 3% TOTAL sched_debug.cfs_rq[31]:/.runnable_load_avg 20 ±18% +40.0% 28 ± 2% TOTAL sched_debug.cpu#5.cpu_load[1] 18 ±22% +56.5% 28 ± 5% TOTAL sched_debug.cpu#8.cpu_load[0] 19 ±16% +49.0% 28 ± 2% TOTAL sched_debug.cpu#25.cpu_load[2] 1040164 ±27% -37.2% 653364 ±10% TOTAL sched_debug.cpu#19.sched_count 80 ±29% -45.9% 43 ±26% TOTAL sched_debug.cfs_rq[37]:/.tg_load_contrib 2953 ±10% +47.1% 4345 ± 2% TOTAL sched_debug.cpu#14.curr->pid 923395 ± 6% -24.6% 695896 ±17% TOTAL sched_debug.cpu#21.sched_count 2800 ± 5% +52.5% 4269 ± 1% TOTAL sched_debug.cpu#11.curr->pid 2746 ±15% +57.0% 4312 ± 2% TOTAL sched_debug.cpu#8.curr->pid 20 ±12% +44.1% 29 ± 1% TOTAL sched_debug.cpu#5.cpu_load[4] 18 ±13% +58.1% 29 ± 1% TOTAL sched_debug.cpu#13.cpu_load[2] 18 ± 7% +54.8% 28 ± 2% TOTAL sched_debug.cpu#11.cpu_load[1] 18 ±13% +56.5% 28 ± 1% TOTAL sched_debug.cpu#13.cpu_load[1] 18 ±22% +60.9% 29 ± 5% TOTAL sched_debug.cpu#8.cpu_load[2] 19 ± 6% +47.9% 28 ± 2% TOTAL sched_debug.cpu#7.cpu_load[2] 19 ±12% +51.6% 28 ± 2% TOTAL sched_debug.cpu#11.cpu_load[0] 18 ±12% +58.7% 29 ± 3% TOTAL sched_debug.cfs_rq[11]:/.runnable_load_avg 18 ±11% +62.6% 29 ± 5% TOTAL sched_debug.cfs_rq[1]:/.runnable_load_avg 19 ± 5% +50.5% 28 ± 1% TOTAL sched_debug.cpu#23.cpu_load[3] 19 ±21% +48.0% 29 ± 3% TOTAL sched_debug.cpu#27.cpu_load[4] 18 ±12% +57.0% 29 ± 2% TOTAL sched_debug.cpu#30.cpu_load[4] 18 ± 7% +53.2% 28 ± 2% TOTAL sched_debug.cpu#11.cpu_load[2] 19 ± 4% +58.9% 30 ± 9% TOTAL sched_debug.cfs_rq[2]:/.runnable_load_avg 1101084 ±30% -44.5% 611225 ± 2% TOTAL sched_debug.cpu#4.sched_count 2942 ±10% +44.9% 4263 ± 2% TOTAL sched_debug.cpu#7.curr->pid 2905 ± 7% +53.9% 4471 ± 2% TOTAL sched_debug.cpu#23.curr->pid 1035532 ±19% -41.9% 601372 ± 1% TOTAL sched_debug.cpu#15.sched_count 663 ± 7% +48.7% 987 ± 0% TOTAL sched_debug.cfs_rq[8]:/.tg_runnable_contrib 30382 ± 7% +49.0% 45275 ± 0% TOTAL sched_debug.cfs_rq[8]:/.avg->runnable_avg_sum 30719 ± 7% +46.5% 44998 ± 0% TOTAL sched_debug.cfs_rq[15]:/.avg->runnable_avg_sum 671 ± 7% +45.8% 979 ± 0% TOTAL sched_debug.cfs_rq[15]:/.tg_runnable_contrib 20 ± 9% +46.1% 29 ± 2% TOTAL sched_debug.cpu#18.cpu_load[4] 20 ± 6% +45.6% 30 ± 4% TOTAL sched_debug.cpu#1.cpu_load[4] 19 ± 9% +52.0% 29 ± 2% TOTAL sched_debug.cpu#14.cpu_load[4] 19 ±10% +57.3% 30 ± 2% TOTAL sched_debug.cpu#13.cpu_load[4] 19 ±12% +55.2% 29 ± 2% TOTAL sched_debug.cpu#13.cpu_load[3] 20 ±13% +35.6% 27 ± 1% TOTAL sched_debug.cfs_rq[22]:/.runnable_load_avg 19 ±11% +43.8% 27 ± 2% TOTAL sched_debug.cfs_rq[5]:/.runnable_load_avg 19 ±13% +42.7% 27 ± 3% TOTAL sched_debug.cpu#31.cpu_load[0] 1045 ± 1% +52.1% 1589 ± 7% TOTAL numa-vmstat.node0.nr_alloc_batch 737654 ± 1% +50.3% 1108371 ± 0% TOTAL softirqs.RCU 2824 ±15% +54.7% 4370 ± 1% TOTAL sched_debug.cpu#31.curr->pid 2900 ± 3% +48.4% 4303 ± 1% TOTAL sched_debug.cpu#17.curr->pid 1117 ± 2% +42.8% 1595 ± 6% TOTAL numa-vmstat.node3.nr_alloc_batch 31475 ± 6% +43.6% 45205 ± 0% TOTAL sched_debug.cfs_rq[26]:/.avg->runnable_avg_sum 2819 ±13% +51.7% 4276 ± 2% TOTAL sched_debug.cpu#26.curr->pid 31215 ± 4% +44.3% 45055 ± 0% TOTAL sched_debug.cfs_rq[30]:/.avg->runnable_avg_sum 30904 ± 4% +46.4% 45232 ± 0% TOTAL sched_debug.cfs_rq[10]:/.avg->runnable_avg_sum 675 ± 4% +45.9% 985 ± 0% TOTAL sched_debug.cfs_rq[10]:/.tg_runnable_contrib 687 ± 6% +42.7% 981 ± 0% TOTAL sched_debug.cfs_rq[26]:/.tg_runnable_contrib 19 ± 3% +47.9% 28 ± 3% TOTAL sched_debug.cpu#4.cpu_load[1] 21 ±18% +32.1% 28 ± 2% TOTAL sched_debug.cpu#31.cpu_load[2] 19 ±12% +44.9% 28 ± 5% TOTAL sched_debug.cpu#15.cpu_load[1] 18 ±21% +57.6% 29 ± 6% TOTAL sched_debug.cpu#8.cpu_load[1] 19 ±17% +47.4% 28 ± 3% TOTAL sched_debug.cpu#25.cpu_load[0] 19 ± 3% +49.0% 28 ± 6% TOTAL sched_debug.cpu#4.cpu_load[0] 20 ±13% +42.0% 28 ± 1% TOTAL sched_debug.cpu#5.cpu_load[3] 20 ±10% +41.0% 28 ± 4% TOTAL sched_debug.cfs_rq[15]:/.runnable_load_avg 19 ± 4% +45.9% 28 ± 2% TOTAL sched_debug.cpu#4.cpu_load[2] 19 ± 7% +47.4% 28 ± 2% TOTAL sched_debug.cpu#23.cpu_load[2] 18 ± 3% +47.9% 27 ± 3% TOTAL sched_debug.cpu#7.cpu_load[0] 19 ±13% +46.9% 28 ± 6% TOTAL sched_debug.cpu#15.cpu_load[0] 19 ± 8% +44.8% 27 ± 2% TOTAL sched_debug.cpu#23.cpu_load[1] 19 ±16% +45.4% 28 ± 1% TOTAL sched_debug.cpu#25.cpu_load[1] 19 ±11% +40.8% 27 ± 1% TOTAL sched_debug.cpu#23.cpu_load[0] 20 ±14% +39.6% 28 ± 1% TOTAL sched_debug.cpu#5.cpu_load[2] 19 ±14% +41.4% 28 ± 2% TOTAL sched_debug.cpu#31.cpu_load[1] 19 ± 6% +45.8% 28 ± 2% TOTAL sched_debug.cpu#7.cpu_load[1] 20 ±13% +40.6% 28 ± 3% TOTAL sched_debug.cfs_rq[25]:/.runnable_load_avg 2943 ± 8% +45.8% 4289 ± 0% TOTAL sched_debug.cpu#1.curr->pid 2703 ±11% +59.2% 4302 ± 1% TOTAL sched_debug.cpu#4.curr->pid 2989 ±13% +42.7% 4265 ± 2% TOTAL sched_debug.cpu#24.curr->pid 3048 ± 8% +42.7% 4349 ± 1% TOTAL sched_debug.cpu#3.curr->pid 30576 ± 6% +47.4% 45060 ± 0% TOTAL sched_debug.cfs_rq[22]:/.avg->runnable_avg_sum 682 ± 4% +43.6% 979 ± 0% TOTAL sched_debug.cfs_rq[30]:/.tg_runnable_contrib 1100 ± 3% +49.4% 1643 ± 4% TOTAL numa-vmstat.node1.nr_alloc_batch 4389 ± 1% +46.7% 6439 ± 1% TOTAL proc-vmstat.nr_alloc_batch 668 ± 6% +46.9% 982 ± 0% TOTAL sched_debug.cfs_rq[22]:/.tg_runnable_contrib 3115 ± 9% +38.4% 4310 ± 1% TOTAL sched_debug.cpu#5.curr->pid 695 ± 4% +41.8% 985 ± 0% TOTAL sched_debug.cfs_rq[18]:/.tg_runnable_contrib 31812 ± 4% +42.5% 45324 ± 0% TOTAL sched_debug.cfs_rq[18]:/.avg->runnable_avg_sum 1157 ± 4% +40.2% 1622 ± 7% TOTAL numa-vmstat.node2.nr_alloc_batch 19 ± 6% +51.6% 28 ± 2% TOTAL sched_debug.cpu#7.cpu_load[3] 19 ± 6% +51.5% 29 ± 1% TOTAL sched_debug.cpu#2.cpu_load[4] 19 ± 6% +53.7% 29 ± 2% TOTAL sched_debug.cpu#7.cpu_load[4] 19 ±13% +51.0% 29 ± 3% TOTAL sched_debug.cpu#19.cpu_load[2] 19 ±10% +49.5% 29 ± 4% TOTAL sched_debug.cpu#2.cpu_load[2] 19 ±13% +50.0% 28 ± 4% TOTAL sched_debug.cpu#19.cpu_load[1] 19 ± 6% +49.5% 29 ± 3% TOTAL sched_debug.cpu#2.cpu_load[3] 20 ± 8% +45.5% 29 ± 1% TOTAL sched_debug.cpu#18.cpu_load[3] 19 ± 5% +51.0% 29 ± 2% TOTAL sched_debug.cpu#20.cpu_load[4] 20 ± 7% +41.2% 28 ± 2% TOTAL sched_debug.cpu#18.cpu_load[2] 20 ± 7% +47.0% 29 ± 2% TOTAL sched_debug.cpu#4.cpu_load[4] 20 ± 5% +45.0% 29 ± 2% TOTAL sched_debug.cpu#4.cpu_load[3] 19 ±16% +50.5% 28 ± 1% TOTAL sched_debug.cpu#27.cpu_load[3] 19 ±12% +44.9% 28 ± 4% TOTAL sched_debug.cpu#16.cpu_load[0] 19 ± 5% +48.5% 29 ± 3% TOTAL sched_debug.cpu#1.cpu_load[3] 19 ± 6% +50.0% 29 ± 1% TOTAL sched_debug.cpu#20.cpu_load[3] 2924 ±10% +48.1% 4330 ± 2% TOTAL sched_debug.cpu#13.curr->pid 2942 ± 7% +45.9% 4291 ± 2% TOTAL sched_debug.cpu#15.curr->pid 2957 ± 5% +44.4% 4269 ± 1% TOTAL sched_debug.cpu#9.curr->pid 2930 ±12% +47.6% 4326 ± 3% TOTAL sched_debug.cpu#16.curr->pid 684 ± 7% +44.1% 986 ± 0% TOTAL sched_debug.cfs_rq[27]:/.tg_runnable_contrib 32158 ± 4% +40.1% 45057 ± 0% TOTAL sched_debug.cfs_rq[24]:/.avg->runnable_avg_sum 2949 ± 9% +46.3% 4315 ± 2% TOTAL sched_debug.cpu#19.curr->pid 20 ± 9% +46.5% 29 ± 1% TOTAL sched_debug.cpu#16.cpu_load[4] 19 ±14% +51.0% 29 ± 5% TOTAL sched_debug.cpu#19.cpu_load[0] 31356 ± 7% +43.8% 45098 ± 0% TOTAL sched_debug.cfs_rq[27]:/.avg->runnable_avg_sum 33090 ± 7% +36.2% 45061 ± 0% TOTAL sched_debug.cfs_rq[29]:/.avg->runnable_avg_sum 721 ± 7% +35.7% 979 ± 0% TOTAL sched_debug.cfs_rq[29]:/.tg_runnable_contrib 31372 ± 5% +43.7% 45081 ± 0% TOTAL sched_debug.cfs_rq[11]:/.avg->runnable_avg_sum 701 ± 4% +39.8% 980 ± 0% TOTAL sched_debug.cfs_rq[24]:/.tg_runnable_contrib 3005 ± 6% +44.1% 4331 ± 2% TOTAL sched_debug.cpu#28.curr->pid 722 ± 5% +36.2% 983 ± 0% TOTAL sched_debug.cfs_rq[28]:/.tg_runnable_contrib 32215 ± 6% +40.3% 45183 ± 0% TOTAL sched_debug.cfs_rq[19]:/.avg->runnable_avg_sum 32108 ± 3% +40.5% 45102 ± 0% TOTAL sched_debug.cfs_rq[13]:/.avg->runnable_avg_sum 703 ± 6% +39.9% 983 ± 0% TOTAL sched_debug.cfs_rq[19]:/.tg_runnable_contrib 33049 ± 5% +36.4% 45063 ± 0% TOTAL sched_debug.cfs_rq[28]:/.avg->runnable_avg_sum 685 ± 5% +43.1% 981 ± 0% TOTAL sched_debug.cfs_rq[11]:/.tg_runnable_contrib 702 ± 3% +39.7% 981 ± 0% TOTAL sched_debug.cfs_rq[13]:/.tg_runnable_contrib 32365 ± 1% +39.3% 45086 ± 0% TOTAL sched_debug.cfs_rq[20]:/.avg->runnable_avg_sum 706 ± 1% +38.9% 982 ± 0% TOTAL sched_debug.cfs_rq[20]:/.tg_runnable_contrib 1120854 ±41% -45.0% 616182 ± 4% TOTAL sched_debug.cpu#5.sched_count 31452 ± 3% +44.5% 45437 ± 1% TOTAL sched_debug.cfs_rq[9]:/.avg->runnable_avg_sum 32534 ± 5% +38.4% 45040 ± 0% TOTAL sched_debug.cfs_rq[31]:/.avg->runnable_avg_sum 687 ± 3% +44.1% 990 ± 1% TOTAL sched_debug.cfs_rq[9]:/.tg_runnable_contrib 702 ± 7% +40.4% 985 ± 0% TOTAL sched_debug.cfs_rq[25]:/.tg_runnable_contrib 694 ± 5% +41.6% 983 ± 0% TOTAL sched_debug.cfs_rq[14]:/.tg_runnable_contrib 19 ±24% +50.0% 28 ± 7% TOTAL sched_debug.cfs_rq[8]:/.runnable_load_avg 19 ± 9% +46.9% 28 ± 3% TOTAL sched_debug.cpu#20.cpu_load[2] 19 ±12% +42.4% 28 ± 4% TOTAL sched_debug.cpu#16.cpu_load[1] 19 ±10% +47.5% 29 ± 6% TOTAL sched_debug.cpu#2.cpu_load[1] 19 ±14% +43.9% 28 ± 2% TOTAL sched_debug.cpu#20.cpu_load[0] 20 ±13% +33.7% 27 ± 2% TOTAL sched_debug.cpu#24.cpu_load[1] 20 ±13% +50.0% 30 ±11% TOTAL sched_debug.cpu#2.cpu_load[0] 18 ±16% +48.9% 28 ± 3% TOTAL sched_debug.cpu#27.cpu_load[2] 18 ±19% +46.8% 27 ± 5% TOTAL sched_debug.cpu#26.cpu_load[1] 19 ±14% +43.3% 27 ± 4% TOTAL sched_debug.cfs_rq[30]:/.runnable_load_avg 20 ± 6% +41.6% 28 ± 4% TOTAL sched_debug.cfs_rq[4]:/.runnable_load_avg 20 ±11% +40.0% 28 ± 2% TOTAL sched_debug.cfs_rq[16]:/.runnable_load_avg 20 ± 4% +36.3% 27 ± 2% TOTAL sched_debug.cfs_rq[20]:/.runnable_load_avg 19 ±13% +44.9% 28 ± 1% TOTAL sched_debug.cpu#16.cpu_load[2] 710 ± 5% +38.1% 981 ± 0% TOTAL sched_debug.cfs_rq[31]:/.tg_runnable_contrib 32116 ± 7% +40.5% 45109 ± 0% TOTAL sched_debug.cfs_rq[25]:/.avg->runnable_avg_sum 31789 ± 5% +41.7% 45041 ± 0% TOTAL sched_debug.cfs_rq[14]:/.avg->runnable_avg_sum 31727 ± 4% +42.1% 45080 ± 0% TOTAL sched_debug.cfs_rq[17]:/.avg->runnable_avg_sum 696 ± 6% +41.2% 983 ± 0% TOTAL sched_debug.cfs_rq[0]:/.tg_runnable_contrib 31871 ± 6% +41.4% 45051 ± 0% TOTAL sched_debug.cfs_rq[0]:/.avg->runnable_avg_sum 32292 ± 5% +39.4% 45011 ± 0% TOTAL sched_debug.cfs_rq[7]:/.avg->runnable_avg_sum 699 ± 3% +40.5% 982 ± 0% TOTAL sched_debug.cfs_rq[16]:/.tg_runnable_contrib 706 ± 5% +39.0% 981 ± 0% TOTAL sched_debug.cfs_rq[7]:/.tg_runnable_contrib 3025 ±16% +43.2% 4334 ± 2% TOTAL sched_debug.cpu#10.curr->pid 32043 ± 3% +40.6% 45060 ± 0% TOTAL sched_debug.cfs_rq[16]:/.avg->runnable_avg_sum 705 ± 4% +39.8% 985 ± 0% TOTAL sched_debug.cfs_rq[3]:/.tg_runnable_contrib 32260 ± 4% +39.8% 45100 ± 0% TOTAL sched_debug.cfs_rq[3]:/.avg->runnable_avg_sum 694 ± 4% +41.3% 980 ± 0% TOTAL sched_debug.cfs_rq[17]:/.tg_runnable_contrib 32722 ± 3% +38.1% 45195 ± 0% TOTAL sched_debug.cfs_rq[2]:/.avg->runnable_avg_sum 2869 ±11% +52.0% 4362 ± 3% TOTAL sched_debug.cpu#30.curr->pid 698 ± 5% +40.6% 982 ± 0% TOTAL sched_debug.cfs_rq[23]:/.tg_runnable_contrib 32007 ± 5% +40.7% 45031 ± 0% TOTAL sched_debug.cfs_rq[23]:/.avg->runnable_avg_sum 21 ± 7% +34.3% 29 ± 3% TOTAL sched_debug.cfs_rq[18]:/.runnable_load_avg 20 ±11% +40.8% 29 ± 2% TOTAL sched_debug.cpu#24.cpu_load[3] 21 ±10% +34.6% 28 ± 4% TOTAL sched_debug.cpu#29.cpu_load[3] 21 ±10% +36.8% 29 ± 3% TOTAL sched_debug.cpu#29.cpu_load[4] 21 ± 9% +39.0% 29 ± 1% TOTAL sched_debug.cpu#24.cpu_load[4] 21 ±12% +34.9% 28 ± 5% TOTAL sched_debug.cpu#29.cpu_load[2] 20 ±11% +41.7% 29 ± 2% TOTAL sched_debug.cpu#21.cpu_load[3] 20 ±10% +45.0% 29 ± 2% TOTAL sched_debug.cpu#14.cpu_load[3] 20 ±15% +38.2% 28 ± 3% TOTAL sched_debug.cpu#14.cpu_load[1] 20 ±11% +45.0% 29 ± 2% TOTAL sched_debug.cpu#16.cpu_load[3] 23 ±10% +25.2% 28 ± 2% TOTAL sched_debug.cpu#55.load 21 ± 6% +34.6% 28 ± 2% TOTAL sched_debug.cpu#28.cpu_load[4] 20 ± 6% +38.8% 28 ± 3% TOTAL sched_debug.cfs_rq[7]:/.runnable_load_avg 20 ±11% +44.1% 29 ± 1% TOTAL sched_debug.cpu#21.cpu_load[4] 2959 ± 8% +45.8% 4315 ± 1% TOTAL sched_debug.cpu#6.curr->pid 33054 ± 2% +36.8% 45208 ± 0% TOTAL sched_debug.cfs_rq[1]:/.avg->runnable_avg_sum 715 ± 2% +37.4% 983 ± 0% TOTAL sched_debug.cfs_rq[2]:/.tg_runnable_contrib 32267 ± 5% +39.7% 45074 ± 0% TOTAL sched_debug.cfs_rq[4]:/.avg->runnable_avg_sum 705 ± 4% +38.9% 979 ± 0% TOTAL sched_debug.cfs_rq[4]:/.tg_runnable_contrib 2983 ±12% +47.3% 4394 ± 2% TOTAL sched_debug.cpu#2.curr->pid 32679 ± 5% +37.9% 45073 ± 0% TOTAL sched_debug.cfs_rq[6]:/.avg->runnable_avg_sum 722 ± 2% +35.8% 981 ± 0% TOTAL sched_debug.cfs_rq[1]:/.tg_runnable_contrib 2841 ±20% +54.9% 4399 ± 1% TOTAL sched_debug.cpu#0.curr->pid 32684 ± 5% +37.8% 45024 ± 0% TOTAL sched_debug.cfs_rq[21]:/.avg->runnable_avg_sum 714 ± 5% +37.3% 980 ± 0% TOTAL sched_debug.cfs_rq[6]:/.tg_runnable_contrib 22 ±10% +33.3% 29 ± 1% TOTAL sched_debug.cpu#12.cpu_load[4] 1044460 ±29% -42.8% 597073 ± 1% TOTAL sched_debug.cpu#31.sched_count 2962 ±10% +45.6% 4314 ± 2% TOTAL sched_debug.cpu#21.curr->pid 107755 ± 0% +35.8% 146314 ± 0% TOTAL sched_debug.cfs_rq[16]:/.exec_clock 3040 ±14% +45.8% 4431 ± 3% TOTAL sched_debug.cpu#18.curr->pid 713 ± 5% +37.3% 980 ± 0% TOTAL sched_debug.cfs_rq[21]:/.tg_runnable_contrib 107444 ± 0% +36.2% 146370 ± 0% TOTAL sched_debug.cfs_rq[8]:/.exec_clock 33921 ± 7% +32.9% 45070 ± 0% TOTAL sched_debug.cfs_rq[5]:/.avg->runnable_avg_sum 19 ±16% +41.7% 27 ± 4% TOTAL sched_debug.cpu#27.cpu_load[0] 18 ±16% +46.8% 27 ± 4% TOTAL sched_debug.cpu#27.cpu_load[1] 20 ±13% +35.3% 27 ± 4% TOTAL sched_debug.cpu#24.cpu_load[0] 110484 ± 1% +34.1% 148181 ± 0% TOTAL sched_debug.cfs_rq[10]:/.exec_clock 108447 ± 0% +34.7% 146038 ± 0% TOTAL sched_debug.cfs_rq[17]:/.exec_clock 741 ± 7% +32.3% 980 ± 0% TOTAL sched_debug.cfs_rq[5]:/.tg_runnable_contrib 3131 ±10% +40.9% 4412 ± 5% TOTAL sched_debug.cpu#29.curr->pid 108894 ± 1% +33.9% 145821 ± 0% TOTAL sched_debug.cfs_rq[24]:/.exec_clock 108897 ± 0% +34.3% 146281 ± 0% TOTAL sched_debug.cfs_rq[9]:/.exec_clock 110117 ± 2% +32.5% 145856 ± 0% TOTAL sched_debug.cfs_rq[25]:/.exec_clock 109871 ± 1% +32.8% 145922 ± 0% TOTAL sched_debug.cfs_rq[15]:/.exec_clock 110887 ± 1% +32.8% 147227 ± 0% TOTAL sched_debug.cfs_rq[12]:/.exec_clock 20 ±13% +40.2% 28 ± 4% TOTAL sched_debug.cpu#0.cpu_load[3] 21 ± 7% +30.5% 27 ± 2% TOTAL sched_debug.cfs_rq[28]:/.runnable_load_avg 20 ± 7% +37.9% 28 ± 1% TOTAL sched_debug.cpu#18.cpu_load[1] 21 ± 9% +30.6% 28 ± 2% TOTAL sched_debug.cfs_rq[29]:/.runnable_load_avg 19 ±12% +43.4% 28 ± 1% TOTAL sched_debug.cpu#20.cpu_load[1] 20 ± 9% +38.8% 28 ± 2% TOTAL sched_debug.cpu#21.cpu_load[2] 21 ± 5% +33.0% 28 ± 2% TOTAL sched_debug.cpu#28.cpu_load[3] 21 ± 5% +32.4% 27 ± 2% TOTAL sched_debug.cpu#28.cpu_load[1] 20 ± 5% +33.7% 27 ± 2% TOTAL sched_debug.cpu#28.cpu_load[2] 21 ± 9% +29.0% 27 ± 5% TOTAL sched_debug.cpu#29.cpu_load[0] 21 ±13% +35.2% 28 ± 3% TOTAL sched_debug.cpu#24.cpu_load[2] 21 ± 9% +34.3% 28 ± 1% TOTAL sched_debug.cpu#18.cpu_load[0] 3098 ±11% +39.9% 4335 ± 2% TOTAL sched_debug.cpu#25.curr->pid 109876 ± 0% +33.3% 146491 ± 0% TOTAL sched_debug.cfs_rq[11]:/.exec_clock 268016 ± 3% +32.8% 355948 ± 9% TOTAL meminfo.Committed_AS 109686 ± 1% +33.3% 146181 ± 0% TOTAL sched_debug.cfs_rq[13]:/.exec_clock 109930 ± 1% +33.3% 146529 ± 0% TOTAL sched_debug.cfs_rq[18]:/.exec_clock 110814 ± 2% +31.8% 146073 ± 0% TOTAL sched_debug.cfs_rq[22]:/.exec_clock 111231 ± 1% +31.2% 145955 ± 0% TOTAL sched_debug.cfs_rq[26]:/.exec_clock 792742 ± 2% -24.3% 599925 ± 1% TOTAL sched_debug.cpu#8.nr_switches 20 ±13% +40.6% 28 ± 2% TOTAL sched_debug.cpu#14.cpu_load[2] 22 ±11% +30.4% 29 ± 1% TOTAL sched_debug.cpu#12.cpu_load[3] 22 ±10% +32.7% 29 ± 3% TOTAL sched_debug.cfs_rq[21]:/.runnable_load_avg 22 ±11% +26.5% 28 ± 1% TOTAL sched_debug.cpu#12.cpu_load[2] 20 ±11% +43.1% 29 ± 5% TOTAL sched_debug.cpu#0.cpu_load[4] 22 ±11% +29.1% 28 ± 2% TOTAL sched_debug.cpu#12.cpu_load[1] 22 ±21% +30.0% 28 ± 2% TOTAL sched_debug.cpu#17.cpu_load[3] 896719 ±18% -31.3% 615816 ± 4% TOTAL sched_debug.cpu#30.sched_count 110467 ± 1% +32.0% 145847 ± 0% TOTAL sched_debug.cfs_rq[19]:/.exec_clock 110856 ± 1% +31.7% 145977 ± 0% TOTAL sched_debug.cfs_rq[20]:/.exec_clock 112749 ± 1% +32.1% 148960 ± 0% TOTAL sched_debug.cfs_rq[2]:/.exec_clock 110628 ± 1% +32.0% 146083 ± 0% TOTAL sched_debug.cfs_rq[23]:/.exec_clock 116500 ± 1% +31.0% 152586 ± 0% TOTAL sched_debug.cfs_rq[0]:/.exec_clock 33803 ± 6% +33.2% 45033 ± 0% TOTAL sched_debug.cfs_rq[12]:/.avg->runnable_avg_sum 738 ± 6% +32.9% 981 ± 0% TOTAL sched_debug.cfs_rq[12]:/.tg_runnable_contrib 111685 ± 0% +30.4% 145683 ± 0% TOTAL sched_debug.cfs_rq[4]:/.exec_clock 111045 ± 1% +31.3% 145845 ± 0% TOTAL sched_debug.cfs_rq[21]:/.exec_clock 111156 ± 0% +31.3% 145923 ± 0% TOTAL sched_debug.cfs_rq[14]:/.exec_clock 111731 ± 0% +30.3% 145623 ± 0% TOTAL sched_debug.cfs_rq[5]:/.exec_clock 111147 ± 1% +31.0% 145641 ± 0% TOTAL sched_debug.cfs_rq[6]:/.exec_clock 3172 ± 3% +33.8% 4245 ± 0% TOTAL sched_debug.cpu#12.curr->pid 111847 ± 0% +30.4% 145836 ± 0% TOTAL sched_debug.cfs_rq[1]:/.exec_clock 20 ±39% +43.1% 29 ± 3% TOTAL sched_debug.cpu#63.load 111241 ± 0% +31.0% 145727 ± 0% TOTAL sched_debug.cfs_rq[3]:/.exec_clock 111457 ± 1% +30.7% 145630 ± 0% TOTAL sched_debug.cfs_rq[31]:/.exec_clock 112620 ± 1% +29.4% 145693 ± 0% TOTAL sched_debug.cfs_rq[27]:/.exec_clock 112963 ± 1% +28.9% 145645 ± 0% TOTAL sched_debug.cfs_rq[29]:/.exec_clock 112421 ± 0% +29.6% 145673 ± 0% TOTAL sched_debug.cfs_rq[7]:/.exec_clock 7338 ±11% +23.3% 9050 ± 7% TOTAL numa-vmstat.node3.nr_anon_pages 29353 ±11% +23.3% 36204 ± 7% TOTAL numa-meminfo.node3.AnonPages 766344 ± 4% -22.2% 596356 ± 1% TOTAL sched_debug.cpu#12.nr_switches 112871 ± 0% +29.0% 145613 ± 0% TOTAL sched_debug.cfs_rq[28]:/.exec_clock 785724 ± 2% -22.0% 612543 ± 0% TOTAL sched_debug.cpu#0.nr_switches 770632 ± 2% -22.3% 598517 ± 1% TOTAL sched_debug.cpu#15.nr_switches 113328 ± 0% +28.5% 145614 ± 0% TOTAL sched_debug.cfs_rq[30]:/.exec_clock 19 ±16% +40.2% 27 ± 2% TOTAL sched_debug.cfs_rq[0]:/.runnable_load_avg 19 ±23% +39.2% 27 ± 4% TOTAL sched_debug.cfs_rq[26]:/.runnable_load_avg 763775 ± 3% -21.8% 596955 ± 1% TOTAL sched_debug.cpu#13.nr_switches 21 ±10% +30.6% 28 ± 4% TOTAL sched_debug.cpu#29.cpu_load[1] 20 ±10% +36.5% 28 ± 3% TOTAL sched_debug.cpu#21.cpu_load[1] 21 ±18% +31.1% 27 ± 1% TOTAL sched_debug.cpu#0.cpu_load[1] 22 ± 4% +30.0% 28 ± 4% TOTAL sched_debug.cpu#52.load 21 ±11% +30.8% 28 ± 3% TOTAL sched_debug.cpu#21.cpu_load[0] 21 ± 6% +28.4% 28 ± 3% TOTAL sched_debug.cpu#28.cpu_load[0] 21 ±19% +31.8% 28 ± 2% TOTAL sched_debug.cpu#17.cpu_load[2] 19 ±20% +45.8% 28 ± 3% TOTAL sched_debug.cpu#26.cpu_load[0] 22 ±16% +27.3% 28 ± 5% TOTAL sched_debug.cfs_rq[27]:/.runnable_load_avg 21 ±17% +29.4% 28 ± 2% TOTAL sched_debug.cpu#17.cpu_load[0] 21 ±12% +32.1% 28 ± 2% TOTAL sched_debug.cfs_rq[24]:/.runnable_load_avg 20 ±17% +34.6% 28 ± 2% TOTAL sched_debug.cpu#0.cpu_load[2] 773695 ± 2% -22.5% 599935 ± 1% TOTAL sched_debug.cpu#11.nr_switches 760832 ± 2% -21.3% 598498 ± 1% TOTAL sched_debug.cpu#9.nr_switches 23 ±17% +20.3% 28 ± 5% TOTAL sched_debug.cfs_rq[52]:/.load 761399 ± 1% -21.2% 600107 ± 1% TOTAL sched_debug.cpu#10.nr_switches 770683 ± 1% -21.1% 607908 ± 1% TOTAL sched_debug.cpu#2.nr_switches 757803 ± 4% -20.7% 600923 ± 1% TOTAL sched_debug.cpu#6.nr_switches 750665 ± 1% -19.1% 606952 ± 0% TOTAL sched_debug.cpu#1.nr_switches 744077 ± 2% -19.6% 598358 ± 0% TOTAL sched_debug.cpu#16.nr_switches 753828 ± 1% -19.6% 605817 ± 1% TOTAL sched_debug.cpu#3.nr_switches 742810 ± 2% -18.9% 602251 ± 0% TOTAL sched_debug.cpu#5.nr_switches 732991 ± 2% -18.5% 597448 ± 1% TOTAL sched_debug.cpu#17.nr_switches 725349 ± 2% -17.9% 595187 ± 1% TOTAL sched_debug.cpu#14.nr_switches 21 ±18% +29.6% 28 ± 2% TOTAL sched_debug.cpu#17.cpu_load[1] 22 ±14% +28.2% 28 ± 2% TOTAL sched_debug.cpu#12.cpu_load[0] 736992 ± 2% -17.8% 605506 ± 1% TOTAL sched_debug.cpu#4.nr_switches 724402 ± 1% -16.7% 603369 ± 0% TOTAL sched_debug.cpu#7.nr_switches 712130 ± 2% -15.5% 602079 ± 1% TOTAL sched_debug.cpu#24.nr_switches 715409 ± 5% -16.1% 600473 ± 2% TOTAL sched_debug.cpu#22.nr_switches 169672 ± 0% +19.0% 201867 ± 0% TOTAL meminfo.Active(anon) 42418 ± 0% +18.9% 50446 ± 0% TOTAL proc-vmstat.nr_active_anon 710553 ± 3% -16.0% 597062 ± 1% TOTAL sched_debug.cpu#31.nr_switches 723251 ± 2% -17.1% 599549 ± 1% TOTAL sched_debug.cpu#18.nr_switches 706093 ± 4% -14.9% 601183 ± 1% TOTAL sched_debug.cpu#26.nr_switches 704669 ± 2% -14.9% 599338 ± 1% TOTAL sched_debug.cpu#20.nr_switches 703753 ± 3% -15.3% 596238 ± 1% TOTAL sched_debug.cpu#21.nr_switches 702440 ± 5% -15.1% 596380 ± 1% TOTAL sched_debug.cpu#25.nr_switches 702660 ± 4% -14.7% 599141 ± 1% TOTAL sched_debug.cpu#19.nr_switches 22 ±14% +28.2% 28 ± 4% TOTAL sched_debug.cfs_rq[12]:/.runnable_load_avg 22 ±22% +32.4% 29 ± 1% TOTAL sched_debug.cpu#34.load 701182 ± 4% -14.6% 598606 ± 1% TOTAL sched_debug.cpu#23.nr_switches 0.23 ± 5% -15.8% 0.19 ± 6% TOTAL turbostat.%pc3 238557 ± 0% +13.5% 270767 ± 0% TOTAL meminfo.Active 135003 ± 0% +12.7% 152182 ± 0% TOTAL sched_debug.cpu#56.nr_load_updates 135518 ± 1% +12.2% 152028 ± 0% TOTAL sched_debug.cpu#63.nr_load_updates 668473 ± 4% -10.5% 598372 ± 1% TOTAL sched_debug.cpu#29.nr_switches 137963 ± 0% +11.9% 154381 ± 0% TOTAL sched_debug.cpu#57.nr_load_updates 135802 ± 0% +12.1% 152194 ± 0% TOTAL sched_debug.cpu#58.nr_load_updates 673342 ± 3% -10.3% 604167 ± 2% TOTAL sched_debug.cpu#30.nr_switches 136075 ± 0% +11.8% 152117 ± 0% TOTAL sched_debug.cpu#60.nr_load_updates 136265 ± 0% +11.6% 152124 ± 0% TOTAL sched_debug.cpu#59.nr_load_updates 136657 ± 0% +11.5% 152399 ± 0% TOTAL sched_debug.cpu#48.nr_load_updates 672367 ± 4% -10.6% 600775 ± 2% TOTAL sched_debug.cpu#27.nr_switches 136400 ± 0% +11.6% 152176 ± 0% TOTAL sched_debug.cpu#61.nr_load_updates 136146 ± 0% +11.7% 152103 ± 0% TOTAL sched_debug.cpu#62.nr_load_updates 1848 ± 2% +10.9% 2049 ± 5% TOTAL numa-meminfo.node0.Mapped 136683 ± 0% +11.4% 152199 ± 0% TOTAL sched_debug.cpu#55.nr_load_updates 137175 ± 0% +11.0% 152310 ± 0% TOTAL sched_debug.cpu#51.nr_load_updates 139156 ± 0% +10.8% 154236 ± 0% TOTAL sched_debug.cpu#49.nr_load_updates 137076 ± 0% +11.1% 152253 ± 0% TOTAL sched_debug.cpu#53.nr_load_updates 672781 ± 3% -10.5% 601982 ± 1% TOTAL sched_debug.cpu#28.nr_switches 136873 ± 1% +11.2% 152248 ± 0% TOTAL sched_debug.cpu#52.nr_load_updates 137601 ± 0% +10.8% 152404 ± 0% TOTAL sched_debug.cpu#50.nr_load_updates 137409 ± 0% +10.8% 152228 ± 0% TOTAL sched_debug.cpu#54.nr_load_updates 608198 ± 1% +5895.2% 36462984 ± 0% TOTAL time.voluntary_context_switches 62.13 ± 1% +247.4% 215.86 ± 1% TOTAL time.user_time 560843 ± 0% -53.2% 262235 ± 0% TOTAL vmstat.system.cs 54.79 ± 0% +81.7% 99.56 ± 0% TOTAL turbostat.%c0 38022 ± 0% +76.6% 67144 ± 0% TOTAL vmstat.system.in 2726 ± 0% +68.6% 4596 ± 0% TOTAL time.percent_of_cpu_this_job_got 8155 ± 0% +67.2% 13638 ± 0% TOTAL time.system_time 951709 ± 0% +53.4% 1459554 ± 0% TOTAL time.involuntary_context_switches 227030 ± 1% -16.1% 190393 ± 0% TOTAL time.minor_page_faults qperf.sctp.bw 1.3e+09 ++---------------------------------------------------------------+ 1.2e+09 ++ O O O | O OO O O O O O O OO OO OO O O O O O | 1.1e+09 ++ OO OO O O OO OO O O O O OO O O 1e+09 ++ O | | | 9e+08 ++ | 8e+08 ++ | 7e+08 ++ | | | 6e+08 ++ | 5e+08 ++ | | .**. *. *. | 4e+08 *+**.**.*. *.**.**.*.**.**.*.**.**.**.* **.*.**.* * *.** | 3e+08 ++--------*------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample To reproduce: apt-get install ruby ruby-oj git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/setup-local job.yaml # the job file attached in this email bin/run-local job.yaml Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com

7 years, 7 months

1
0
0 / 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP October 2014