Greeting, FYI, we noticed a -5.1% regression of will-it-scale.per_thread_ops due to commit: commit: 00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f ("locking/rwsem: Always release wait_lock before waking up tasks") https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale on test machine: 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory with following parameters: nr_task: 50% mode: thread test: mmap2 ucode: 0x400001c cpufreq_governor: performance test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale In addition to that, the commit also has significant impact on the following tests: +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: boot-time.boot 2.6% regression | | test machine | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory | | test parameters | cpufreq_governor=performance | | | iterations=30 | | | nr_task=1600% | | | test=new_dbase | | | ucode=0x14 | +------------------+------------------------------------------------------------------------+ | testcase: change | aim7: | | test machine | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory | | test parameters | cpufreq_governor=performance | | | disk=4BRD_12G | | | fs=ext4 | | | load=3000 | | | md=RAID1 | | | test=disk_src | +------------------+------------------------------------------------------------------------+ | testcase: change | aim7: | | test machine | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory | | test parameters | cpufreq_governor=performance | | | disk=4BRD_12G | | | fs=xfs | | | load=3000 | | | md=RAID1 | | | test=disk_wrt | +------------------+------------------------------------------------------------------------+ | testcase: change | aim7: iostat.md0.wkB/s 56.8% improvement | | test machine | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory | | test parameters | cpufreq_governor=performance | | | disk=4BRD_12G | | | fs=xfs | | | load=3000 | | | md=RAID1 | | | test=disk_wrt | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: reaim.child_utime -1.6% undefined | | test machine | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory | | test parameters | cpufreq_governor=performance | | | iterations=20 | | | nr_task=1600% | | | test=aim9 | | | ucode=0x14 | +------------------+------------------------------------------------------------------------+ | testcase: change | unixbench: | | test machine | 192 threads Skylake-SP with 256G memory | | test parameters | cpufreq_governor=performance | | | nr_task=30% | | | runtime=300s | | | test=shell8 | +------------------+------------------------------------------------------------------------+ | testcase: change | will-it-scale: | | test machine | 192 threads Skylake-SP with 256G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=100% | | | test=pwrite2 | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: boot-time.boot -4.1% improvement | | test machine | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory | | test parameters | cpufreq_governor=performance | | | iterations=10 | | | nr_task=1600% | | | test=custom | | | ucode=0x14 | +------------------+------------------------------------------------------------------------+ | testcase: change | aim7: | | test machine | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory | | test parameters | cpufreq_governor=performance | | | load=2000 | | | test=fork_test | | | ucode=0xb000036 | +------------------+------------------------------------------------------------------------+ | testcase: change | aim7: | | test machine | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory | | test parameters | cpufreq_governor=performance | | | load=8000 | | | test=misc_rtns_1 | | | ucode=0xb000036 | +------------------+------------------------------------------------------------------------+ | testcase: change | will-it-scale: | | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=100% | | | test=unlink1 | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: reaim.child_systime 23.3% undefined | | test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | nr_task=1600% | | | runtime=300s | | | test=alltests | | | ucode=0x43 | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: reaim.child_systime -2.7% undefined | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory | | test parameters | cpufreq_governor=performance | | | nr_task=900t | | | runtime=300s | | | test=disk_cp | | | ucode=0x42e | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: | | test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | nr_task=1600% | | | runtime=300s | | | test=new_dbase | | | ucode=0x43 | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory | | test parameters | cpufreq_governor=performance | | | nr_task=300t | | | runtime=300s | | | test=disk_wrt | | | ucode=0x42e | +------------------+------------------------------------------------------------------------+ | testcase: change | vm-scalability: boot-time.boot 2.0% regression | | test machine | 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | runtime=300s | | | test=small-allocs | | | ucode=0xb000036 | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: boot-time.boot 3.3% regression | | test machine | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory | | test parameters | cpufreq_governor=performance | | | iterations=30 | | | nr_task=1600% | | | test=dbase | | | ucode=0x14 | +------------------+------------------------------------------------------------------------+ | testcase: change | aim7: | | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | load=2000 | | | test=fork_test | | | ucode=0xb000036 | +------------------+------------------------------------------------------------------------+ | testcase: change | aim7: | | test machine | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory | | test parameters | cpufreq_governor=performance | | | load=8000 | | | test=disk_src | | | ucode=0xb000036 | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory | | test parameters | cpufreq_governor=performance | | | nr_task=7000t | | | runtime=300s | | | test=shared_memory | | | ucode=0x42e | +------------------+------------------------------------------------------------------------+ | testcase: change | reaim: | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory | | test parameters | cpufreq_governor=performance | | | nr_task=400t | | | runtime=300s | | | test=disk_src | | | ucode=0x42e | +------------------+------------------------------------------------------------------------+ | testcase: change | fxmark: iostat.sda.wkB/s 1.8% improvement | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory | | test parameters | cpufreq_governor=performance | | | directio=directio | | | disk=1HDD | | | fstype=xfs | | | media=hdd | | | test=MWUL | | | ucode=0x42e | +------------------+------------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com&gt; Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.2-clear/thread/50%/clear-ota-25590-x86_64-2018-10-18.cgz/lkp-csl-2sp4/mmap2/will-it-scale/0x400001c commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 5762 -5.1% 5467 will-it-scale.per_thread_ops 176.00 +1.3% 178.25 will-it-scale.time.percent_of_cpu_this_job_got 521.49 +1.2% 527.84 will-it-scale.time.system_time 56180812 -2.1% 55004407 will-it-scale.time.voluntary_context_switches 276626 -5.1% 262480 will-it-scale.workload 6836 -1.8% 6710 boot-time.idle 4584795 -12.7% 4004010 meminfo.Committed_AS 1501 ± 5% -15.4% 1270 ± 5% slabinfo.task_group.active_objs 1501 ± 5% -15.4% 1270 ± 5% slabinfo.task_group.num_objs 371197 -4.5% 354429 ± 2% vmstat.system.cs 206566 -1.9% 202660 vmstat.system.in 4.91e+08 ± 11% -23.4% 3.762e+08 ± 4% cpuidle.C1.time 5335198 ± 3% -9.2% 4845346 ± 4% cpuidle.C1.usage 9660501 ± 2% -12.4% 8459367 ± 6% cpuidle.POLL.usage 5332157 ± 3% -9.2% 4843565 ± 4% turbostat.C1 1.69 ± 11% -0.4 1.26 ± 3% turbostat.C1% 0.39 ± 5% +184.1% 1.12 ± 59% turbostat.CPU%c6 2637533 ± 8% +18.5% 3124786 ± 5% numa-vmstat.node0.numa_hit 2621013 ± 8% +18.8% 3114498 ± 5% numa-vmstat.node0.numa_local 2195535 ± 12% -19.2% 1773999 ± 10% numa-vmstat.node1.numa_hit 986533 ± 27% -43.4% 558697 ± 33% numa-vmstat.node1.numa_local 258.33 ± 16% +27.8% 330.25 ± 8% sched_debug.cfs_rq:/.runnable_load_avg.max 256.92 ± 17% +28.5% 330.25 ± 8% sched_debug.cpu.cpu_load[0].max 276147 ± 30% +41.1% 389714 ± 7% sched_debug.cpu.load.max 59345 ± 3% +9.7% 65073 ± 9% sched_debug.cpu.max_idle_balance_cost.avg 2.60 ± 16% -17.3% 2.15 ± 13% sched_debug.cpu.nr_uninterruptible.stddev 8182 ±133% +541.6% 52501 ± 50% interrupts.CPU10.RES:Rescheduling_interrupts 72730 ± 18% -79.8% 14676 ± 60% interrupts.CPU30.RES:Rescheduling_interrupts 68164 ± 32% -56.6% 29579 ± 55% interrupts.CPU49.RES:Rescheduling_interrupts 115055 ± 36% -71.6% 32664 ± 49% interrupts.CPU52.RES:Rescheduling_interrupts 29073 ± 71% +150.2% 72747 ± 43% interrupts.CPU54.RES:Rescheduling_interrupts 54063 ± 38% -61.0% 21064 ± 54% interrupts.CPU67.RES:Rescheduling_interrupts 82518 ± 29% -69.0% 25542 ± 76% interrupts.CPU73.RES:Rescheduling_interrupts 2782 ± 31% +39.5% 3880 ± 8% interrupts.CPU77.CAL:Function_call_interrupts 24230 ± 22% +71.2% 41476 ± 33% interrupts.CPU79.RES:Rescheduling_interrupts 3298 ± 11% +18.4% 3906 ± 8% interrupts.CPU85.CAL:Function_call_interrupts 3200 ± 6% +22.0% 3903 ± 8% interrupts.CPU86.CAL:Function_call_interrupts 641.50 ± 27% +37.2% 880.00 ± 11% interrupts.CPU89.NMI:Non-maskable_interrupts 641.50 ± 27% +37.2% 880.00 ± 11% interrupts.CPU89.PMI:Performance_monitoring_interrupts 3241 ± 13% +20.3% 3900 ± 8% interrupts.CPU92.CAL:Function_call_interrupts 674.00 ± 20% +25.2% 844.00 ± 7% interrupts.CPU95.NMI:Non-maskable_interrupts 674.00 ± 20% +25.2% 844.00 ± 7% interrupts.CPU95.PMI:Performance_monitoring_interrupts 4571304 ± 9% -23.3% 3507482 ± 5% interrupts.RES:Rescheduling_interrupts 306.75 ± 8% +16.7% 358.00 ± 5% interrupts.TLB:TLB_shootdowns 10.88 +13.6% 12.36 ± 9% perf-stat.i.MPKI 1.174e+09 -3.6% 1.131e+09 ± 2% perf-stat.i.branch-instructions 375713 -4.2% 359844 ± 2% perf-stat.i.context-switches 1.88 +9.1% 2.05 ± 7% perf-stat.i.cpi 1.466e+09 -3.6% 1.413e+09 ± 2% perf-stat.i.dTLB-loads 8.112e+08 -4.2% 7.773e+08 ± 2% perf-stat.i.dTLB-stores 71.13 -1.2 69.98 perf-stat.i.iTLB-load-miss-rate% 5.361e+09 -3.8% 5.159e+09 ± 2% perf-stat.i.instructions 0.53 -3.9% 0.51 ± 2% perf-stat.i.ipc 1.86 +3.7% 1.93 ± 2% perf-stat.overall.cpi 71.15 -0.8 70.34 perf-stat.overall.iTLB-load-miss-rate% 0.54 -3.5% 0.52 ± 2% perf-stat.overall.ipc 5789520 +3.8% 6012086 perf-stat.overall.path-length 1.17e+09 -3.6% 1.128e+09 ± 2% perf-stat.ps.branch-instructions 374455 -4.2% 358648 ± 2% perf-stat.ps.context-switches 1.461e+09 -3.6% 1.408e+09 ± 2% perf-stat.ps.dTLB-loads 8.085e+08 -4.2% 7.747e+08 ± 2% perf-stat.ps.dTLB-stores 5.343e+09 -3.8% 5.142e+09 ± 2% perf-stat.ps.instructions 100352 ± 2% +13.0% 113434 ± 6% softirqs.CPU10.TIMER 101777 ± 3% +11.5% 113486 ± 5% softirqs.CPU11.TIMER 100055 ± 3% +10.0% 110075 ± 5% softirqs.CPU13.TIMER 101021 ± 3% +32.8% 134115 ± 16% softirqs.CPU15.TIMER 99491 ± 2% +13.5% 112882 ± 9% softirqs.CPU16.TIMER 100003 ± 2% +9.9% 109934 ± 5% softirqs.CPU17.TIMER 37527 ± 2% +15.4% 43321 ± 12% softirqs.CPU19.SCHED 103667 ± 4% +11.6% 115707 ± 4% softirqs.CPU2.TIMER 101609 ± 4% +9.0% 110781 ± 7% softirqs.CPU23.TIMER 97414 ± 2% +12.8% 109926 ± 7% softirqs.CPU25.TIMER 99800 ± 5% +10.8% 110574 ± 8% softirqs.CPU26.TIMER 99683 ± 5% +13.5% 113100 ± 6% softirqs.CPU27.TIMER 97869 ± 2% +10.8% 108412 ± 5% softirqs.CPU30.TIMER 98332 ± 3% +14.8% 112847 ± 7% softirqs.CPU33.TIMER 37531 ± 2% +12.7% 42283 ± 10% softirqs.CPU34.SCHED 99845 ± 6% +11.6% 111447 ± 6% softirqs.CPU34.TIMER 100067 ± 6% +12.8% 112851 ± 6% softirqs.CPU35.TIMER 98909 ± 4% +11.5% 110324 ± 6% softirqs.CPU36.TIMER 99760 ± 4% +11.1% 110797 ± 6% softirqs.CPU37.TIMER 99036 ± 2% +12.8% 111684 ± 6% softirqs.CPU38.TIMER 100075 ± 2% +11.4% 111475 ± 4% softirqs.CPU4.TIMER 100876 ± 6% +12.9% 113938 ± 7% softirqs.CPU41.TIMER 98943 ± 5% +12.1% 110934 ± 6% softirqs.CPU42.TIMER 37554 ± 2% +9.8% 41249 ± 4% softirqs.CPU43.SCHED 100322 ± 5% +13.3% 113660 ± 8% softirqs.CPU43.TIMER 39193 ± 6% +11.8% 43810 ± 11% softirqs.CPU44.SCHED 37486 ± 2% +12.0% 41992 ± 7% softirqs.CPU45.SCHED 99330 ± 5% +14.4% 113605 ± 5% softirqs.CPU45.TIMER 100692 ± 5% +12.0% 112743 ± 5% softirqs.CPU46.TIMER 37703 ± 3% +6.7% 40218 ± 3% softirqs.CPU68.SCHED 98362 ± 2% +10.5% 108699 ± 6% softirqs.CPU68.TIMER 36569 ± 4% +10.3% 40344 ± 6% softirqs.CPU74.SCHED 94622 ± 2% +11.3% 105361 ± 5% softirqs.CPU76.TIMER 35949 ± 3% +9.7% 39444 ± 3% softirqs.CPU78.SCHED 96464 +10.2% 106340 ± 4% softirqs.CPU80.TIMER 95793 ± 4% +9.2% 104568 ± 3% softirqs.CPU88.TIMER 96624 ± 2% +7.8% 104154 ± 5% softirqs.CPU89.TIMER 101648 ± 3% +10.5% 112304 ± 6% softirqs.CPU9.TIMER 36172 ± 3% +8.0% 39083 ± 6% softirqs.CPU91.SCHED 97133 ± 4% +11.4% 108210 ± 7% softirqs.CPU92.TIMER 35978 ± 3% +11.3% 40060 ± 4% softirqs.CPU93.SCHED 97301 ± 3% +20.1% 116847 ± 19% softirqs.CPU93.TIMER 95831 ± 2% +9.4% 104867 ± 4% softirqs.CPU95.TIMER 66.97 ± 2% -2.0 65.01 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 67.56 ± 2% -1.9 65.61 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 77.08 -1.9 75.15 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 77.07 -1.9 75.15 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 77.02 -1.9 75.10 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 78.27 -1.8 76.45 perf-profile.calltrace.cycles-pp.secondary_startup_64 59.20 -1.7 57.45 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 1.64 -0.2 1.40 ± 4% perf-profile.calltrace.cycles-pp.schedule.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap 1.57 ± 2% -0.2 1.34 ± 4% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.rwsem_down_write_slowpath.down_write_killable.__vm_munmap 1.18 ± 4% -0.1 1.06 ± 5% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 0.56 ± 4% +0.0 0.60 ± 4% perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 0.82 ± 2% +0.1 0.91 ± 6% perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.80 ± 6% +0.1 0.90 ± 7% perf-profile.calltrace.cycles-pp.find_vma.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 1.12 ± 3% +0.1 1.24 ± 2% perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 1.39 +0.1 1.53 ± 2% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending 0.86 ± 9% +0.2 1.01 ± 2% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap 1.52 +0.2 1.68 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle 1.54 +0.2 1.70 ± 3% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry 1.54 +0.2 1.71 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary 0.41 ± 58% +0.2 0.62 perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff 0.66 ± 8% +0.2 0.89 ± 5% perf-profile.calltrace.cycles-pp.unmapped_area_topdown.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap 2.28 ± 3% +0.2 2.51 ± 2% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.77 ± 9% +0.3 1.03 ± 3% perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff 0.88 ± 9% +0.3 1.13 ± 2% perf-profile.calltrace.cycles-pp.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 1.13 ± 4% +0.3 1.39 ± 2% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff 1.17 ± 5% +0.3 1.44 ± 2% perf-profile.calltrace.cycles-pp.schedule.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff 0.98 ± 7% +0.3 1.27 ± 2% perf-profile.calltrace.cycles-pp.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 3.32 ± 4% +0.3 3.65 ± 2% perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.13 ±173% +0.4 0.52 ± 2% perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 3.63 ± 4% +0.4 4.08 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.55 ± 3% perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 2.87 ± 4% +0.7 3.58 ± 2% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 3.10 ± 4% +0.7 3.83 ± 2% perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.07 ± 4% +1.3 11.35 ± 2% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.22 ± 4% +1.3 11.51 ± 2% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.46 ± 4% +1.8 22.31 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 20.40 ± 4% +1.8 22.24 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 77.08 -1.9 75.15 perf-profile.children.cycles-pp.start_secondary 68.72 -1.8 66.88 perf-profile.children.cycles-pp.cpuidle_enter 68.69 -1.8 66.86 perf-profile.children.cycles-pp.cpuidle_enter_state 78.27 -1.8 76.45 perf-profile.children.cycles-pp.secondary_startup_64 78.27 -1.8 76.45 perf-profile.children.cycles-pp.cpu_startup_entry 78.29 -1.8 76.48 perf-profile.children.cycles-pp.do_idle 59.20 -1.7 57.46 perf-profile.children.cycles-pp.intel_idle 0.64 ± 10% -0.2 0.47 ± 28% perf-profile.children.cycles-pp.tick_nohz_irq_exit 1.18 ± 4% -0.1 1.06 ± 5% perf-profile.children.cycles-pp.poll_idle 0.47 ± 10% -0.1 0.39 ± 13% perf-profile.children.cycles-pp.reschedule_interrupt 0.41 ± 11% -0.1 0.33 ± 13% perf-profile.children.cycles-pp.scheduler_ipi 0.12 ± 6% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.run_timer_softirq 0.26 -0.0 0.23 ± 6% perf-profile.children.cycles-pp.rcu_eqs_enter 0.36 -0.0 0.34 ± 4% perf-profile.children.cycles-pp.irq_enter 0.09 ± 5% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.kfree 0.07 ± 7% -0.0 0.05 perf-profile.children.cycles-pp.tlb_finish_mmu 0.07 ± 16% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.rcu_all_qs 0.06 ± 14% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.__list_del_entry_valid 0.15 ± 7% +0.0 0.17 ± 4% perf-profile.children.cycles-pp.fput_many 0.11 ± 6% +0.0 0.13 ± 12% perf-profile.children.cycles-pp.vma_merge 0.39 ± 3% +0.0 0.41 ± 2% perf-profile.children.cycles-pp.lapic_next_deadline 0.61 ± 2% +0.0 0.64 perf-profile.children.cycles-pp.native_write_msr 0.32 ± 2% +0.0 0.35 ± 3% perf-profile.children.cycles-pp.vma_compute_subtree_gap 0.19 ± 7% +0.0 0.22 ± 4% perf-profile.children.cycles-pp.__vm_enough_memory 0.48 ± 5% +0.0 0.52 perf-profile.children.cycles-pp.vma_link 0.36 ± 4% +0.0 0.40 ± 4% perf-profile.children.cycles-pp.find_next_bit 0.13 ± 14% +0.0 0.18 ± 2% perf-profile.children.cycles-pp._cond_resched 0.18 ± 12% +0.0 0.23 ± 2% perf-profile.children.cycles-pp.___might_sleep 1.02 ± 3% +0.1 1.12 ± 3% perf-profile.children.cycles-pp.rwsem_mark_wake 0.80 ± 6% +0.1 0.91 ± 7% perf-profile.children.cycles-pp.find_vma 1.13 ± 3% +0.1 1.25 ± 2% perf-profile.children.cycles-pp.unmap_region 0.67 ± 9% +0.1 0.78 perf-profile.children.cycles-pp._raw_spin_lock_irq 0.44 ± 4% +0.1 0.57 ± 2% perf-profile.children.cycles-pp.unmap_vmas 0.38 ± 6% +0.1 0.51 ± 4% perf-profile.children.cycles-pp.unmap_page_range 0.20 ± 35% +0.2 0.36 ± 6% perf-profile.children.cycles-pp.run_local_timers 0.66 ± 8% +0.2 0.89 ± 5% perf-profile.children.cycles-pp.unmapped_area_topdown 1.39 ± 8% +0.2 1.64 perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.78 ± 8% +0.3 1.04 ± 3% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.88 ± 9% +0.3 1.13 ± 2% perf-profile.children.cycles-pp.shmem_get_unmapped_area 0.99 ± 7% +0.3 1.27 ± 2% perf-profile.children.cycles-pp.get_unmapped_area 3.34 ± 4% +0.3 3.67 ± 2% perf-profile.children.cycles-pp.__do_munmap 3.64 ± 4% +0.5 4.09 perf-profile.children.cycles-pp.do_mmap 6.32 ± 4% +0.8 7.07 ± 2% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 6.81 ± 4% +0.8 7.62 ± 2% perf-profile.children.cycles-pp.down_write_killable 10.07 ± 4% +1.3 11.35 ± 2% perf-profile.children.cycles-pp.vm_mmap_pgoff 10.22 ± 4% +1.3 11.51 ± 2% perf-profile.children.cycles-pp.ksys_mmap_pgoff 20.46 ± 4% +1.8 22.31 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 20.41 ± 4% +1.8 22.26 ± 2% perf-profile.children.cycles-pp.do_syscall_64 59.19 -1.7 57.44 perf-profile.self.cycles-pp.intel_idle 1.14 ± 3% -0.1 1.03 ± 5% perf-profile.self.cycles-pp.poll_idle 0.15 ± 5% -0.0 0.12 ± 8% perf-profile.self.cycles-pp.call_cpuidle 0.06 ± 14% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.__list_del_entry_valid 0.11 ± 6% +0.0 0.13 ± 12% perf-profile.self.cycles-pp.vma_merge 0.04 ± 57% +0.0 0.06 ± 6% perf-profile.self.cycles-pp.rcu_all_qs 0.61 ± 2% +0.0 0.64 perf-profile.self.cycles-pp.native_write_msr 0.30 ± 2% +0.0 0.34 ± 2% perf-profile.self.cycles-pp.vma_compute_subtree_gap 0.28 ± 5% +0.0 0.32 ± 3% perf-profile.self.cycles-pp.__do_munmap 0.17 ± 13% +0.0 0.21 ± 3% perf-profile.self.cycles-pp.___might_sleep 0.38 ± 6% +0.0 0.42 perf-profile.self.cycles-pp.mmap_region 0.12 ± 9% +0.0 0.16 ± 22% perf-profile.self.cycles-pp.__account_scheduler_latency 0.44 ± 6% +0.0 0.48 ± 5% perf-profile.self.cycles-pp.down_write_killable 0.23 ± 9% +0.0 0.28 ± 4% perf-profile.self.cycles-pp.rwsem_mark_wake 0.22 ± 6% +0.1 0.29 ± 5% perf-profile.self.cycles-pp.unmap_page_range 0.65 ± 7% +0.1 0.74 ± 8% perf-profile.self.cycles-pp.find_vma 0.37 ± 9% +0.1 0.47 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.74 ± 6% +0.1 0.89 ± 4% perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.20 ± 35% +0.2 0.35 ± 8% perf-profile.self.cycles-pp.run_local_timers 0.66 ± 8% +0.2 0.88 ± 5% perf-profile.self.cycles-pp.unmapped_area_topdown 1.39 ± 8% +0.2 1.63 perf-profile.self.cycles-pp.rwsem_spin_on_owner will-it-scale.per_thread_ops 7000 +-+------------------------------------------------------------------+ | | 6000 +-++.+..+.+..+.+..+.+..+..+.+..+.+..+.+..+.+..+..+.+..+.+.. | | : O O O.O O O O 5000 +-+: | O : O O O O O O O O O O O O O O O | 4000 +-+O O O O O | | : | 3000 +-+ | |: | 2000 +-+ | |: | 1000 +-+ | | | 0 +-+------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory ========================================================================================= compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/30/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/new_dbase/reaim/0x14 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :2 dmesg.WARNING:at_ip_do_semtimedop/0x :4 50% 2:2 dmesg.WARNING:at_ip_perf_event_mmap_output/0x 1:4 -14% 1:2 perf-profile.children.cycles-pp.error_entry 1:4 -11% 0:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 1046913 -1.5% 1031046 reaim.max_jobs_per_min 193655 ± 19% -18.9% 157005 ± 2% softirqs.CPU43.TIMER 5.00 ± 61% +220.0% 16.00 ± 18% vmstat.procs.b 1.29 ± 70% +1.2 2.51 ± 4% turbostat.C1E% 66.48 +1.2% 67.31 turbostat.RAMWatt 37.58 ± 2% +2.6% 38.57 boot-time.boot 23.94 +0.7% 24.11 boot-time.dhcp 4655 ± 3% +3.2% 4801 boot-time.idle 9479168 ± 12% +27.5% 12090368 ± 8% meminfo.DirectMap2M 1601 ± 28% -50.9% 787.00 meminfo.Mlocked 114587 ± 7% +13.5% 130108 meminfo.Shmem 4681 ± 11% +17.6% 5507 ± 7% slabinfo.kmalloc-rcl-64.active_objs 4681 ± 11% +17.6% 5507 ± 7% slabinfo.kmalloc-rcl-64.num_objs 1076 ± 7% -15.7% 908.00 slabinfo.xfrm_state.active_objs 1076 ± 7% -15.7% 908.00 slabinfo.xfrm_state.num_objs 294089 ± 8% +7.1% 315115 ± 8% numa-meminfo.node1.FilePages 1741 ± 55% +80.2% 3139 ± 3% numa-meminfo.node1.Inactive 1559 ± 61% +88.6% 2939 ± 3% numa-meminfo.node1.Inactive(anon) 20963 ± 16% +65.1% 34612 ± 29% numa-meminfo.node1.KReclaimable 20963 ± 16% +65.1% 34612 ± 29% numa-meminfo.node1.SReclaimable 21840 ±132% +132.2% 50711 ± 86% numa-meminfo.node1.Shmem 446783 ± 17% -24.6% 337072 ± 3% numa-meminfo.node3.AnonPages 110.50 ± 32% -50.2% 55.00 ± 29% numa-vmstat.node0.nr_mlock 73537 ± 8% +7.0% 78692 ± 8% numa-vmstat.node1.nr_file_pages 388.50 ± 61% +89.6% 736.50 ± 3% numa-vmstat.node1.nr_inactive_anon 1668 ± 3% +9.5% 1827 ± 7% numa-vmstat.node1.nr_mapped 102.75 ± 44% -61.1% 40.00 ± 2% numa-vmstat.node1.nr_mlock 5475 ±132% +130.0% 12591 ± 86% numa-vmstat.node1.nr_shmem 5242 ± 16% +65.1% 8653 ± 29% numa-vmstat.node1.nr_slab_reclaimable 388.50 ± 61% +89.6% 736.50 ± 3% numa-vmstat.node1.nr_zone_inactive_anon 83.50 ± 28% -52.7% 39.50 numa-vmstat.node2.nr_mlock 112398 ± 16% -24.1% 85258 ± 4% numa-vmstat.node3.nr_anon_pages 284891 +1.3% 288657 proc-vmstat.nr_file_pages 5637 +2.4% 5770 proc-vmstat.nr_inactive_anon 401.25 ± 28% -52.0% 192.50 proc-vmstat.nr_mlock 28688 ± 7% +13.3% 32516 ± 2% proc-vmstat.nr_shmem 5637 +2.4% 5770 proc-vmstat.nr_zone_inactive_anon 7134 ± 61% +78.7% 12750 ± 6% proc-vmstat.numa_hint_faults 1.194e+09 +0.7% 1.203e+09 proc-vmstat.numa_hit 1.194e+09 +0.7% 1.203e+09 proc-vmstat.numa_local 54546 ± 2% +7.2% 58468 ± 3% proc-vmstat.pgactivate 1.204e+09 +0.7% 1.212e+09 proc-vmstat.pgalloc_normal 1.203e+09 +0.7% 1.212e+09 proc-vmstat.pgfree 571.58 ± 5% +15.9% 662.44 ± 2% sched_debug.cfs_rq:/.load_avg.max 0.42 ± 29% +100.0% 0.83 ± 6% sched_debug.cfs_rq:/.load_avg.min 0.52 ± 15% +24.0% 0.65 ± 4% sched_debug.cfs_rq:/.nr_running.avg 628.88 ± 13% +19.7% 752.45 ± 4% sched_debug.cfs_rq:/.util_avg.avg 1383 ± 9% +11.5% 1542 ± 6% sched_debug.cfs_rq:/.util_avg.max 248.75 ± 15% +33.8% 332.72 ± 8% sched_debug.cfs_rq:/.util_avg.min 119.07 ± 27% +35.0% 160.71 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.avg 0.06 ±173% +500.0% 0.33 ± 33% sched_debug.cfs_rq:/.util_est_enqueued.min 146725 ± 15% +34.6% 197510 ± 3% sched_debug.cpu.avg_idle.min 85.85 ± 10% +25.7% 107.90 ± 10% sched_debug.cpu.clock.stddev 85.85 ± 10% +25.7% 107.90 ± 10% sched_debug.cpu.clock_task.stddev 25195 ± 12% +18.2% 29793 ± 5% sched_debug.cpu.curr->pid.stddev 0.00 ± 9% +22.5% 0.00 ± 9% sched_debug.cpu.next_balance.stddev 10.13 ± 2% +13.9% 11.54 ± 5% sched_debug.cpu.nr_uninterruptible.avg 15162686 ± 75% +707.2% 1.224e+08 ± 79% syscalls.sys_brk.max 1.345e+08 ± 10% +2.6e+07 1.602e+08 ± 3% syscalls.sys_brk.noise.100% 1.471e+08 ± 12% +2.7e+07 1.741e+08 ± 2% syscalls.sys_brk.noise.2% 1.437e+08 ± 11% +2.7e+07 1.706e+08 ± 2% syscalls.sys_brk.noise.25% 1.467e+08 ± 12% +2.7e+07 1.736e+08 ± 2% syscalls.sys_brk.noise.5% 1.397e+08 ± 11% +2.7e+07 1.663e+08 ± 2% syscalls.sys_brk.noise.50% 1.367e+08 ± 11% +2.6e+07 1.626e+08 ± 2% syscalls.sys_brk.noise.75% 6.435e+08 ± 78% +108.2% 1.339e+09 syscalls.sys_mmap.max 320983 ±138% +5348.2% 17487893 ± 13% syscalls.sys_newstat.max 3342 ±101% +130.9% 7718 syscalls.sys_newstat.med 1.073e+08 ±100% +1.2e+08 2.271e+08 ± 3% syscalls.sys_newstat.noise.2% 1.01e+08 ±100% +1.1e+08 2.106e+08 ± 3% syscalls.sys_newstat.noise.25% 1.071e+08 ±100% +1.2e+08 2.263e+08 ± 3% syscalls.sys_newstat.noise.5% 87529237 ±100% +9.3e+07 1.802e+08 ± 3% syscalls.sys_newstat.noise.50% 23.08 ± 4% -12.8% 20.13 ± 3% perf-stat.i.MPKI 1.38 +0.1 1.45 perf-stat.i.cache-miss-rate% 84942 ± 2% +3.4% 87812 perf-stat.i.context-switches 1.89 ± 7% -17.3% 1.56 perf-stat.i.cpi 8139 ± 11% -24.0% 6183 ± 2% perf-stat.i.cycles-between-cache-misses 0.28 ± 12% -0.1 0.21 ± 6% perf-stat.i.dTLB-load-miss-rate% 0.14 -0.0 0.13 ± 2% perf-stat.i.dTLB-store-miss-rate% 58.16 ± 2% -2.2 55.97 perf-stat.i.iTLB-load-miss-rate% 10359697 ± 3% +6.6% 11042136 ± 3% perf-stat.i.iTLB-loads 0.79 +3.3% 0.82 perf-stat.i.ipc 6896869 ± 2% +2.9% 7098370 perf-stat.i.node-loads 8004284 ± 3% +4.2% 8341263 perf-stat.i.node-store-misses 7417146 ± 5% +8.7% 8061221 ± 3% perf-stat.i.node-stores 1.42 +0.0 1.44 perf-stat.overall.cache-miss-rate% 4567 -1.6% 4493 perf-stat.overall.cycles-between-cache-misses 10320774 ± 3% +6.5% 10994114 ± 3% perf-stat.ps.iTLB-loads 6871230 ± 2% +2.9% 7067176 perf-stat.ps.node-loads 7969678 ± 3% +4.1% 8297859 perf-stat.ps.node-store-misses 7413339 ± 5% +8.7% 8057926 ± 3% perf-stat.ps.node-stores 1296 ±106% -67.7% 418.50 ± 2% interrupts.45:PCI-MSI.1572864-edge.eth0-TxRx-0 440.50 ± 59% +324.0% 1867 ± 36% interrupts.48:PCI-MSI.1572867-edge.eth0-TxRx-3 266.50 ± 5% +51.4% 403.50 ± 28% interrupts.54:PCI-MSI.1572873-edge.eth0-TxRx-9 279.25 ± 8% +62.4% 453.50 ± 26% interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15 279.25 ± 6% -11.2% 248.00 interrupts.63:PCI-MSI.1572882-edge.eth0-TxRx-18 929442 ± 2% +3.4% 960727 interrupts.CAL:Function_call_interrupts 1296 ±106% -67.7% 418.50 ± 2% interrupts.CPU0.45:PCI-MSI.1572864-edge.eth0-TxRx-0 1788 ± 3% +13.5% 2030 ± 2% interrupts.CPU0.TLB:TLB_shootdowns 1773 ± 2% +21.2% 2149 ± 2% interrupts.CPU1.TLB:TLB_shootdowns 6356 ± 5% +7.1% 6808 ± 3% interrupts.CPU10.CAL:Function_call_interrupts 677.00 ±172% +408.1% 3440 interrupts.CPU10.NMI:Non-maskable_interrupts 677.00 ±172% +408.1% 3440 interrupts.CPU10.PMI:Performance_monitoring_interrupts 1766 +22.5% 2162 interrupts.CPU10.TLB:TLB_shootdowns 1834 ± 4% +15.6% 2120 ± 3% interrupts.CPU100.TLB:TLB_shootdowns 1539 ± 76% +235.8% 5168 ± 30% interrupts.CPU101.NMI:Non-maskable_interrupts 1539 ± 76% +235.8% 5168 ± 30% interrupts.CPU101.PMI:Performance_monitoring_interrupts 1810 ± 3% +15.4% 2088 ± 3% interrupts.CPU101.TLB:TLB_shootdowns 1842 ± 8% +11.8% 2060 ± 6% interrupts.CPU103.TLB:TLB_shootdowns 1796 ± 4% +13.0% 2029 ± 3% interrupts.CPU104.TLB:TLB_shootdowns 1835 ± 4% +16.2% 2132 interrupts.CPU105.TLB:TLB_shootdowns 1848 ± 5% +14.7% 2120 ± 2% interrupts.CPU106.TLB:TLB_shootdowns 1834 ± 2% +15.0% 2110 interrupts.CPU107.TLB:TLB_shootdowns 1825 ± 3% +14.5% 2090 interrupts.CPU108.TLB:TLB_shootdowns 1844 ± 4% +14.0% 2103 ± 2% interrupts.CPU109.TLB:TLB_shootdowns 6380 ± 6% +8.9% 6950 ± 2% interrupts.CPU11.CAL:Function_call_interrupts 694.50 ±166% +633.8% 5096 ± 32% interrupts.CPU11.NMI:Non-maskable_interrupts 694.50 ±166% +633.8% 5096 ± 32% interrupts.CPU11.PMI:Performance_monitoring_interrupts 1828 ± 4% +15.1% 2105 ± 4% interrupts.CPU11.TLB:TLB_shootdowns 1819 ± 5% +18.0% 2147 ± 6% interrupts.CPU110.TLB:TLB_shootdowns 3634 ± 58% -99.0% 36.00 ± 97% interrupts.CPU111.NMI:Non-maskable_interrupts 3634 ± 58% -99.0% 36.00 ± 97% interrupts.CPU111.PMI:Performance_monitoring_interrupts 1876 ± 8% +13.8% 2135 ± 5% interrupts.CPU111.TLB:TLB_shootdowns 5937 ± 12% -100.0% 0.50 ±100% interrupts.CPU112.NMI:Non-maskable_interrupts 5937 ± 12% -100.0% 0.50 ±100% interrupts.CPU112.PMI:Performance_monitoring_interrupts 1835 ± 6% +13.7% 2087 ± 4% interrupts.CPU112.TLB:TLB_shootdowns 5033 ± 18% -33.0% 3374 ± 2% interrupts.CPU113.NMI:Non-maskable_interrupts 5033 ± 18% -33.0% 3374 ± 2% interrupts.CPU113.PMI:Performance_monitoring_interrupts 1798 ± 4% +17.8% 2118 ± 4% interrupts.CPU113.TLB:TLB_shootdowns 1818 ± 3% +19.9% 2180 ± 3% interrupts.CPU114.TLB:TLB_shootdowns 1787 ± 3% +17.1% 2093 ± 4% interrupts.CPU115.TLB:TLB_shootdowns 5083 ± 28% +35.1% 6868 ± 3% interrupts.CPU116.NMI:Non-maskable_interrupts 5083 ± 28% +35.1% 6868 ± 3% interrupts.CPU116.PMI:Performance_monitoring_interrupts 2939 ± 60% +134.2% 6885 ± 2% interrupts.CPU117.NMI:Non-maskable_interrupts 2939 ± 60% +134.2% 6885 ± 2% interrupts.CPU117.PMI:Performance_monitoring_interrupts 1777 ± 2% +18.4% 2104 interrupts.CPU117.TLB:TLB_shootdowns 1799 ± 5% +22.8% 2209 ± 3% interrupts.CPU118.TLB:TLB_shootdowns 1861 ± 3% +17.8% 2193 interrupts.CPU119.TLB:TLB_shootdowns 1895 ± 7% +11.9% 2120 ± 2% interrupts.CPU12.TLB:TLB_shootdowns 1846 ± 3% +19.5% 2206 ± 4% interrupts.CPU120.TLB:TLB_shootdowns 1806 ± 5% +18.4% 2139 interrupts.CPU121.TLB:TLB_shootdowns 1856 ± 6% +15.7% 2148 ± 2% interrupts.CPU122.TLB:TLB_shootdowns 1768 ± 4% +21.7% 2152 ± 2% interrupts.CPU123.TLB:TLB_shootdowns 1821 ± 3% +13.5% 2067 ± 4% interrupts.CPU124.TLB:TLB_shootdowns 733.75 ±168% +354.2% 3333 ± 97% interrupts.CPU125.NMI:Non-maskable_interrupts 733.75 ±168% +354.2% 3333 ± 97% interrupts.CPU125.PMI:Performance_monitoring_interrupts 1828 ± 3% +14.4% 2091 interrupts.CPU125.TLB:TLB_shootdowns 1806 ± 4% +12.1% 2024 ± 4% interrupts.CPU126.TLB:TLB_shootdowns 1359 ±172% +279.4% 5156 ± 30% interrupts.CPU127.NMI:Non-maskable_interrupts 1359 ±172% +279.4% 5156 ± 30% interrupts.CPU127.PMI:Performance_monitoring_interrupts 1834 ± 5% +15.9% 2125 ± 3% interrupts.CPU127.TLB:TLB_shootdowns 1810 ± 3% +14.8% 2078 interrupts.CPU128.TLB:TLB_shootdowns 1841 ± 5% +14.3% 2105 interrupts.CPU129.TLB:TLB_shootdowns 5913 ± 12% -70.4% 1748 ± 99% interrupts.CPU13.NMI:Non-maskable_interrupts 5913 ± 12% -70.4% 1748 ± 99% interrupts.CPU13.PMI:Performance_monitoring_interrupts 1778 ± 4% +18.9% 2114 interrupts.CPU13.TLB:TLB_shootdowns 1861 ± 6% +11.7% 2080 ± 6% interrupts.CPU130.TLB:TLB_shootdowns 1768 +25.3% 2216 ± 2% interrupts.CPU131.TLB:TLB_shootdowns 1810 ± 3% +16.9% 2115 ± 4% interrupts.CPU132.TLB:TLB_shootdowns 2343 ± 87% +119.8% 5152 ± 30% interrupts.CPU133.NMI:Non-maskable_interrupts 2343 ± 87% +119.8% 5152 ± 30% interrupts.CPU133.PMI:Performance_monitoring_interrupts 1842 ± 5% +14.8% 2115 ± 3% interrupts.CPU133.TLB:TLB_shootdowns 1866 ± 6% +12.0% 2090 ± 3% interrupts.CPU134.TLB:TLB_shootdowns 1829 ± 3% +15.5% 2113 ± 2% interrupts.CPU135.TLB:TLB_shootdowns 1898 ± 5% +13.8% 2159 ± 3% interrupts.CPU136.TLB:TLB_shootdowns 1778 ± 3% +24.8% 2219 ± 6% interrupts.CPU137.TLB:TLB_shootdowns 1887 ± 5% +13.3% 2138 ± 3% interrupts.CPU138.TLB:TLB_shootdowns 9349 ± 2% +10.8% 10359 interrupts.CPU139.RES:Rescheduling_interrupts 1824 ± 3% +21.0% 2206 ± 2% interrupts.CPU139.TLB:TLB_shootdowns 1815 ± 3% +19.0% 2161 ± 3% interrupts.CPU14.TLB:TLB_shootdowns 1815 ± 6% +18.1% 2143 ± 4% interrupts.CPU140.TLB:TLB_shootdowns 1815 ± 3% +16.3% 2110 ± 3% interrupts.CPU141.TLB:TLB_shootdowns 1843 ± 5% +16.4% 2145 interrupts.CPU142.TLB:TLB_shootdowns 9028 +8.2% 9767 interrupts.CPU143.RES:Rescheduling_interrupts 1782 ± 3% +20.4% 2147 ± 2% interrupts.CPU143.TLB:TLB_shootdowns 279.25 ± 8% +62.4% 453.50 ± 26% interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15 6215 ± 2% +12.7% 7006 interrupts.CPU15.CAL:Function_call_interrupts 1608 ±141% +328.5% 6893 interrupts.CPU15.NMI:Non-maskable_interrupts 1608 ±141% +328.5% 6893 interrupts.CPU15.PMI:Performance_monitoring_interrupts 1885 ± 7% +12.8% 2128 ± 3% interrupts.CPU15.TLB:TLB_shootdowns 1784 ± 2% +21.3% 2165 ± 2% interrupts.CPU16.TLB:TLB_shootdowns 6329 ± 5% +9.3% 6917 ± 2% interrupts.CPU17.CAL:Function_call_interrupts 1808 ± 3% +17.2% 2120 ± 2% interrupts.CPU17.TLB:TLB_shootdowns 279.25 ± 6% -11.2% 248.00 interrupts.CPU18.63:PCI-MSI.1572882-edge.eth0-TxRx-18 1783 ± 5% +19.8% 2136 ± 6% interrupts.CPU18.TLB:TLB_shootdowns 6202 ± 3% +7.9% 6692 ± 3% interrupts.CPU19.CAL:Function_call_interrupts 1314 ±173% +165.9% 3495 ± 99% interrupts.CPU19.NMI:Non-maskable_interrupts 1314 ±173% +165.9% 3495 ± 99% interrupts.CPU19.PMI:Performance_monitoring_interrupts 1814 ± 7% +19.1% 2161 ± 3% interrupts.CPU19.TLB:TLB_shootdowns 6427 ± 5% +5.9% 6806 ± 3% interrupts.CPU2.CAL:Function_call_interrupts 697.75 ±162% +875.9% 6809 ± 2% interrupts.CPU2.NMI:Non-maskable_interrupts 697.75 ±162% +875.9% 6809 ± 2% interrupts.CPU2.PMI:Performance_monitoring_interrupts 1824 ± 4% +17.3% 2141 ± 5% interrupts.CPU2.TLB:TLB_shootdowns 1843 ± 6% +16.6% 2149 ± 3% interrupts.CPU20.TLB:TLB_shootdowns 3475 ± 65% +100.0% 6949 ± 2% interrupts.CPU21.NMI:Non-maskable_interrupts 3475 ± 65% +100.0% 6949 ± 2% interrupts.CPU21.PMI:Performance_monitoring_interrupts 1813 ± 4% +18.7% 2153 ± 2% interrupts.CPU21.TLB:TLB_shootdowns 1859 ± 3% +14.3% 2124 interrupts.CPU22.TLB:TLB_shootdowns 6549 ± 3% +6.3% 6962 ± 2% interrupts.CPU23.CAL:Function_call_interrupts 2528 ± 63% +170.2% 6833 ± 2% interrupts.CPU23.NMI:Non-maskable_interrupts 2528 ± 63% +170.2% 6833 ± 2% interrupts.CPU23.PMI:Performance_monitoring_interrupts 1837 ± 6% +17.0% 2150 ± 4% interrupts.CPU23.TLB:TLB_shootdowns 1808 ± 3% +15.7% 2093 ± 4% interrupts.CPU24.TLB:TLB_shootdowns 1816 ± 4% +18.0% 2142 ± 2% interrupts.CPU25.TLB:TLB_shootdowns 1812 ± 5% +17.0% 2120 interrupts.CPU26.TLB:TLB_shootdowns 1769 ± 3% +19.5% 2115 ± 7% interrupts.CPU27.TLB:TLB_shootdowns 1862 +12.5% 2094 interrupts.CPU28.TLB:TLB_shootdowns 1838 ± 5% +11.9% 2057 ± 5% interrupts.CPU29.TLB:TLB_shootdowns 440.50 ± 59% +324.0% 1867 ± 36% interrupts.CPU3.48:PCI-MSI.1572867-edge.eth0-TxRx-3 6171 ± 2% +9.6% 6764 ± 5% interrupts.CPU3.CAL:Function_call_interrupts 1280 ±173% +302.0% 5146 ± 32% interrupts.CPU3.NMI:Non-maskable_interrupts 1280 ±173% +302.0% 5146 ± 32% interrupts.CPU3.PMI:Performance_monitoring_interrupts 1769 ± 3% +20.8% 2137 ± 4% interrupts.CPU3.TLB:TLB_shootdowns 905.25 ±169% +471.1% 5169 ± 34% interrupts.CPU30.NMI:Non-maskable_interrupts 905.25 ±169% +471.1% 5169 ± 34% interrupts.CPU30.PMI:Performance_monitoring_interrupts 1798 ± 5% +19.2% 2145 ± 4% interrupts.CPU31.TLB:TLB_shootdowns 1821 ± 3% +19.5% 2176 ± 5% interrupts.CPU33.TLB:TLB_shootdowns 6256 ± 3% +12.4% 7031 interrupts.CPU34.CAL:Function_call_interrupts 252.00 ±172% +1926.4% 5106 ± 31% interrupts.CPU34.NMI:Non-maskable_interrupts 252.00 ±172% +1926.4% 5106 ± 31% interrupts.CPU34.PMI:Performance_monitoring_interrupts 1877 ± 5% +14.3% 2145 ± 8% interrupts.CPU34.TLB:TLB_shootdowns 1804 ± 6% +17.5% 2120 ± 7% interrupts.CPU35.TLB:TLB_shootdowns 6188 ± 4% +11.6% 6909 interrupts.CPU36.CAL:Function_call_interrupts 1347 ±173% +280.4% 5126 ± 36% interrupts.CPU36.NMI:Non-maskable_interrupts 1347 ±173% +280.4% 5126 ± 36% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1751 ± 4% +19.7% 2096 ± 3% interrupts.CPU36.TLB:TLB_shootdowns 1772 ± 4% +18.1% 2093 ± 5% interrupts.CPU37.TLB:TLB_shootdowns 1875 ± 3% +12.6% 2111 interrupts.CPU38.TLB:TLB_shootdowns 900.75 ±172% +675.1% 6981 ± 3% interrupts.CPU39.NMI:Non-maskable_interrupts 900.75 ±172% +675.1% 6981 ± 3% interrupts.CPU39.PMI:Performance_monitoring_interrupts 1818 ± 4% +15.9% 2106 ± 3% interrupts.CPU39.TLB:TLB_shootdowns 1827 ± 6% +17.2% 2142 ± 3% interrupts.CPU4.TLB:TLB_shootdowns 6197 ± 2% +13.4% 7027 interrupts.CPU40.CAL:Function_call_interrupts 1.25 ± 34% +5.4e+05% 6793 ± 3% interrupts.CPU40.NMI:Non-maskable_interrupts 1.25 ± 34% +5.4e+05% 6793 ± 3% interrupts.CPU40.PMI:Performance_monitoring_interrupts 1811 ± 3% +22.0% 2210 ± 2% interrupts.CPU40.TLB:TLB_shootdowns 1372 ±168% +406.5% 6951 interrupts.CPU41.NMI:Non-maskable_interrupts 1372 ±168% +406.5% 6951 interrupts.CPU41.PMI:Performance_monitoring_interrupts 1827 ± 6% +13.9% 2081 ± 4% interrupts.CPU41.TLB:TLB_shootdowns 2099 ±108% +227.5% 6874 ± 2% interrupts.CPU42.NMI:Non-maskable_interrupts 2099 ±108% +227.5% 6874 ± 2% interrupts.CPU42.PMI:Performance_monitoring_interrupts 1824 ± 5% +14.7% 2091 ± 4% interrupts.CPU42.TLB:TLB_shootdowns 2912 ± 92% +139.0% 6960 ± 2% interrupts.CPU43.NMI:Non-maskable_interrupts 2912 ± 92% +139.0% 6960 ± 2% interrupts.CPU43.PMI:Performance_monitoring_interrupts 1775 ± 2% +20.1% 2133 ± 2% interrupts.CPU43.TLB:TLB_shootdowns 1815 ± 4% +13.4% 2058 interrupts.CPU44.TLB:TLB_shootdowns 1803 ± 2% +19.9% 2162 ± 4% interrupts.CPU45.TLB:TLB_shootdowns 722.00 ±149% +856.6% 6906 ± 2% interrupts.CPU46.NMI:Non-maskable_interrupts 722.00 ±149% +856.6% 6906 ± 2% interrupts.CPU46.PMI:Performance_monitoring_interrupts 1821 ± 2% +20.6% 2196 ± 2% interrupts.CPU46.TLB:TLB_shootdowns 1350 ±173% +403.6% 6802 ± 3% interrupts.CPU47.NMI:Non-maskable_interrupts 1350 ±173% +403.6% 6802 ± 3% interrupts.CPU47.PMI:Performance_monitoring_interrupts 1784 ± 5% +16.2% 2073 ± 3% interrupts.CPU47.TLB:TLB_shootdowns 1.00 ± 70% +3.4e+05% 3397 ± 93% interrupts.CPU48.NMI:Non-maskable_interrupts 1.00 ± 70% +3.4e+05% 3397 ± 93% interrupts.CPU48.PMI:Performance_monitoring_interrupts 1919 ± 8% +10.4% 2119 ± 4% interrupts.CPU48.TLB:TLB_shootdowns 6184 +8.5% 6707 ± 3% interrupts.CPU49.CAL:Function_call_interrupts 6.50 ±155% +1.1e+05% 6876 interrupts.CPU49.NMI:Non-maskable_interrupts 6.50 ±155% +1.1e+05% 6876 interrupts.CPU49.PMI:Performance_monitoring_interrupts 1774 +20.6% 2139 ± 4% interrupts.CPU49.TLB:TLB_shootdowns 1809 ± 2% +15.6% 2091 ± 2% interrupts.CPU5.TLB:TLB_shootdowns 6218 +8.9% 6771 ± 4% interrupts.CPU50.CAL:Function_call_interrupts 4.25 ± 87% +1.6e+05% 6759 ± 3% interrupts.CPU50.NMI:Non-maskable_interrupts 4.25 ± 87% +1.6e+05% 6759 ± 3% interrupts.CPU50.PMI:Performance_monitoring_interrupts 1802 ± 3% +19.5% 2154 ± 2% interrupts.CPU50.TLB:TLB_shootdowns 1885 ± 2% +10.6% 2085 interrupts.CPU51.TLB:TLB_shootdowns 1818 ± 3% +20.2% 2187 interrupts.CPU52.TLB:TLB_shootdowns 1839 ± 6% +14.0% 2096 ± 8% interrupts.CPU53.TLB:TLB_shootdowns 6314 ± 4% +5.6% 6669 ± 3% interrupts.CPU54.CAL:Function_call_interrupts 1735 ± 4% +22.1% 2118 ± 2% interrupts.CPU54.TLB:TLB_shootdowns 4565 ± 59% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts 4565 ± 59% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts 1792 ± 4% +16.9% 2095 ± 3% interrupts.CPU55.TLB:TLB_shootdowns 10308 ± 2% +8.8% 11220 ± 2% interrupts.CPU56.RES:Rescheduling_interrupts 1848 ± 6% +13.7% 2102 ± 2% interrupts.CPU56.TLB:TLB_shootdowns 6284 ± 4% +9.1% 6854 interrupts.CPU57.CAL:Function_call_interrupts 1791 ±131% +283.6% 6871 interrupts.CPU57.NMI:Non-maskable_interrupts 1791 ±131% +283.6% 6871 interrupts.CPU57.PMI:Performance_monitoring_interrupts 10680 ± 4% +11.1% 11866 ± 2% interrupts.CPU57.RES:Rescheduling_interrupts 1776 +19.1% 2116 ± 3% interrupts.CPU57.TLB:TLB_shootdowns 1849 ± 3% +11.7% 2066 ± 2% interrupts.CPU58.TLB:TLB_shootdowns 1774 ± 6% +27.3% 2258 ± 4% interrupts.CPU59.TLB:TLB_shootdowns 1799 ± 4% +24.7% 2243 ± 7% interrupts.CPU6.TLB:TLB_shootdowns 1839 ± 5% +14.6% 2107 interrupts.CPU60.TLB:TLB_shootdowns 1819 ± 3% +11.4% 2027 ± 2% interrupts.CPU61.TLB:TLB_shootdowns 9953 ± 2% +14.5% 11400 ± 8% interrupts.CPU62.RES:Rescheduling_interrupts 1839 ± 2% +15.4% 2123 interrupts.CPU62.TLB:TLB_shootdowns 1840 ± 3% +13.8% 2093 ± 2% interrupts.CPU63.TLB:TLB_shootdowns 1829 ± 3% +20.9% 2212 ± 4% interrupts.CPU64.TLB:TLB_shootdowns 1802 ± 2% +16.6% 2100 interrupts.CPU65.TLB:TLB_shootdowns 1850 ± 4% +10.8% 2050 ± 7% interrupts.CPU66.TLB:TLB_shootdowns 1816 ± 5% +13.6% 2064 ± 4% interrupts.CPU67.TLB:TLB_shootdowns 1841 ± 4% +12.9% 2079 ± 2% interrupts.CPU68.TLB:TLB_shootdowns 6374 ± 2% +9.4% 6974 interrupts.CPU7.CAL:Function_call_interrupts 789.25 ±153% +531.0% 4980 ± 31% interrupts.CPU7.NMI:Non-maskable_interrupts 789.25 ±153% +531.0% 4980 ± 31% interrupts.CPU7.PMI:Performance_monitoring_interrupts 1886 ± 6% +12.4% 2119 ± 3% interrupts.CPU7.TLB:TLB_shootdowns 1841 ± 7% +13.0% 2080 interrupts.CPU70.TLB:TLB_shootdowns 1781 ± 3% +25.0% 2227 interrupts.CPU71.TLB:TLB_shootdowns 1805 ± 3% +17.9% 2129 ± 3% interrupts.CPU72.TLB:TLB_shootdowns 1844 ± 5% +16.8% 2153 interrupts.CPU73.TLB:TLB_shootdowns 1848 ± 5% +16.2% 2148 ± 2% interrupts.CPU74.TLB:TLB_shootdowns 1850 ± 3% +15.4% 2136 ± 5% interrupts.CPU75.TLB:TLB_shootdowns 1872 ± 4% +18.0% 2210 interrupts.CPU76.TLB:TLB_shootdowns 3894 ± 61% +75.1% 6818 ± 2% interrupts.CPU77.NMI:Non-maskable_interrupts 3894 ± 61% +75.1% 6818 ± 2% interrupts.CPU77.PMI:Performance_monitoring_interrupts 1778 ± 3% +18.2% 2102 ± 3% interrupts.CPU77.TLB:TLB_shootdowns 4300 ± 24% +58.6% 6819 interrupts.CPU78.NMI:Non-maskable_interrupts 4300 ± 24% +58.6% 6819 interrupts.CPU78.PMI:Performance_monitoring_interrupts 1799 +18.2% 2127 ± 4% interrupts.CPU78.TLB:TLB_shootdowns 4562 ± 41% -98.5% 69.50 ± 51% interrupts.CPU79.NMI:Non-maskable_interrupts 4562 ± 41% -98.5% 69.50 ± 51% interrupts.CPU79.PMI:Performance_monitoring_interrupts 1783 ± 3% +20.7% 2152 interrupts.CPU79.TLB:TLB_shootdowns 1843 +14.2% 2106 interrupts.CPU8.TLB:TLB_shootdowns 1769 ± 2% +19.6% 2115 interrupts.CPU80.TLB:TLB_shootdowns 1828 ± 5% +14.9% 2101 interrupts.CPU81.TLB:TLB_shootdowns 1844 +15.9% 2137 ± 4% interrupts.CPU82.TLB:TLB_shootdowns 1793 ± 5% +19.4% 2141 interrupts.CPU83.TLB:TLB_shootdowns 47.25 ±120% +704.2% 380.00 ± 80% interrupts.CPU84.NMI:Non-maskable_interrupts 47.25 ±120% +704.2% 380.00 ± 80% interrupts.CPU84.PMI:Performance_monitoring_interrupts 1834 ± 7% +19.1% 2184 interrupts.CPU85.TLB:TLB_shootdowns 1855 ± 3% +15.1% 2135 ± 3% interrupts.CPU86.TLB:TLB_shootdowns 3879 ± 69% -100.0% 1.50 ± 33% interrupts.CPU87.NMI:Non-maskable_interrupts 3879 ± 69% -100.0% 1.50 ± 33% interrupts.CPU87.PMI:Performance_monitoring_interrupts 1852 ± 2% +15.7% 2143 ± 4% interrupts.CPU87.TLB:TLB_shootdowns 1790 ± 5% +19.1% 2131 ± 4% interrupts.CPU88.TLB:TLB_shootdowns 1848 ± 6% +18.1% 2183 ± 4% interrupts.CPU89.TLB:TLB_shootdowns 266.50 ± 5% +51.4% 403.50 ± 28% interrupts.CPU9.54:PCI-MSI.1572873-edge.eth0-TxRx-9 1816 ± 3% +19.1% 2162 ± 2% interrupts.CPU9.TLB:TLB_shootdowns 1842 ± 4% +15.4% 2125 ± 5% interrupts.CPU90.TLB:TLB_shootdowns 1795 ± 4% +21.1% 2174 interrupts.CPU91.TLB:TLB_shootdowns 1804 ± 4% +17.5% 2120 interrupts.CPU92.TLB:TLB_shootdowns 1791 ± 5% +23.8% 2218 interrupts.CPU93.TLB:TLB_shootdowns 1874 ± 6% +14.3% 2142 ± 3% interrupts.CPU94.TLB:TLB_shootdowns 5182 ± 29% -100.0% 1.00 interrupts.CPU95.NMI:Non-maskable_interrupts 5182 ± 29% -100.0% 1.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts 1841 ± 5% +16.0% 2135 ± 4% interrupts.CPU95.TLB:TLB_shootdowns 1821 ± 4% +15.8% 2109 ± 3% interrupts.CPU96.TLB:TLB_shootdowns 1789 ± 2% +19.6% 2140 interrupts.CPU97.TLB:TLB_shootdowns 1789 ± 4% +22.3% 2188 ± 3% interrupts.CPU98.TLB:TLB_shootdowns 1797 ± 5% +19.4% 2146 ± 3% interrupts.CPU99.TLB:TLB_shootdowns 101.25 ± 23% +46.7% 148.50 ± 7% interrupts.IWI:IRQ_work_interrupts 401089 ± 11% +20.2% 481954 ± 6% interrupts.NMI:Non-maskable_interrupts 401089 ± 11% +20.2% 481954 ± 6% interrupts.PMI:Performance_monitoring_interrupts 262999 ± 3% +16.5% 306361 ± 3% interrupts.TLB:TLB_shootdowns 9.50 ±103% -9.5 0.00 perf-profile.calltrace.cycles-pp.sieve 0.51 ± 60% +0.3 0.81 perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit 0.55 ± 61% +0.3 0.89 perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.32 ±100% +0.4 0.68 ± 3% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit 0.31 ±100% +0.4 0.68 ± 3% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit 1.00 ± 31% +0.4 1.38 ± 6% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.unmap_region 1.02 ± 31% +0.4 1.41 ± 7% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.__do_munmap.mmap_region 1.02 ± 31% +0.4 1.40 ± 7% perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.unmap_region.__do_munmap 0.37 ±100% +0.4 0.77 ± 2% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit 0.14 ±173% +0.4 0.54 ± 3% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput 0.32 ±100% +0.4 0.73 ± 4% perf-profile.calltrace.cycles-pp.clear_page_erms.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma 0.14 ±173% +0.4 0.55 ± 2% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap 0.36 ±100% +0.4 0.79 perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu 1.10 ± 31% +0.4 1.52 ± 7% perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.mmap_region.do_mmap 0.37 ±100% +0.4 0.80 perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region 0.34 ±100% +0.4 0.77 ± 3% perf-profile.calltrace.cycles-pp.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page 1.14 ± 31% +0.4 1.58 ± 6% perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff 0.61 ± 63% +0.4 1.06 ± 3% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault 0.35 ±100% +0.4 0.79 ± 3% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma.__do_munmap 0.38 ±100% +0.4 0.83 perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap 0.35 ±100% +0.4 0.80 ± 3% perf-profile.calltrace.cycles-pp.down_write.__vma_adjust.__split_vma.__do_munmap.mmap_region 0.96 ± 38% +0.5 1.44 perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk 0.41 ±100% +0.5 0.90 ± 3% perf-profile.calltrace.cycles-pp.__split_vma.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff 0.40 ±100% +0.5 0.89 ± 4% perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.__do_munmap.mmap_region.do_mmap 0.97 ± 39% +0.5 1.46 perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 0.79 ± 64% +0.5 1.32 perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 0.76 ± 63% +0.5 1.31 ± 3% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault 0.83 ± 63% +0.6 1.43 perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault 1.26 ± 38% +0.6 1.90 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.93 ± 66% +0.7 1.64 perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.77 ± 31% +0.7 2.50 ± 5% perf-profile.calltrace.cycles-pp.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 1.98 ± 25% +0.8 2.74 perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 1.18 ± 67% +0.9 2.09 perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.50 ± 21% +1.1 4.61 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 3.34 ± 34% +1.5 4.86 perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 2.51 ± 49% +1.6 4.06 perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.87 ± 35% +1.8 5.63 perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 4.00 ± 35% +1.8 5.81 perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 4.04 ± 35% +1.8 5.87 perf-profile.calltrace.cycles-pp.page_fault 5.13 ± 31% +2.0 7.17 perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 5.16 ± 31% +2.0 7.21 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.18 ± 31% +2.1 7.23 perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.19 ± 31% +2.1 7.25 perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.34 ± 30% +2.1 7.41 ± 3% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit 5.58 ± 30% +2.1 7.73 ± 3% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit 6.82 ± 29% +2.5 9.36 ± 2% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 6.81 ± 29% +2.5 9.35 ± 2% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 7.25 ± 28% +2.7 9.93 ± 2% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.25 ± 28% +2.7 9.93 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.25 ± 28% +2.7 9.93 ± 2% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.01 ± 33% +3.3 11.32 ± 2% perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.12 ± 33% +3.4 11.47 ± 2% perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.72 ± 34% +3.4 12.12 ± 2% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.dup_mm.copy_process 8.99 ± 33% +3.5 12.52 ± 2% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.dup_mm.copy_process._do_fork 9.03 ± 33% +3.5 12.58 ± 2% perf-profile.calltrace.cycles-pp.down_write.dup_mm.copy_process._do_fork.do_syscall_64 10.29 ± 33% +4.0 14.31 ± 2% perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.92 ± 38% +15.3 49.23 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.96 ± 38% +15.3 49.30 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 9.51 ±103% -9.5 0.00 perf-profile.children.cycles-pp.sieve 0.07 ± 19% +0.0 0.09 perf-profile.children.cycles-pp.strlcpy 0.06 ± 20% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.__do_fault 0.06 ± 20% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.__tlb_remove_page_size 0.10 ± 12% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.__mod_lruvec_state 0.07 ± 17% +0.0 0.09 ± 11% perf-profile.children.cycles-pp.__list_add_valid 0.07 ± 17% +0.0 0.09 perf-profile.children.cycles-pp.mem_cgroup_charge_statistics 0.07 ± 20% +0.0 0.10 perf-profile.children.cycles-pp.__check_object_size 0.04 ± 58% +0.0 0.07 perf-profile.children.cycles-pp.mem_cgroup_update_lru_size 0.09 ± 24% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.down_read 0.08 ± 19% +0.0 0.11 ± 9% perf-profile.children.cycles-pp.strncpy_from_user 0.11 ± 21% +0.0 0.14 ± 11% perf-profile.children.cycles-pp.new_sync_read 0.08 ± 22% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.finish_task_switch 0.08 ± 21% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.down_write_killable 0.07 ± 20% +0.0 0.10 ± 14% perf-profile.children.cycles-pp.do_notify_parent 0.08 ± 23% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.try_charge 0.09 ± 20% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.__fxstat64 0.08 ± 23% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.mem_cgroup_uncharge_list 0.07 ± 26% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.find_get_entry 0.05 ± 60% +0.0 0.08 perf-profile.children.cycles-pp.free_pgd_range 0.08 ± 26% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.__count_memcg_events 0.05 ± 61% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.d_alloc_parallel 0.03 ±100% +0.0 0.06 perf-profile.children.cycles-pp.inode_permission 0.03 ±100% +0.0 0.06 perf-profile.children.cycles-pp.move_queued_task 0.11 ± 15% +0.0 0.14 ± 7% perf-profile.children.cycles-pp.free_pcppages_bulk 0.11 ± 20% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.page_add_file_rmap 0.10 ± 25% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.memcpy_erms 0.10 ± 28% +0.0 0.13 ± 7% perf-profile.children.cycles-pp.stack_trace_save_tsk 0.04 ± 59% +0.0 0.08 perf-profile.children.cycles-pp.shift_arg_pages 0.10 ± 23% +0.0 0.14 ± 7% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.11 ± 22% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.do_open_execat 0.10 ± 23% +0.0 0.14 ± 7% perf-profile.children.cycles-pp.sched_ttwu_pending 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.free_p4d_range 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.__update_load_avg_se 0.12 ± 27% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair 0.12 ± 26% +0.0 0.16 ± 6% perf-profile.children.cycles-pp.dput 0.11 ± 28% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.__d_lookup_rcu 0.10 ± 25% +0.0 0.14 perf-profile.children.cycles-pp.time 0.10 ± 17% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.__x64_sys_munmap 0.06 ± 58% +0.0 0.10 perf-profile.children.cycles-pp.cap_vm_enough_memory 0.11 ± 28% +0.0 0.15 ± 6% perf-profile.children.cycles-pp.__account_scheduler_latency 0.11 ± 23% +0.0 0.15 perf-profile.children.cycles-pp.__might_sleep 0.07 ± 24% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.lookup_slow 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.__ctype_init 0.09 ± 29% +0.0 0.13 perf-profile.children.cycles-pp.setup_arg_pages 0.03 ±100% +0.0 0.08 ± 6% perf-profile.children.cycles-pp._IO_file_open 0.12 ± 24% +0.0 0.17 ± 5% perf-profile.children.cycles-pp.do_task_dead 0.10 ± 23% +0.0 0.15 perf-profile.children.cycles-pp.__rb_insert_augmented 0.05 ± 62% +0.0 0.10 perf-profile.children.cycles-pp.__lookup_slow 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.__wake_up_common_lock 0.01 ±173% +0.0 0.06 perf-profile.children.cycles-pp.khugepaged_enter_vma_merge 0.15 ± 21% +0.0 0.20 perf-profile.children.cycles-pp.copy_user_highpage 0.15 ± 21% +0.0 0.20 perf-profile.children.cycles-pp.copy_page 0.03 ±100% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.__wake_up_common 0.11 ± 25% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.refcount_dec_not_one 0.04 ±100% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.rcu_all_qs 0.05 ± 61% +0.1 0.10 ± 19% perf-profile.children.cycles-pp.__vma_link_rb 0.00 +0.1 0.05 perf-profile.children.cycles-pp.filemap_fault 0.00 +0.1 0.05 perf-profile.children.cycles-pp.mark_page_accessed 0.00 +0.1 0.05 perf-profile.children.cycles-pp._exit 0.00 +0.1 0.05 perf-profile.children.cycles-pp.memmem 0.00 +0.1 0.05 perf-profile.children.cycles-pp.perf_exclude_event 0.14 ± 27% +0.1 0.19 ± 5% perf-profile.children.cycles-pp._cond_resched 0.14 ± 26% +0.1 0.20 ± 2% perf-profile.children.cycles-pp.selinux_inode_permission 0.09 ± 31% +0.1 0.15 ± 31% perf-profile.children.cycles-pp.__generic_file_write_iter 0.01 ±173% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.__slab_alloc 0.15 ± 24% +0.1 0.20 ± 4% perf-profile.children.cycles-pp.update_curr 0.10 ± 29% +0.1 0.15 ± 6% perf-profile.children.cycles-pp.free_uid 0.10 ± 29% +0.1 0.15 ± 6% perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave 0.14 ± 26% +0.1 0.20 ± 23% perf-profile.children.cycles-pp.new_sync_write 0.10 ± 32% +0.1 0.15 ± 29% perf-profile.children.cycles-pp.generic_file_write_iter 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.__mod_zone_page_state 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.___slab_alloc 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.uncharge_batch 0.16 ± 27% +0.1 0.22 ± 20% perf-profile.children.cycles-pp.vfs_write 0.14 ± 23% +0.1 0.20 perf-profile.children.cycles-pp.do_faccessat 0.13 ± 25% +0.1 0.19 perf-profile.children.cycles-pp.remove_vma 0.12 ± 24% +0.1 0.18 perf-profile.children.cycles-pp.strnlen_user 0.12 ± 22% +0.1 0.17 ± 2% perf-profile.children.cycles-pp.mem_cgroup_commit_charge 0.15 ± 29% +0.1 0.21 ± 4% perf-profile.children.cycles-pp.security_inode_permission 0.16 ± 29% +0.1 0.21 ± 2% perf-profile.children.cycles-pp.anon_vma_interval_tree_insert 0.17 ± 28% +0.1 0.23 ± 2% perf-profile.children.cycles-pp.__clear_user 0.15 ± 27% +0.1 0.21 ± 4% perf-profile.children.cycles-pp.wake_up_new_task 0.12 ± 22% +0.1 0.18 ± 11% perf-profile.children.cycles-pp.anon_vma_interval_tree_remove 0.00 +0.1 0.06 perf-profile.children.cycles-pp.unmap_single_vma 0.17 ± 26% +0.1 0.23 perf-profile.children.cycles-pp.lookup_fast 0.16 ± 27% +0.1 0.22 ± 6% perf-profile.children.cycles-pp.sched_exec 0.12 ± 24% +0.1 0.18 ± 5% perf-profile.children.cycles-pp.perf_event_mmap_output 0.16 ± 25% +0.1 0.23 ± 20% perf-profile.children.cycles-pp.ksys_write 0.20 ± 26% +0.1 0.26 perf-profile.children.cycles-pp.free_pages_and_swap_cache 0.20 ± 16% +0.1 0.27 ± 5% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.20 ± 16% +0.1 0.27 ± 5% perf-profile.children.cycles-pp.lru_add_drain 0.24 ± 19% +0.1 0.31 ± 9% perf-profile.children.cycles-pp.free_unref_page_list 0.20 ± 29% +0.1 0.27 perf-profile.children.cycles-pp.enqueue_entity 0.18 ± 25% +0.1 0.25 perf-profile.children.cycles-pp.___might_sleep 0.25 ± 20% +0.1 0.33 ± 3% perf-profile.children.cycles-pp.get_unmapped_area 0.21 ± 28% +0.1 0.30 ± 5% perf-profile.children.cycles-pp.kmem_cache_alloc 0.23 ± 29% +0.1 0.31 ± 3% perf-profile.children.cycles-pp.enqueue_task_fair 0.19 ± 31% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.smpboot_thread_fn 0.31 ± 20% +0.1 0.41 ± 2% perf-profile.children.cycles-pp.wp_page_copy 0.23 ± 24% +0.1 0.33 perf-profile.children.cycles-pp.alloc_set_pte 0.29 ± 28% +0.1 0.39 perf-profile.children.cycles-pp.mmap64 0.25 ± 24% +0.1 0.35 perf-profile.children.cycles-pp.walk_component 0.31 ± 25% +0.1 0.42 ± 3% perf-profile.children.cycles-pp.find_vma 0.34 ± 23% +0.1 0.45 perf-profile.children.cycles-pp.link_path_walk 0.17 ± 25% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.put_cred_rcu 0.31 ± 30% +0.1 0.42 ± 4% perf-profile.children.cycles-pp._dl_addr 0.29 ± 25% +0.1 0.41 ± 3% perf-profile.children.cycles-pp.select_task_rq_fair 0.30 ± 26% +0.1 0.42 ± 3% perf-profile.children.cycles-pp.copy_strings 0.39 ± 18% +0.1 0.50 ± 2% perf-profile.children.cycles-pp.do_wp_page 0.31 ± 19% +0.1 0.42 ± 3% perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.31 ± 23% +0.1 0.43 ± 4% perf-profile.children.cycles-pp.perf_iterate_sb 0.36 ± 21% +0.1 0.48 ± 8% perf-profile.children.cycles-pp.iterate_supers 0.46 ± 19% +0.1 0.58 ± 6% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.38 ± 22% +0.1 0.50 ± 8% perf-profile.children.cycles-pp.__x64_sys_sync 0.38 ± 22% +0.1 0.50 ± 8% perf-profile.children.cycles-pp.ksys_sync 0.39 ± 17% +0.1 0.52 ± 2% perf-profile.children.cycles-pp.__lru_cache_add 0.38 ± 23% +0.1 0.51 ± 7% perf-profile.children.cycles-pp.sync 0.35 ± 29% +0.1 0.49 ± 2% perf-profile.children.cycles-pp.vma_interval_tree_insert 0.47 ± 22% +0.1 0.61 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.36 ± 21% +0.1 0.51 ± 6% perf-profile.children.cycles-pp.vma_merge 0.33 ± 27% +0.2 0.48 ± 2% perf-profile.children.cycles-pp.rcu_core 0.43 ± 27% +0.2 0.58 perf-profile.children.cycles-pp.copy_p4d_range 0.44 ± 28% +0.2 0.60 perf-profile.children.cycles-pp.copy_page_range 0.54 ± 15% +0.2 0.71 perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.57 ± 22% +0.2 0.77 perf-profile.children.cycles-pp.perf_event_mmap 0.61 ± 27% +0.2 0.81 ± 6% perf-profile.children.cycles-pp.load_balance 0.68 ± 21% +0.2 0.89 perf-profile.children.cycles-pp.native_flush_tlb_one_user 0.57 ± 29% +0.2 0.78 ± 3% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.70 ± 20% +0.2 0.92 perf-profile.children.cycles-pp.flush_tlb_func_common 0.76 ± 20% +0.2 1.00 ± 2% perf-profile.children.cycles-pp.flush_tlb_mm_range 0.64 ± 27% +0.2 0.89 ± 9% perf-profile.children.cycles-pp.pick_next_task_fair 0.60 ± 26% +0.3 0.85 ± 12% perf-profile.children.cycles-pp.do_sys_open 0.79 ± 20% +0.3 1.06 ± 3% perf-profile.children.cycles-pp.clear_page_erms 0.63 ± 26% +0.3 0.89 ± 12% perf-profile.children.cycles-pp.do_filp_open 0.62 ± 27% +0.3 0.88 ± 12% perf-profile.children.cycles-pp.path_openat 0.68 ± 27% +0.3 0.95 ± 8% perf-profile.children.cycles-pp.schedule 0.79 ± 25% +0.3 1.06 ± 4% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.84 ± 20% +0.3 1.11 ± 3% perf-profile.children.cycles-pp.prep_new_page 0.77 ± 33% +0.3 1.08 ± 5% perf-profile.children.cycles-pp.rwsem_wake 1.09 ± 22% +0.3 1.40 perf-profile.children.cycles-pp.prepare_exit_to_usermode 1.15 ± 20% +0.3 1.46 perf-profile.children.cycles-pp.release_pages 0.70 ± 32% +0.3 1.05 ± 4% perf-profile.children.cycles-pp.wake_up_q 0.94 ± 27% +0.4 1.29 ± 8% perf-profile.children.cycles-pp.__schedule 1.22 ± 19% +0.4 1.57 perf-profile.children.cycles-pp.alloc_pages_vma 1.22 ± 21% +0.4 1.59 ± 3% perf-profile.children.cycles-pp.get_page_from_freelist 0.76 ± 31% +0.4 1.13 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 1.29 ± 20% +0.4 1.69 perf-profile.children.cycles-pp.do_brk_flags 1.19 ± 26% +0.4 1.60 perf-profile.children.cycles-pp.filemap_map_pages 1.46 ± 21% +0.4 1.91 ± 2% perf-profile.children.cycles-pp.__alloc_pages_nodemask 1.27 ± 24% +0.4 1.71 perf-profile.children.cycles-pp.unmap_page_range 1.33 ± 24% +0.5 1.81 perf-profile.children.cycles-pp.unmap_vmas 1.98 ± 21% +0.6 2.54 perf-profile.children.cycles-pp.tlb_finish_mmu 1.95 ± 21% +0.6 2.52 perf-profile.children.cycles-pp.tlb_flush_mmu 2.19 ± 19% +0.7 2.85 perf-profile.children.cycles-pp.do_anonymous_page 2.88 ± 24% +0.9 3.78 ± 3% perf-profile.children.cycles-pp.unmap_region 3.15 ± 20% +1.0 4.11 perf-profile.children.cycles-pp.__x64_sys_brk 3.91 ± 25% +1.3 5.20 ± 2% perf-profile.children.cycles-pp.__do_munmap 4.39 ± 21% +1.4 5.79 perf-profile.children.cycles-pp.__handle_mm_fault 4.61 ± 21% +1.5 6.08 perf-profile.children.cycles-pp.handle_mm_fault 4.53 ± 31% +1.5 6.04 ± 3% perf-profile.children.cycles-pp.__vma_adjust 5.54 ± 19% +1.6 7.16 perf-profile.children.cycles-pp.page_fault 5.20 ± 22% +1.6 6.85 perf-profile.children.cycles-pp.__do_page_fault 5.35 ± 22% +1.7 7.04 perf-profile.children.cycles-pp.do_page_fault 5.42 ± 31% +2.2 7.58 perf-profile.children.cycles-pp.ksys_mmap_pgoff 7.28 ± 28% +2.7 9.98 ± 2% perf-profile.children.cycles-pp.__x64_sys_exit_group 7.28 ± 28% +2.7 9.98 ± 2% perf-profile.children.cycles-pp.do_group_exit 7.28 ± 28% +2.7 9.98 ± 2% perf-profile.children.cycles-pp.do_exit 10.30 ± 33% +4.0 14.31 ± 2% perf-profile.children.cycles-pp.dup_mm 10.75 ± 33% +4.1 14.87 ± 2% perf-profile.children.cycles-pp.copy_process 10.93 ± 33% +4.2 15.12 ± 2% perf-profile.children.cycles-pp._do_fork 29.33 ± 31% +9.7 39.08 perf-profile.children.cycles-pp.rwsem_down_write_slowpath 29.57 ± 31% +9.9 39.42 perf-profile.children.cycles-pp.down_write 45.34 ± 29% +15.0 60.32 perf-profile.children.cycles-pp.do_syscall_64 45.39 ± 29% +15.0 60.38 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 9.39 ±103% -9.4 0.00 perf-profile.self.cycles-pp.sieve 0.08 ± 19% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.sync_regs 0.09 ± 21% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.page_add_file_rmap 0.08 ± 19% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.unlock_page 0.08 ± 16% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.do_brk_flags 0.05 ± 58% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.__mod_memcg_state 0.07 ± 23% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.unlink_anon_vmas 0.08 ± 24% +0.0 0.11 perf-profile.self.cycles-pp.__count_memcg_events 0.08 ± 24% +0.0 0.11 ± 9% perf-profile.self.cycles-pp.perf_event_mmap_output 0.09 ± 20% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.update_curr 0.10 ± 23% +0.0 0.13 ± 7% perf-profile.self.cycles-pp.memcpy_erms 0.05 ± 60% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.free_pcppages_bulk 0.09 ± 29% +0.0 0.12 ± 12% perf-profile.self.cycles-pp.vma_compute_subtree_gap 0.03 ±100% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.rcu_all_qs 0.10 ± 27% +0.0 0.14 perf-profile.self.cycles-pp.__might_sleep 0.03 ±100% +0.0 0.07 perf-profile.self.cycles-pp.dup_mm 0.13 ± 26% +0.0 0.17 ± 5% perf-profile.self.cycles-pp.vma_interval_tree_remove 0.04 ± 58% +0.0 0.08 ± 5% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.down_write_killable 0.09 ± 25% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.strnlen_user 0.09 ± 22% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.__rb_insert_augmented 0.16 ± 18% +0.0 0.21 ± 2% perf-profile.self.cycles-pp.perf_iterate_sb 0.14 ± 23% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.perf_event_mmap 0.15 ± 20% +0.0 0.20 ± 2% perf-profile.self.cycles-pp.copy_page 0.19 ± 20% +0.0 0.24 ± 4% perf-profile.self.cycles-pp.__list_del_entry_valid 0.05 ± 62% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.put_cred_rcu 0.13 ± 18% +0.0 0.18 perf-profile.self.cycles-pp.__vma_adjust 0.11 ± 25% +0.0 0.15 ± 3% perf-profile.self.cycles-pp.refcount_dec_not_one 0.00 +0.1 0.05 perf-profile.self.cycles-pp.___slab_alloc 0.00 +0.1 0.05 perf-profile.self.cycles-pp.pmd_devmap_trans_unstable 0.00 +0.1 0.05 perf-profile.self.cycles-pp._vm_normal_page 0.00 +0.1 0.05 perf-profile.self.cycles-pp.mark_page_accessed 0.11 ± 24% +0.1 0.16 ± 9% perf-profile.self.cycles-pp.anon_vma_interval_tree_remove 0.20 ± 20% +0.1 0.25 ± 4% perf-profile.self.cycles-pp.ftrace_syscall_enter 0.15 ± 23% +0.1 0.20 perf-profile.self.cycles-pp.find_vma 0.15 ± 25% +0.1 0.21 ± 2% perf-profile.self.cycles-pp.__slab_free 0.12 ± 17% +0.1 0.17 ± 2% perf-profile.self.cycles-pp.handle_mm_fault 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.__mod_zone_page_state 0.17 ± 20% +0.1 0.23 ± 2% perf-profile.self.cycles-pp.__do_page_fault 0.00 +0.1 0.06 perf-profile.self.cycles-pp.unmap_single_vma 0.15 ± 31% +0.1 0.21 ± 2% perf-profile.self.cycles-pp.anon_vma_interval_tree_insert 0.19 ± 20% +0.1 0.26 perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.17 ± 26% +0.1 0.24 perf-profile.self.cycles-pp.___might_sleep 0.18 ± 24% +0.1 0.25 perf-profile.self.cycles-pp.down_write 0.13 ± 28% +0.1 0.20 ± 4% perf-profile.self.cycles-pp.try_to_wake_up 0.21 ± 25% +0.1 0.29 perf-profile.self.cycles-pp.select_task_rq_fair 0.23 ± 27% +0.1 0.33 perf-profile.self.cycles-pp.copy_p4d_range 0.28 ± 19% +0.1 0.38 perf-profile.self.cycles-pp.__handle_mm_fault 0.29 ± 30% +0.1 0.41 ± 3% perf-profile.self.cycles-pp._dl_addr 0.35 ± 29% +0.1 0.48 ± 2% perf-profile.self.cycles-pp.vma_interval_tree_insert 0.45 ± 21% +0.1 0.57 ± 6% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.37 ± 23% +0.1 0.51 perf-profile.self.cycles-pp._raw_spin_lock 0.37 ± 27% +0.1 0.51 ± 8% perf-profile.self.cycles-pp.update_sd_lb_stats 0.47 ± 22% +0.1 0.61 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.45 ± 28% +0.2 0.62 ± 3% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.55 ± 23% +0.2 0.73 perf-profile.self.cycles-pp.release_pages 0.68 ± 21% +0.2 0.89 perf-profile.self.cycles-pp.native_flush_tlb_one_user 0.79 ± 21% +0.3 1.04 ± 3% perf-profile.self.cycles-pp.clear_page_erms 0.72 ± 30% +0.3 0.98 ± 3% perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.70 ± 26% +0.3 0.97 perf-profile.self.cycles-pp.unmap_page_range 0.82 ± 27% +0.3 1.10 perf-profile.self.cycles-pp.filemap_map_pages 1.04 ± 22% +0.3 1.35 perf-profile.self.cycles-pp.prepare_exit_to_usermode 1.28 ± 22% +0.4 1.67 perf-profile.self.cycles-pp.do_syscall_64 *************************************************************************************************** lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-7/performance/4BRD_12G/ext4/x86_64-rhel-7.6/3000/RAID1/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_src/aim7 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:2 dmesg.WARNING:at#for_ip_interrupt_entry/0x 2:4 -50% :2 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x %stddev %change %stddev \ | \ 2930 ± 2% -4.4% 2802 aim7.time.maximum_resident_set_size 309.25 ±173% +304.2% 1250 meminfo.Mlocked 0.03 ± 31% +0.0 0.04 ± 25% mpstat.cpu.all.iowait% 2.54 ± 76% -2.0 0.53 ± 5% turbostat.C1% 1468703 ± 75% +176.4% 4059526 ± 62% turbostat.C3 5.75 ± 93% +8.9 14.62 ± 72% turbostat.C3% 4.132e+08 ± 95% +160.6% 1.077e+09 ± 71% cpuidle.C3.time 1469107 ± 75% +176.4% 4059905 ± 62% cpuidle.C3.usage 106863 ± 3% +13.0% 120776 cpuidle.POLL.time 6635 ± 5% +12.3% 7451 ± 4% cpuidle.POLL.usage 109506 +1.1% 110712 proc-vmstat.nr_active_anon 107428 +1.0% 108475 proc-vmstat.nr_anon_pages 77.25 ±173% +303.9% 312.00 proc-vmstat.nr_mlock 109506 +1.1% 110712 proc-vmstat.nr_zone_active_anon 1778 ± 5% -79.3% 368.50 ± 50% numa-meminfo.node0.Active(file) 356.25 ± 26% +399.8% 1780 ± 9% numa-meminfo.node1.Active(file) 55033 ± 37% +93.4% 106429 ± 15% numa-meminfo.node1.AnonHugePages 214912 ± 14% +15.9% 249074 ± 7% numa-meminfo.node1.AnonPages 1237733 ± 12% +52.8% 1890859 ± 29% numa-meminfo.node1.MemUsed 5280 ± 15% +16.2% 6133 ± 10% slabinfo.dmaengine-unmap-16.active_objs 5283 ± 15% +16.2% 6140 ± 10% slabinfo.dmaengine-unmap-16.num_objs 20271 ± 6% +8.8% 22058 ± 2% slabinfo.kmalloc-32.active_objs 191.25 ± 35% +49.0% 285.00 ± 10% slabinfo.request_queue.num_objs 1732 ± 10% +19.2% 2064 slabinfo.skbuff_head_cache.active_objs 1840 ± 10% +19.1% 2192 slabinfo.skbuff_head_cache.num_objs 0.75 ± 3% -0.1 0.65 ± 3% perf-stat.i.dTLB-load-miss-rate% 17735473 ± 11% -19.1% 14349602 ± 4% perf-stat.i.dTLB-load-misses 0.19 ± 4% -0.0 0.15 ± 3% perf-stat.i.dTLB-store-miss-rate% 1897982 ± 4% -20.9% 1500903 ± 3% perf-stat.i.dTLB-store-misses 2188000 ± 3% -4.4% 2091270 perf-stat.i.iTLB-load-misses 0.74 ± 3% -0.1 0.64 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.19 ± 5% -0.0 0.15 ± 3% perf-stat.overall.dTLB-store-miss-rate% 17630676 ± 11% -19.1% 14270976 ± 4% perf-stat.ps.dTLB-load-misses 1886940 ± 4% -20.9% 1492696 ± 3% perf-stat.ps.dTLB-store-misses 434.30 ± 16% +31.3% 570.43 ± 6% sched_debug.cfs_rq:/.exec_clock.stddev 20136 ± 5% +17.9% 23747 ± 7% sched_debug.cfs_rq:/.load.avg 2.00 ± 40% +75.0% 3.50 ± 7% sched_debug.cfs_rq:/.load_avg.min 0.31 ± 12% +14.2% 0.36 ± 8% sched_debug.cfs_rq:/.nr_running.avg 2.06 ± 5% +33.3% 2.75 ± 9% sched_debug.cfs_rq:/.nr_spread_over.max 7.64 ± 8% +11.2% 8.50 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.avg 142834 ± 14% +14.9% 164118 ± 4% sched_debug.cfs_rq:/.spread0.max 151.79 ± 18% +21.5% 184.50 ± 5% sched_debug.cfs_rq:/.util_avg.min 9.16 ± 4% -11.3% 8.12 ± 2% sched_debug.cpu.cpu_load[1].avg 2364 ± 10% +25.7% 2971 ± 9% sched_debug.cpu.nr_switches.stddev -0.23 -5554.5% 12.50 ± 20% sched_debug.cpu.nr_uninterruptible.min 126.66 ± 16% +17.4% 148.73 sched_debug.cpu.ttwu_local.stddev 176.50 ± 61% +46.2% 258.00 ± 46% interrupts.42:IR-PCI-MSI.524292-edge.eth0-TxRx-3 135.50 ± 26% +60.5% 217.50 ± 5% interrupts.43:IR-PCI-MSI.524293-edge.eth0-TxRx-4 784.00 ± 86% -85.9% 110.50 ± 2% interrupts.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6 319.50 ± 45% -68.7% 100.00 ± 5% interrupts.46:IR-PCI-MSI.524296-edge.eth0-TxRx-7 2847 ± 12% +18.4% 3371 ± 5% interrupts.CPU0.RES:Rescheduling_interrupts 1236 ± 3% -100.0% 0.00 interrupts.CPU12.10:IR-IO-APIC.10-edge.ipmi_si 2600 ± 2% +10.0% 2861 ± 3% interrupts.CPU18.RES:Rescheduling_interrupts 477.75 ± 96% +108.3% 995.00 ± 9% interrupts.CPU21.NMI:Non-maskable_interrupts 477.75 ± 96% +108.3% 995.00 ± 9% interrupts.CPU21.PMI:Performance_monitoring_interrupts 176.50 ± 61% +46.2% 258.00 ± 46% interrupts.CPU36.42:IR-PCI-MSI.524292-edge.eth0-TxRx-3 135.50 ± 26% +60.5% 217.50 ± 5% interrupts.CPU38.43:IR-PCI-MSI.524293-edge.eth0-TxRx-4 784.00 ± 86% -85.9% 110.50 ± 2% interrupts.CPU4.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6 319.50 ± 45% -68.7% 100.00 ± 5% interrupts.CPU6.46:IR-PCI-MSI.524296-edge.eth0-TxRx-7 444.50 ± 5% -79.3% 92.00 ± 50% numa-vmstat.node0.nr_active_file 17870 -97.4% 462.00 ± 40% numa-vmstat.node0.nr_dirtied 44.50 ±173% +250.6% 156.00 ± 16% numa-vmstat.node0.nr_mlock 17843 -97.6% 432.50 ± 41% numa-vmstat.node0.nr_written 444.50 ± 5% -79.3% 92.00 ± 50% numa-vmstat.node0.nr_zone_active_file 831590 ± 3% -33.4% 553775 ± 22% numa-vmstat.node0.numa_hit 819665 ± 4% -39.6% 494911 ± 15% numa-vmstat.node0.numa_local 11924 ±108% +393.6% 58863 ± 77% numa-vmstat.node0.numa_other 88.75 ± 26% +400.8% 444.50 ± 9% numa-vmstat.node1.nr_active_file 53723 ± 14% +15.9% 62267 ± 7% numa-vmstat.node1.nr_anon_pages 694.50 ± 28% +2499.2% 18051 numa-vmstat.node1.nr_dirtied 32.75 ±173% +374.8% 155.50 ± 15% numa-vmstat.node1.nr_mlock 665.75 ± 29% +2605.9% 18014 numa-vmstat.node1.nr_written 88.75 ± 26% +400.8% 444.50 ± 9% numa-vmstat.node1.nr_zone_active_file 510127 ± 7% +55.2% 791681 ± 15% numa-vmstat.node1.numa_hit 343005 ± 12% +95.7% 671266 ± 11% numa-vmstat.node1.numa_local 52986 +7.8% 57135 ± 2% softirqs.CPU0.RCU 54275 +8.0% 58613 ± 4% softirqs.CPU1.RCU 28664 ± 3% +10.6% 31701 ± 2% softirqs.CPU1.SCHED 74527 ± 5% -17.7% 61372 ± 4% softirqs.CPU1.TIMER 52906 +11.4% 58960 softirqs.CPU11.RCU 52231 +12.0% 58488 ± 6% softirqs.CPU13.RCU 28678 ± 3% +9.8% 31491 ± 4% softirqs.CPU13.SCHED 52884 ± 2% +11.4% 58899 softirqs.CPU18.RCU 52822 +9.9% 58040 ± 2% softirqs.CPU19.RCU 52881 ± 2% +9.9% 58100 ± 4% softirqs.CPU2.RCU 52515 ± 2% +9.2% 57356 ± 2% softirqs.CPU20.RCU 54018 +14.4% 61810 ± 4% softirqs.CPU21.RCU 28115 +12.9% 31748 softirqs.CPU21.SCHED 54773 ± 2% +6.5% 58316 ± 3% softirqs.CPU22.RCU 53536 +11.4% 59660 ± 5% softirqs.CPU23.RCU 29616 ± 5% +7.1% 31705 ± 5% softirqs.CPU23.SCHED 27797 +9.0% 30292 ± 2% softirqs.CPU24.SCHED 55008 +8.5% 59698 ± 4% softirqs.CPU25.RCU 55186 ± 2% +7.3% 59236 ± 4% softirqs.CPU27.RCU 54285 +9.1% 59232 ± 5% softirqs.CPU29.RCU 53456 +8.1% 57762 ± 3% softirqs.CPU3.RCU 53396 +10.8% 59144 ± 3% softirqs.CPU31.RCU 54044 +10.2% 59576 ± 5% softirqs.CPU5.RCU 1.22 ± 2% -0.1 1.11 ± 4% perf-profile.calltrace.cycles-pp.ext4_orphan_add.ext4_unlink.vfs_unlink.do_unlinkat.do_syscall_64 0.65 ± 6% +0.1 0.73 ± 3% perf-profile.calltrace.cycles-pp.crc32c_pcl_intel_update.creat 1.16 ± 9% +0.1 1.29 perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.00 ± 10% +0.2 1.16 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 4.58 ± 3% +0.2 4.78 ± 2% perf-profile.calltrace.cycles-pp.ext4_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.89 ± 3% +0.2 5.08 ± 2% perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 0.47 ± 57% +0.2 0.71 ± 3% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.13 ±173% +0.5 0.60 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.00 +0.5 0.54 ± 6% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.24 ± 22% -0.1 0.17 perf-profile.children.cycles-pp.mutex_lock 0.39 ± 7% -0.1 0.33 perf-profile.children.cycles-pp.dput 0.12 ± 13% -0.1 0.07 ± 7% perf-profile.children.cycles-pp.tick_nohz_next_event 0.41 ± 3% -0.0 0.37 ± 2% perf-profile.children.cycles-pp.inode_permission 0.18 ± 8% -0.0 0.15 ± 10% perf-profile.children.cycles-pp.may_delete 0.09 ± 17% -0.0 0.06 perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.12 ± 12% -0.0 0.10 ± 5% perf-profile.children.cycles-pp.find_inode_bit 0.11 ± 7% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.find_next_zero_bit 0.05 ± 8% +0.0 0.07 perf-profile.children.cycles-pp.open64 0.09 ± 12% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.set_next_entity 0.14 ± 10% +0.0 0.16 ± 6% perf-profile.children.cycles-pp.mark_page_accessed 0.06 ± 14% +0.0 0.08 perf-profile.children.cycles-pp.generic_file_read_iter 0.18 ± 8% +0.0 0.20 perf-profile.children.cycles-pp.switch_fpu_return 0.08 ± 8% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.fsnotify_grab_connector 0.07 ± 14% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.security_file_alloc 0.12 ± 19% +0.0 0.15 ± 17% perf-profile.children.cycles-pp.vfs_read 0.10 ± 8% +0.0 0.13 ± 7% perf-profile.children.cycles-pp.run_timer_softirq 0.07 ± 15% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.note_gp_changes 0.13 ± 16% +0.0 0.16 ± 9% perf-profile.children.cycles-pp.ksys_read 0.04 ± 58% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.__ext4_handle_dirty_super 0.04 ± 60% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.update_cfs_rq_h_load 0.08 ± 6% +0.0 0.11 ± 9% perf-profile.children.cycles-pp.new_sync_read 0.12 ± 17% +0.0 0.16 ± 12% perf-profile.children.cycles-pp.rcu_all_qs 0.13 ± 10% +0.0 0.17 ± 3% perf-profile.children.cycles-pp.ext4_release_file 0.04 ± 59% +0.0 0.08 ± 12% perf-profile.children.cycles-pp.__srcu_read_lock 0.16 ± 6% +0.0 0.20 ± 4% perf-profile.children.cycles-pp.avc_has_perm 0.29 ± 5% +0.0 0.33 perf-profile.children.cycles-pp.d_lookup 0.44 ± 5% +0.0 0.48 ± 2% perf-profile.children.cycles-pp.__fput 0.05 ± 61% +0.0 0.09 ± 11% perf-profile.children.cycles-pp.__kernel_text_address 0.11 ± 17% +0.0 0.15 ± 13% perf-profile.children.cycles-pp.ext4_inode_attach_jinode 0.19 ± 11% +0.0 0.23 ± 2% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.03 ±100% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.kernel_text_address 0.03 ±102% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.15 ± 5% +0.0 0.20 ± 7% perf-profile.children.cycles-pp.read 0.01 ±173% +0.0 0.06 perf-profile.children.cycles-pp.check_preempt_curr 0.22 ± 14% +0.0 0.27 ± 3% perf-profile.children.cycles-pp.update_blocked_averages 0.20 ± 19% +0.0 0.24 ± 6% perf-profile.children.cycles-pp.__alloc_file 0.17 ± 15% +0.0 0.21 ± 2% perf-profile.children.cycles-pp._raw_spin_trylock 0.00 +0.1 0.05 perf-profile.children.cycles-pp.native_apic_mem_write 0.00 +0.1 0.05 perf-profile.children.cycles-pp.__note_gp_changes 0.23 ± 15% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.run_rebalance_domains 0.30 ± 14% +0.1 0.35 ± 4% perf-profile.children.cycles-pp.clockevents_program_event 0.27 ± 7% +0.1 0.33 perf-profile.children.cycles-pp.__d_lookup 0.23 ± 15% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.alloc_empty_file 0.38 ± 11% +0.1 0.45 ± 4% perf-profile.children.cycles-pp.jbd2_journal_stop 0.52 ± 6% +0.1 0.59 ± 5% perf-profile.children.cycles-pp.rcu_core 0.41 ± 7% +0.1 0.49 ± 7% perf-profile.children.cycles-pp.__ext4_journal_stop 0.14 ± 11% +0.1 0.22 ± 28% perf-profile.children.cycles-pp.__virt_addr_valid 0.50 ± 4% +0.1 0.60 ± 8% perf-profile.children.cycles-pp.ext4_map_blocks 0.72 ± 7% +0.1 0.82 ± 7% perf-profile.children.cycles-pp.getname_flags 0.47 ± 7% +0.1 0.57 ± 7% perf-profile.children.cycles-pp.strncpy_from_user 0.33 ± 6% +0.1 0.43 ± 12% perf-profile.children.cycles-pp.ext4_es_lookup_extent 0.23 ± 35% +0.1 0.35 ± 28% perf-profile.children.cycles-pp.io_serial_in 0.89 ± 9% +0.1 1.01 ± 7% perf-profile.children.cycles-pp.tick_sched_handle 1.03 ± 9% +0.1 1.16 ± 5% perf-profile.children.cycles-pp.tick_sched_timer 1.38 ± 6% +0.1 1.52 perf-profile.children.cycles-pp.irq_exit 1.21 ± 8% +0.2 1.38 ± 2% perf-profile.children.cycles-pp.__softirqentry_text_start 1.55 ± 5% +0.2 1.73 perf-profile.children.cycles-pp.crc32c_pcl_intel_update 4.59 ± 3% +0.2 4.79 ± 2% perf-profile.children.cycles-pp.ext4_evict_inode 4.89 ± 3% +0.2 5.08 ± 2% perf-profile.children.cycles-pp.evict 4.23 ± 4% +0.2 4.46 perf-profile.children.cycles-pp.smp_apic_timer_interrupt 4.49 ± 4% +0.2 4.72 perf-profile.children.cycles-pp.apic_timer_interrupt 0.20 ± 18% -0.0 0.15 ± 3% perf-profile.self.cycles-pp.mutex_lock 0.07 ± 12% -0.0 0.03 ±100% perf-profile.self.cycles-pp.tick_nohz_next_event 0.13 ± 6% -0.0 0.10 ± 10% perf-profile.self.cycles-pp.jbd2__journal_start 0.09 ± 17% -0.0 0.06 perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.09 ± 9% -0.0 0.06 perf-profile.self.cycles-pp.inode_init_always 0.11 ± 7% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.find_next_zero_bit 0.06 ± 6% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.ext4_alloc_inode 0.13 ± 10% +0.0 0.15 ± 3% perf-profile.self.cycles-pp.__virt_addr_valid 0.13 ± 12% +0.0 0.16 ± 6% perf-profile.self.cycles-pp.mark_page_accessed 0.07 ± 11% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.update_blocked_averages 0.11 ± 14% +0.0 0.15 ± 10% perf-profile.self.cycles-pp.ext4_mark_inode_dirty 0.04 ± 60% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.04 ± 59% +0.0 0.08 ± 12% perf-profile.self.cycles-pp.__srcu_read_lock 0.22 ± 10% +0.0 0.26 ± 7% perf-profile.self.cycles-pp.strncpy_from_user 0.16 ± 12% +0.0 0.20 ± 7% perf-profile.self.cycles-pp.ktime_get 0.15 ± 10% +0.0 0.20 ± 2% perf-profile.self.cycles-pp.avc_has_perm 0.11 ± 14% +0.0 0.15 ± 16% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.32 ± 9% +0.0 0.37 ± 2% perf-profile.self.cycles-pp.jbd2_journal_stop 0.16 ± 14% +0.0 0.21 perf-profile.self.cycles-pp._raw_spin_trylock 0.21 ± 12% +0.0 0.26 ± 3% perf-profile.self.cycles-pp.__d_lookup 0.13 ± 17% +0.0 0.18 ± 16% perf-profile.self.cycles-pp.__schedule 0.03 ±100% +0.1 0.08 ± 6% perf-profile.self.cycles-pp.__getblk_gfp 0.24 ± 6% +0.1 0.30 ± 8% perf-profile.self.cycles-pp.find_get_entry 0.10 ± 22% +0.1 0.15 ± 35% perf-profile.self.cycles-pp.add_dirent_to_buf 0.21 ± 8% +0.1 0.32 ± 15% perf-profile.self.cycles-pp.ext4_es_lookup_extent 1.21 ± 4% +0.2 1.40 perf-profile.self.cycles-pp.crc32c_pcl_intel_update *************************************************************************************************** lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID1/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/disk_wrt/aim7 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :2 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 63.75 +2.0% 65.00 turbostat.PkgTmp 280.00 ± 12% +37.1% 384.00 ± 8% slabinfo.biovec-64.active_objs 280.00 ± 12% +37.1% 384.00 ± 8% slabinfo.biovec-64.num_objs 20468 +0.6% 20586 perf-stat.i.context-switches 2875 ± 4% +5.8% 3043 perf-stat.overall.instructions-per-iTLB-miss 0.74 +0.8% 0.74 perf-stat.overall.ipc 20280 +0.6% 20397 perf-stat.ps.context-switches 124.21 ± 11% +21.6% 150.99 sched_debug.cfs_rq:/.load_avg.stddev 87668 ± 31% +30.1% 114018 ± 9% sched_debug.cfs_rq:/.spread0.max 47.30 ± 9% -31.3% 32.48 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.avg 884.66 ± 20% +28.0% 1132 ± 7% sched_debug.cpu.curr->pid.avg 1213 ± 11% +20.2% 1458 sched_debug.cpu.curr->pid.stddev 122.38 ± 5% +13.8% 139.25 sched_debug.cpu.ttwu_local.min 1870 -1.5% 1843 proc-vmstat.nr_dirty 2284 -1.8% 2242 proc-vmstat.nr_inactive_file 2284 -1.8% 2242 proc-vmstat.nr_zone_inactive_file 1673 ± 2% -4.9% 1591 proc-vmstat.nr_zone_write_pending 562.25 ±158% +366.7% 2624 ± 19% proc-vmstat.numa_pages_migrated 1286 ±130% +2496.0% 33384 proc-vmstat.numa_pte_updates 562.25 ±158% +366.7% 2624 ± 19% proc-vmstat.pgmigrate_success 50132 ± 12% +26.4% 63389 ± 7% softirqs.CPU1.TIMER 19663 ± 10% +12.7% 22151 ± 8% softirqs.CPU12.RCU 18448 ± 2% +8.3% 19976 ± 3% softirqs.CPU18.RCU 49885 ± 5% -8.7% 45564 softirqs.CPU20.TIMER 18240 ± 4% +14.6% 20897 ± 8% softirqs.CPU22.RCU 18703 ± 2% +8.1% 20219 ± 4% softirqs.CPU24.RCU 16173 +7.7% 17418 ± 4% softirqs.CPU25.SCHED 161721 ± 24% +34.3% 217174 ± 2% numa-meminfo.node0.Active 161373 ± 24% +33.7% 215820 ± 2% numa-meminfo.node0.Active(anon) 348.00 ±166% +289.1% 1354 numa-meminfo.node0.Active(file) 156218 ± 24% +33.7% 208918 ± 2% numa-meminfo.node0.AnonPages 12329 ± 41% +35.8% 16737 ± 10% numa-meminfo.node1.Inactive(anon) 4700 ± 14% -24.2% 3561 numa-meminfo.node1.Inactive(file) 52814 ± 87% -96.4% 1915 ± 9% numa-meminfo.node1.PageTables 82374 ± 46% -47.9% 42926 numa-meminfo.node1.SUnreclaim 40389 ± 24% +33.9% 54061 ± 2% numa-vmstat.node0.nr_active_anon 87.00 ±166% +289.1% 338.50 numa-vmstat.node0.nr_active_file 39099 ± 24% +33.9% 52339 ± 2% numa-vmstat.node0.nr_anon_pages 1076 ± 16% +28.0% 1377 ± 2% numa-vmstat.node0.nr_inactive_file 40389 ± 24% +33.9% 54061 ± 2% numa-vmstat.node0.nr_zone_active_anon 87.00 ±166% +289.1% 338.50 numa-vmstat.node0.nr_zone_active_file 1076 ± 15% +27.5% 1372 ± 2% numa-vmstat.node0.nr_zone_inactive_file 41725 ±129% +78.0% 74289 ± 82% numa-vmstat.node0.numa_other 3083 ± 41% +35.7% 4184 ± 10% numa-vmstat.node1.nr_inactive_anon 1183 ± 12% -22.4% 918.50 ± 5% numa-vmstat.node1.nr_inactive_file 13205 ± 87% -96.4% 479.00 ± 9% numa-vmstat.node1.nr_page_table_pages 20593 ± 46% -47.9% 10739 numa-vmstat.node1.nr_slab_unreclaimable 3083 ± 41% +35.7% 4184 ± 10% numa-vmstat.node1.nr_zone_inactive_anon 1181 ± 12% -22.4% 917.50 ± 4% numa-vmstat.node1.nr_zone_inactive_file 67.00 ± 9% +50.7% 101.00 ± 28% interrupts.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5 305.50 ±144% +435.4% 1635 ± 31% interrupts.CPU11.NMI:Non-maskable_interrupts 305.50 ±144% +435.4% 1635 ± 31% interrupts.CPU11.PMI:Performance_monitoring_interrupts 845.00 ± 17% -27.0% 617.00 ± 7% interrupts.CPU12.RES:Rescheduling_interrupts 973.75 ± 17% -28.4% 697.00 ± 4% interrupts.CPU14.RES:Rescheduling_interrupts 706.75 ± 21% +20.1% 849.00 ± 14% interrupts.CPU16.RES:Rescheduling_interrupts 1656 ± 32% -67.1% 544.50 ± 99% interrupts.CPU18.NMI:Non-maskable_interrupts 1656 ± 32% -67.1% 544.50 ± 99% interrupts.CPU18.PMI:Performance_monitoring_interrupts 67.00 ± 9% +50.7% 101.00 ± 28% interrupts.CPU2.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5 1195 ± 12% +16.7% 1395 ± 12% interrupts.CPU2.CAL:Function_call_interrupts 95.75 ±173% +1649.9% 1675 ± 32% interrupts.CPU2.NMI:Non-maskable_interrupts 95.75 ±173% +1649.9% 1675 ± 32% interrupts.CPU2.PMI:Performance_monitoring_interrupts 1197 ± 11% +10.7% 1325 ± 8% interrupts.CPU20.CAL:Function_call_interrupts 1633 ± 33% -69.9% 492.00 ± 92% interrupts.CPU22.NMI:Non-maskable_interrupts 1633 ± 33% -69.9% 492.00 ± 92% interrupts.CPU22.PMI:Performance_monitoring_interrupts 860.75 ± 13% -32.7% 579.50 ± 12% interrupts.CPU22.RES:Rescheduling_interrupts 1652 ± 33% -95.2% 79.00 ±100% interrupts.CPU24.NMI:Non-maskable_interrupts 1652 ± 33% -95.2% 79.00 ±100% interrupts.CPU24.PMI:Performance_monitoring_interrupts 1193 ± 9% +11.6% 1331 ± 10% interrupts.CPU28.CAL:Function_call_interrupts 636.25 ± 16% +30.4% 829.50 ± 10% interrupts.CPU30.RES:Rescheduling_interrupts 784.00 ± 8% +20.5% 944.50 interrupts.CPU31.RES:Rescheduling_interrupts 1174 ± 12% +19.5% 1403 ± 13% interrupts.CPU32.CAL:Function_call_interrupts 7.75 ±173% +14248.4% 1112 interrupts.CPU32.NMI:Non-maskable_interrupts 7.75 ±173% +14248.4% 1112 interrupts.CPU32.PMI:Performance_monitoring_interrupts 276.00 ±172% +211.1% 858.50 ± 95% interrupts.CPU33.NMI:Non-maskable_interrupts 276.00 ±172% +211.1% 858.50 ± 95% interrupts.CPU33.PMI:Performance_monitoring_interrupts 280.00 ±166% +320.4% 1177 ± 85% interrupts.CPU34.NMI:Non-maskable_interrupts 280.00 ±166% +320.4% 1177 ± 85% interrupts.CPU34.PMI:Performance_monitoring_interrupts 83.00 ±172% +608.4% 588.00 ± 80% interrupts.CPU38.NMI:Non-maskable_interrupts 83.00 ±172% +608.4% 588.00 ± 80% interrupts.CPU38.PMI:Performance_monitoring_interrupts 572.75 ± 18% +22.9% 704.00 ± 5% interrupts.CPU38.RES:Rescheduling_interrupts 278.25 ±173% +490.7% 1643 ± 31% interrupts.CPU4.NMI:Non-maskable_interrupts 278.25 ±173% +490.7% 1643 ± 31% interrupts.CPU4.PMI:Performance_monitoring_interrupts 663.50 ± 14% +24.6% 826.50 ± 12% interrupts.CPU4.RES:Rescheduling_interrupts 1306 ± 62% -46.5% 699.00 interrupts.CPU7.RES:Rescheduling_interrupts 1090 ± 70% +53.6% 1674 ± 31% interrupts.CPU9.NMI:Non-maskable_interrupts 1090 ± 70% +53.6% 1674 ± 31% interrupts.CPU9.PMI:Performance_monitoring_interrupts 42.50 -1.0 41.50 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 42.85 -1.0 41.87 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 36.02 -0.8 35.22 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 34.94 -0.8 34.18 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 30.66 -0.7 29.95 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 28.84 -0.7 28.16 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 23.88 -0.7 23.21 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write 23.66 -0.6 23.02 perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write 1.48 ± 2% -0.1 1.39 ± 3% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin 0.55 ± 3% +0.1 0.60 ± 7% perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open 0.54 ± 2% +0.1 0.60 ± 7% perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat 0.72 ± 4% +0.1 0.79 ± 4% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ± 5% +0.1 0.80 ± 4% perf-profile.calltrace.cycles-pp.down_write.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 1.31 ± 2% +0.1 1.39 ± 3% perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_open 0.59 ± 2% +0.1 0.67 ± 4% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write.do_unlinkat.do_syscall_64 0.90 ± 8% +0.1 1.02 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 2.34 ± 4% +0.1 2.47 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink 2.34 ± 4% +0.1 2.47 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 2.74 ± 2% +0.1 2.87 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write 1.73 ± 2% +0.1 1.86 ± 4% perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin 2.30 ± 4% +0.1 2.44 perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 2.39 ± 3% +0.2 2.54 perf-profile.calltrace.cycles-pp.unlink 3.35 ± 3% +0.2 3.59 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write 36.04 -0.8 35.23 perf-profile.children.cycles-pp.ksys_write 52.54 -0.8 51.76 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 52.20 -0.8 51.43 perf-profile.children.cycles-pp.do_syscall_64 34.98 -0.8 34.22 perf-profile.children.cycles-pp.vfs_write 30.68 -0.7 29.97 perf-profile.children.cycles-pp.new_sync_write 28.89 -0.7 28.20 perf-profile.children.cycles-pp.xfs_file_buffered_aio_write 23.89 -0.7 23.23 perf-profile.children.cycles-pp.iomap_file_buffered_write 23.71 -0.7 23.05 perf-profile.children.cycles-pp.iomap_apply 0.37 ± 5% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.node_dirty_ok 0.14 ± 16% -0.1 0.04 ±100% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm 1.51 ± 2% -0.1 1.43 ± 2% perf-profile.children.cycles-pp.get_page_from_freelist 0.56 ± 4% -0.1 0.48 perf-profile.children.cycles-pp._cond_resched 0.28 ± 10% -0.0 0.23 ± 4% perf-profile.children.cycles-pp.rcu_all_qs 0.16 ± 13% -0.0 0.13 perf-profile.children.cycles-pp.__xfs_btree_check_sblock 0.19 ± 2% -0.0 0.17 ± 3% perf-profile.children.cycles-pp.xfs_da3_node_read 0.15 ± 10% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.xfs_get_extsz_hint 0.10 ± 5% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.orc_find 0.11 ± 4% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair 0.15 ± 7% +0.0 0.17 perf-profile.children.cycles-pp.xfs_btree_increment 0.06 ± 20% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.xfs_btree_read_buf_block 0.09 ± 20% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.xfs_btree_lookup_get_block 0.04 ± 58% +0.0 0.07 perf-profile.children.cycles-pp.xas_find 0.26 ± 8% +0.0 0.29 perf-profile.children.cycles-pp.iov_iter_advance 0.16 ± 5% +0.0 0.20 ± 2% perf-profile.children.cycles-pp.xfs_dir_ino_validate 0.04 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.__x86_indirect_thunk_r15 0.14 ± 9% +0.0 0.18 ± 5% perf-profile.children.cycles-pp.xfs_buf_item_format 0.08 ± 19% +0.0 0.12 ± 8% perf-profile.children.cycles-pp._xfs_trans_bjoin 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.__inc_numa_state 0.03 ±100% +0.0 0.07 perf-profile.children.cycles-pp.xfs_trans_free 0.06 ± 67% +0.0 0.10 ± 14% perf-profile.children.cycles-pp.xfs_find_daxdev_for_inode 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.__unwind_start 0.55 ± 3% +0.1 0.60 ± 7% perf-profile.children.cycles-pp.xfs_dir_ialloc 0.34 ± 6% +0.1 0.40 ± 10% perf-profile.children.cycles-pp.__generic_write_end 0.54 ± 2% +0.1 0.60 ± 7% perf-profile.children.cycles-pp.xfs_ialloc 0.47 ± 3% +0.1 0.54 ± 7% perf-profile.children.cycles-pp.xfs_dialloc 1.31 ± 2% +0.1 1.39 ± 3% perf-profile.children.cycles-pp.xfs_create 1.01 ± 4% +0.1 1.09 ± 3% perf-profile.children.cycles-pp.___might_sleep 0.95 ± 4% +0.1 1.03 ± 4% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.53 ± 12% +0.1 0.62 ± 6% perf-profile.children.cycles-pp.__xfs_dir3_data_check 0.53 ± 12% +0.1 0.62 ± 5% perf-profile.children.cycles-pp.xfs_dir3_data_check 2.31 ± 4% +0.1 2.44 perf-profile.children.cycles-pp.do_unlinkat 2.40 ± 3% +0.2 2.55 perf-profile.children.cycles-pp.unlink 3.07 +0.2 3.25 perf-profile.children.cycles-pp.syscall_return_via_sysret 3.39 ± 3% +0.2 3.63 perf-profile.children.cycles-pp.entry_SYSCALL_64 0.99 ± 4% -0.1 0.86 perf-profile.self.cycles-pp.xfs_file_buffered_aio_write 0.14 ± 16% -0.1 0.04 ±100% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm 0.18 ± 30% -0.1 0.09 ± 11% perf-profile.self.cycles-pp.lock_page_memcg 0.47 ± 4% -0.1 0.40 ± 3% perf-profile.self.cycles-pp.iomap_write_end 0.25 ± 7% -0.1 0.19 ± 5% perf-profile.self.cycles-pp.node_dirty_ok 0.43 ± 7% -0.0 0.38 perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.27 ± 4% -0.0 0.22 perf-profile.self.cycles-pp.xfs_bmbt_to_iomap 0.11 ± 9% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.node_page_state 0.26 -0.0 0.23 ± 2% perf-profile.self.cycles-pp.file_update_time 0.11 ± 12% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.xfs_get_extsz_hint 0.08 -0.0 0.06 ± 16% perf-profile.self.cycles-pp.xfs_agino_range 0.08 ± 5% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.mem_cgroup_try_charge 0.08 ± 10% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.truncate_cleanup_page 0.09 ± 12% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.xas_create 0.06 ± 17% +0.0 0.09 ± 17% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.04 ± 58% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.free_unref_page_list 0.15 ± 5% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.__fdget_pos 0.07 ± 17% +0.0 0.10 ± 14% perf-profile.self.cycles-pp.memcpy_erms 0.17 ± 4% +0.0 0.21 ± 2% perf-profile.self.cycles-pp.__mod_lruvec_state 0.31 ± 7% +0.0 0.35 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.27 ± 7% +0.0 0.31 ± 6% perf-profile.self.cycles-pp.__add_to_page_cache_locked 0.26 ± 10% +0.0 0.30 ± 3% perf-profile.self.cycles-pp.free_pcppages_bulk 0.05 ± 74% +0.0 0.10 ± 19% perf-profile.self.cycles-pp.xfs_find_daxdev_for_inode 0.01 ±173% +0.0 0.06 ± 16% perf-profile.self.cycles-pp.hrtimer_interrupt 0.00 +0.1 0.05 perf-profile.self.cycles-pp.xfs_buf_item_init 0.03 ±100% +0.1 0.08 ± 6% perf-profile.self.cycles-pp.__x86_indirect_thunk_r15 0.19 ± 27% +0.1 0.24 ± 4% perf-profile.self.cycles-pp.__sb_end_write 0.12 ± 16% +0.1 0.18 ± 2% perf-profile.self.cycles-pp.mem_cgroup_commit_charge 0.97 ± 3% +0.1 1.04 ± 4% perf-profile.self.cycles-pp.___might_sleep 0.61 ± 9% +0.1 0.69 ± 5% perf-profile.self.cycles-pp.xas_load 0.51 ± 3% +0.1 0.59 ± 2% perf-profile.self.cycles-pp.write 0.91 ± 3% +0.1 1.00 ± 4% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.26 ± 9% +0.1 0.36 ± 6% perf-profile.self.cycles-pp.__alloc_pages_nodemask 3.06 +0.2 3.24 perf-profile.self.cycles-pp.syscall_return_via_sysret 3.03 ± 3% +0.2 3.25 perf-profile.self.cycles-pp.entry_SYSCALL_64 *************************************************************************************************** lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID1/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_wrt/aim7 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- %stddev %change %stddev \ | \ 1304 -3.1% 1263 aim7.time.system_time 5.18 ± 4% +0.7 5.91 mpstat.cpu.all.usr% 69.05 -5.1% 65.53 iostat.cpu.idle 25.78 ± 3% +10.8% 28.56 iostat.cpu.system 5.11 ± 4% +14.2% 5.83 iostat.cpu.user 1024 ± 54% +56.8% 1607 ± 5% iostat.md0.wkB/s 68.50 -5.1% 65.00 vmstat.cpu.id 1024 ± 54% +56.7% 1604 ± 5% vmstat.io.bo 4.00 +500.0% 24.00 vmstat.memory.buff 17890 ± 3% +13.5% 20313 vmstat.system.cs 1101 ± 2% +7.3% 1181 ± 2% slabinfo.UNIX.active_objs 1101 ± 2% +7.3% 1181 ± 2% slabinfo.UNIX.num_objs 4346 ± 4% +11.5% 4846 ± 2% slabinfo.kmalloc-1k.active_objs 4378 ± 4% +11.0% 4859 ± 2% slabinfo.kmalloc-1k.num_objs 1952 ± 18% +31.1% 2560 slabinfo.skbuff_head_cache.active_objs 2040 ± 17% +31.8% 2688 slabinfo.skbuff_head_cache.num_objs 194684 ± 7% -49.0% 99382 meminfo.AnonHugePages 2519569 +9.0% 2747468 meminfo.Committed_AS 6016512 +25.8% 7570432 ± 6% meminfo.DirectMap2M 6647 ± 7% +11.7% 7424 ± 2% meminfo.Dirty 8149 ± 5% +11.5% 9090 ± 2% meminfo.Inactive(file) 1977 ± 11% -100.0% 0.00 meminfo.Mlocked 20229 ± 6% +15.5% 23363 meminfo.max_used_kB 167735 ± 28% +20.1% 201529 ± 28% numa-meminfo.node0.Active(anon) 106332 ± 16% -61.9% 40537 ± 27% numa-meminfo.node0.AnonHugePages 162893 ± 28% +22.3% 199190 ± 29% numa-meminfo.node0.AnonPages 8638 ± 28% +59.4% 13773 ± 16% numa-meminfo.node0.Inactive 4576 ± 56% +110.7% 9640 ± 30% numa-meminfo.node0.Inactive(anon) 1140 ± 10% -100.0% 0.00 numa-meminfo.node0.Mlocked 88219 ± 15% -33.5% 58632 ± 17% numa-meminfo.node1.AnonHugePages 3251 ± 6% +10.0% 3575 ± 2% numa-meminfo.node1.Dirty 4053 ± 18% +18.0% 4781 ± 14% numa-meminfo.node1.Inactive(file) 407.00 ± 3% +11.2% 452.50 turbostat.Avg_MHz 0.86 ±112% +176.5% 2.38 ± 82% turbostat.CPU%c3 33.81 ± 3% +17.8% 39.83 turbostat.CorWatt 51.75 ± 5% +23.7% 64.00 turbostat.CoreTmp 10.75 ± 34% -57.4% 4.58 turbostat.Pkg%pc2 0.03 ±173% +483.3% 0.18 ± 94% turbostat.Pkg%pc3 2.94 ± 39% -90.3% 0.29 ± 96% turbostat.Pkg%pc6 52.00 ± 5% +23.1% 64.00 turbostat.PkgTmp 59.26 +11.4% 66.03 turbostat.PkgWatt 10206 ± 3% -13.0% 8884 turbostat.SMI 337.75 +1.0% 341.00 proc-vmstat.nr_active_file 94.25 ± 7% -49.1% 48.00 proc-vmstat.nr_anon_transparent_hugepages 1610 ± 5% +12.8% 1816 proc-vmstat.nr_dirty 1979 ± 4% +12.2% 2220 proc-vmstat.nr_inactive_file 48494 ± 3% +12.0% 54334 proc-vmstat.nr_kernel_stack 494.75 ± 11% -100.0% 0.00 proc-vmstat.nr_mlock 39652 +5.8% 41942 proc-vmstat.nr_slab_unreclaimable 28998 ± 58% +44.2% 41804 ± 4% proc-vmstat.nr_written 337.75 +1.0% 341.00 proc-vmstat.nr_zone_active_file 1978 ± 4% +12.2% 2220 proc-vmstat.nr_zone_inactive_file 1419 ± 7% +11.7% 1586 ± 3% proc-vmstat.nr_zone_write_pending 23827 ± 6% -94.6% 1292 ± 99% proc-vmstat.numa_pte_updates 483395 -9.5% 437383 proc-vmstat.pgfault 1724 ± 2% -2.9% 1675 proc-vmstat.unevictable_pgs_culled 40697 ± 28% +22.3% 49761 ± 29% numa-vmstat.node0.nr_anon_pages 19766117 ± 3% -12.6% 17268001 numa-vmstat.node0.nr_dirtied 820.75 ± 3% +13.6% 932.00 numa-vmstat.node0.nr_dirty 1146 ± 56% +110.2% 2409 ± 30% numa-vmstat.node0.nr_inactive_anon 285.50 ± 10% -100.0% 0.00 numa-vmstat.node0.nr_mlock 1146 ± 56% +110.2% 2409 ± 30% numa-vmstat.node0.nr_zone_inactive_anon 732.25 ± 2% +11.4% 816.00 numa-vmstat.node0.nr_zone_write_pending 20278495 ± 3% -12.6% 17727979 numa-vmstat.node0.numa_hit 20276149 ± 3% -12.9% 17656262 numa-vmstat.node0.numa_local 2353 ± 90% +2947.3% 71725 ± 89% numa-vmstat.node0.numa_other 19859884 ± 3% -13.8% 17124582 numa-vmstat.node1.nr_dirtied 817.25 ± 4% +11.3% 909.50 numa-vmstat.node1.nr_dirty 1006 ± 17% +19.1% 1198 ± 11% numa-vmstat.node1.nr_inactive_file 208.75 ± 11% -100.0% 0.00 numa-vmstat.node1.nr_mlock 1006 ± 17% +19.3% 1201 ± 10% numa-vmstat.node1.nr_zone_inactive_file 723.25 ± 3% +11.1% 803.50 numa-vmstat.node1.nr_zone_write_pending 20396643 ± 3% -13.3% 17691336 numa-vmstat.node1.numa_hit 20225241 ± 3% -13.0% 17589172 numa-vmstat.node1.numa_local 52229 ± 7% -13.5% 45200 interrupts.CAL:Function_call_interrupts 1037 ± 10% +61.8% 1678 ± 5% interrupts.CPU0.RES:Rescheduling_interrupts 1301 ± 6% -17.5% 1073 interrupts.CPU10.CAL:Function_call_interrupts 1289 ± 2% -15.7% 1087 ± 8% interrupts.CPU12.CAL:Function_call_interrupts 1322 ± 7% -14.5% 1131 ± 4% interrupts.CPU19.CAL:Function_call_interrupts 17.00 ±169% +6250.0% 1079 ± 2% interrupts.CPU2.NMI:Non-maskable_interrupts 17.00 ±169% +6250.0% 1079 ± 2% interrupts.CPU2.PMI:Performance_monitoring_interrupts 1923 ± 23% -73.2% 515.00 ± 99% interrupts.CPU21.NMI:Non-maskable_interrupts 1923 ± 23% -73.2% 515.00 ± 99% interrupts.CPU21.PMI:Performance_monitoring_interrupts 1350 ± 5% -27.6% 978.00 ± 6% interrupts.CPU22.CAL:Function_call_interrupts 1959 ± 24% -91.7% 162.50 ± 99% interrupts.CPU22.NMI:Non-maskable_interrupts 1959 ± 24% -91.7% 162.50 ± 99% interrupts.CPU22.PMI:Performance_monitoring_interrupts 1046 ± 21% -41.0% 617.50 ± 2% interrupts.CPU22.RES:Rescheduling_interrupts 1341 ± 8% -18.0% 1100 interrupts.CPU23.CAL:Function_call_interrupts 880.75 ± 12% -29.0% 625.50 interrupts.CPU23.RES:Rescheduling_interrupts 1323 ± 10% -15.9% 1112 interrupts.CPU25.CAL:Function_call_interrupts 872.75 ± 12% -29.5% 615.00 ± 2% interrupts.CPU25.RES:Rescheduling_interrupts 347.75 ±133% +210.7% 1080 ± 5% interrupts.CPU30.NMI:Non-maskable_interrupts 347.75 ±133% +210.7% 1080 ± 5% interrupts.CPU30.PMI:Performance_monitoring_interrupts 1316 ± 10% -15.8% 1108 interrupts.CPU31.CAL:Function_call_interrupts 721.75 ± 5% -14.6% 616.50 ± 3% interrupts.CPU31.RES:Rescheduling_interrupts 1304 ± 15% -21.7% 1021 ± 2% interrupts.CPU36.CAL:Function_call_interrupts 711.75 ± 13% +162.6% 1869 ± 54% interrupts.CPU5.RES:Rescheduling_interrupts 1331 ± 7% -20.3% 1061 ± 2% interrupts.CPU6.CAL:Function_call_interrupts 802.75 ± 11% -21.8% 628.00 interrupts.CPU6.RES:Rescheduling_interrupts 1325 ± 5% -14.2% 1138 ± 4% interrupts.CPU7.CAL:Function_call_interrupts 1314 ± 6% -15.8% 1106 interrupts.CPU9.CAL:Function_call_interrupts 9961 +11.7% 11124 sched_debug.cfs_rq:/.exec_clock.max 162.13 ± 11% +102.1% 327.63 ± 7% sched_debug.cfs_rq:/.exec_clock.stddev 237563 ± 5% +6.4% 252874 ± 3% sched_debug.cfs_rq:/.min_vruntime.max 0.39 ± 7% -14.5% 0.33 sched_debug.cfs_rq:/.nr_running.avg 12.54 +148.2% 31.13 ± 21% sched_debug.cfs_rq:/.removed.load_avg.avg 78.34 +52.8% 119.68 ± 11% sched_debug.cfs_rq:/.removed.load_avg.stddev 577.48 +149.3% 1439 ± 21% sched_debug.cfs_rq:/.removed.runnable_sum.avg 3606 +53.4% 5533 ± 11% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 3.67 ± 28% +203.5% 11.12 ± 16% sched_debug.cfs_rq:/.removed.util_avg.avg 146.62 ± 28% +43.4% 210.25 ± 9% sched_debug.cfs_rq:/.removed.util_avg.max 22.89 ± 28% +91.2% 43.76 ± 4% sched_debug.cfs_rq:/.removed.util_avg.stddev 10.87 ± 9% +69.8% 18.47 ± 39% sched_debug.cfs_rq:/.runnable_load_avg.stddev 39019 +663.6% 297940 ± 86% sched_debug.cfs_rq:/.runnable_weight.max 17929 +218.3% 57063 ± 68% sched_debug.cfs_rq:/.runnable_weight.stddev 60838 ± 26% +49.2% 90778 ± 11% sched_debug.cfs_rq:/.spread0.max 371.50 ± 21% +31.5% 488.50 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.max 477488 ± 75% -84.5% 74077 ± 2% sched_debug.cpu.clock.avg 477508 ± 75% -84.5% 74095 ± 2% sched_debug.cpu.clock.max 477467 ± 75% -84.5% 74058 ± 2% sched_debug.cpu.clock.min 477488 ± 75% -84.5% 74077 ± 2% sched_debug.cpu.clock_task.avg 477508 ± 75% -84.5% 74095 ± 2% sched_debug.cpu.clock_task.max 477467 ± 75% -84.5% 74058 ± 2% sched_debug.cpu.clock_task.min 7.81 ± 8% +19.0% 9.29 ± 5% sched_debug.cpu.cpu_load[0].avg 8.75 ± 7% +18.3% 10.34 ± 6% sched_debug.cpu.cpu_load[1].avg 9.42 ± 7% +18.1% 11.12 ± 2% sched_debug.cpu.cpu_load[2].avg 43.38 ± 11% +91.4% 83.00 ± 43% sched_debug.cpu.cpu_load[2].max 8.80 ± 9% +67.0% 14.70 ± 35% sched_debug.cpu.cpu_load[2].stddev 9.72 ± 9% +23.6% 12.01 sched_debug.cpu.cpu_load[3].avg 46.62 ± 18% +71.6% 80.00 ± 19% sched_debug.cpu.cpu_load[3].max 8.60 ± 14% +69.9% 14.61 ± 17% sched_debug.cpu.cpu_load[3].stddev 8.47 ± 10% +61.0% 13.64 ± 7% sched_debug.cpu.cpu_load[4].avg 41.12 ± 15% +336.5% 179.50 ± 12% sched_debug.cpu.cpu_load[4].max 8.28 ± 13% +248.2% 28.83 ± 12% sched_debug.cpu.cpu_load[4].stddev 42318 ± 9% -11.5% 37470 sched_debug.cpu.nr_load_updates.avg 39974 ± 9% -13.0% 34783 sched_debug.cpu.nr_load_updates.min 0.38 ± 8% -16.4% 0.32 sched_debug.cpu.nr_running.avg 15.56 ± 3% +15.8% 18.02 ± 4% sched_debug.cpu.nr_uninterruptible.stddev 8499 +6.5% 9050 ± 4% sched_debug.cpu.ttwu_count.max 455.17 ± 8% +14.3% 520.19 ± 5% sched_debug.cpu.ttwu_count.stddev 121.12 ± 3% +7.3% 130.00 ± 4% sched_debug.cpu.ttwu_local.min 477466 ± 75% -84.5% 74058 ± 2% sched_debug.cpu_clk 473896 ± 75% -85.1% 70485 ± 2% sched_debug.ktime 477908 ± 74% -84.4% 74502 ± 2% sched_debug.sched_clk 18.97 ± 14% -39.7% 11.43 ± 8% perf-stat.i.MPKI 2.325e+09 ± 3% +14.8% 2.669e+09 perf-stat.i.branch-instructions 3.65 ± 9% -1.2 2.47 ± 3% perf-stat.i.branch-miss-rate% 40992858 ± 4% +17.0% 47962008 perf-stat.i.branch-misses 9.45 ± 14% -2.3 7.17 perf-stat.i.cache-miss-rate% 5752408 ± 2% +7.9% 6205826 ± 3% perf-stat.i.cache-misses 99449480 ± 4% +12.5% 1.119e+08 ± 2% perf-stat.i.cache-references 18273 ± 3% +12.5% 20561 perf-stat.i.context-switches 2.48 ± 8% -31.7% 1.70 ± 3% perf-stat.i.cpi 1.629e+10 ± 4% +11.1% 1.809e+10 perf-stat.i.cpu-cycles 945.35 ± 4% +9.1% 1031 perf-stat.i.cpu-migrations 2944 ± 3% +4.2% 3067 perf-stat.i.cycles-between-cache-misses 3.591e+09 ± 3% +14.3% 4.102e+09 perf-stat.i.dTLB-loads 7154623 ± 8% +13.3% 8105429 perf-stat.i.dTLB-store-misses 2.473e+09 ± 3% +14.4% 2.83e+09 perf-stat.i.dTLB-stores 82.21 ± 2% -3.2 79.06 perf-stat.i.iTLB-load-miss-rate% 1050734 ± 12% +15.3% 1210985 perf-stat.i.iTLB-loads 1.172e+10 ± 3% +14.8% 1.345e+10 perf-stat.i.instructions 2449 +27.2% 3114 perf-stat.i.instructions-per-iTLB-miss 0.62 ± 2% +14.3% 0.71 perf-stat.i.ipc 1295155 ± 7% +17.5% 1522168 ± 3% perf-stat.i.node-load-misses 1692798 ± 5% +17.0% 1980114 perf-stat.i.node-loads 761603 ± 8% +15.8% 882204 perf-stat.i.node-store-misses 1197239 ± 9% +17.3% 1403801 perf-stat.i.node-stores 1.76 +0.0 1.80 perf-stat.overall.branch-miss-rate% 1.39 -3.2% 1.35 perf-stat.overall.cpi 79.96 -1.9 78.04 perf-stat.overall.iTLB-load-miss-rate% 2811 ± 2% +11.2% 3125 perf-stat.overall.instructions-per-iTLB-miss 0.72 +3.3% 0.74 perf-stat.overall.ipc 2.306e+09 ± 3% +14.7% 2.645e+09 perf-stat.ps.branch-instructions 40663464 ± 4% +16.9% 47517919 perf-stat.ps.branch-misses 5707130 ± 2% +7.7% 6149393 ± 3% perf-stat.ps.cache-misses 98651423 ± 4% +12.4% 1.109e+08 ± 2% perf-stat.ps.cache-references 18126 ± 3% +12.4% 20371 perf-stat.ps.context-switches 1.616e+10 ± 4% +10.9% 1.793e+10 perf-stat.ps.cpu-cycles 937.88 ± 4% +9.0% 1022 perf-stat.ps.cpu-migrations 3.562e+09 ± 3% +14.1% 4.064e+09 perf-stat.ps.dTLB-loads 7096976 ± 8% +13.2% 8030377 perf-stat.ps.dTLB-store-misses 2.454e+09 ± 3% +14.3% 2.804e+09 perf-stat.ps.dTLB-stores 1042256 ± 12% +15.1% 1199770 perf-stat.ps.iTLB-loads 1.162e+10 ± 3% +14.6% 1.332e+10 perf-stat.ps.instructions 1284746 ± 7% +17.4% 1508066 ± 3% perf-stat.ps.node-load-misses 1679220 ± 5% +16.8% 1961768 perf-stat.ps.node-loads 755495 ± 8% +15.7% 874037 perf-stat.ps.node-store-misses 1187940 ± 9% +17.1% 1390847 perf-stat.ps.node-stores 54528 ± 2% -22.6% 42229 ± 6% softirqs.CPU0.TIMER 19067 ± 4% -10.6% 17047 softirqs.CPU1.SCHED 53953 -17.8% 44323 ± 7% softirqs.CPU1.TIMER 18615 ± 5% -9.4% 16866 softirqs.CPU10.SCHED 18576 ± 3% -7.5% 17176 softirqs.CPU11.SCHED 49349 ± 6% -15.8% 41544 ± 7% softirqs.CPU11.TIMER 18891 ± 5% -10.9% 16837 ± 2% softirqs.CPU12.SCHED 57456 ± 14% -28.2% 41276 ± 4% softirqs.CPU12.TIMER 18784 ± 2% -10.1% 16884 softirqs.CPU13.SCHED 49424 ± 5% -16.5% 41289 ± 9% softirqs.CPU13.TIMER 19223 ± 2% -11.2% 17070 softirqs.CPU14.SCHED 51678 ± 8% -19.0% 41856 ± 6% softirqs.CPU14.TIMER 19010 ± 2% -9.7% 17160 softirqs.CPU15.SCHED 19306 ± 2% -12.1% 16971 softirqs.CPU16.SCHED 52068 ± 8% -20.7% 41291 ± 7% softirqs.CPU16.TIMER 19982 ± 7% +7.6% 21503 ± 5% softirqs.CPU17.RCU 18737 ± 3% -8.4% 17166 softirqs.CPU17.SCHED 19111 ± 3% -11.8% 16858 ± 2% softirqs.CPU18.SCHED 18896 ± 4% -9.2% 17159 softirqs.CPU19.SCHED 52848 ± 5% -21.2% 41638 ± 5% softirqs.CPU19.TIMER 19445 ± 2% -11.8% 17151 softirqs.CPU2.SCHED 53871 ± 11% -23.9% 40994 ± 7% softirqs.CPU2.TIMER 52506 ± 12% -21.1% 41435 ± 6% softirqs.CPU20.TIMER 20471 ± 6% +7.7% 22044 ± 2% softirqs.CPU21.RCU 18435 ± 4% -10.0% 16590 softirqs.CPU22.SCHED 49458 ± 3% -18.2% 40465 ± 4% softirqs.CPU22.TIMER 20835 ± 5% +15.4% 24044 ± 8% softirqs.CPU23.RCU 18719 ± 4% -9.2% 16999 softirqs.CPU24.SCHED 49855 ± 4% -16.3% 41723 ± 6% softirqs.CPU24.TIMER 49582 ± 5% -15.7% 41774 ± 5% softirqs.CPU25.TIMER 18825 ± 4% -9.7% 16996 softirqs.CPU26.SCHED 50142 ± 4% -17.3% 41468 ± 6% softirqs.CPU26.TIMER 49521 ± 6% -16.6% 41297 ± 6% softirqs.CPU27.TIMER 19061 ± 4% -11.3% 16913 softirqs.CPU28.SCHED 56138 ± 16% -26.7% 41134 ± 7% softirqs.CPU28.TIMER 49305 ± 7% -16.3% 41278 ± 7% softirqs.CPU29.TIMER 50274 ± 6% -17.8% 41305 ± 5% softirqs.CPU3.TIMER 19289 ± 5% -13.7% 16652 softirqs.CPU30.SCHED 19318 ± 7% -12.3% 16948 softirqs.CPU32.SCHED 18563 ± 3% -8.1% 17055 softirqs.CPU33.SCHED 19555 ± 3% -13.5% 16910 softirqs.CPU34.SCHED 54208 ± 9% -23.6% 41413 ± 7% softirqs.CPU34.TIMER 18777 ± 3% -9.6% 16972 softirqs.CPU35.SCHED 18985 ± 3% -10.1% 17066 softirqs.CPU36.SCHED 52365 ± 9% -19.2% 42299 ± 8% softirqs.CPU36.TIMER 18686 ± 4% -8.9% 17030 softirqs.CPU37.SCHED 51287 ± 5% -17.9% 42102 ± 7% softirqs.CPU37.TIMER 19511 ± 2% -14.0% 16783 softirqs.CPU38.SCHED 58324 ± 11% -29.9% 40875 ± 8% softirqs.CPU38.TIMER 18670 ± 3% -8.6% 17072 softirqs.CPU39.SCHED 52113 ± 4% -19.7% 41828 ± 7% softirqs.CPU39.TIMER 49903 ± 5% -16.3% 41768 ± 8% softirqs.CPU4.TIMER 49614 ± 6% -16.3% 41528 ± 5% softirqs.CPU5.TIMER 18943 ± 6% -9.5% 17144 softirqs.CPU6.SCHED 49359 ± 5% -15.6% 41661 ± 7% softirqs.CPU6.TIMER 49260 ± 6% -14.8% 41974 ± 4% softirqs.CPU7.TIMER 19050 ± 5% -10.9% 16982 softirqs.CPU8.SCHED 18876 ± 3% -8.1% 17339 softirqs.CPU9.SCHED 760956 ± 3% -9.5% 688784 softirqs.SCHED 2064576 ± 3% -17.5% 1704059 ± 6% softirqs.TIMER 9.99 -0.8 9.21 ± 3% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 7.40 -0.7 6.73 ± 4% perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write 7.11 -0.7 6.45 ± 3% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply 1.78 ± 3% -0.3 1.44 ± 5% perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin 2.94 ± 2% -0.3 2.60 ± 4% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor 0.55 -0.3 0.27 ±100% perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode 0.87 ± 16% -0.2 0.62 ± 2% perf-profile.calltrace.cycles-pp.__sb_start_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.50 ± 2% -0.2 1.34 ± 2% perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor 0.82 ± 5% -0.1 0.74 perf-profile.calltrace.cycles-pp.xfs_file_iomap_end.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write 0.60 +0.0 0.61 perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write.do_unlinkat.do_syscall_64 0.44 ± 59% +0.2 0.68 ± 10% perf-profile.calltrace.cycles-pp.xfs_iext_lookup_extent.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write 0.14 ±173% +0.4 0.58 ± 13% perf-profile.calltrace.cycles-pp.iov_iter_fault_in_readable.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 1.12 ± 8% +0.5 1.64 ± 3% perf-profile.calltrace.cycles-pp.xfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 10.03 -0.8 9.25 ± 3% perf-profile.children.cycles-pp.iomap_write_begin 7.43 -0.7 6.76 ± 4% perf-profile.children.cycles-pp.grab_cache_page_write_begin 7.14 -0.7 6.48 ± 4% perf-profile.children.cycles-pp.pagecache_get_page 1.82 ± 3% -0.4 1.46 ± 5% perf-profile.children.cycles-pp.__add_to_page_cache_locked 2.95 ± 2% -0.3 2.60 ± 4% perf-profile.children.cycles-pp.add_to_page_cache_lru 0.95 ± 14% -0.3 0.68 ± 5% perf-profile.children.cycles-pp.__sb_start_write 0.97 ± 3% -0.2 0.78 ± 9% perf-profile.children.cycles-pp.xas_load 1.52 -0.2 1.36 ± 2% perf-profile.children.cycles-pp.find_get_entry 0.93 ± 5% -0.1 0.80 ± 4% perf-profile.children.cycles-pp.xfs_iunlock 0.43 ± 4% -0.1 0.33 ± 9% perf-profile.children.cycles-pp.xfs_errortag_test 0.67 ± 2% -0.1 0.56 ± 3% perf-profile.children.cycles-pp.up_write 0.38 ± 8% -0.1 0.29 ± 6% perf-profile.children.cycles-pp.xas_start 0.36 ± 14% -0.1 0.27 ± 7% perf-profile.children.cycles-pp.node_dirty_ok 0.86 ± 6% -0.1 0.78 perf-profile.children.cycles-pp.xfs_file_iomap_end 1.06 ± 3% -0.1 0.97 perf-profile.children.cycles-pp.___might_sleep 0.49 ± 4% -0.1 0.41 perf-profile.children.cycles-pp.xfs_trans_read_buf_map 0.45 ± 16% -0.1 0.39 perf-profile.children.cycles-pp.scheduler_tick 0.36 ± 15% -0.1 0.30 ± 5% perf-profile.children.cycles-pp.xfs_trans_alloc 0.21 ± 5% -0.1 0.15 ± 10% perf-profile.children.cycles-pp.workingset_update_node 0.35 ± 4% -0.1 0.29 perf-profile.children.cycles-pp.xfs_buf_read_map 0.23 ± 16% -0.1 0.17 ± 8% perf-profile.children.cycles-pp.xfs_log_reserve 0.19 ± 9% -0.1 0.13 perf-profile.children.cycles-pp.lock_page_memcg 0.55 -0.0 0.50 ± 6% perf-profile.children.cycles-pp.xfs_ifree 0.08 ± 5% -0.0 0.03 ±100% perf-profile.children.cycles-pp.__update_load_avg_se 0.33 ± 3% -0.0 0.28 ± 3% perf-profile.children.cycles-pp.xfs_buf_get_map 0.07 ± 10% -0.0 0.03 ±100% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.24 ± 20% -0.0 0.20 perf-profile.children.cycles-pp.task_tick_fair 0.31 -0.0 0.27 perf-profile.children.cycles-pp.xfs_buf_find 0.27 ± 4% -0.0 0.23 ± 4% perf-profile.children.cycles-pp.xfs_read_agi 0.12 ± 15% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.xas_create 0.09 ± 16% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.memcg_check_events 0.14 ± 12% -0.0 0.11 perf-profile.children.cycles-pp.update_load_avg 0.08 ± 8% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.__x86_indirect_thunk_r10 0.35 ± 3% -0.0 0.33 perf-profile.children.cycles-pp.xfs_dir_createname 0.08 ± 11% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.xfs_btree_read_buf_block 0.07 ± 5% -0.0 0.06 perf-profile.children.cycles-pp.xfs_dir3_data_get_ftype 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.xfs_perag_get 0.04 ± 60% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.uncharge_page 0.09 ± 37% +0.0 0.13 perf-profile.children.cycles-pp.ret_from_fork 0.08 ± 35% +0.0 0.13 perf-profile.children.cycles-pp.kthread 0.49 ± 7% +0.0 0.54 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.00 +0.1 0.05 perf-profile.children.cycles-pp.__generic_file_write_iter 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.free_unref_page_prepare 0.22 ± 12% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.xfs_bmapi_reserve_delalloc 1.29 ± 6% +0.1 1.36 ± 5% perf-profile.children.cycles-pp.iomap_set_range_uptodate 0.34 ± 4% +0.1 0.42 perf-profile.children.cycles-pp.__fsnotify_parent 0.11 ± 10% +0.1 0.19 ± 5% perf-profile.children.cycles-pp.delay_tsc 0.64 ± 9% +0.1 0.72 perf-profile.children.cycles-pp._raw_spin_lock 0.28 ± 19% +0.1 0.37 perf-profile.children.cycles-pp.uart_console_write 0.49 ± 6% +0.1 0.58 ± 13% perf-profile.children.cycles-pp.iov_iter_fault_in_readable 0.28 ± 19% +0.1 0.37 perf-profile.children.cycles-pp.serial8250_console_putchar 0.58 ± 12% +0.1 0.71 ± 8% perf-profile.children.cycles-pp.xfs_iext_lookup_extent 1.13 ± 8% +0.5 1.64 ± 3% perf-profile.children.cycles-pp.xfs_file_write_iter 0.99 ± 3% -0.2 0.83 ± 4% perf-profile.self.cycles-pp.xfs_file_buffered_aio_write 0.40 ± 14% -0.1 0.26 ± 3% perf-profile.self.cycles-pp.security_file_permission 0.59 ± 3% -0.1 0.48 ± 12% perf-profile.self.cycles-pp.xas_load 0.40 ± 6% -0.1 0.29 ± 8% perf-profile.self.cycles-pp.xfs_errortag_test 0.60 ± 3% -0.1 0.51 perf-profile.self.cycles-pp.up_write 1.02 ± 2% -0.1 0.94 perf-profile.self.cycles-pp.___might_sleep 0.31 ± 12% -0.1 0.23 perf-profile.self.cycles-pp.__add_to_page_cache_locked 0.30 ± 4% -0.1 0.22 ± 6% perf-profile.self.cycles-pp.file_update_time 0.19 ± 6% -0.1 0.12 ± 8% perf-profile.self.cycles-pp.workingset_update_node 0.48 ± 3% -0.1 0.42 ± 6% perf-profile.self.cycles-pp.xfs_file_iomap_begin 0.29 ± 8% -0.1 0.23 ± 4% perf-profile.self.cycles-pp.xfs_fs_dirty_inode 0.17 ± 12% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.lock_page_memcg 0.07 ± 11% -0.0 0.03 ±100% perf-profile.self.cycles-pp.__update_load_avg_se 0.14 ± 7% -0.0 0.10 ± 15% perf-profile.self.cycles-pp.mem_cgroup_commit_charge 0.10 ± 18% -0.0 0.06 perf-profile.self.cycles-pp.xas_create 0.15 ± 7% -0.0 0.12 ± 8% perf-profile.self.cycles-pp.__fdget_pos 0.09 ± 21% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.mem_cgroup_try_charge 0.07 ± 12% -0.0 0.05 perf-profile.self.cycles-pp.__percpu_counter_sum 0.07 -0.0 0.06 perf-profile.self.cycles-pp.xfs_dir3_data_get_ftype 0.06 ± 17% +0.0 0.08 ± 12% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.36 ± 4% +0.0 0.39 ± 2% perf-profile.self.cycles-pp.xas_store 0.05 ± 58% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.xfs_agino_range 0.47 ± 3% +0.0 0.51 perf-profile.self.cycles-pp.__list_del_entry_valid 0.25 ± 11% +0.0 0.29 ± 6% perf-profile.self.cycles-pp.__alloc_pages_nodemask 0.04 ± 58% +0.0 0.08 ± 12% perf-profile.self.cycles-pp.uncharge_page 0.42 ± 8% +0.0 0.46 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.41 ± 4% +0.0 0.46 perf-profile.self.cycles-pp.__mark_inode_dirty 0.32 ± 5% +0.0 0.36 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.11 ± 10% +0.0 0.15 ± 6% perf-profile.self.cycles-pp.delay_tsc 0.08 ± 23% +0.1 0.13 ± 7% perf-profile.self.cycles-pp.__lru_cache_add 0.47 ± 6% +0.1 0.53 ± 4% perf-profile.self.cycles-pp.iov_iter_fault_in_readable 0.41 ± 3% +0.1 0.47 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.31 ± 10% +0.1 0.38 ± 5% perf-profile.self.cycles-pp.new_sync_write 0.32 ± 4% +0.1 0.40 ± 3% perf-profile.self.cycles-pp.__fsnotify_parent 0.57 ± 13% +0.1 0.66 ± 6% perf-profile.self.cycles-pp.xfs_iext_lookup_extent 0.55 ± 8% +0.1 0.65 ± 5% perf-profile.self.cycles-pp.iomap_write_actor 1.08 ± 9% +0.5 1.56 ± 2% perf-profile.self.cycles-pp.xfs_file_write_iter *************************************************************************************************** lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory ========================================================================================= compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/20/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/aim9/reaim/0x14 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :2 dmesg.WARNING:at#for_ip_interrupt_entry/0x 0:4 -4% 0:2 perf-profile.children.cycles-pp.error_entry 0:4 -3% 0:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 1589 -1.6% 1564 reaim.child_utime 92.73 +3.1% 95.62 reaim.jti 6.68 ± 20% -39.9% 4.02 ± 4% reaim.std_dev_percent 2.29 ± 25% -37.7% 1.43 ± 2% reaim.std_dev_time 9735811 ± 5% -20.4% 7751707 reaim.time.involuntary_context_switches 4456 +2.0% 4544 reaim.time.maximum_resident_set_size 9.237e+08 +1.4% 9.371e+08 reaim.time.minor_page_faults 31822 -1.6% 31324 reaim.time.user_time 39972125 ± 3% +7.1% 42816711 reaim.time.voluntary_context_switches 5363 ± 9% -12.5% 4693 boot-time.idle 18335 ± 68% -98.1% 355.00 numa-numastat.node0.other_node 866.09 ± 3% +4.9% 908.47 pmeter.Average_Active_Power 374417 ± 9% -25.0% 280892 ± 5% meminfo.DirectMap4k 244169 ± 3% +9.3% 266898 ± 2% meminfo.PageTables 0.08 ± 16% -0.0 0.04 ± 2% mpstat.cpu.all.iowait% 0.02 ± 10% +0.0 0.02 ± 8% mpstat.cpu.all.soft% 2.67 ± 17% +68.8% 4.50 ± 11% vmstat.procs.b 102516 ± 5% +15.0% 117851 vmstat.system.cs 306586 ± 9% -14.3% 262885 ± 2% softirqs.CPU133.TIMER 301177 ± 10% -14.4% 257932 ± 2% softirqs.CPU134.TIMER 348974 ± 20% -25.6% 259770 ± 2% softirqs.CPU61.TIMER 299486 ± 9% -15.4% 253362 softirqs.CPU64.TIMER 80298 ± 3% +9.2% 87708 ± 3% softirqs.CPU94.NET_RX 27.45 ± 4% -5.2 22.24 ± 11% turbostat.C6% 0.85 ±107% +292.7% 3.32 ± 90% turbostat.CPU%c3 16.26 ± 4% -43.9% 9.12 ± 34% turbostat.CPU%c6 52.33 ± 4% +11.8% 58.50 turbostat.CoreTmp 11.62 ± 7% -30.6% 8.06 ± 2% turbostat.Pkg%pc2 0.59 ± 17% -61.0% 0.23 ±100% turbostat.Pkg%pc6 56.67 ± 3% +8.5% 61.50 turbostat.PkgTmp 496.32 ± 3% +6.4% 528.04 turbostat.PkgWatt 72.66 +4.6% 76.01 turbostat.RAMWatt 10083 ± 3% +50.5% 15171 ± 28% numa-vmstat.node0.nr_slab_reclaimable 19593 ± 64% -93.1% 1353 ± 30% numa-vmstat.node0.numa_other 126905 ± 7% -14.4% 108652 numa-vmstat.node1.nr_file_pages 22060 ± 13% +42.3% 31389 ± 26% numa-vmstat.node1.nr_kernel_stack 11946 ± 13% +48.1% 17697 ± 25% numa-vmstat.node1.nr_page_table_pages 62212 ± 8% -32.9% 41751 numa-vmstat.node1.nr_shmem 51065 ± 15% -18.8% 41449 ± 4% numa-vmstat.node2.nr_shmem 152702 ± 12% +20.5% 184056 ± 6% numa-vmstat.node3.nr_active_anon 95312 ± 15% +34.3% 128000 numa-vmstat.node3.nr_anon_pages 47.00 ± 51% +124.5% 105.50 ± 22% numa-vmstat.node3.nr_dirtied 47.00 ± 51% +124.5% 105.50 ± 22% numa-vmstat.node3.nr_written 152702 ± 12% +20.5% 184051 ± 6% numa-vmstat.node3.nr_zone_active_anon 17552894 +1.7e+06 19268273 syscalls.sys_close.noise.2% 15759501 ± 2% +1.9e+06 17627007 syscalls.sys_close.noise.25% 17442873 +1.7e+06 19155505 syscalls.sys_close.noise.5% 13169085 ± 2% +1.9e+06 15084330 syscalls.sys_close.noise.50% 10857762 ± 5% +1.1e+06 11964287 ± 3% syscalls.sys_close.noise.75% 1.219e+09 ± 93% -93.4% 80431405 ± 21% syscalls.sys_read.max 2.569e+09 ±123% -2.3e+09 2.578e+08 ± 10% syscalls.sys_read.noise.100% 2.596e+09 ±122% -2.3e+09 2.88e+08 ± 9% syscalls.sys_read.noise.2% 2.591e+09 ±122% -2.3e+09 2.817e+08 ± 10% syscalls.sys_read.noise.25% 2.596e+09 ±122% -2.3e+09 2.878e+08 ± 9% syscalls.sys_read.noise.5% 2.581e+09 ±123% -2.3e+09 2.71e+08 ± 10% syscalls.sys_read.noise.50% 2.574e+09 ±123% -2.3e+09 2.622e+08 ± 10% syscalls.sys_read.noise.75% 399915 ± 3% +9.9% 439446 proc-vmstat.nr_anon_pages 176.67 ± 3% +6.1% 187.50 proc-vmstat.nr_dirtied 487035 ± 5% -6.0% 457770 proc-vmstat.nr_file_pages 110310 ± 3% +7.8% 118870 proc-vmstat.nr_kernel_stack 6819 ± 2% +3.8% 7080 proc-vmstat.nr_mapped 60748 ± 3% +9.9% 66752 ± 2% proc-vmstat.nr_page_table_pages 231076 ± 11% -12.5% 202112 proc-vmstat.nr_shmem 118658 +2.4% 121531 proc-vmstat.nr_slab_unreclaimable 175.33 ± 3% +7.2% 188.00 proc-vmstat.nr_written 226291 ± 21% -32.6% 152628 ± 3% proc-vmstat.numa_hint_faults 175326 ± 20% -30.7% 121440 proc-vmstat.numa_pages_migrated 9.266e+08 +1.4% 9.397e+08 proc-vmstat.pgfault 175326 ± 20% -30.7% 121440 proc-vmstat.pgmigrate_success 40337 ± 3% +50.6% 60759 ± 28% numa-meminfo.node0.KReclaimable 8745 ± 28% +26.4% 11057 ± 3% numa-meminfo.node0.Mapped 40337 ± 3% +50.6% 60759 ± 28% numa-meminfo.node0.SReclaimable 31869 ± 79% -100.0% 0.00 numa-meminfo.node1.AnonHugePages 508265 ± 7% -14.6% 434111 numa-meminfo.node1.FilePages 22056 ± 13% +41.2% 31138 ± 25% numa-meminfo.node1.KernelStack 47686 ± 13% +47.1% 70141 ± 25% numa-meminfo.node1.PageTables 249498 ± 8% -33.3% 166508 numa-meminfo.node1.Shmem 204458 ± 13% -19.1% 165313 ± 5% numa-meminfo.node2.Shmem 613353 ± 12% +19.6% 733539 ± 6% numa-meminfo.node3.Active 613302 ± 12% +19.6% 733387 ± 6% numa-meminfo.node3.Active(anon) 48847 ± 73% +189.7% 141490 ± 2% numa-meminfo.node3.AnonHugePages 382035 ± 15% +33.5% 510087 numa-meminfo.node3.AnonPages 5411 +7.5% 5817 ± 4% numa-meminfo.node3.Mapped 171762 ± 27% +41.6% 243282 ± 2% slabinfo.Acpi-Parse.active_objs 2382 ± 27% +41.9% 3380 ± 2% slabinfo.Acpi-Parse.active_slabs 173961 ± 27% +41.9% 246813 ± 2% slabinfo.Acpi-Parse.num_objs 2382 ± 27% +41.9% 3380 ± 2% slabinfo.Acpi-Parse.num_slabs 987.67 ± 2% +19.1% 1176 slabinfo.biovec-128.active_objs 987.67 ± 2% +19.1% 1176 slabinfo.biovec-128.num_objs 1358 ± 8% -14.9% 1155 slabinfo.kmalloc-rcl-96.active_objs 1358 ± 8% -14.9% 1155 slabinfo.kmalloc-rcl-96.num_objs 191139 ± 8% -28.7% 136191 ± 6% slabinfo.lsm_file_cache.active_objs 746.00 ± 8% -28.8% 531.50 ± 6% slabinfo.lsm_file_cache.active_slabs 191139 ± 8% -28.7% 136191 ± 6% slabinfo.lsm_file_cache.num_objs 746.00 ± 8% -28.8% 531.50 ± 6% slabinfo.lsm_file_cache.num_slabs 1638 ± 7% +15.0% 1883 ± 6% slabinfo.nfs_read_data.active_objs 1638 ± 7% +15.0% 1883 ± 6% slabinfo.nfs_read_data.num_objs 767.69 ± 15% +28.1% 983.45 ± 4% sched_debug.cfs_rq:/.exec_clock.stddev 0.31 ± 5% +19.5% 0.37 sched_debug.cfs_rq:/.nr_running.stddev 13.37 ± 8% -16.3% 11.19 ± 3% sched_debug.cfs_rq:/.removed.load_avg.avg 716.88 ± 16% +19.6% 857.73 ± 3% sched_debug.cfs_rq:/.removed.load_avg.max 611.11 ± 7% -15.6% 515.87 ± 3% sched_debug.cfs_rq:/.removed.runnable_sum.avg 33176 ± 16% +19.6% 39678 ± 3% sched_debug.cfs_rq:/.removed.runnable_sum.max 2.24 ± 21% -44.5% 1.24 ± 14% sched_debug.cfs_rq:/.removed.util_avg.avg 16.35 ± 12% -29.0% 11.62 ± 16% sched_debug.cfs_rq:/.removed.util_avg.stddev 5.39 ± 9% -32.7% 3.63 sched_debug.cfs_rq:/.runnable_load_avg.avg 369.63 ± 29% -44.1% 206.73 ± 8% sched_debug.cfs_rq:/.runnable_load_avg.max 36.03 ± 19% -41.6% 21.04 sched_debug.cfs_rq:/.runnable_load_avg.stddev 225.53 ± 4% +34.9% 304.13 ± 3% sched_debug.cfs_rq:/.util_avg.min 228.37 ± 5% -12.8% 199.11 sched_debug.cfs_rq:/.util_avg.stddev 66833 ± 10% +72.7% 115413 ± 2% sched_debug.cpu.avg_idle.min 5.81 ± 22% -35.2% 3.76 ± 8% sched_debug.cpu.cpu_load[0].avg 445.72 ± 16% -25.6% 331.77 ± 4% sched_debug.cpu.cpu_load[0].max 42.06 ± 20% -32.8% 28.27 ± 5% sched_debug.cpu.cpu_load[0].stddev 5.69 ± 19% -35.5% 3.67 ± 5% sched_debug.cpu.cpu_load[1].avg 337.12 ± 10% -25.8% 250.03 ± 5% sched_debug.cpu.cpu_load[1].max 32.57 ± 14% -32.0% 22.16 ± 3% sched_debug.cpu.cpu_load[1].stddev 5.52 ± 16% -34.0% 3.65 sched_debug.cpu.cpu_load[2].avg 257.90 ± 6% -22.0% 201.17 ± 12% sched_debug.cpu.cpu_load[2].max 26.25 ± 9% -27.9% 18.93 ± 8% sched_debug.cpu.cpu_load[2].stddev 5.15 ± 14% -33.3% 3.43 sched_debug.cpu.cpu_load[3].avg 19.84 ± 7% -23.9% 15.10 ± 11% sched_debug.cpu.cpu_load[3].stddev 4.52 ± 14% -32.9% 3.03 ± 2% sched_debug.cpu.cpu_load[4].avg 28161 ± 5% +9.7% 30894 ± 4% sched_debug.cpu.curr->pid.stddev 597.66 ± 39% +150.9% 1499 ± 26% sched_debug.cpu.max_idle_balance_cost.stddev 6.97 ± 5% +19.7% 8.35 sched_debug.cpu.nr_uninterruptible.avg 385919 ± 5% +9.4% 422383 ± 6% sched_debug.cpu.sched_count.max 104453 ± 7% +21.8% 127211 sched_debug.cpu.sched_goidle.avg 110916 ± 7% +19.5% 132580 sched_debug.cpu.sched_goidle.max 100330 ± 7% +22.9% 123301 sched_debug.cpu.sched_goidle.min 157482 ± 4% +13.6% 178917 sched_debug.cpu.ttwu_count.avg 163535 ± 4% +13.5% 185634 sched_debug.cpu.ttwu_count.max 152786 ± 5% +14.1% 174393 sched_debug.cpu.ttwu_count.min 673.39 ± 6% -20.9% 532.74 ± 5% sched_debug.cpu.ttwu_local.stddev 15.49 ± 17% -32.2% 10.50 ± 5% perf-stat.i.MPKI 2.531e+10 ± 7% +6.3% 2.692e+10 perf-stat.i.branch-instructions 2.16 ± 19% -0.7 1.46 ± 4% perf-stat.i.branch-miss-rate% 2.407e+08 ± 7% +6.3% 2.559e+08 perf-stat.i.branch-misses 7.10 ± 3% +1.1 8.17 perf-stat.i.cache-miss-rate% 88290679 ± 7% +13.3% 1e+08 perf-stat.i.cache-misses 1.015e+09 ± 7% +7.1% 1.087e+09 perf-stat.i.cache-references 103162 ± 5% +14.9% 118526 perf-stat.i.context-switches 2.68 ± 13% -23.2% 2.05 perf-stat.i.cpi 35426 ± 6% +15.7% 40979 perf-stat.i.cpu-migrations 10332 ± 27% -45.3% 5653 ± 4% perf-stat.i.cycles-between-cache-misses 0.39 ± 7% -0.1 0.31 ± 12% perf-stat.i.dTLB-load-miss-rate% 4.634e+10 ± 7% +6.3% 4.926e+10 perf-stat.i.dTLB-loads 0.12 ± 3% -0.0 0.11 perf-stat.i.dTLB-store-miss-rate% 7036966 ± 7% +8.8% 7657668 perf-stat.i.iTLB-loads 1.595e+11 ± 7% +6.3% 1.695e+11 perf-stat.i.instructions 7596 +11.9% 8503 perf-stat.i.instructions-per-iTLB-miss 0.53 ± 2% +7.9% 0.57 perf-stat.i.ipc 50913654 ± 7% +15.4% 58779174 perf-stat.i.node-load-misses 16212935 ± 7% +11.2% 18030635 perf-stat.i.node-loads 10815686 ± 7% +10.5% 11953107 perf-stat.i.node-store-misses 9536170 ± 7% +7.9% 10287859 perf-stat.i.node-stores 8.69 +0.5 9.20 perf-stat.overall.cache-miss-rate% 1.68 -1.8% 1.65 perf-stat.overall.cpi 3030 -7.9% 2789 perf-stat.overall.cycles-between-cache-misses 0.11 ± 3% -0.0 0.10 perf-stat.overall.dTLB-store-miss-rate% 72.17 -1.6 70.56 perf-stat.overall.iTLB-load-miss-rate% 8739 ± 2% +5.7% 9236 perf-stat.overall.instructions-per-iTLB-miss 0.60 +1.9% 0.61 perf-stat.overall.ipc 75.84 +0.7 76.52 perf-stat.overall.node-load-miss-rate% 53.12 +0.6 53.73 perf-stat.overall.node-store-miss-rate% 2.523e+10 ± 7% +6.4% 2.685e+10 perf-stat.ps.branch-instructions 2.4e+08 ± 7% +6.4% 2.553e+08 perf-stat.ps.branch-misses 87961660 ± 7% +13.4% 99747792 perf-stat.ps.cache-misses 1.012e+09 ± 7% +7.2% 1.084e+09 perf-stat.ps.cache-references 102771 ± 5% +15.0% 118185 perf-stat.ps.context-switches 35287 ± 6% +15.8% 40858 perf-stat.ps.cpu-migrations 4.619e+10 ± 7% +6.4% 4.914e+10 perf-stat.ps.dTLB-loads 7013138 ± 7% +8.9% 7637838 perf-stat.ps.iTLB-loads 1.59e+11 ± 7% +6.4% 1.691e+11 perf-stat.ps.instructions 50716358 ± 7% +15.6% 58608142 perf-stat.ps.node-load-misses 16157092 ± 7% +11.3% 17983416 perf-stat.ps.node-loads 10773759 ± 7% +10.6% 11918485 perf-stat.ps.node-store-misses 9505149 ± 7% +8.0% 10262547 perf-stat.ps.node-stores 1.62 ± 36% -1.3 0.28 ±100% perf-profile.calltrace.cycles-pp.ftrace_syscall_enter.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.96 ± 30% -1.2 0.75 ± 5% perf-profile.calltrace.cycles-pp.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.10 ± 16% -1.2 3.92 perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write.ksys_write 4.84 ± 16% -1.1 3.72 perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write 1.56 ± 33% -1.0 0.59 perf-profile.calltrace.cycles-pp.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.36 ± 38% +0.6 1.96 perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.00 +0.6 0.65 ± 9% perf-profile.calltrace.cycles-pp.down_write.__vma_adjust.__split_vma.__do_munmap.mmap_region 0.00 +0.6 0.65 ± 9% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma.__do_munmap 0.00 +0.7 0.69 ± 8% perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.__do_munmap.mmap_region.do_mmap 0.00 +0.7 0.69 ± 8% perf-profile.calltrace.cycles-pp.__split_vma.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff 2.64 ± 33% +0.7 3.33 ± 2% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 2.71 ± 33% +0.7 3.42 ± 2% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.36 ± 70% +0.9 1.28 ± 8% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.unmap_region 0.36 ± 70% +0.9 1.29 ± 8% perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.unmap_region.__do_munmap 0.36 ± 70% +0.9 1.30 ± 8% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.__do_munmap.mmap_region 0.38 ± 70% +1.0 1.34 ± 7% perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.mmap_region.do_mmap 0.42 ± 70% +1.0 1.39 ± 6% perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff 1.12 ± 34% +1.0 2.13 ± 2% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.vma_link.mmap_region 2.79 ± 30% +1.1 3.84 ± 4% perf-profile.calltrace.cycles-pp.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve 2.77 ± 30% +1.1 3.83 ± 4% perf-profile.calltrace.cycles-pp.mmput.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file 1.16 ± 35% +1.1 2.21 ± 2% perf-profile.calltrace.cycles-pp.down_write.vma_link.mmap_region.do_mmap.vm_mmap_pgoff 1.15 ± 34% +1.1 2.21 ± 2% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.vma_link.mmap_region.do_mmap 1.29 ± 34% +1.1 2.37 ± 2% perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 1.08 ± 39% +1.3 2.43 ± 6% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma.mprotect_fixup 1.10 ± 38% +1.4 2.45 ± 6% perf-profile.calltrace.cycles-pp.down_write.__vma_adjust.__split_vma.mprotect_fixup.do_mprotect_pkey 1.29 ± 37% +1.4 2.69 ± 6% perf-profile.calltrace.cycles-pp.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64 1.27 ± 37% +1.4 2.67 ± 6% perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect 1.36 ± 37% +1.4 2.77 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.36 ± 37% +1.4 2.77 ± 6% perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.34 ± 37% +1.4 2.75 ± 6% perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.96 ± 37% +1.4 4.39 ± 4% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary 0.64 ± 70% +1.4 2.09 ± 7% perf-profile.calltrace.cycles-pp.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 3.32 ± 37% +1.5 4.79 ± 3% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler 2.58 ± 31% +1.5 4.11 ± 5% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec 0.87 ± 71% +2.1 2.96 ± 7% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma 2.20 ± 37% +2.4 4.62 ± 4% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.16 ± 37% +2.4 4.59 ± 4% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.19 ± 37% +2.4 4.62 ± 4% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.12 ± 37% +2.4 4.55 ± 4% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 10.81 ± 51% +8.5 19.27 ± 4% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit 12.55 ± 49% +8.7 21.27 ± 3% perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.39 ± 48% +8.8 22.17 ± 2% perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.62 ± 48% +8.8 22.43 ± 2% perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.40 ± 39% +9.0 17.41 ± 5% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.dup_mm.copy_process 12.77 ± 51% +9.2 21.96 ± 3% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 12.78 ± 51% +9.2 21.98 ± 3% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 8.56 ± 40% +9.3 17.88 ± 4% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.dup_mm.copy_process._do_fork 8.63 ± 40% +9.4 17.98 ± 4% perf-profile.calltrace.cycles-pp.down_write.dup_mm.copy_process._do_fork.do_syscall_64 13.46 ± 51% +9.4 22.87 ± 2% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.46 ± 51% +9.4 22.87 ± 2% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.46 ± 51% +9.4 22.88 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.45 ± 44% +9.6 18.02 ± 4% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit 10.85 ± 40% +11.0 21.84 ± 5% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.exit_mmap 10.94 ± 40% +11.0 21.97 ± 5% perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput 10.75 ± 41% +11.7 22.43 ± 5% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables 6.37 ± 66% -4.6 1.77 ± 2% perf-profile.children.cycles-pp.apic_timer_interrupt 6.00 ± 66% -4.3 1.71 ± 2% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 4.04 ± 68% -2.9 1.12 perf-profile.children.cycles-pp.hrtimer_interrupt 2.89 ± 59% -1.9 0.99 perf-profile.children.cycles-pp.__hrtimer_run_queues 1.89 ± 91% -1.8 0.07 ± 7% perf-profile.children.cycles-pp.menu_select 1.97 ± 30% -1.2 0.76 ± 4% perf-profile.children.cycles-pp.syscall_trace_enter 4.86 ± 16% -1.1 3.74 perf-profile.children.cycles-pp.generic_perform_write 1.64 ± 35% -1.1 0.53 ± 9% perf-profile.children.cycles-pp.ftrace_syscall_enter 1.22 ± 58% -1.1 0.17 ± 5% perf-profile.children.cycles-pp.trace_buffer_lock_reserve 1.89 ± 48% -1.0 0.86 perf-profile.children.cycles-pp.tick_sched_timer 1.57 ± 32% -1.0 0.60 perf-profile.children.cycles-pp.syscall_slow_exit_work 1.39 ± 37% -0.9 0.46 ± 2% perf-profile.children.cycles-pp.ftrace_syscall_exit 1.71 ± 44% -0.9 0.84 perf-profile.children.cycles-pp.tick_sched_handle 0.97 ± 61% -0.9 0.11 ± 9% perf-profile.children.cycles-pp.ring_buffer_lock_reserve 0.86 ± 95% -0.8 0.06 ± 9% perf-profile.children.cycles-pp.ktime_get 1.59 ± 40% -0.8 0.83 perf-profile.children.cycles-pp.update_process_times 1.30 ± 48% -0.8 0.54 ± 10% perf-profile.children.cycles-pp.irq_exit 0.70 ± 91% -0.6 0.07 ± 7% perf-profile.children.cycles-pp.clockevents_program_event 1.08 ± 31% -0.5 0.59 ± 8% perf-profile.children.cycles-pp.__softirqentry_text_start 0.41 ± 83% -0.4 0.03 ±100% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 1.08 ± 20% -0.3 0.73 ± 2% perf-profile.children.cycles-pp.scheduler_tick 0.32 ± 82% -0.3 0.03 ±100% perf-profile.children.cycles-pp.native_write_msr 0.33 ± 67% -0.3 0.06 ± 16% perf-profile.children.cycles-pp.rebalance_domains 0.41 ± 37% -0.2 0.21 ± 4% perf-profile.children.cycles-pp.native_irq_return_iret 0.60 ± 17% -0.2 0.41 ± 7% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.38 ± 26% -0.2 0.20 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.51 ± 15% -0.2 0.34 ± 8% perf-profile.children.cycles-pp.copyin 0.24 ± 35% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.memcpy_erms 0.30 ± 33% -0.1 0.18 perf-profile.children.cycles-pp.elf_map 0.15 ± 45% -0.1 0.04 ±100% perf-profile.children.cycles-pp.run_rebalance_domains 0.41 ± 14% -0.1 0.30 ± 4% perf-profile.children.cycles-pp.shmem_alloc_page 0.18 ± 44% -0.1 0.08 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.15 ± 35% -0.1 0.07 ± 28% perf-profile.children.cycles-pp.update_blocked_averages 0.23 ± 9% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.get_unmapped_area 0.14 ± 22% -0.1 0.09 ± 17% perf-profile.children.cycles-pp.update_rq_clock 0.15 ± 10% -0.1 0.10 ± 10% perf-profile.children.cycles-pp.vfs_statx 0.18 ± 14% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.__x64_sys_getdents 0.18 ± 12% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.iterate_dir 0.13 ± 6% -0.0 0.10 ± 5% perf-profile.children.cycles-pp.__do_sys_newstat 0.12 ± 6% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.sock_write_iter 0.06 -0.0 0.03 ±100% perf-profile.children.cycles-pp.__ip_queue_xmit 0.06 -0.0 0.03 ±100% perf-profile.children.cycles-pp.sock_read_iter 0.12 ± 10% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.sock_sendmsg 0.08 -0.0 0.05 perf-profile.children.cycles-pp.__x64_sys_kill 0.08 ± 5% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.iov_iter_advance 0.09 ± 10% -0.0 0.06 perf-profile.children.cycles-pp.__set_page_dirty_no_writeback 0.08 ± 6% -0.0 0.05 perf-profile.children.cycles-pp.__x64_sys_write 0.10 ± 12% -0.0 0.07 perf-profile.children.cycles-pp.do_signal 0.09 ± 13% -0.0 0.07 perf-profile.children.cycles-pp.security_mmap_addr 0.09 ± 9% -0.0 0.07 perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.08 ± 17% -0.0 0.06 perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.08 ± 6% -0.0 0.06 perf-profile.children.cycles-pp.ip_output 0.07 ± 7% -0.0 0.05 perf-profile.children.cycles-pp.__tcp_transmit_skb 0.04 ± 71% +0.0 0.07 perf-profile.children.cycles-pp.pipe_read 0.02 ±141% +0.0 0.06 perf-profile.children.cycles-pp.pipe_wait 0.07 ± 70% +0.0 0.12 perf-profile.children.cycles-pp.osq_unlock 0.07 ± 71% +0.1 0.14 ± 3% perf-profile.children.cycles-pp.dequeue_entity 0.15 ± 35% +0.1 0.22 ± 6% perf-profile.children.cycles-pp.__pte_alloc 0.13 ± 39% +0.1 0.19 ± 5% perf-profile.children.cycles-pp.put_cred_rcu 0.07 ± 72% +0.1 0.14 ± 3% perf-profile.children.cycles-pp.schedule_idle 0.13 ± 73% +0.1 0.27 ± 5% perf-profile.children.cycles-pp.do_task_dead 0.50 ± 5% +0.1 0.63 ± 4% perf-profile.children.cycles-pp.load_balance 0.33 ± 8% +0.2 0.49 ± 4% perf-profile.children.cycles-pp.find_busiest_group 0.31 ± 12% +0.2 0.48 ± 4% perf-profile.children.cycles-pp.update_sd_lb_stats 0.72 ± 26% +0.2 0.95 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.41 ± 37% +0.2 0.64 ± 2% perf-profile.children.cycles-pp.schedule 0.37 ± 49% +0.3 0.69 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 0.65 ± 42% +0.4 1.05 ± 3% perf-profile.children.cycles-pp.__schedule 0.34 ± 62% +0.5 0.79 perf-profile.children.cycles-pp.wake_up_q 0.42 ± 59% +0.5 0.88 perf-profile.children.cycles-pp.rwsem_wake 0.51 ± 53% +0.5 0.99 perf-profile.children.cycles-pp.try_to_wake_up 1.39 ± 37% +0.6 1.99 perf-profile.children.cycles-pp.filemap_map_pages 0.79 ± 35% +0.8 1.63 ± 4% perf-profile.children.cycles-pp.unmap_region 1.43 ± 33% +1.0 2.42 ± 2% perf-profile.children.cycles-pp.vma_link 1.22 ± 34% +1.3 2.47 ± 5% perf-profile.children.cycles-pp.__do_munmap 3.51 ± 30% +1.3 4.80 ± 3% perf-profile.children.cycles-pp.flush_old_exec 1.36 ± 37% +1.4 2.77 ± 6% perf-profile.children.cycles-pp.__x64_sys_mprotect 1.36 ± 37% +1.4 2.77 ± 6% perf-profile.children.cycles-pp.do_mprotect_pkey 1.35 ± 37% +1.4 2.77 ± 6% perf-profile.children.cycles-pp.mprotect_fixup 1.64 ± 37% +1.8 3.47 ± 6% perf-profile.children.cycles-pp.__split_vma 1.81 ± 35% +1.8 3.65 ± 5% perf-profile.children.cycles-pp.__vma_adjust 2.37 ± 35% +2.4 4.72 ± 4% perf-profile.children.cycles-pp.vm_mmap_pgoff 2.28 ± 36% +2.4 4.64 ± 4% perf-profile.children.cycles-pp.mmap_region 2.32 ± 36% +2.4 4.69 ± 4% perf-profile.children.cycles-pp.do_mmap 2.20 ± 37% +2.4 4.62 ± 4% perf-profile.children.cycles-pp.ksys_mmap_pgoff 12.55 ± 49% +8.7 21.27 ± 3% perf-profile.children.cycles-pp.dup_mm 13.39 ± 48% +8.8 22.17 ± 2% perf-profile.children.cycles-pp.copy_process 13.62 ± 48% +8.8 22.43 ± 2% perf-profile.children.cycles-pp._do_fork 13.46 ± 51% +9.4 22.87 ± 2% perf-profile.children.cycles-pp.do_group_exit 13.46 ± 51% +9.4 22.87 ± 2% perf-profile.children.cycles-pp.do_exit 13.46 ± 51% +9.4 22.88 ± 2% perf-profile.children.cycles-pp.__x64_sys_exit_group 16.26 ± 46% +10.5 26.75 ± 3% perf-profile.children.cycles-pp.exit_mmap 16.27 ± 46% +10.5 26.77 ± 3% perf-profile.children.cycles-pp.mmput 14.39 ± 46% +10.6 25.03 ± 4% perf-profile.children.cycles-pp.free_pgtables 11.62 ± 38% +11.8 23.45 ± 5% perf-profile.children.cycles-pp.unlink_file_vma 26.18 ± 46% +19.5 45.70 ± 5% perf-profile.children.cycles-pp.osq_lock 27.24 ± 46% +20.3 47.53 ± 4% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 27.66 ± 46% +20.4 48.02 ± 4% perf-profile.children.cycles-pp.down_write 0.32 ± 82% -0.3 0.03 ±100% perf-profile.self.cycles-pp.native_write_msr 0.34 ± 43% -0.2 0.10 ± 10% perf-profile.self.cycles-pp.ring_buffer_lock_reserve 0.66 ± 3% -0.2 0.44 ± 11% perf-profile.self.cycles-pp.ftrace_syscall_enter 0.41 ± 37% -0.2 0.21 ± 4% perf-profile.self.cycles-pp.native_irq_return_iret 0.34 ± 26% -0.2 0.17 ± 5% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.50 ± 15% -0.1 0.35 perf-profile.self.cycles-pp.ftrace_syscall_exit 0.23 ± 37% -0.1 0.10 perf-profile.self.cycles-pp.memcpy_erms 0.32 ± 11% -0.1 0.22 ± 6% perf-profile.self.cycles-pp.syscall_trace_enter 0.24 ± 7% -0.1 0.18 ± 11% perf-profile.self.cycles-pp.get_page_from_freelist 0.11 ± 25% -0.0 0.07 perf-profile.self.cycles-pp.trace_buffer_lock_reserve 0.13 ± 13% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.generic_file_write_iter 0.13 ± 12% -0.0 0.10 perf-profile.self.cycles-pp.generic_perform_write 0.08 ± 10% -0.0 0.05 perf-profile.self.cycles-pp.__set_page_dirty_no_writeback 0.08 ± 10% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.ksys_write 0.08 ± 10% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.__x64_sys_brk 0.08 ± 6% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.iov_iter_advance 0.09 ± 10% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.shmem_write_begin 0.07 ± 11% -0.0 0.05 perf-profile.self.cycles-pp.__fdget_pos 0.07 ± 11% -0.0 0.05 perf-profile.self.cycles-pp.file_update_time 0.06 ± 7% -0.0 0.05 perf-profile.self.cycles-pp.vma_merge 0.03 ± 70% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.put_cred_rcu 0.04 ± 71% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.vm_area_dup 0.07 ± 71% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.osq_unlock 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__anon_vma_interval_tree_augment_rotate 0.05 ± 74% +0.1 0.11 perf-profile.self.cycles-pp.try_to_wake_up 0.24 ± 17% +0.1 0.38 ± 5% perf-profile.self.cycles-pp.update_sd_lb_stats 0.39 ± 42% +0.3 0.69 perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.98 ± 39% +0.5 1.48 perf-profile.self.cycles-pp.filemap_map_pages 25.86 ± 46% +19.3 45.15 ± 5% perf-profile.self.cycles-pp.osq_lock 739.00 ± 42% -39.4% 448.00 ± 2% interrupts.51:PCI-MSI.1572870-edge.eth0-TxRx-6 477.67 ± 9% +245.7% 1651 ± 7% interrupts.53:PCI-MSI.1572872-edge.eth0-TxRx-8 683.00 ± 32% -33.6% 453.50 interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15 499.67 ± 9% -13.4% 432.50 interrupts.72:PCI-MSI.1572891-edge.eth0-TxRx-27 24632 ± 6% -12.4% 21588 ± 2% interrupts.CPU0.RES:Rescheduling_interrupts 22338 ± 4% -12.9% 19463 interrupts.CPU1.RES:Rescheduling_interrupts 2217 ± 9% +12.7% 2498 ± 2% interrupts.CPU1.TLB:TLB_shootdowns 21081 ± 3% -17.2% 17455 ± 2% interrupts.CPU10.RES:Rescheduling_interrupts 2023 ± 11% +14.2% 2311 ± 4% interrupts.CPU10.TLB:TLB_shootdowns 10088 ± 3% -10.4% 9042 ± 2% interrupts.CPU100.CAL:Function_call_interrupts 2940 ± 84% -99.5% 15.00 ±100% interrupts.CPU100.NMI:Non-maskable_interrupts 2940 ± 84% -99.5% 15.00 ±100% interrupts.CPU100.PMI:Performance_monitoring_interrupts 21754 -18.6% 17712 ± 3% interrupts.CPU100.RES:Rescheduling_interrupts 2052 ± 11% +17.6% 2413 ± 2% interrupts.CPU100.TLB:TLB_shootdowns 1161 ± 40% -77.6% 260.00 ±100% interrupts.CPU101.NMI:Non-maskable_interrupts 1161 ± 40% -77.6% 260.00 ±100% interrupts.CPU101.PMI:Performance_monitoring_interrupts 20386 -28.0% 14672 interrupts.CPU101.RES:Rescheduling_interrupts 2093 ± 13% +16.0% 2428 ± 2% interrupts.CPU101.TLB:TLB_shootdowns 20075 -23.1% 15434 interrupts.CPU102.RES:Rescheduling_interrupts 1991 ± 9% +20.3% 2396 interrupts.CPU102.TLB:TLB_shootdowns 20283 ± 4% -20.8% 16055 interrupts.CPU103.RES:Rescheduling_interrupts 20393 -23.8% 15537 interrupts.CPU104.RES:Rescheduling_interrupts 20123 -23.5% 15401 ± 3% interrupts.CPU105.RES:Rescheduling_interrupts 2049 ± 9% +17.4% 2405 ± 2% interrupts.CPU105.TLB:TLB_shootdowns 20391 ± 2% -25.1% 15272 ± 5% interrupts.CPU106.RES:Rescheduling_interrupts 2079 ± 6% +18.2% 2457 ± 2% interrupts.CPU106.TLB:TLB_shootdowns 19707 -22.7% 15239 ± 5% interrupts.CPU107.RES:Rescheduling_interrupts 2143 ± 7% +16.1% 2487 ± 3% interrupts.CPU107.TLB:TLB_shootdowns 3426 ± 68% -99.7% 11.00 ±100% interrupts.CPU108.NMI:Non-maskable_interrupts 3426 ± 68% -99.7% 11.00 ±100% interrupts.CPU108.PMI:Performance_monitoring_interrupts 21621 -18.6% 17600 interrupts.CPU108.RES:Rescheduling_interrupts 1947 ± 10% +19.0% 2318 interrupts.CPU108.TLB:TLB_shootdowns 2418 ±121% +195.3% 7141 interrupts.CPU109.NMI:Non-maskable_interrupts 2418 ±121% +195.3% 7141 interrupts.CPU109.PMI:Performance_monitoring_interrupts 21399 ± 2% -13.9% 18430 ± 2% interrupts.CPU109.RES:Rescheduling_interrupts 1978 ± 9% +25.1% 2474 interrupts.CPU109.TLB:TLB_shootdowns 20465 ± 2% -22.3% 15895 interrupts.CPU11.RES:Rescheduling_interrupts 21345 ± 2% -18.3% 17435 ± 4% interrupts.CPU110.RES:Rescheduling_interrupts 1971 ± 8% +18.6% 2338 ± 4% interrupts.CPU110.TLB:TLB_shootdowns 20086 ± 3% -19.1% 16258 ± 2% interrupts.CPU111.RES:Rescheduling_interrupts 2047 ± 5% +17.2% 2399 ± 13% interrupts.CPU111.TLB:TLB_shootdowns 1472 ±109% +380.9% 7082 ± 2% interrupts.CPU112.NMI:Non-maskable_interrupts 1472 ±109% +380.9% 7082 ± 2% interrupts.CPU112.PMI:Performance_monitoring_interrupts 20424 -20.0% 16331 interrupts.CPU112.RES:Rescheduling_interrupts 2094 ± 11% +12.6% 2358 ± 3% interrupts.CPU112.TLB:TLB_shootdowns 20019 -21.5% 15722 ± 4% interrupts.CPU113.RES:Rescheduling_interrupts 2022 ± 13% +17.1% 2368 ± 3% interrupts.CPU113.TLB:TLB_shootdowns 20327 -19.4% 16374 ± 2% interrupts.CPU114.RES:Rescheduling_interrupts 2022 ± 12% +19.0% 2407 interrupts.CPU114.TLB:TLB_shootdowns 20429 ± 2% -21.3% 16079 ± 5% interrupts.CPU115.RES:Rescheduling_interrupts 2091 ± 9% +15.5% 2415 interrupts.CPU115.TLB:TLB_shootdowns 20389 ± 2% -13.6% 17614 interrupts.CPU116.RES:Rescheduling_interrupts 2041 ± 11% +17.8% 2405 ± 2% interrupts.CPU116.TLB:TLB_shootdowns 20250 ± 2% -18.7% 16460 ± 2% interrupts.CPU117.RES:Rescheduling_interrupts 20035 -20.8% 15878 interrupts.CPU118.RES:Rescheduling_interrupts 1960 ± 8% +16.7% 2288 ± 7% interrupts.CPU118.TLB:TLB_shootdowns 19647 ± 2% -21.1% 15504 ± 5% interrupts.CPU119.RES:Rescheduling_interrupts 2159 ± 6% +7.4% 2319 ± 2% interrupts.CPU119.TLB:TLB_shootdowns 2359 ±123% +200.0% 7078 interrupts.CPU12.NMI:Non-maskable_interrupts 2359 ±123% +200.0% 7078 interrupts.CPU12.PMI:Performance_monitoring_interrupts 20907 ± 3% -13.9% 17995 interrupts.CPU12.RES:Rescheduling_interrupts 3773 ± 54% +87.1% 7060 interrupts.CPU120.NMI:Non-maskable_interrupts 3773 ± 54% +87.1% 7060 interrupts.CPU120.PMI:Performance_monitoring_interrupts 20164 -19.9% 16159 interrupts.CPU120.RES:Rescheduling_interrupts 2107 ± 10% +14.7% 2417 ± 4% interrupts.CPU120.TLB:TLB_shootdowns 1520 ± 87% +365.3% 7076 interrupts.CPU121.NMI:Non-maskable_interrupts 1520 ± 87% +365.3% 7076 interrupts.CPU121.PMI:Performance_monitoring_interrupts 19674 ± 3% -17.3% 16266 interrupts.CPU121.RES:Rescheduling_interrupts 1950 ± 11% +23.2% 2403 ± 2% interrupts.CPU121.TLB:TLB_shootdowns 264.67 ±100% +1873.2% 5222 ± 30% interrupts.CPU122.NMI:Non-maskable_interrupts 264.67 ±100% +1873.2% 5222 ± 30% interrupts.CPU122.PMI:Performance_monitoring_interrupts 19613 ± 2% -17.5% 16190 interrupts.CPU122.RES:Rescheduling_interrupts 2100 ± 10% +20.5% 2530 ± 10% interrupts.CPU122.TLB:TLB_shootdowns 1136 ± 54% -93.0% 80.00 ±100% interrupts.CPU123.NMI:Non-maskable_interrupts 1136 ± 54% -93.0% 80.00 ±100% interrupts.CPU123.PMI:Performance_monitoring_interrupts 20297 -26.2% 14973 interrupts.CPU123.RES:Rescheduling_interrupts 20612 ± 3% -19.7% 16553 ± 2% interrupts.CPU124.RES:Rescheduling_interrupts 2179 ± 39% +142.7% 5289 ± 31% interrupts.CPU125.NMI:Non-maskable_interrupts 2179 ± 39% +142.7% 5289 ± 31% interrupts.CPU125.PMI:Performance_monitoring_interrupts 20258 -20.6% 16085 interrupts.CPU125.RES:Rescheduling_interrupts 2161 ± 39% +225.4% 7032 interrupts.CPU126.NMI:Non-maskable_interrupts 2161 ± 39% +225.4% 7032 interrupts.CPU126.PMI:Performance_monitoring_interrupts 22110 ± 3% -13.9% 19029 interrupts.CPU126.RES:Rescheduling_interrupts 2050 ± 9% +16.7% 2391 ± 5% interrupts.CPU126.TLB:TLB_shootdowns 21262 -12.1% 18698 interrupts.CPU127.RES:Rescheduling_interrupts 2097 ± 10% +14.6% 2404 interrupts.CPU127.TLB:TLB_shootdowns 21213 ± 3% -15.8% 17857 ± 2% interrupts.CPU128.RES:Rescheduling_interrupts 1999 ± 12% +21.0% 2418 ± 4% interrupts.CPU128.TLB:TLB_shootdowns 20900 ± 2% -20.3% 16649 ± 4% interrupts.CPU129.RES:Rescheduling_interrupts 2038 ± 12% +11.9% 2280 ± 4% interrupts.CPU129.TLB:TLB_shootdowns 2670 ±106% +99.1% 5316 ± 35% interrupts.CPU13.NMI:Non-maskable_interrupts 2670 ±106% +99.1% 5316 ± 35% interrupts.CPU13.PMI:Performance_monitoring_interrupts 20950 ± 3% -14.5% 17911 interrupts.CPU13.RES:Rescheduling_interrupts 2117 ± 11% +9.3% 2315 ± 7% interrupts.CPU13.TLB:TLB_shootdowns 1031 ± 73% +411.8% 5280 ± 31% interrupts.CPU130.NMI:Non-maskable_interrupts 1031 ± 73% +411.8% 5280 ± 31% interrupts.CPU130.PMI:Performance_monitoring_interrupts 20559 ± 2% -19.7% 16516 interrupts.CPU130.RES:Rescheduling_interrupts 1991 ± 9% +16.8% 2326 interrupts.CPU130.TLB:TLB_shootdowns 2630 ±108% +100.7% 5279 ± 30% interrupts.CPU131.NMI:Non-maskable_interrupts 2630 ±108% +100.7% 5279 ± 30% interrupts.CPU131.PMI:Performance_monitoring_interrupts 20323 ± 2% -17.9% 16683 interrupts.CPU131.RES:Rescheduling_interrupts 2094 ± 10% +12.0% 2347 ± 4% interrupts.CPU131.TLB:TLB_shootdowns 2703 ± 38% +157.0% 6949 interrupts.CPU132.NMI:Non-maskable_interrupts 2703 ± 38% +157.0% 6949 interrupts.CPU132.PMI:Performance_monitoring_interrupts 21400 ± 2% -11.1% 19014 interrupts.CPU132.RES:Rescheduling_interrupts 2116 ± 8% +13.5% 2402 ± 9% interrupts.CPU132.TLB:TLB_shootdowns 20862 ± 2% -21.9% 16294 ± 2% interrupts.CPU133.RES:Rescheduling_interrupts 2036 ± 8% +12.1% 2283 ± 6% interrupts.CPU133.TLB:TLB_shootdowns 1662 ± 77% +215.9% 5252 ± 31% interrupts.CPU134.NMI:Non-maskable_interrupts 1662 ± 77% +215.9% 5252 ± 31% interrupts.CPU134.PMI:Performance_monitoring_interrupts 20319 -19.9% 16272 interrupts.CPU134.RES:Rescheduling_interrupts 2045 ± 10% +10.0% 2249 ± 5% interrupts.CPU134.TLB:TLB_shootdowns 20071 ± 2% -19.9% 16068 ± 4% interrupts.CPU135.RES:Rescheduling_interrupts 2031 ± 10% +12.1% 2276 ± 5% interrupts.CPU135.TLB:TLB_shootdowns 20438 -19.5% 16448 ± 3% interrupts.CPU136.RES:Rescheduling_interrupts 2045 ± 10% +23.0% 2515 interrupts.CPU136.TLB:TLB_shootdowns 788.00 ± 72% +797.5% 7072 ± 2% interrupts.CPU137.NMI:Non-maskable_interrupts 788.00 ± 72% +797.5% 7072 ± 2% interrupts.CPU137.PMI:Performance_monitoring_interrupts 20227 -18.2% 16546 interrupts.CPU137.RES:Rescheduling_interrupts 2074 ± 10% +12.9% 2341 ± 4% interrupts.CPU137.TLB:TLB_shootdowns 1653 ± 90% +332.2% 7147 ± 2% interrupts.CPU138.NMI:Non-maskable_interrupts 1653 ± 90% +332.2% 7147 ± 2% interrupts.CPU138.PMI:Performance_monitoring_interrupts 20251 ± 2% -18.0% 16607 interrupts.CPU138.RES:Rescheduling_interrupts 19889 ± 2% -22.3% 15451 interrupts.CPU139.RES:Rescheduling_interrupts 2040 ± 7% +19.8% 2444 ± 4% interrupts.CPU139.TLB:TLB_shootdowns 3569 ± 67% -100.0% 0.50 ±100% interrupts.CPU14.NMI:Non-maskable_interrupts 3569 ± 67% -100.0% 0.50 ±100% interrupts.CPU14.PMI:Performance_monitoring_interrupts 20557 ± 2% -19.9% 16471 interrupts.CPU14.RES:Rescheduling_interrupts 482.00 ±122% +1360.9% 7041 ± 2% interrupts.CPU140.NMI:Non-maskable_interrupts 482.00 ±122% +1360.9% 7041 ± 2% interrupts.CPU140.PMI:Performance_monitoring_interrupts 21150 -15.1% 17964 ± 2% interrupts.CPU140.RES:Rescheduling_interrupts 872.33 ± 69% +505.7% 5283 ± 31% interrupts.CPU141.NMI:Non-maskable_interrupts 872.33 ± 69% +505.7% 5283 ± 31% interrupts.CPU141.PMI:Performance_monitoring_interrupts 20878 ± 3% -12.5% 18258 interrupts.CPU141.RES:Rescheduling_interrupts 1889 ± 10% +23.5% 2333 ± 4% interrupts.CPU141.TLB:TLB_shootdowns 2737 ± 37% +93.6% 5300 ± 32% interrupts.CPU142.NMI:Non-maskable_interrupts 2737 ± 37% +93.6% 5300 ± 32% interrupts.CPU142.PMI:Performance_monitoring_interrupts 20634 -20.8% 16339 interrupts.CPU142.RES:Rescheduling_interrupts 2004 ± 12% +24.1% 2487 ± 2% interrupts.CPU142.TLB:TLB_shootdowns 2756 ± 38% -98.3% 47.00 ± 23% interrupts.CPU143.NMI:Non-maskable_interrupts 2756 ± 38% -98.3% 47.00 ± 23% interrupts.CPU143.PMI:Performance_monitoring_interrupts 19788 ± 2% -25.4% 14763 interrupts.CPU143.RES:Rescheduling_interrupts 2147 ± 10% +17.3% 2518 interrupts.CPU143.TLB:TLB_shootdowns 683.00 ± 32% -33.6% 453.50 interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15 20730 ± 2% -18.7% 16860 interrupts.CPU15.RES:Rescheduling_interrupts 20331 ± 2% -23.6% 15523 interrupts.CPU16.RES:Rescheduling_interrupts 1949 ± 14% +23.5% 2407 interrupts.CPU16.TLB:TLB_shootdowns 20198 ± 3% -20.5% 16063 interrupts.CPU17.RES:Rescheduling_interrupts 2090 ± 12% +15.2% 2409 ± 2% interrupts.CPU17.TLB:TLB_shootdowns 22240 ± 5% -16.8% 18511 ± 3% interrupts.CPU19.RES:Rescheduling_interrupts 1074 ±141% +395.0% 5316 ± 34% interrupts.CPU2.NMI:Non-maskable_interrupts 1074 ±141% +395.0% 5316 ± 34% interrupts.CPU2.PMI:Performance_monitoring_interrupts 21389 ± 3% -8.4% 19582 interrupts.CPU2.RES:Rescheduling_interrupts 21922 ± 2% -16.9% 18226 ± 2% interrupts.CPU20.RES:Rescheduling_interrupts 22730 ± 4% -17.2% 18809 ± 5% interrupts.CPU21.RES:Rescheduling_interrupts 21374 ± 3% -16.8% 17777 ± 2% interrupts.CPU22.RES:Rescheduling_interrupts 2034 ± 9% +16.3% 2365 ± 3% interrupts.CPU22.TLB:TLB_shootdowns 2148 ±141% +144.1% 5244 ± 35% interrupts.CPU23.NMI:Non-maskable_interrupts 2148 ±141% +144.1% 5244 ± 35% interrupts.CPU23.PMI:Performance_monitoring_interrupts 20429 ± 4% -14.3% 17512 interrupts.CPU23.RES:Rescheduling_interrupts 1964 ± 9% +21.5% 2386 ± 2% interrupts.CPU23.TLB:TLB_shootdowns 20429 -15.6% 17238 ± 2% interrupts.CPU24.RES:Rescheduling_interrupts 1925 ± 12% +25.1% 2408 ± 2% interrupts.CPU24.TLB:TLB_shootdowns 20575 -16.3% 17216 ± 5% interrupts.CPU25.RES:Rescheduling_interrupts 2015 ± 10% +19.3% 2405 interrupts.CPU25.TLB:TLB_shootdowns 20893 ± 3% -18.9% 16938 ± 5% interrupts.CPU26.RES:Rescheduling_interrupts 1972 ± 9% +19.5% 2356 ± 4% interrupts.CPU26.TLB:TLB_shootdowns 499.67 ± 9% -13.4% 432.50 interrupts.CPU27.72:PCI-MSI.1572891-edge.eth0-TxRx-27 1707 ± 79% +205.5% 5217 ± 31% interrupts.CPU27.NMI:Non-maskable_interrupts 1707 ± 79% +205.5% 5217 ± 31% interrupts.CPU27.PMI:Performance_monitoring_interrupts 20676 ± 3% -17.7% 17015 interrupts.CPU27.RES:Rescheduling_interrupts 612.67 ±136% +751.8% 5219 ± 32% interrupts.CPU28.NMI:Non-maskable_interrupts 612.67 ±136% +751.8% 5219 ± 32% interrupts.CPU28.PMI:Performance_monitoring_interrupts 2018 ± 14% +19.1% 2404 ± 2% interrupts.CPU28.TLB:TLB_shootdowns 2331 ±130% +129.0% 5340 ± 34% interrupts.CPU29.NMI:Non-maskable_interrupts 2331 ±130% +129.0% 5340 ± 34% interrupts.CPU29.PMI:Performance_monitoring_interrupts 2056 ± 10% +16.1% 2386 ± 3% interrupts.CPU29.TLB:TLB_shootdowns 0.67 ±141% +5.7e+05% 3796 ± 86% interrupts.CPU3.NMI:Non-maskable_interrupts 0.67 ±141% +5.7e+05% 3796 ± 86% interrupts.CPU3.PMI:Performance_monitoring_interrupts 21956 -14.3% 18823 interrupts.CPU3.RES:Rescheduling_interrupts 2066 ± 12% +14.1% 2356 ± 4% interrupts.CPU3.TLB:TLB_shootdowns 20772 ± 4% -21.9% 16220 ± 5% interrupts.CPU30.RES:Rescheduling_interrupts 2015 ± 12% +18.5% 2388 ± 6% interrupts.CPU30.TLB:TLB_shootdowns 20566 -21.4% 16163 interrupts.CPU31.RES:Rescheduling_interrupts 20370 ± 3% -19.6% 16372 ± 2% interrupts.CPU32.RES:Rescheduling_interrupts 21077 ± 4% -19.6% 16949 interrupts.CPU33.RES:Rescheduling_interrupts 20812 -20.3% 16588 ± 2% interrupts.CPU34.RES:Rescheduling_interrupts 2005 ± 15% +18.7% 2381 ± 3% interrupts.CPU34.TLB:TLB_shootdowns 20487 ± 3% -22.1% 15957 ± 2% interrupts.CPU35.RES:Rescheduling_interrupts 2015 ± 7% +16.9% 2357 interrupts.CPU35.TLB:TLB_shootdowns 948.67 ± 75% +648.6% 7101 ± 2% interrupts.CPU36.NMI:Non-maskable_interrupts 948.67 ± 75% +648.6% 7101 ± 2% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1991 ± 12% +21.4% 2417 ± 8% interrupts.CPU36.TLB:TLB_shootdowns 21592 -16.4% 18055 interrupts.CPU37.RES:Rescheduling_interrupts 2080 ± 10% +15.8% 2408 ± 2% interrupts.CPU37.TLB:TLB_shootdowns 2071 ± 11% +15.3% 2389 ± 4% interrupts.CPU38.TLB:TLB_shootdowns 21635 ± 2% -14.2% 18572 ± 3% interrupts.CPU39.RES:Rescheduling_interrupts 2047 ± 8% +17.7% 2409 ± 5% interrupts.CPU39.TLB:TLB_shootdowns 22078 ± 6% -15.4% 18681 interrupts.CPU4.RES:Rescheduling_interrupts 1926 ± 12% +18.8% 2288 interrupts.CPU4.TLB:TLB_shootdowns 20553 -20.6% 16318 interrupts.CPU40.RES:Rescheduling_interrupts 1973 ± 15% +21.2% 2391 ± 4% interrupts.CPU40.TLB:TLB_shootdowns 20719 ± 2% -18.5% 16878 ± 7% interrupts.CPU41.RES:Rescheduling_interrupts 1942 ± 13% +21.7% 2364 ± 3% interrupts.CPU41.TLB:TLB_shootdowns 21622 ± 4% -21.1% 17064 ± 4% interrupts.CPU42.RES:Rescheduling_interrupts 21437 ± 2% -20.5% 17049 ± 6% interrupts.CPU43.RES:Rescheduling_interrupts 1996 ± 10% +24.1% 2477 ± 2% interrupts.CPU43.TLB:TLB_shootdowns 21385 -16.5% 17866 interrupts.CPU44.RES:Rescheduling_interrupts 21222 ± 3% -17.4% 17519 ± 5% interrupts.CPU45.RES:Rescheduling_interrupts 21934 -21.4% 17233 interrupts.CPU46.RES:Rescheduling_interrupts 2117 ± 7% +15.4% 2443 ± 5% interrupts.CPU46.TLB:TLB_shootdowns 20216 ± 2% -19.0% 16366 ± 2% interrupts.CPU47.RES:Rescheduling_interrupts 19863 -21.9% 15513 interrupts.CPU48.RES:Rescheduling_interrupts 2055 ± 8% +16.7% 2398 ± 2% interrupts.CPU48.TLB:TLB_shootdowns 20442 ± 3% -22.6% 15830 interrupts.CPU49.RES:Rescheduling_interrupts 21033 ± 4% -15.7% 17732 interrupts.CPU5.RES:Rescheduling_interrupts 20830 ± 2% -23.9% 15855 interrupts.CPU50.RES:Rescheduling_interrupts 2060 ± 9% +13.6% 2341 ± 2% interrupts.CPU50.TLB:TLB_shootdowns 1199 ±120% +201.4% 3615 interrupts.CPU51.NMI:Non-maskable_interrupts 1199 ±120% +201.4% 3615 interrupts.CPU51.PMI:Performance_monitoring_interrupts 20247 ± 4% -16.4% 16927 interrupts.CPU51.RES:Rescheduling_interrupts 2070 ± 10% +17.6% 2434 ± 7% interrupts.CPU51.TLB:TLB_shootdowns 1618 ± 91% +227.9% 5307 ± 30% interrupts.CPU52.NMI:Non-maskable_interrupts 1618 ± 91% +227.9% 5307 ± 30% interrupts.CPU52.PMI:Performance_monitoring_interrupts 21352 -17.4% 17631 interrupts.CPU52.RES:Rescheduling_interrupts 20631 -17.6% 17002 interrupts.CPU53.RES:Rescheduling_interrupts 2143 ± 6% +8.8% 2333 ± 4% interrupts.CPU53.TLB:TLB_shootdowns 23539 ± 2% -14.1% 20216 ± 3% interrupts.CPU54.RES:Rescheduling_interrupts 2037 ± 9% +11.4% 2269 ± 4% interrupts.CPU54.TLB:TLB_shootdowns 22685 -19.3% 18306 ± 4% interrupts.CPU55.RES:Rescheduling_interrupts 2107 ± 10% +9.0% 2296 ± 5% interrupts.CPU55.TLB:TLB_shootdowns 21677 -15.8% 18247 ± 4% interrupts.CPU56.RES:Rescheduling_interrupts 2008 ± 9% +17.3% 2355 ± 7% interrupts.CPU56.TLB:TLB_shootdowns 21908 ± 2% -15.3% 18548 interrupts.CPU57.RES:Rescheduling_interrupts 2047 ± 12% +13.4% 2321 ± 7% interrupts.CPU57.TLB:TLB_shootdowns 2270 ±130% -99.9% 1.50 ± 33% interrupts.CPU58.NMI:Non-maskable_interrupts 2270 ±130% -99.9% 1.50 ± 33% interrupts.CPU58.PMI:Performance_monitoring_interrupts 20646 -19.4% 16650 interrupts.CPU58.RES:Rescheduling_interrupts 21046 -21.5% 16522 ± 2% interrupts.CPU59.RES:Rescheduling_interrupts 1992 ± 14% +22.0% 2430 interrupts.CPU59.TLB:TLB_shootdowns 739.00 ± 42% -39.4% 448.00 ± 2% interrupts.CPU6.51:PCI-MSI.1572870-edge.eth0-TxRx-6 919.33 ±141% +86.4% 1713 ± 99% interrupts.CPU6.NMI:Non-maskable_interrupts 919.33 ±141% +86.4% 1713 ± 99% interrupts.CPU6.PMI:Performance_monitoring_interrupts 1926 ± 9% +16.8% 2251 ± 5% interrupts.CPU6.TLB:TLB_shootdowns 21628 ± 2% -14.4% 18524 interrupts.CPU60.RES:Rescheduling_interrupts 2054 ± 12% +17.5% 2413 interrupts.CPU60.TLB:TLB_shootdowns 1723 ± 82% +311.3% 7088 ± 2% interrupts.CPU61.NMI:Non-maskable_interrupts 1723 ± 82% +311.3% 7088 ± 2% interrupts.CPU61.PMI:Performance_monitoring_interrupts 21809 -14.8% 18586 interrupts.CPU61.RES:Rescheduling_interrupts 2111 ± 10% +15.6% 2440 ± 3% interrupts.CPU61.TLB:TLB_shootdowns 20756 -23.3% 15923 ± 2% interrupts.CPU62.RES:Rescheduling_interrupts 21028 -19.4% 16955 ± 2% interrupts.CPU63.RES:Rescheduling_interrupts 2187 ± 4% +11.1% 2430 ± 7% interrupts.CPU63.TLB:TLB_shootdowns 20854 ± 2% -15.7% 17577 interrupts.CPU64.RES:Rescheduling_interrupts 2076 ± 8% +10.7% 2297 ± 5% interrupts.CPU64.TLB:TLB_shootdowns 20753 -22.6% 16067 interrupts.CPU65.RES:Rescheduling_interrupts 20470 ± 2% -23.3% 15699 interrupts.CPU66.RES:Rescheduling_interrupts 2014 ± 11% +15.1% 2319 interrupts.CPU66.TLB:TLB_shootdowns 1691 ± 79% +211.8% 5272 ± 31% interrupts.CPU67.NMI:Non-maskable_interrupts 1691 ± 79% +211.8% 5272 ± 31% interrupts.CPU67.PMI:Performance_monitoring_interrupts 21412 ± 2% -15.3% 18129 interrupts.CPU67.RES:Rescheduling_interrupts 2057 ± 10% +16.8% 2404 ± 2% interrupts.CPU67.TLB:TLB_shootdowns 21473 ± 2% -16.0% 18042 ± 2% interrupts.CPU68.RES:Rescheduling_interrupts 2056 ± 11% +16.3% 2391 ± 3% interrupts.CPU68.TLB:TLB_shootdowns 21768 ± 3% -19.6% 17502 interrupts.CPU69.RES:Rescheduling_interrupts 2102 ± 7% +15.1% 2419 ± 2% interrupts.CPU69.TLB:TLB_shootdowns 2637 ±103% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts 2637 ±103% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts 21803 -20.8% 17265 interrupts.CPU7.RES:Rescheduling_interrupts 2027 ± 10% +25.0% 2534 interrupts.CPU7.TLB:TLB_shootdowns 20582 -20.4% 16392 interrupts.CPU70.RES:Rescheduling_interrupts 1253 ±141% +465.2% 7086 ± 2% interrupts.CPU71.NMI:Non-maskable_interrupts 1253 ±141% +465.2% 7086 ± 2% interrupts.CPU71.PMI:Performance_monitoring_interrupts 1417 ± 93% +396.7% 7042 interrupts.CPU72.NMI:Non-maskable_interrupts 1417 ± 93% +396.7% 7042 interrupts.CPU72.PMI:Performance_monitoring_interrupts 1976 ± 9% +20.7% 2385 interrupts.CPU72.TLB:TLB_shootdowns 21584 ± 2% -14.4% 18475 interrupts.CPU73.RES:Rescheduling_interrupts 1936 ± 11% +22.0% 2362 ± 2% interrupts.CPU73.TLB:TLB_shootdowns 21004 ± 2% -20.2% 16753 interrupts.CPU74.RES:Rescheduling_interrupts 2068 ± 6% +16.7% 2415 interrupts.CPU74.TLB:TLB_shootdowns 21152 -19.9% 16943 ± 5% interrupts.CPU75.RES:Rescheduling_interrupts 2022 ± 12% +12.2% 2268 ± 4% interrupts.CPU75.TLB:TLB_shootdowns 2334 ±119% +124.7% 5244 ± 31% interrupts.CPU76.NMI:Non-maskable_interrupts 2334 ±119% +124.7% 5244 ± 31% interrupts.CPU76.PMI:Performance_monitoring_interrupts 21464 -13.7% 18515 interrupts.CPU76.RES:Rescheduling_interrupts 1645 ± 75% +330.1% 7078 interrupts.CPU77.NMI:Non-maskable_interrupts 1645 ± 75% +330.1% 7078 interrupts.CPU77.PMI:Performance_monitoring_interrupts 21447 ± 2% -18.9% 17392 interrupts.CPU77.RES:Rescheduling_interrupts 20921 ± 5% -16.2% 17536 interrupts.CPU78.RES:Rescheduling_interrupts 1969 ± 10% +21.8% 2398 ± 11% interrupts.CPU78.TLB:TLB_shootdowns 593.33 ±141% +789.1% 5275 ± 31% interrupts.CPU79.NMI:Non-maskable_interrupts 593.33 ±141% +789.1% 5275 ± 31% interrupts.CPU79.PMI:Performance_monitoring_interrupts 20456 ± 3% -14.8% 17424 ± 2% interrupts.CPU79.RES:Rescheduling_interrupts 2007 ± 5% +18.2% 2373 ± 5% interrupts.CPU79.TLB:TLB_shootdowns 477.67 ± 9% +245.7% 1651 ± 7% interrupts.CPU8.53:PCI-MSI.1572872-edge.eth0-TxRx-8 2284 ± 65% +134.0% 5346 ± 34% interrupts.CPU8.NMI:Non-maskable_interrupts 2284 ± 65% +134.0% 5346 ± 34% interrupts.CPU8.PMI:Performance_monitoring_interrupts 20918 ± 4% -16.7% 17427 ± 4% interrupts.CPU8.RES:Rescheduling_interrupts 20186 ± 2% -20.3% 16087 ± 5% interrupts.CPU80.RES:Rescheduling_interrupts 2751 ± 38% -99.8% 5.50 ± 81% interrupts.CPU81.NMI:Non-maskable_interrupts 2751 ± 38% -99.8% 5.50 ± 81% interrupts.CPU81.PMI:Performance_monitoring_interrupts 20462 -25.4% 15275 interrupts.CPU81.RES:Rescheduling_interrupts 2053 ± 10% +22.0% 2505 ± 4% interrupts.CPU81.TLB:TLB_shootdowns 20707 ± 3% -19.2% 16737 interrupts.CPU82.RES:Rescheduling_interrupts 3209 ± 71% +121.7% 7115 ± 2% interrupts.CPU83.NMI:Non-maskable_interrupts 3209 ± 71% +121.7% 7115 ± 2% interrupts.CPU83.PMI:Performance_monitoring_interrupts 20304 -21.3% 15973 interrupts.CPU83.RES:Rescheduling_interrupts 2031 ± 13% +15.0% 2337 ± 6% interrupts.CPU83.TLB:TLB_shootdowns 2143 ± 38% -95.8% 91.00 ± 86% interrupts.CPU84.NMI:Non-maskable_interrupts 2143 ± 38% -95.8% 91.00 ± 86% interrupts.CPU84.PMI:Performance_monitoring_interrupts 21467 ± 2% -22.0% 16750 interrupts.CPU84.RES:Rescheduling_interrupts 2054 ± 14% +17.9% 2421 ± 4% interrupts.CPU84.TLB:TLB_shootdowns 20823 ± 2% -22.3% 16175 interrupts.CPU85.RES:Rescheduling_interrupts 2156 ± 12% +18.4% 2553 ± 16% interrupts.CPU85.TLB:TLB_shootdowns 1525 ± 16% +247.0% 5292 ± 31% interrupts.CPU86.NMI:Non-maskable_interrupts 1525 ± 16% +247.0% 5292 ± 31% interrupts.CPU86.PMI:Performance_monitoring_interrupts 20385 -18.1% 16698 interrupts.CPU86.RES:Rescheduling_interrupts 20292 -21.2% 15997 ± 4% interrupts.CPU87.RES:Rescheduling_interrupts 2020 ± 16% +24.2% 2510 ± 3% interrupts.CPU87.TLB:TLB_shootdowns 1071 ± 73% +402.3% 5380 ± 34% interrupts.CPU88.NMI:Non-maskable_interrupts 1071 ± 73% +402.3% 5380 ± 34% interrupts.CPU88.PMI:Performance_monitoring_interrupts 20153 ± 2% -17.3% 16669 interrupts.CPU88.RES:Rescheduling_interrupts 2125 ± 12% +14.3% 2429 ± 4% interrupts.CPU88.TLB:TLB_shootdowns 20208 -24.7% 15207 ± 8% interrupts.CPU89.RES:Rescheduling_interrupts 2167 ± 12% +13.9% 2468 ± 3% interrupts.CPU89.TLB:TLB_shootdowns 2132 ±141% +149.4% 5318 ± 35% interrupts.CPU9.NMI:Non-maskable_interrupts 2132 ±141% +149.4% 5318 ± 35% interrupts.CPU9.PMI:Performance_monitoring_interrupts 20990 ± 3% -15.7% 17687 interrupts.CPU9.RES:Rescheduling_interrupts 2037 ± 13% +13.3% 2308 ± 6% interrupts.CPU9.TLB:TLB_shootdowns 3629 ± 57% +46.6% 5321 ± 31% interrupts.CPU90.NMI:Non-maskable_interrupts 3629 ± 57% +46.6% 5321 ± 31% interrupts.CPU90.PMI:Performance_monitoring_interrupts 22658 ± 3% -16.2% 18998 interrupts.CPU90.RES:Rescheduling_interrupts 2055 ± 12% +16.1% 2386 ± 8% interrupts.CPU90.TLB:TLB_shootdowns 1761 ± 89% +201.8% 5314 ± 34% interrupts.CPU91.NMI:Non-maskable_interrupts 1761 ± 89% +201.8% 5314 ± 34% interrupts.CPU91.PMI:Performance_monitoring_interrupts 21470 ± 2% -14.4% 18381 interrupts.CPU91.RES:Rescheduling_interrupts 2008 ± 12% +18.0% 2371 ± 5% interrupts.CPU91.TLB:TLB_shootdowns 20890 ± 3% -18.1% 17101 ± 2% interrupts.CPU92.RES:Rescheduling_interrupts 2194 ± 11% +11.9% 2456 ± 4% interrupts.CPU92.TLB:TLB_shootdowns 22379 -15.5% 18910 ± 4% interrupts.CPU93.RES:Rescheduling_interrupts 1974 ± 8% +18.9% 2346 ± 2% interrupts.CPU93.TLB:TLB_shootdowns 21107 -20.7% 16740 ± 6% interrupts.CPU94.RES:Rescheduling_interrupts 20073 -20.6% 15938 interrupts.CPU95.RES:Rescheduling_interrupts 2011 ± 11% +18.4% 2382 ± 2% interrupts.CPU95.TLB:TLB_shootdowns 20381 -18.5% 16618 ± 4% interrupts.CPU96.RES:Rescheduling_interrupts 20631 -20.2% 16466 ± 3% interrupts.CPU97.RES:Rescheduling_interrupts 2062 ± 6% +13.8% 2347 interrupts.CPU97.TLB:TLB_shootdowns 20499 -25.1% 15362 ± 2% interrupts.CPU98.RES:Rescheduling_interrupts 1521 ± 85% -100.0% 0.50 ±100% interrupts.CPU99.NMI:Non-maskable_interrupts 1521 ± 85% -100.0% 0.50 ±100% interrupts.CPU99.PMI:Performance_monitoring_interrupts 20320 -26.1% 15026 interrupts.CPU99.RES:Rescheduling_interrupts 3022635 -18.3% 2470301 interrupts.RES:Rescheduling_interrupts 296038 ± 10% +15.1% 340639 ± 3% interrupts.TLB:TLB_shootdowns *************************************************************************************************** lkp-skl-4sp1: 192 threads Skylake-SP with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-2019-05-14.cgz/300s/lkp-skl-4sp1/shell8/unixbench commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:4 45% 2:2 perf-profile.calltrace.cycles-pp.error_entry 2:4 -2% 2:2 perf-profile.children.cycles-pp.error_entry 1:4 20% 2:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 4.215e+09 ± 33% +43.4% 6.046e+09 ± 31% cpuidle.C1E.time 0.00 ±135% +0.0 0.01 ± 80% mpstat.cpu.all.soft% 1047 ±155% -99.5% 5.00 proc-vmstat.numa_hint_faults_local 30037 ± 45% -96.4% 1092 ± 64% proc-vmstat.numa_pte_updates 32.20 ± 31% +13.2 45.39 ± 28% turbostat.C1E% 62.11 ± 8% +9.8% 68.22 ± 7% turbostat.CPU%c1 516.50 ± 9% +17.3% 606.00 ± 5% slabinfo.skbuff_fclone_cache.active_objs 516.50 ± 9% +17.3% 606.00 ± 5% slabinfo.skbuff_fclone_cache.num_objs 26982 ± 4% -7.6% 24934 slabinfo.vmap_area.active_objs 63208 ± 5% +7.2% 67766 ± 4% numa-vmstat.node0.nr_file_pages 13378 ± 9% -14.6% 11428 ± 2% numa-vmstat.node1.nr_kernel_stack 6558 ± 28% -29.8% 4601 numa-vmstat.node1.nr_page_table_pages 5421 ± 11% +20.5% 6535 ± 3% numa-vmstat.node3.nr_page_table_pages 272.75 ± 34% +191.3% 794.50 ± 42% numa-vmstat.node3.nr_shmem 6026 ± 23% +30.4% 7856 ± 17% numa-vmstat.node3.nr_slab_reclaimable 3.11 ± 6% -0.2 2.92 perf-stat.i.branch-miss-rate% 19.24 ± 6% +2.3 21.55 ± 8% perf-stat.i.cache-miss-rate% 30.43 ± 3% -1.4 29.05 perf-stat.i.iTLB-load-miss-rate% 90.26 +0.5 90.81 perf-stat.i.node-load-miss-rate% 20.05 ± 6% +2.4 22.49 ± 10% perf-stat.overall.cache-miss-rate% 78.86 +0.9 79.75 perf-stat.overall.node-store-miss-rate% 49116 ± 58% -98.8% 612.50 ± 58% numa-meminfo.node0.AnonHugePages 252747 ± 5% +7.3% 271178 ± 4% numa-meminfo.node0.FilePages 3256 ± 40% -67.0% 1075 ± 13% numa-meminfo.node1.Inactive 829650 ± 10% -21.7% 649923 numa-meminfo.node1.MemUsed 26061 ± 29% -29.2% 18456 numa-meminfo.node1.PageTables 5095 ±168% +311.5% 20968 ± 85% numa-meminfo.node2.AnonHugePages 24108 ± 23% +30.3% 31425 ± 17% numa-meminfo.node3.KReclaimable 21780 ± 11% +21.2% 26407 ± 2% numa-meminfo.node3.PageTables 24108 ± 23% +30.3% 31425 ± 17% numa-meminfo.node3.SReclaimable 1094 ± 34% +190.6% 3179 ± 42% numa-meminfo.node3.Shmem 100972 ± 8% +9.4% 110438 ± 6% numa-meminfo.node3.Slab 2534 ±100% +203.6% 7695 ± 29% sched_debug.cfs_rq:/.MIN_vruntime.avg 486638 ±100% +105.9% 1001882 ± 3% sched_debug.cfs_rq:/.MIN_vruntime.max 35028 ±100% +145.6% 86026 ± 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev 2534 ±100% +203.6% 7695 ± 29% sched_debug.cfs_rq:/.max_vruntime.avg 486638 ±100% +105.9% 1001882 ± 3% sched_debug.cfs_rq:/.max_vruntime.max 35028 ±100% +145.6% 86026 ± 13% sched_debug.cfs_rq:/.max_vruntime.stddev 0.29 ± 15% +16.7% 0.34 ± 5% sched_debug.cfs_rq:/.nr_spread_over.stddev 10223 ±107% +180.5% 28676 ± 46% sched_debug.cfs_rq:/.spread0.avg 82274 ± 16% +26.7% 104241 ± 6% sched_debug.cfs_rq:/.spread0.max -41611 -39.3% -25237 sched_debug.cfs_rq:/.spread0.min 1.47 ± 72% -54.7% 0.67 ± 2% sched_debug.cpu.cpu_load[0].avg 11.54 ±130% -78.0% 2.54 ± 3% sched_debug.cpu.cpu_load[0].stddev 60.88 ± 26% +38.8% 84.50 ± 8% sched_debug.cpu.cpu_load[3].max 9596 ± 20% +26.1% 12101 ± 14% sched_debug.cpu.curr->pid.avg 280526 ± 91% -92.8% 20294 sched_debug.cpu.load.max 50817 ± 6% +17.6% 59740 ± 6% sched_debug.cpu.nr_load_updates.max 1454 ± 11% +34.6% 1957 ± 7% sched_debug.cpu.nr_load_updates.stddev 14922 ± 6% +12.0% 16718 softirqs.CPU0.SCHED 17109 ± 2% +17.3% 20074 ± 11% softirqs.CPU10.RCU 16556 ± 2% +10.4% 18285 ± 4% softirqs.CPU109.RCU 10912 ± 4% +6.6% 11636 ± 3% softirqs.CPU109.SCHED 26380 ± 5% +19.5% 31531 ± 13% softirqs.CPU109.TIMER 17052 ± 4% +34.2% 22878 ± 26% softirqs.CPU11.RCU 10926 ± 4% +27.8% 13963 ± 12% softirqs.CPU11.SCHED 27695 ± 5% +44.6% 40059 ± 23% softirqs.CPU11.TIMER 17034 +6.5% 18135 ± 3% softirqs.CPU119.RCU 10905 ± 3% +7.3% 11705 ± 5% softirqs.CPU119.SCHED 10900 ± 5% +5.2% 11462 ± 5% softirqs.CPU12.SCHED 10705 ± 3% +4.6% 11195 ± 4% softirqs.CPU121.SCHED 10596 ± 4% +5.1% 11139 ± 5% softirqs.CPU138.SCHED 16059 ± 7% +14.7% 18424 ± 10% softirqs.CPU141.RCU 10472 ± 5% +5.5% 11044 ± 5% softirqs.CPU143.SCHED 10663 ± 4% +3.7% 11059 ± 4% softirqs.CPU159.SCHED 10779 ± 5% +13.0% 12177 ± 4% softirqs.CPU180.SCHED 17166 ± 2% +10.7% 19002 ± 5% softirqs.CPU19.RCU 27980 ± 5% +15.6% 32334 ± 2% softirqs.CPU2.TIMER 16568 ± 3% +10.7% 18348 ± 3% softirqs.CPU24.RCU 16860 ± 2% +14.6% 19322 ± 10% softirqs.CPU25.RCU 10843 ± 2% +5.7% 11459 ± 3% softirqs.CPU38.SCHED 10689 ± 3% +4.4% 11163 ± 5% softirqs.CPU46.SCHED 11435 ± 12% +16.7% 13347 ± 13% softirqs.CPU55.SCHED 28041 ± 11% +13.1% 31725 ± 10% softirqs.CPU55.TIMER 10694 ± 4% +10.6% 11832 ± 3% softirqs.CPU61.SCHED 11175 ± 6% +10.5% 12354 ± 8% softirqs.CPU72.SCHED 0.72 ±115% +1.1 1.85 ± 28% perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ±115% +1.1 1.85 ± 28% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ±115% +1.1 1.85 ± 28% perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ±115% +1.1 1.85 ± 28% perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.47 ±100% +1.4 1.85 ± 28% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 0.43 ±173% +1.4 1.85 ± 28% perf-profile.calltrace.cycles-pp.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64 0.00 +1.9 1.85 ± 28% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 44.74 ± 11% +8.3 53.00 ± 5% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 47.19 ± 14% +10.0 57.23 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 47.15 ± 14% +11.3 58.42 perf-profile.calltrace.cycles-pp.secondary_startup_64 47.15 ± 14% +11.3 58.42 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 47.15 ± 14% +11.3 58.42 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 47.15 ± 14% +11.3 58.42 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 46.93 ± 14% +11.5 58.42 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 40.50 ± 19% -10.9 29.62 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 40.29 ± 19% -10.7 29.62 ± 3% perf-profile.children.cycles-pp.do_syscall_64 2.91 ± 47% -2.2 0.67 ±100% perf-profile.children.cycles-pp.mutex_lock 0.72 ±115% +1.1 1.85 ± 28% perf-profile.children.cycles-pp.__x64_sys_mprotect 0.72 ±115% +1.1 1.85 ± 28% perf-profile.children.cycles-pp.do_mmap 0.72 ±115% +1.1 1.85 ± 28% perf-profile.children.cycles-pp.do_mprotect_pkey 0.72 ±115% +1.1 1.85 ± 28% perf-profile.children.cycles-pp.ksys_mmap_pgoff 0.72 ±115% +1.1 1.85 ± 28% perf-profile.children.cycles-pp.mprotect_fixup 0.72 ±115% +1.1 1.85 ± 28% perf-profile.children.cycles-pp.vm_mmap_pgoff 0.47 ±100% +1.4 1.85 ± 28% perf-profile.children.cycles-pp.mmap_region 0.43 ±173% +1.4 1.85 ± 28% perf-profile.children.cycles-pp.do_dentry_open 0.00 +1.9 1.85 ± 28% perf-profile.children.cycles-pp.clockevents_program_event 0.00 +3.7 3.71 ± 28% perf-profile.children.cycles-pp._raw_spin_lock 44.74 ± 11% +8.3 53.00 ± 5% perf-profile.children.cycles-pp.intel_idle 47.15 ± 14% +11.3 58.42 perf-profile.children.cycles-pp.secondary_startup_64 47.15 ± 14% +11.3 58.42 perf-profile.children.cycles-pp.start_secondary 47.15 ± 14% +11.3 58.42 perf-profile.children.cycles-pp.cpu_startup_entry 47.15 ± 14% +11.3 58.42 perf-profile.children.cycles-pp.do_idle 46.93 ± 14% +11.5 58.42 perf-profile.children.cycles-pp.cpuidle_enter 46.93 ± 14% +11.5 58.42 perf-profile.children.cycles-pp.cpuidle_enter_state 2.91 ± 47% -2.2 0.67 ±100% perf-profile.self.cycles-pp.mutex_lock 0.00 +3.7 3.71 ± 28% perf-profile.self.cycles-pp._raw_spin_lock 44.74 ± 11% +8.3 53.00 ± 5% perf-profile.self.cycles-pp.intel_idle 1846 ± 5% +11.6% 2060 ± 7% interrupts.CPU0.TLB:TLB_shootdowns 2000 ± 2% +10.2% 2205 ± 7% interrupts.CPU1.TLB:TLB_shootdowns 2086 ± 4% +9.8% 2291 ± 8% interrupts.CPU100.TLB:TLB_shootdowns 2030 ± 5% +12.5% 2285 ± 8% interrupts.CPU103.TLB:TLB_shootdowns 746.75 ± 8% -14.1% 641.50 ± 3% interrupts.CPU104.RES:Rescheduling_interrupts 1989 ± 5% +13.6% 2259 ± 4% interrupts.CPU107.TLB:TLB_shootdowns 2014 ± 5% +11.6% 2247 ± 3% interrupts.CPU108.TLB:TLB_shootdowns 2024 ± 4% +9.7% 2222 ± 4% interrupts.CPU109.TLB:TLB_shootdowns 2008 ± 2% +16.0% 2330 ± 6% interrupts.CPU11.TLB:TLB_shootdowns 2013 ± 4% +9.9% 2212 ± 7% interrupts.CPU113.TLB:TLB_shootdowns 1967 ± 5% +14.0% 2243 ± 4% interrupts.CPU114.TLB:TLB_shootdowns 1989 ± 5% +9.6% 2180 ± 8% interrupts.CPU117.TLB:TLB_shootdowns 2015 ± 2% +10.0% 2216 ± 4% interrupts.CPU118.TLB:TLB_shootdowns 2006 ± 5% +14.0% 2286 ± 4% interrupts.CPU119.TLB:TLB_shootdowns 899.25 ± 4% +11.0% 998.00 ± 3% interrupts.CPU120.RES:Rescheduling_interrupts 1881 ± 6% +12.9% 2124 ± 8% interrupts.CPU120.TLB:TLB_shootdowns 2070 ± 6% +7.9% 2233 ± 6% interrupts.CPU121.TLB:TLB_shootdowns 134762 ± 4% +16.7% 157232 ± 4% interrupts.CPU128.LOC:Local_timer_interrupts 1999 ± 4% +12.2% 2244 ± 10% interrupts.CPU128.TLB:TLB_shootdowns 2039 ± 8% +8.4% 2212 ± 6% interrupts.CPU129.TLB:TLB_shootdowns 2019 ± 5% +17.3% 2369 ± 7% interrupts.CPU13.TLB:TLB_shootdowns 133696 ± 4% +6.1% 141813 ± 4% interrupts.CPU130.LOC:Local_timer_interrupts 2071 ± 7% +8.0% 2236 ± 8% interrupts.CPU131.TLB:TLB_shootdowns 2063 ± 4% +10.0% 2270 ± 6% interrupts.CPU135.TLB:TLB_shootdowns 2033 ± 5% +11.5% 2266 ± 6% interrupts.CPU136.TLB:TLB_shootdowns 132973 ± 4% +6.2% 141217 ± 4% interrupts.CPU139.LOC:Local_timer_interrupts 2041 ± 6% +10.9% 2265 ± 9% interrupts.CPU140.TLB:TLB_shootdowns 2036 ± 5% +10.8% 2255 ± 8% interrupts.CPU141.TLB:TLB_shootdowns 2023 ± 7% +11.9% 2263 ± 5% interrupts.CPU143.TLB:TLB_shootdowns 2030 ± 6% +10.0% 2233 ± 9% interrupts.CPU145.TLB:TLB_shootdowns 1991 ± 4% +18.5% 2360 ± 5% interrupts.CPU159.TLB:TLB_shootdowns 2091 ± 5% +7.9% 2256 ± 7% interrupts.CPU160.TLB:TLB_shootdowns 2070 ± 4% +10.6% 2290 ± 5% interrupts.CPU161.TLB:TLB_shootdowns 2064 ± 7% +11.9% 2310 ± 5% interrupts.CPU162.TLB:TLB_shootdowns 2062 ± 5% +13.2% 2336 ± 7% interrupts.CPU163.TLB:TLB_shootdowns 2064 ± 8% +12.7% 2326 ± 10% interrupts.CPU165.TLB:TLB_shootdowns 2033 ± 6% +13.2% 2302 ± 9% interrupts.CPU166.TLB:TLB_shootdowns 2088 ± 6% +9.7% 2292 ± 5% interrupts.CPU167.TLB:TLB_shootdowns 2004 ± 6% +9.8% 2201 ± 10% interrupts.CPU174.TLB:TLB_shootdowns 2047 ± 2% +12.2% 2297 ± 9% interrupts.CPU176.TLB:TLB_shootdowns 763.00 ± 6% -10.4% 683.50 interrupts.CPU181.RES:Rescheduling_interrupts 2087 ± 7% +7.5% 2244 ± 7% interrupts.CPU182.TLB:TLB_shootdowns 2101 ± 3% +11.5% 2342 ± 9% interrupts.CPU186.TLB:TLB_shootdowns 2009 ± 4% +14.4% 2299 ± 9% interrupts.CPU187.TLB:TLB_shootdowns 2073 ± 2% +6.2% 2201 ± 6% interrupts.CPU189.TLB:TLB_shootdowns 2070 ± 9% +8.5% 2246 ± 6% interrupts.CPU190.TLB:TLB_shootdowns 1995 ± 4% +11.4% 2223 ± 5% interrupts.CPU23.TLB:TLB_shootdowns 890.25 ± 5% +8.4% 965.00 ± 4% interrupts.CPU24.RES:Rescheduling_interrupts 133004 ± 4% +6.7% 141981 ± 5% interrupts.CPU35.LOC:Local_timer_interrupts 2057 ± 6% +9.7% 2257 ± 4% interrupts.CPU36.TLB:TLB_shootdowns 2033 ± 3% +10.0% 2237 ± 7% interrupts.CPU37.TLB:TLB_shootdowns 133340 ± 5% +5.9% 141238 ± 4% interrupts.CPU42.LOC:Local_timer_interrupts 134080 ± 5% +6.2% 142377 ± 5% interrupts.CPU44.LOC:Local_timer_interrupts 2075 ± 2% +12.6% 2335 ± 6% interrupts.CPU44.TLB:TLB_shootdowns 2018 ± 4% +16.7% 2355 ± 9% interrupts.CPU45.TLB:TLB_shootdowns 2029 ± 4% +16.1% 2356 ± 3% interrupts.CPU5.TLB:TLB_shootdowns 2125 ± 4% +8.0% 2296 ± 4% interrupts.CPU50.TLB:TLB_shootdowns 2054 +8.6% 2231 ± 7% interrupts.CPU54.TLB:TLB_shootdowns 2043 ± 5% +12.9% 2306 ± 9% interrupts.CPU56.TLB:TLB_shootdowns 2098 ± 5% +8.4% 2274 ± 2% interrupts.CPU61.TLB:TLB_shootdowns 2027 ± 2% +13.7% 2306 ± 7% interrupts.CPU63.TLB:TLB_shootdowns 2018 ± 9% +13.1% 2283 ± 7% interrupts.CPU66.TLB:TLB_shootdowns 772.25 ± 6% -10.1% 694.50 interrupts.CPU69.RES:Rescheduling_interrupts 2050 ± 5% +12.8% 2312 ± 10% interrupts.CPU70.TLB:TLB_shootdowns 2114 ± 3% +8.5% 2295 ± 6% interrupts.CPU74.TLB:TLB_shootdowns 2179 ± 6% +8.4% 2361 ± 5% interrupts.CPU86.TLB:TLB_shootdowns 2043 ± 5% +11.3% 2273 ± 7% interrupts.CPU9.TLB:TLB_shootdowns 2078 ± 2% +10.8% 2301 ± 7% interrupts.CPU90.TLB:TLB_shootdowns 2036 ± 4% +8.2% 2204 ± 6% interrupts.CPU92.TLB:TLB_shootdowns 2063 ± 8% +11.0% 2289 ± 9% interrupts.CPU93.TLB:TLB_shootdowns 2008 ± 5% +10.7% 2223 ± 6% interrupts.CPU96.TLB:TLB_shootdowns 1985 ± 6% +11.7% 2218 ± 7% interrupts.CPU98.TLB:TLB_shootdowns 395318 ± 4% +8.3% 428205 ± 8% interrupts.TLB:TLB_shootdowns *************************************************************************************************** lkp-skl-4sp1: 192 threads Skylake-SP with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-2019-05-14.cgz/lkp-skl-4sp1/pwrite2/will-it-scale commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- %stddev %change %stddev \ | \ 407.25 ± 5% -21.8% 318.50 ± 16% kthread_noise.total_time 0.00 ± 6% +0.0 0.00 ± 61% mpstat.cpu.all.soft% 0.05 ± 3% +0.0 0.06 ± 9% mpstat.cpu.all.usr% 6.41 ± 12% -36.3% 4.08 ± 26% irq_exception_noise.__do_page_fault.95th 11.54 ± 5% -30.4% 8.02 ± 6% irq_exception_noise.__do_page_fault.99th 201126 ± 4% +35.7% 272858 ± 24% irq_exception_noise.softirq_time 25808 +1.6% 26225 proc-vmstat.nr_slab_reclaimable 472.50 ± 56% +628.3% 3441 ± 80% proc-vmstat.numa_hint_faults_local 1555625 -2.4% 1517554 proc-vmstat.numa_hit 1410822 ± 2% -2.7% 1372898 proc-vmstat.numa_local 2706 ± 7% +13.1% 3060 ± 8% proc-vmstat.pgactivate 67.00 +28.4% 86.00 ± 19% turbostat.Avg_MHz 4.69 +1.2 5.85 ± 18% turbostat.Busy% 0.09 ± 28% +1883.3% 1.79 ± 92% turbostat.Pkg%pc2 37.00 +5.4% 39.00 turbostat.PkgTmp 191.96 +4.7% 200.91 ± 3% turbostat.PkgWatt 52.35 +17.2% 61.37 ± 12% turbostat.RAMWatt 16717 ± 5% +11.2% 18597 slabinfo.cred_jar.active_objs 16717 ± 5% +11.2% 18597 slabinfo.cred_jar.num_objs 4493 ± 3% -10.8% 4007 ± 2% slabinfo.eventpoll_pwq.active_objs 4493 ± 3% -10.8% 4007 ± 2% slabinfo.eventpoll_pwq.num_objs 51900 ± 4% +6.3% 55166 ± 3% slabinfo.filp.active_objs 52293 ± 4% +6.9% 55879 ± 3% slabinfo.filp.num_objs 1299 ± 6% +11.8% 1453 ± 4% slabinfo.task_group.active_objs 1299 ± 6% +11.8% 1453 ± 4% slabinfo.task_group.num_objs 10619 ± 85% -95.4% 485.50 ± 63% numa-meminfo.node1.Inactive(anon) 1041 ±112% +892.0% 10332 ± 75% numa-meminfo.node2.Inactive 320.75 ± 34% +3121.2% 10332 ± 75% numa-meminfo.node2.Inactive(anon) 15907 ± 21% +60.7% 25570 ± 17% numa-meminfo.node2.KReclaimable 6626 ± 6% +15.9% 7683 ± 7% numa-meminfo.node2.KernelStack 8908 ± 8% +53.3% 13657 ± 31% numa-meminfo.node2.PageTables 15907 ± 21% +60.7% 25570 ± 17% numa-meminfo.node2.SReclaimable 53362 ± 10% +25.5% 66971 numa-meminfo.node2.SUnreclaim 927.75 ± 95% +1203.5% 12093 ± 53% numa-meminfo.node2.Shmem 69269 ± 12% +33.6% 92541 ± 5% numa-meminfo.node2.Slab 2654 ± 85% -95.4% 121.00 ± 63% numa-vmstat.node1.nr_inactive_anon 2161 ± 23% -40.1% 1294 numa-vmstat.node1.nr_mapped 2654 ± 85% -95.4% 121.00 ± 63% numa-vmstat.node1.nr_zone_inactive_anon 79.75 ± 34% +3138.2% 2582 ± 75% numa-vmstat.node2.nr_inactive_anon 6622 ± 6% +15.9% 7676 ± 7% numa-vmstat.node2.nr_kernel_stack 1465 ± 21% +38.6% 2030 ± 30% numa-vmstat.node2.nr_mapped 2224 ± 8% +53.4% 3412 ± 30% numa-vmstat.node2.nr_page_table_pages 231.25 ± 95% +1207.0% 3022 ± 53% numa-vmstat.node2.nr_shmem 3976 ± 21% +60.8% 6392 ± 17% numa-vmstat.node2.nr_slab_reclaimable 13340 ± 10% +25.5% 16744 numa-vmstat.node2.nr_slab_unreclaimable 79.75 ± 34% +3138.2% 2582 ± 75% numa-vmstat.node2.nr_zone_inactive_anon 465714 ± 32% +62.3% 755809 ± 36% numa-vmstat.node2.numa_hit 336044 ± 46% +90.6% 640510 ± 46% numa-vmstat.node2.numa_local 2005 ± 2% +11.6% 2238 ± 10% sched_debug.cfs_rq:/.exec_clock.avg 14.00 ± 4% -12.0% 12.31 sched_debug.cpu.clock.stddev 14.00 ± 4% -12.0% 12.31 sched_debug.cpu.clock_task.stddev 1.37 ± 4% +17.5% 1.61 ± 4% sched_debug.cpu.cpu_load[1].avg 199.10 ± 2% +24.5% 247.97 ± 3% sched_debug.cpu.cpu_load[1].max 14.80 ± 2% +21.4% 17.97 ± 4% sched_debug.cpu.cpu_load[1].stddev 1.31 ± 5% +11.0% 1.46 ± 2% sched_debug.cpu.cpu_load[2].avg 195.25 ± 2% +13.3% 221.23 sched_debug.cpu.cpu_load[2].max 14.48 ± 2% +10.9% 16.06 sched_debug.cpu.cpu_load[2].stddev 5483 ± 15% +201.0% 16507 ± 60% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 8% -11.9% 0.00 sched_debug.cpu.next_balance.stddev 130966 +11.5% 146000 ± 10% sched_debug.cpu.nr_load_updates.avg 139797 +11.3% 155637 ± 9% sched_debug.cpu.nr_load_updates.max 127184 +12.5% 143112 ± 9% sched_debug.cpu.nr_load_updates.min 2.20 +3.6 5.82 ± 61% perf-stat.i.branch-miss-rate% 17065871 +128.2% 38940396 ± 54% perf-stat.i.branch-misses 15171379 ± 4% -11.5% 13428822 ± 5% perf-stat.i.cache-misses 3.61 +34.7% 4.87 ± 23% perf-stat.i.cpi 1.207e+10 +26.4% 1.525e+10 ± 18% perf-stat.i.cpu-cycles 873.59 ± 11% +32.5% 1157 ± 22% perf-stat.i.cycles-between-cache-misses 0.00 ± 4% +0.1 0.09 ± 94% perf-stat.i.dTLB-store-miss-rate% 24641 ± 3% +1481.0% 389570 ± 92% perf-stat.i.dTLB-store-misses 0.28 -21.4% 0.22 ± 23% perf-stat.i.ipc 33574 ± 6% +17.3% 39391 ± 9% perf-stat.i.node-loads 1501402 ± 11% -16.1% 1260418 ± 4% perf-stat.i.node-store-misses 2.35 +3.5 5.82 ± 58% perf-stat.overall.branch-miss-rate% 3.54 +33.5% 4.72 ± 23% perf-stat.overall.cpi 798.25 ± 5% +44.0% 1149 ± 23% perf-stat.overall.cycles-between-cache-misses 0.00 ± 3% +0.1 0.08 ± 93% perf-stat.overall.dTLB-store-miss-rate% 0.28 -20.8% 0.22 ± 23% perf-stat.overall.ipc 17098757 +127.2% 38846297 ± 54% perf-stat.ps.branch-misses 15119630 ± 4% -11.5% 13383715 ± 5% perf-stat.ps.cache-misses 1.204e+10 +26.3% 1.52e+10 ± 17% perf-stat.ps.cpu-cycles 24654 ± 3% +1475.3% 388374 ± 92% perf-stat.ps.dTLB-store-misses 33486 ± 6% +17.3% 39294 ± 8% perf-stat.ps.node-loads 1496284 ± 11% -16.1% 1256127 ± 4% perf-stat.ps.node-store-misses 19.96 ± 21% -7.3 12.65 ± 10% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.20 ± 21% -7.3 12.89 ± 9% perf-profile.calltrace.cycles-pp.vfs_write.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pwrite 20.25 ± 21% -7.3 12.95 ± 9% perf-profile.calltrace.cycles-pp.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pwrite 20.35 ± 21% -7.3 13.05 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pwrite 20.36 ± 21% -7.3 13.07 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_pwrite 19.85 ± 21% -7.3 12.56 ± 10% perf-profile.calltrace.cycles-pp.generic_file_write_iter.new_sync_write.vfs_write.ksys_pwrite64.do_syscall_64 20.92 ± 20% -7.3 13.66 ± 8% perf-profile.calltrace.cycles-pp.__libc_pwrite 14.24 ± 31% -6.7 7.52 ± 9% perf-profile.calltrace.cycles-pp.down_write.generic_file_write_iter.new_sync_write.vfs_write.ksys_pwrite64 13.94 ± 32% -6.7 7.26 ± 8% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.generic_file_write_iter.new_sync_write.vfs_write 0.54 ± 4% +0.0 0.58 ± 2% perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task 1.04 ± 12% +0.5 1.50 perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.00 +0.5 0.53 ± 3% perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 1.23 ± 11% +0.7 1.88 ± 8% perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 3.32 ± 14% +0.7 4.05 ± 9% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.06 ± 9% +0.8 1.86 perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 +0.9 0.93 ± 11% perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 4.92 ± 12% +2.2 7.08 ± 12% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 9.38 ± 14% +2.8 12.21 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 20.57 ± 10% +3.0 23.55 ± 8% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 17.96 ± 10% +3.0 21.00 ± 9% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 43.67 ± 5% +5.1 48.73 ± 7% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 77.15 ± 7% +8.3 85.41 perf-profile.calltrace.cycles-pp.secondary_startup_64 76.62 ± 7% +8.4 85.03 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 76.62 ± 7% +8.4 85.03 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 76.58 ± 7% +8.4 85.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 64.90 ± 6% +8.9 73.77 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 64.56 ± 6% +9.0 73.60 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 21.55 ± 26% -8.2 13.31 ± 9% perf-profile.children.cycles-pp.do_syscall_64 21.57 ± 26% -8.2 13.33 ± 9% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 19.97 ± 21% -7.3 12.66 ± 10% perf-profile.children.cycles-pp.new_sync_write 20.25 ± 21% -7.3 12.95 ± 9% perf-profile.children.cycles-pp.ksys_pwrite64 20.21 ± 21% -7.3 12.91 ± 9% perf-profile.children.cycles-pp.vfs_write 19.85 ± 21% -7.3 12.56 ± 10% perf-profile.children.cycles-pp.generic_file_write_iter 20.96 ± 20% -7.3 13.70 ± 8% perf-profile.children.cycles-pp.__libc_pwrite 14.29 ± 32% -6.8 7.52 ± 9% perf-profile.children.cycles-pp.down_write 14.02 ± 32% -6.7 7.29 ± 8% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 2.11 ± 35% -0.7 1.39 ± 15% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.47 ± 17% -0.2 0.30 ± 4% perf-profile.children.cycles-pp.arch_cpu_idle_enter 0.41 ± 19% -0.2 0.25 ± 4% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust 0.53 ± 8% -0.1 0.41 ± 14% perf-profile.children.cycles-pp.update_load_avg 0.22 ± 11% -0.1 0.15 ± 20% perf-profile.children.cycles-pp.__update_load_avg_se 0.47 ± 3% -0.1 0.42 ± 6% perf-profile.children.cycles-pp.run_timer_softirq 0.10 ± 8% +0.0 0.13 ± 7% perf-profile.children.cycles-pp.__orc_find 0.06 ± 11% +0.0 0.09 ± 11% perf-profile.children.cycles-pp.get_cpu_device 0.14 ± 6% +0.0 0.17 ± 5% perf-profile.children.cycles-pp.rcu_irq_exit 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.update_rt_rq_load_avg 0.08 ± 12% +0.0 0.12 ± 19% perf-profile.children.cycles-pp.intel_pmu_disable_all 0.00 +0.1 0.07 ± 14% perf-profile.children.cycles-pp.rcu_needs_cpu 0.58 ± 4% +0.1 0.67 ± 8% perf-profile.children.cycles-pp.sched_clock 0.41 ± 6% +0.1 0.49 ± 11% perf-profile.children.cycles-pp.unlock_page 0.56 ± 4% +0.1 0.65 ± 9% perf-profile.children.cycles-pp.native_sched_clock 0.05 ± 8% +0.1 0.15 ± 58% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.53 ± 5% +0.1 0.66 ± 15% perf-profile.children.cycles-pp.sched_clock_cpu 0.11 ± 4% +0.2 0.27 ± 32% perf-profile.children.cycles-pp.rcu_irq_enter 0.03 ±100% +0.2 0.18 ± 67% perf-profile.children.cycles-pp.hrtimer_forward 0.06 ± 14% +0.3 0.33 ± 78% perf-profile.children.cycles-pp.tick_check_broadcast_expired 1.06 ± 12% +0.5 1.52 perf-profile.children.cycles-pp.tick_irq_enter 1.26 ± 11% +0.7 1.93 ± 8% perf-profile.children.cycles-pp.irq_enter 1.09 ± 9% +0.8 1.92 perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 5.15 ± 10% +2.1 7.21 ± 12% perf-profile.children.cycles-pp.__hrtimer_run_queues 9.72 ± 12% +2.7 12.41 ± 10% perf-profile.children.cycles-pp.hrtimer_interrupt 18.39 ± 10% +2.9 21.27 ± 9% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 19.78 ± 10% +2.9 22.66 ± 8% perf-profile.children.cycles-pp.apic_timer_interrupt 43.98 ± 4% +5.0 48.98 ± 7% perf-profile.children.cycles-pp.intel_idle 77.21 ± 7% +8.2 85.46 perf-profile.children.cycles-pp.do_idle 77.15 ± 7% +8.3 85.41 perf-profile.children.cycles-pp.secondary_startup_64 77.15 ± 7% +8.3 85.41 perf-profile.children.cycles-pp.cpu_startup_entry 76.62 ± 7% +8.4 85.03 perf-profile.children.cycles-pp.start_secondary 65.36 ± 6% +8.7 74.10 ± 2% perf-profile.children.cycles-pp.cpuidle_enter 65.33 ± 6% +8.8 74.08 ± 2% perf-profile.children.cycles-pp.cpuidle_enter_state 0.39 ± 19% -0.2 0.20 perf-profile.self.cycles-pp.tsc_verify_tsc_adjust 0.47 ± 9% -0.1 0.32 ± 10% perf-profile.self.cycles-pp.__softirqentry_text_start 0.23 ± 4% -0.1 0.14 ± 42% perf-profile.self.cycles-pp.shmem_write_end 0.20 ± 10% -0.1 0.14 ± 21% perf-profile.self.cycles-pp.__update_load_avg_se 0.22 ± 10% -0.1 0.16 ± 6% perf-profile.self.cycles-pp.trace_event_raw_event_softirq 0.07 ± 17% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.intel_pmu_disable_all 0.04 ± 58% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.hrtimer_next_event_without 0.10 ± 8% +0.0 0.12 ± 12% perf-profile.self.cycles-pp.__orc_find 0.23 +0.0 0.26 ± 3% perf-profile.self.cycles-pp.lapic_next_event 0.13 ± 5% +0.0 0.17 ± 5% perf-profile.self.cycles-pp.rcu_irq_exit 0.04 ± 59% +0.0 0.09 ± 11% perf-profile.self.cycles-pp.get_cpu_device 0.28 ± 10% +0.1 0.35 ± 17% perf-profile.self.cycles-pp.hrtimer_interrupt 0.00 +0.1 0.07 ± 14% perf-profile.self.cycles-pp.rcu_needs_cpu 0.40 ± 5% +0.1 0.48 ± 11% perf-profile.self.cycles-pp.unlock_page 0.05 ± 8% +0.1 0.15 ± 58% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.07 ± 11% +0.1 0.17 ± 41% perf-profile.self.cycles-pp.scheduler_tick 0.15 ± 17% +0.1 0.24 ± 26% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.01 ±173% +0.1 0.11 ± 54% perf-profile.self.cycles-pp.timerqueue_del 0.01 ±173% +0.1 0.12 ± 47% perf-profile.self.cycles-pp.irq_enter 0.01 ±173% +0.1 0.12 ± 58% perf-profile.self.cycles-pp.sched_clock 0.13 ± 5% +0.1 0.27 ± 47% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.00 +0.1 0.14 ± 44% perf-profile.self.cycles-pp.rcu_irq_enter 0.00 +0.2 0.17 ± 65% perf-profile.self.cycles-pp.hrtimer_forward 0.04 ± 58% +0.3 0.32 ± 81% perf-profile.self.cycles-pp.tick_check_broadcast_expired 43.97 ± 4% +4.9 48.83 ± 7% perf-profile.self.cycles-pp.intel_idle 31419 ± 15% -24.2% 23802 softirqs.CPU1.RCU 30103 ± 22% -25.2% 22515 ± 3% softirqs.CPU100.RCU 30145 ± 22% -25.7% 22383 ± 3% softirqs.CPU101.RCU 29952 ± 21% -24.9% 22503 ± 3% softirqs.CPU102.RCU 29935 ± 22% -25.4% 22342 ± 4% softirqs.CPU103.RCU 30697 ± 19% -27.5% 22263 ± 4% softirqs.CPU104.RCU 29884 ± 22% -22.8% 23067 softirqs.CPU105.RCU 30351 ± 22% -26.2% 22407 ± 4% softirqs.CPU106.RCU 30122 ± 22% -25.9% 22317 ± 3% softirqs.CPU107.RCU 30203 ± 21% -26.8% 22111 ± 5% softirqs.CPU108.RCU 29667 ± 22% -25.0% 22235 ± 3% softirqs.CPU109.RCU 30310 ± 25% -26.3% 22324 ± 2% softirqs.CPU11.RCU 29722 ± 22% -25.9% 22034 ± 3% softirqs.CPU110.RCU 30799 ± 19% -28.2% 22126 ± 3% softirqs.CPU111.RCU 29287 ± 22% -27.6% 21205 ± 5% softirqs.CPU112.RCU 29356 ± 23% -27.8% 21186 ± 7% softirqs.CPU113.RCU 29350 ± 23% -29.4% 20727 ± 9% softirqs.CPU114.RCU 29271 ± 22% -29.2% 20733 ± 10% softirqs.CPU115.RCU 29889 ± 22% -28.7% 21315 ± 6% softirqs.CPU116.RCU 29521 ± 24% -28.1% 21217 ± 6% softirqs.CPU117.RCU 29332 ± 22% -27.3% 21338 ± 6% softirqs.CPU118.RCU 29485 ± 22% -27.9% 21262 ± 6% softirqs.CPU119.RCU 29350 ± 21% -24.5% 22155 ± 3% softirqs.CPU12.RCU 29902 ± 26% -26.4% 21994 softirqs.CPU129.RCU 29654 ± 21% -25.6% 22076 ± 2% softirqs.CPU13.RCU 29163 ± 21% -28.0% 21003 ± 8% softirqs.CPU132.RCU 28891 ± 21% -25.2% 21622 ± 2% softirqs.CPU141.RCU 29024 ± 25% -27.7% 20971 ± 3% softirqs.CPU146.RCU 29658 ± 24% -26.6% 21768 ± 2% softirqs.CPU149.RCU 29962 ± 19% -24.4% 22650 softirqs.CPU15.RCU 29597 ± 23% -26.4% 21786 ± 2% softirqs.CPU153.RCU 29312 ± 24% -26.4% 21562 ± 3% softirqs.CPU154.RCU 29587 ± 23% -26.7% 21700 ± 2% softirqs.CPU155.RCU 31189 ± 18% -29.1% 22113 ± 4% softirqs.CPU16.RCU 28949 ± 24% -28.6% 20677 ± 3% softirqs.CPU160.RCU 29190 ± 23% -29.5% 20579 ± 4% softirqs.CPU161.RCU 29015 ± 25% -32.1% 19710 ± 6% softirqs.CPU162.RCU 28928 ± 24% -28.8% 20599 ± 3% softirqs.CPU164.RCU 28777 ± 24% -29.7% 20239 ± 4% softirqs.CPU165.RCU 28013 ± 23% -29.2% 19826 ± 8% softirqs.CPU168.RCU 28886 ± 23% -28.7% 20589 ± 7% softirqs.CPU169.RCU 28894 ± 23% -29.3% 20436 ± 9% softirqs.CPU170.RCU 29063 ± 23% -29.4% 20528 ± 7% softirqs.CPU172.RCU 29041 ± 21% -29.7% 20408 ± 7% softirqs.CPU174.RCU 29317 ± 22% -30.1% 20500 ± 7% softirqs.CPU175.RCU 28672 ± 23% -25.0% 21504 ± 3% softirqs.CPU178.RCU 28472 ± 23% -27.3% 20700 ± 5% softirqs.CPU181.RCU 28458 ± 23% -25.6% 21161 ± 3% softirqs.CPU182.RCU 28616 ± 23% -26.5% 21035 ± 4% softirqs.CPU183.RCU 28791 ± 23% -25.9% 21339 ± 3% softirqs.CPU184.RCU 29064 ± 22% -26.4% 21381 ± 4% softirqs.CPU185.RCU 28681 ± 23% -27.2% 20885 ± 4% softirqs.CPU186.RCU 28700 ± 23% -26.3% 21163 ± 3% softirqs.CPU187.RCU 28551 ± 23% -26.5% 20974 softirqs.CPU188.RCU 28521 ± 23% -25.6% 21214 ± 2% softirqs.CPU189.RCU 29479 ± 22% -27.4% 21403 ± 7% softirqs.CPU19.RCU 28570 ± 24% -25.6% 21255 ± 3% softirqs.CPU190.RCU 28954 ± 22% -26.3% 21340 ± 3% softirqs.CPU191.RCU 29496 ± 22% -26.8% 21604 ± 6% softirqs.CPU20.RCU 29924 ± 23% -28.0% 21549 ± 6% softirqs.CPU21.RCU 29499 ± 21% -26.2% 21768 ± 6% softirqs.CPU23.RCU 29133 ± 20% -25.2% 21795 softirqs.CPU33.RCU 103342 +9.2% 112860 softirqs.CPU33.TIMER 29694 ± 21% -24.0% 22576 ± 3% softirqs.CPU4.RCU 29369 ± 19% -23.8% 22377 ± 2% softirqs.CPU45.RCU 30046 ± 23% -28.7% 21416 ± 6% softirqs.CPU49.RCU 29434 ± 22% -24.1% 22330 ± 2% softirqs.CPU5.RCU 29605 ± 23% -26.1% 21874 ± 2% softirqs.CPU54.RCU 29537 ± 22% -24.8% 22222 softirqs.CPU55.RCU 29138 ± 23% -24.1% 22112 softirqs.CPU56.RCU 29665 ± 23% -25.5% 22090 softirqs.CPU57.RCU 29672 ± 23% -25.6% 22074 ± 2% softirqs.CPU58.RCU 29388 ± 21% -23.8% 22407 ± 3% softirqs.CPU6.RCU 29921 ± 23% -27.7% 21641 ± 4% softirqs.CPU60.RCU 29906 ± 24% -27.5% 21696 ± 2% softirqs.CPU61.RCU 29619 ± 23% -25.5% 22072 ± 2% softirqs.CPU62.RCU 29673 ± 23% -25.7% 22053 ± 2% softirqs.CPU63.RCU 29114 ± 22% -24.4% 21998 ± 2% softirqs.CPU64.RCU 29106 ± 23% -28.4% 20827 ± 2% softirqs.CPU65.RCU 28473 ± 22% -29.9% 19972 ± 4% softirqs.CPU66.RCU 28763 ± 24% -27.8% 20761 ± 2% softirqs.CPU67.RCU 28660 ± 23% -27.2% 20876 ± 2% softirqs.CPU68.RCU 28962 ± 23% -29.5% 20431 ± 3% softirqs.CPU69.RCU 28894 ± 24% -27.2% 21033 ± 2% softirqs.CPU70.RCU 29887 ± 28% -28.4% 21401 softirqs.CPU71.RCU 28072 ± 20% -28.8% 19980 ± 7% softirqs.CPU72.RCU 28418 ± 23% -27.7% 20550 ± 6% softirqs.CPU73.RCU 28601 ± 22% -29.0% 20316 ± 8% softirqs.CPU74.RCU 28833 ± 22% -28.6% 20575 ± 7% softirqs.CPU75.RCU 28743 ± 23% -28.0% 20690 ± 7% softirqs.CPU76.RCU 102826 ± 3% +30.7% 134349 ± 23% softirqs.CPU77.TIMER 28531 ± 23% -28.6% 20368 ± 6% softirqs.CPU78.RCU 28899 ± 21% -29.2% 20464 ± 6% softirqs.CPU79.RCU 28775 ± 23% -26.4% 21190 ± 3% softirqs.CPU80.RCU 28592 ± 23% -26.5% 21007 ± 2% softirqs.CPU81.RCU 29934 ± 24% -30.4% 20825 ± 6% softirqs.CPU84.RCU 28802 ± 22% -27.2% 20978 ± 5% softirqs.CPU85.RCU 28764 ± 22% -25.3% 21500 ± 2% softirqs.CPU86.RCU 28839 ± 22% -26.5% 21187 ± 3% softirqs.CPU87.RCU 28545 ± 22% -24.4% 21574 ± 2% softirqs.CPU88.RCU 29941 ± 24% -28.0% 21557 ± 3% softirqs.CPU89.RCU 29404 ± 22% -23.5% 22508 ± 2% softirqs.CPU9.RCU 29032 ± 22% -26.8% 21248 ± 5% softirqs.CPU90.RCU 29070 ± 22% -26.8% 21273 ± 2% softirqs.CPU91.RCU 28791 ± 22% -25.1% 21564 ± 2% softirqs.CPU92.RCU 28534 ± 22% -24.3% 21596 ± 2% softirqs.CPU93.RCU 28411 ± 22% -23.7% 21689 ± 2% softirqs.CPU94.RCU 28526 ± 22% -24.2% 21609 ± 3% softirqs.CPU95.RCU 31022 ± 19% -30.6% 21526 ± 7% softirqs.CPU96.RCU 29709 ± 22% -29.4% 20981 ± 9% softirqs.CPU97.RCU 29889 ± 22% -25.8% 22190 ± 4% softirqs.CPU99.RCU 5598226 ± 22% -26.0% 4141614 ± 4% softirqs.RCU 210.75 ± 20% +128.2% 481.00 ± 35% interrupts.34:PCI-MSI.26738691-edge.eth0-TxRx-2 1224085 ± 7% -7.2% 1135614 interrupts.CAL:Function_call_interrupts 7650 ± 5% -50.3% 3803 ± 47% interrupts.CPU1.RES:Rescheduling_interrupts 210.75 ± 20% +128.2% 481.00 ± 35% interrupts.CPU10.34:PCI-MSI.26738691-edge.eth0-TxRx-2 514.50 ± 7% -31.5% 352.50 ± 18% interrupts.CPU10.NMI:Non-maskable_interrupts 514.50 ± 7% -31.5% 352.50 ± 18% interrupts.CPU10.PMI:Performance_monitoring_interrupts 369.75 ± 31% +41.7% 524.00 ± 9% interrupts.CPU100.NMI:Non-maskable_interrupts 369.75 ± 31% +41.7% 524.00 ± 9% interrupts.CPU100.PMI:Performance_monitoring_interrupts 337.75 ± 31% +49.8% 506.00 ± 12% interrupts.CPU101.NMI:Non-maskable_interrupts 337.75 ± 31% +49.8% 506.00 ± 12% interrupts.CPU101.PMI:Performance_monitoring_interrupts 6985 ± 5% -41.6% 4081 ± 47% interrupts.CPU101.RES:Rescheduling_interrupts 292.75 ± 29% +74.0% 509.50 ± 10% interrupts.CPU102.NMI:Non-maskable_interrupts 292.75 ± 29% +74.0% 509.50 ± 10% interrupts.CPU102.PMI:Performance_monitoring_interrupts 7501 ± 11% -43.2% 4259 ± 46% interrupts.CPU102.RES:Rescheduling_interrupts 360.25 ± 33% +46.8% 529.00 ± 10% interrupts.CPU103.NMI:Non-maskable_interrupts 360.25 ± 33% +46.8% 529.00 ± 10% interrupts.CPU103.PMI:Performance_monitoring_interrupts 342.25 ± 27% +47.0% 503.00 ± 10% interrupts.CPU104.NMI:Non-maskable_interrupts 342.25 ± 27% +47.0% 503.00 ± 10% interrupts.CPU104.PMI:Performance_monitoring_interrupts 7726 ± 7% -43.0% 4402 ± 46% interrupts.CPU104.RES:Rescheduling_interrupts 350.50 ± 32% +54.9% 543.00 ± 10% interrupts.CPU106.NMI:Non-maskable_interrupts 350.50 ± 32% +54.9% 543.00 ± 10% interrupts.CPU106.PMI:Performance_monitoring_interrupts 7558 ± 8% -45.0% 4155 ± 54% interrupts.CPU106.RES:Rescheduling_interrupts 366.25 ± 32% +49.9% 549.00 ± 13% interrupts.CPU107.NMI:Non-maskable_interrupts 366.25 ± 32% +49.9% 549.00 ± 13% interrupts.CPU107.PMI:Performance_monitoring_interrupts 7650 ± 5% -43.8% 4300 ± 51% interrupts.CPU108.RES:Rescheduling_interrupts 338.00 ± 34% +62.9% 550.50 ± 3% interrupts.CPU109.NMI:Non-maskable_interrupts 338.00 ± 34% +62.9% 550.50 ± 3% interrupts.CPU109.PMI:Performance_monitoring_interrupts 7347 ± 4% -45.4% 4015 ± 41% interrupts.CPU109.RES:Rescheduling_interrupts 7213 ± 6% -38.5% 4438 ± 46% interrupts.CPU110.RES:Rescheduling_interrupts 340.00 ± 30% +56.3% 531.50 ± 9% interrupts.CPU111.NMI:Non-maskable_interrupts 340.00 ± 30% +56.3% 531.50 ± 9% interrupts.CPU111.PMI:Performance_monitoring_interrupts 7143 ± 3% -39.0% 4360 ± 49% interrupts.CPU112.RES:Rescheduling_interrupts 368.25 ± 27% +49.9% 552.00 ± 4% interrupts.CPU113.NMI:Non-maskable_interrupts 368.25 ± 27% +49.9% 552.00 ± 4% interrupts.CPU113.PMI:Performance_monitoring_interrupts 369.50 ± 27% +41.7% 523.50 ± 10% interrupts.CPU114.NMI:Non-maskable_interrupts 369.50 ± 27% +41.7% 523.50 ± 10% interrupts.CPU114.PMI:Performance_monitoring_interrupts 349.25 ± 30% +47.9% 516.50 ± 14% interrupts.CPU116.NMI:Non-maskable_interrupts 349.25 ± 30% +47.9% 516.50 ± 14% interrupts.CPU116.PMI:Performance_monitoring_interrupts 7127 ± 2% -38.7% 4370 ± 49% interrupts.CPU116.RES:Rescheduling_interrupts 347.50 ± 34% +47.5% 512.50 ± 7% interrupts.CPU117.NMI:Non-maskable_interrupts 347.50 ± 34% +47.5% 512.50 ± 7% interrupts.CPU117.PMI:Performance_monitoring_interrupts 6399 ± 7% -8.2% 5877 interrupts.CPU118.CAL:Function_call_interrupts 347.25 ± 29% +55.8% 541.00 ± 9% interrupts.CPU118.NMI:Non-maskable_interrupts 347.25 ± 29% +55.8% 541.00 ± 9% interrupts.CPU118.PMI:Performance_monitoring_interrupts 8105 ± 5% -44.4% 4504 ± 49% interrupts.CPU118.RES:Rescheduling_interrupts 6419 ± 7% -7.9% 5909 interrupts.CPU119.CAL:Function_call_interrupts 404.25 ± 21% +31.5% 531.50 ± 9% interrupts.CPU119.NMI:Non-maskable_interrupts 404.25 ± 21% +31.5% 531.50 ± 9% interrupts.CPU119.PMI:Performance_monitoring_interrupts 6957 ± 8% -39.9% 4181 ± 48% interrupts.CPU119.RES:Rescheduling_interrupts 6403 ± 7% -7.6% 5915 interrupts.CPU12.CAL:Function_call_interrupts 6524 ± 14% -50.9% 3201 ± 54% interrupts.CPU120.RES:Rescheduling_interrupts 409.50 ± 23% +31.5% 538.50 ± 3% interrupts.CPU121.NMI:Non-maskable_interrupts 409.50 ± 23% +31.5% 538.50 ± 3% interrupts.CPU121.PMI:Performance_monitoring_interrupts 6429 ± 6% -8.0% 5917 interrupts.CPU125.CAL:Function_call_interrupts 6424 ± 6% -7.7% 5930 interrupts.CPU127.CAL:Function_call_interrupts 6418 ± 6% -7.9% 5913 interrupts.CPU128.CAL:Function_call_interrupts 6413 ± 6% -7.5% 5934 interrupts.CPU129.CAL:Function_call_interrupts 6409 ± 6% -8.5% 5862 interrupts.CPU130.CAL:Function_call_interrupts 7341 ± 6% -42.7% 4204 ± 48% interrupts.CPU130.RES:Rescheduling_interrupts 6409 ± 6% -7.4% 5937 interrupts.CPU131.CAL:Function_call_interrupts 6413 ± 6% -7.6% 5928 interrupts.CPU132.CAL:Function_call_interrupts 385.50 ± 18% +33.7% 515.50 ± 12% interrupts.CPU133.NMI:Non-maskable_interrupts 385.50 ± 18% +33.7% 515.50 ± 12% interrupts.CPU133.PMI:Performance_monitoring_interrupts 403.25 ± 29% +33.4% 538.00 ± 14% interrupts.CPU135.NMI:Non-maskable_interrupts 403.25 ± 29% +33.4% 538.00 ± 14% interrupts.CPU135.PMI:Performance_monitoring_interrupts 7641 ± 2% -39.0% 4662 ± 51% interrupts.CPU137.RES:Rescheduling_interrupts 402.00 ± 29% +29.7% 521.50 ± 15% interrupts.CPU138.NMI:Non-maskable_interrupts 402.00 ± 29% +29.7% 521.50 ± 15% interrupts.CPU138.PMI:Performance_monitoring_interrupts 349.75 ± 32% +45.8% 510.00 ± 7% interrupts.CPU139.NMI:Non-maskable_interrupts 349.75 ± 32% +45.8% 510.00 ± 7% interrupts.CPU139.PMI:Performance_monitoring_interrupts 7371 -39.1% 4487 ± 44% interrupts.CPU14.RES:Rescheduling_interrupts 342.50 ± 33% +55.5% 532.50 ± 11% interrupts.CPU142.NMI:Non-maskable_interrupts 342.50 ± 33% +55.5% 532.50 ± 11% interrupts.CPU142.PMI:Performance_monitoring_interrupts 6402 ± 6% -7.5% 5920 interrupts.CPU144.CAL:Function_call_interrupts 6442 ± 6% -8.0% 5929 interrupts.CPU145.CAL:Function_call_interrupts 333.00 ± 28% +46.5% 488.00 ± 10% interrupts.CPU146.NMI:Non-maskable_interrupts 333.00 ± 28% +46.5% 488.00 ± 10% interrupts.CPU146.PMI:Performance_monitoring_interrupts 351.25 ± 32% +46.0% 513.00 ± 3% interrupts.CPU147.NMI:Non-maskable_interrupts 351.25 ± 32% +46.0% 513.00 ± 3% interrupts.CPU147.PMI:Performance_monitoring_interrupts 348.50 ± 31% +55.2% 541.00 ± 12% interrupts.CPU148.NMI:Non-maskable_interrupts 348.50 ± 31% +55.2% 541.00 ± 12% interrupts.CPU148.PMI:Performance_monitoring_interrupts 350.75 ± 33% +52.8% 536.00 ± 15% interrupts.CPU149.NMI:Non-maskable_interrupts 350.75 ± 33% +52.8% 536.00 ± 15% interrupts.CPU149.PMI:Performance_monitoring_interrupts 7512 ± 2% -45.3% 4107 ± 50% interrupts.CPU15.RES:Rescheduling_interrupts 337.50 ± 31% +48.9% 502.50 ± 7% interrupts.CPU150.NMI:Non-maskable_interrupts 337.50 ± 31% +48.9% 502.50 ± 7% interrupts.CPU150.PMI:Performance_monitoring_interrupts 349.75 ± 28% +47.7% 516.50 ± 10% interrupts.CPU151.NMI:Non-maskable_interrupts 349.75 ± 28% +47.7% 516.50 ± 10% interrupts.CPU151.PMI:Performance_monitoring_interrupts 7436 ± 3% -39.5% 4502 ± 51% interrupts.CPU151.RES:Rescheduling_interrupts 284.75 ± 36% +83.1% 521.50 ± 6% interrupts.CPU155.NMI:Non-maskable_interrupts 284.75 ± 36% +83.1% 521.50 ± 6% interrupts.CPU155.PMI:Performance_monitoring_interrupts 286.25 ± 31% +68.4% 482.00 ± 11% interrupts.CPU156.NMI:Non-maskable_interrupts 286.25 ± 31% +68.4% 482.00 ± 11% interrupts.CPU156.PMI:Performance_monitoring_interrupts 285.25 ± 29% +76.0% 502.00 ± 13% interrupts.CPU157.NMI:Non-maskable_interrupts 285.25 ± 29% +76.0% 502.00 ± 13% interrupts.CPU157.PMI:Performance_monitoring_interrupts 281.75 ± 30% +91.3% 539.00 ± 12% interrupts.CPU158.NMI:Non-maskable_interrupts 281.75 ± 30% +91.3% 539.00 ± 12% interrupts.CPU158.PMI:Performance_monitoring_interrupts 284.25 ± 39% +81.9% 517.00 ± 8% interrupts.CPU159.NMI:Non-maskable_interrupts 284.25 ± 39% +81.9% 517.00 ± 8% interrupts.CPU159.PMI:Performance_monitoring_interrupts 6392 ± 7% -7.6% 5908 interrupts.CPU16.CAL:Function_call_interrupts 286.75 ± 29% +81.5% 520.50 ± 12% interrupts.CPU160.NMI:Non-maskable_interrupts 286.75 ± 29% +81.5% 520.50 ± 12% interrupts.CPU160.PMI:Performance_monitoring_interrupts 287.75 ± 29% +80.0% 518.00 ± 13% interrupts.CPU161.NMI:Non-maskable_interrupts 287.75 ± 29% +80.0% 518.00 ± 13% interrupts.CPU161.PMI:Performance_monitoring_interrupts 279.75 ± 30% +88.2% 526.50 ± 5% interrupts.CPU162.NMI:Non-maskable_interrupts 279.75 ± 30% +88.2% 526.50 ± 5% interrupts.CPU162.PMI:Performance_monitoring_interrupts 6407 ± 7% -7.7% 5912 interrupts.CPU165.CAL:Function_call_interrupts 412.25 ± 13% +24.6% 513.50 ± 11% interrupts.CPU167.NMI:Non-maskable_interrupts 412.25 ± 13% +24.6% 513.50 ± 11% interrupts.CPU167.PMI:Performance_monitoring_interrupts 7213 -35.4% 4657 ± 41% interrupts.CPU17.RES:Rescheduling_interrupts 354.00 ± 29% +37.9% 488.00 ± 12% interrupts.CPU171.NMI:Non-maskable_interrupts 354.00 ± 29% +37.9% 488.00 ± 12% interrupts.CPU171.PMI:Performance_monitoring_interrupts 343.50 ± 32% +50.5% 517.00 ± 12% interrupts.CPU172.NMI:Non-maskable_interrupts 343.50 ± 32% +50.5% 517.00 ± 12% interrupts.CPU172.PMI:Performance_monitoring_interrupts 6444 ± 6% -7.8% 5939 interrupts.CPU174.CAL:Function_call_interrupts 6413 ± 7% -7.7% 5919 interrupts.CPU18.CAL:Function_call_interrupts 6402 ± 7% -7.7% 5910 interrupts.CPU19.CAL:Function_call_interrupts 6840 ± 5% -43.9% 3839 ± 46% interrupts.CPU191.RES:Rescheduling_interrupts 6397 ± 7% -7.6% 5909 interrupts.CPU20.CAL:Function_call_interrupts 6848 ± 8% -41.5% 4003 ± 40% interrupts.CPU20.RES:Rescheduling_interrupts 6417 ± 7% -8.0% 5903 interrupts.CPU21.CAL:Function_call_interrupts 7703 ± 10% -47.5% 4045 ± 47% interrupts.CPU21.RES:Rescheduling_interrupts 7202 ± 8% -44.1% 4025 ± 44% interrupts.CPU22.RES:Rescheduling_interrupts 7280 ± 3% -43.7% 4098 ± 44% interrupts.CPU23.RES:Rescheduling_interrupts 6404 ± 7% -8.0% 5890 interrupts.CPU26.CAL:Function_call_interrupts 6415 ± 7% -16.1% 5384 ± 10% interrupts.CPU3.CAL:Function_call_interrupts 470.00 ± 6% +16.2% 546.00 ± 12% interrupts.CPU32.NMI:Non-maskable_interrupts 470.00 ± 6% +16.2% 546.00 ± 12% interrupts.CPU32.PMI:Performance_monitoring_interrupts 6414 ± 6% -7.6% 5923 interrupts.CPU35.CAL:Function_call_interrupts 6409 ± 6% -7.0% 5958 interrupts.CPU36.CAL:Function_call_interrupts 465.00 ± 8% +14.0% 530.00 ± 13% interrupts.CPU38.NMI:Non-maskable_interrupts 465.00 ± 8% +14.0% 530.00 ± 13% interrupts.CPU38.PMI:Performance_monitoring_interrupts 6412 ± 6% -7.7% 5918 interrupts.CPU39.CAL:Function_call_interrupts 462.50 ± 12% +11.8% 517.00 ± 9% interrupts.CPU39.NMI:Non-maskable_interrupts 462.50 ± 12% +11.8% 517.00 ± 9% interrupts.CPU39.PMI:Performance_monitoring_interrupts 6851 ± 6% -42.2% 3963 ± 49% interrupts.CPU4.RES:Rescheduling_interrupts 455.00 ± 7% +14.6% 521.50 ± 13% interrupts.CPU41.NMI:Non-maskable_interrupts 455.00 ± 7% +14.6% 521.50 ± 13% interrupts.CPU41.PMI:Performance_monitoring_interrupts 6422 ± 6% -7.3% 5951 interrupts.CPU42.CAL:Function_call_interrupts 7549 ± 3% -41.2% 4437 ± 50% interrupts.CPU42.RES:Rescheduling_interrupts 6428 ± 7% -7.6% 5940 interrupts.CPU43.CAL:Function_call_interrupts 6431 ± 7% -7.7% 5938 interrupts.CPU44.CAL:Function_call_interrupts 6376 ± 6% -7.0% 5931 interrupts.CPU47.CAL:Function_call_interrupts 500.00 ± 11% -28.1% 359.50 ± 23% interrupts.CPU49.NMI:Non-maskable_interrupts 500.00 ± 11% -28.1% 359.50 ± 23% interrupts.CPU49.PMI:Performance_monitoring_interrupts 6415 ± 6% -8.2% 5889 interrupts.CPU50.CAL:Function_call_interrupts 6407 ± 7% -9.4% 5805 interrupts.CPU51.CAL:Function_call_interrupts 409.50 ± 22% +39.1% 569.50 ± 5% interrupts.CPU51.NMI:Non-maskable_interrupts 409.50 ± 22% +39.1% 569.50 ± 5% interrupts.CPU51.PMI:Performance_monitoring_interrupts 6408 ± 7% -8.0% 5896 interrupts.CPU52.CAL:Function_call_interrupts 342.75 ± 33% +58.0% 541.50 ± 10% interrupts.CPU52.NMI:Non-maskable_interrupts 342.75 ± 33% +58.0% 541.50 ± 10% interrupts.CPU52.PMI:Performance_monitoring_interrupts 6411 ± 7% -7.8% 5910 interrupts.CPU53.CAL:Function_call_interrupts 348.00 ± 35% +50.6% 524.00 ± 7% interrupts.CPU53.NMI:Non-maskable_interrupts 348.00 ± 35% +50.6% 524.00 ± 7% interrupts.CPU53.PMI:Performance_monitoring_interrupts 6416 ± 6% -7.7% 5919 interrupts.CPU54.CAL:Function_call_interrupts 338.75 ± 31% +51.7% 514.00 ± 12% interrupts.CPU54.NMI:Non-maskable_interrupts 338.75 ± 31% +51.7% 514.00 ± 12% interrupts.CPU54.PMI:Performance_monitoring_interrupts 6419 ± 7% -8.4% 5883 interrupts.CPU55.CAL:Function_call_interrupts 350.00 ± 34% +53.3% 536.50 ± 8% interrupts.CPU55.NMI:Non-maskable_interrupts 350.00 ± 34% +53.3% 536.50 ± 8% interrupts.CPU55.PMI:Performance_monitoring_interrupts 6422 ± 6% -7.9% 5914 interrupts.CPU56.CAL:Function_call_interrupts 401.00 ± 21% +34.8% 540.50 ± 8% interrupts.CPU56.NMI:Non-maskable_interrupts 401.00 ± 21% +34.8% 540.50 ± 8% interrupts.CPU56.PMI:Performance_monitoring_interrupts 6418 ± 7% -7.7% 5926 interrupts.CPU57.CAL:Function_call_interrupts 411.00 ± 21% +30.4% 536.00 ± 10% interrupts.CPU58.NMI:Non-maskable_interrupts 411.00 ± 21% +30.4% 536.00 ± 10% interrupts.CPU58.PMI:Performance_monitoring_interrupts 6410 ± 6% -7.9% 5904 interrupts.CPU59.CAL:Function_call_interrupts 403.00 ± 24% +25.7% 506.50 ± 8% interrupts.CPU59.NMI:Non-maskable_interrupts 403.00 ± 24% +25.7% 506.50 ± 8% interrupts.CPU59.PMI:Performance_monitoring_interrupts 7397 ± 3% -40.2% 4424 ± 50% interrupts.CPU6.RES:Rescheduling_interrupts 6403 ± 7% -7.6% 5917 interrupts.CPU60.CAL:Function_call_interrupts 394.50 ± 22% +35.2% 533.50 ± 14% interrupts.CPU60.NMI:Non-maskable_interrupts 394.50 ± 22% +35.2% 533.50 ± 14% interrupts.CPU60.PMI:Performance_monitoring_interrupts 6406 ± 7% -7.3% 5939 interrupts.CPU62.CAL:Function_call_interrupts 403.25 ± 22% +25.9% 507.50 ± 10% interrupts.CPU62.NMI:Non-maskable_interrupts 403.25 ± 22% +25.9% 507.50 ± 10% interrupts.CPU62.PMI:Performance_monitoring_interrupts 427.25 ± 23% +19.4% 510.00 ± 6% interrupts.CPU63.NMI:Non-maskable_interrupts 427.25 ± 23% +19.4% 510.00 ± 6% interrupts.CPU63.PMI:Performance_monitoring_interrupts 457.50 ± 8% +12.0% 512.50 ± 11% interrupts.CPU64.NMI:Non-maskable_interrupts 457.50 ± 8% +12.0% 512.50 ± 11% interrupts.CPU64.PMI:Performance_monitoring_interrupts 473.25 ± 11% +13.7% 538.00 ± 11% interrupts.CPU71.NMI:Non-maskable_interrupts 473.25 ± 11% +13.7% 538.00 ± 11% interrupts.CPU71.PMI:Performance_monitoring_interrupts 6421 ± 6% -16.0% 5395 ± 4% interrupts.CPU72.CAL:Function_call_interrupts 6311 ± 7% -59.7% 2542 ± 56% interrupts.CPU72.RES:Rescheduling_interrupts 6434 ± 6% -12.8% 5611 ± 6% interrupts.CPU73.CAL:Function_call_interrupts 6931 ± 6% -47.7% 3623 ± 40% interrupts.CPU73.RES:Rescheduling_interrupts 6415 ± 6% -7.3% 5945 interrupts.CPU75.CAL:Function_call_interrupts 6427 ± 6% -7.6% 5941 interrupts.CPU76.CAL:Function_call_interrupts 6428 ± 6% -7.5% 5946 interrupts.CPU78.CAL:Function_call_interrupts 7307 ± 7% -41.7% 4262 ± 50% interrupts.CPU8.RES:Rescheduling_interrupts 6428 ± 6% -7.7% 5930 interrupts.CPU82.CAL:Function_call_interrupts 6402 ± 6% -7.4% 5927 interrupts.CPU86.CAL:Function_call_interrupts 469.75 ± 8% +12.0% 526.00 ± 10% interrupts.CPU86.NMI:Non-maskable_interrupts 469.75 ± 8% +12.0% 526.00 ± 10% interrupts.CPU86.PMI:Performance_monitoring_interrupts 6410 ± 6% -7.7% 5918 interrupts.CPU87.CAL:Function_call_interrupts 6411 ± 6% -7.5% 5928 interrupts.CPU88.CAL:Function_call_interrupts 6413 ± 6% -7.7% 5922 interrupts.CPU89.CAL:Function_call_interrupts 6417 ± 7% -7.6% 5931 interrupts.CPU91.CAL:Function_call_interrupts 6409 ± 7% -7.5% 5926 interrupts.CPU94.CAL:Function_call_interrupts 6405 ± 6% -8.4% 5865 interrupts.CPU96.CAL:Function_call_interrupts 8173 ± 5% -42.6% 4691 ± 38% interrupts.CPU96.RES:Rescheduling_interrupts 6417 ± 7% -7.8% 5918 interrupts.CPU97.CAL:Function_call_interrupts 7758 ± 6% -39.6% 4687 ± 38% interrupts.CPU98.RES:Rescheduling_interrupts 6987 ± 6% -42.4% 4027 ± 28% interrupts.CPU99.RES:Rescheduling_interrupts 80504 ± 7% +15.5% 92961 ± 8% interrupts.NMI:Non-maskable_interrupts 80504 ± 7% +15.5% 92961 ± 8% interrupts.PMI:Performance_monitoring_interrupts *************************************************************************************************** lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory ========================================================================================= compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/10/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/custom/reaim/0x14 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:4 -11% 0:2 perf-profile.children.cycles-pp.error_entry 0:4 -10% 0:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 2549 ± 2% +5.2% 2683 ± 2% reaim.child_systime 25527 ± 2% +5.2% 26866 ± 2% reaim.time.system_time 364348 ± 5% +14.8% 418108 meminfo.DirectMap4k 97407 ± 94% +331.6% 420408 ± 38% syscalls.sys_rt_sigaction.max 18.16 ± 25% -7.9 10.27 ± 38% turbostat.C3% 38.28 ± 2% -4.1% 36.72 boot-time.boot 4721 ± 2% -3.7% 4548 boot-time.idle 2156912 ± 6% +71.9% 3707777 ± 29% cpuidle.POLL.time 468846 ± 11% +47.5% 691572 ± 15% cpuidle.POLL.usage 62557 ± 2% +5.7% 66113 ± 6% numa-vmstat.node1.nr_unevictable 62557 ± 2% +5.7% 66113 ± 6% numa-vmstat.node1.nr_zone_unevictable 5732 ± 29% +41.4% 8104 ± 27% numa-vmstat.node3.nr_slab_reclaimable 7305 ± 29% +22.8% 8969 ± 30% numa-meminfo.node1.Mapped 250228 ± 2% +5.7% 264457 ± 6% numa-meminfo.node1.Unevictable 22894 ± 29% +41.9% 32489 ± 27% numa-meminfo.node3.KReclaimable 6373 ± 5% +15.9% 7384 ± 13% numa-meminfo.node3.Mapped 22894 ± 29% +41.9% 32489 ± 27% numa-meminfo.node3.SReclaimable 21.79 ± 4% -7.6% 20.14 perf-stat.i.MPKI 1.48 ± 11% -0.2 1.25 perf-stat.i.branch-miss-rate% 2626 +3.5% 2720 perf-stat.overall.cycles-between-cache-misses 26402092 ± 2% +3.9% 27418780 perf-stat.overall.path-length 6.083e+13 ± 2% +3.9% 6.317e+13 perf-stat.total.instructions 42744 ± 6% -7.8% 39414 softirqs.CPU124.NET_RX 40812 ± 2% +7.6% 43917 ± 5% softirqs.CPU48.NET_RX 38081 ± 3% +7.8% 41070 ± 4% softirqs.CPU59.NET_RX 40706 ± 4% +9.7% 44636 ± 3% softirqs.CPU64.NET_RX 38992 ± 2% +16.8% 45528 softirqs.CPU83.NET_RX 38859 ± 2% +16.2% 45160 softirqs.CPU88.NET_RX 115563 ± 5% +10.6% 127765 ± 5% slabinfo.kmalloc-32.active_objs 903.00 ± 5% +10.5% 998.00 ± 5% slabinfo.kmalloc-32.active_slabs 115656 ± 5% +10.5% 127819 ± 5% slabinfo.kmalloc-32.num_objs 903.00 ± 5% +10.5% 998.00 ± 5% slabinfo.kmalloc-32.num_slabs 58841 ± 18% -23.3% 45131 slabinfo.lsm_file_cache.active_objs 58841 ± 18% -23.3% 45131 slabinfo.lsm_file_cache.num_objs 888.75 ± 14% +35.1% 1200 ± 6% slabinfo.xfrm_state.active_objs 888.75 ± 14% +35.1% 1200 ± 6% slabinfo.xfrm_state.num_objs 117.75 ± 4% +8.3% 127.50 proc-vmstat.nr_dirtied 280918 +1.4% 284967 proc-vmstat.nr_file_pages 5847 +2.9% 6019 proc-vmstat.nr_inactive_anon 102732 ± 2% +2.2% 105006 proc-vmstat.nr_kernel_stack 7804 ± 2% +4.5% 8154 proc-vmstat.nr_mapped 25837 ± 7% +10.4% 28533 proc-vmstat.nr_shmem 116.75 ± 5% +7.5% 125.50 proc-vmstat.nr_written 5847 +2.9% 6019 proc-vmstat.nr_zone_inactive_anon 4.568e+08 -3.4% 4.414e+08 proc-vmstat.numa_hit 4.566e+08 -3.4% 4.413e+08 proc-vmstat.numa_local 34509 ± 3% +6.9% 36895 proc-vmstat.pgactivate 4.683e+08 -3.3% 4.529e+08 proc-vmstat.pgalloc_normal 4.682e+08 -3.3% 4.528e+08 proc-vmstat.pgfree 673.67 ± 7% -13.4% 583.64 ± 2% sched_debug.cfs_rq:/.exec_clock.stddev 0.93 ± 7% +23.1% 1.14 sched_debug.cfs_rq:/.load_avg.min 1.23 ± 9% -18.4% 1.00 sched_debug.cfs_rq:/.nr_running.max 779.44 ± 9% +15.0% 896.36 ± 2% sched_debug.cfs_rq:/.removed.load_avg.max 35998 ± 9% +15.0% 41403 sched_debug.cfs_rq:/.removed.runnable_sum.max 314.12 ± 26% -49.0% 160.29 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.max 243.17 ± 17% +20.1% 292.07 ± 11% sched_debug.cfs_rq:/.util_avg.min 53.89 ± 15% +23.0% 66.26 ± 5% sched_debug.cpu.clock.stddev 53.89 ± 15% +23.0% 66.26 ± 5% sched_debug.cpu.clock_task.stddev 23.03 ± 8% +13.3% 26.09 ± 8% sched_debug.cpu.cpu_load[2].stddev 185.49 ± 18% +23.0% 228.21 ± 4% sched_debug.cpu.cpu_load[3].max 18.25 ± 14% +27.2% 23.22 ± 3% sched_debug.cpu.cpu_load[3].stddev 3.41 ± 12% +14.7% 3.91 sched_debug.cpu.cpu_load[4].avg 171.67 ± 27% +32.5% 227.50 ± 10% sched_debug.cpu.cpu_load[4].max 16.29 ± 23% +37.1% 22.33 ± 2% sched_debug.cpu.cpu_load[4].stddev 0.00 ± 8% +32.6% 0.00 ± 2% sched_debug.cpu.next_balance.stddev 9.29 ± 12% +22.2% 11.35 sched_debug.cpu.nr_uninterruptible.avg 3.04 ±100% -3.0 0.00 perf-profile.calltrace.cycles-pp.sieve 0.52 ± 60% +0.3 0.80 ± 7% perf-profile.calltrace.cycles-pp.do_wp_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 1.24 ± 18% +0.3 1.55 ± 7% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit 1.30 ± 18% +0.3 1.62 ± 7% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit 0.29 ±100% +0.3 0.64 ± 5% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.56 ± 63% +0.3 0.91 ± 7% perf-profile.calltrace.cycles-pp.copy_p4d_range.copy_page_range.dup_mm.copy_process._do_fork 0.57 ± 63% +0.4 0.94 ± 6% perf-profile.calltrace.cycles-pp.copy_page_range.dup_mm.copy_process._do_fork.do_syscall_64 0.12 ±173% +0.5 0.59 ± 4% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.16 ±173% +0.5 0.65 perf-profile.calltrace.cycles-pp.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.16 ±173% +0.5 0.65 perf-profile.calltrace.cycles-pp.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.38 ± 23% +0.5 1.86 ± 4% perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.75 ± 38% +0.5 1.29 perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mm.copy_process._do_fork.do_syscall_64 0.36 ±105% +0.6 0.94 perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mm.copy_process._do_fork 2.83 ± 20% +0.6 3.48 ± 7% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 2.92 ± 20% +0.7 3.58 ± 7% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 3.26 ± 19% +0.7 3.94 ± 8% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 11.68 ± 25% +3.9 15.58 ± 12% perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.38 ± 34% +4.0 13.43 ± 15% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit 11.97 ± 27% +4.6 16.54 ± 11% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 11.98 ± 27% +4.6 16.55 ± 11% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 12.86 ± 26% +4.6 17.47 ± 10% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.86 ± 26% +4.6 17.47 ± 10% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.86 ± 26% +4.6 17.47 ± 10% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.05 ± 44% +6.0 14.07 ± 14% perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.23 ± 44% +6.1 14.34 ± 13% perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 37.29 ± 20% +11.1 48.38 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 37.31 ± 20% +11.1 48.41 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 3.04 ±100% -3.0 0.00 perf-profile.children.cycles-pp.sieve 0.41 ± 21% -0.2 0.22 ± 13% perf-profile.children.cycles-pp.syscall_trace_enter 0.20 ± 39% -0.2 0.03 ±100% perf-profile.children.cycles-pp.trace_buffer_lock_reserve 0.37 ± 32% -0.2 0.20 ± 2% perf-profile.children.cycles-pp.menu_select 0.33 ± 18% -0.2 0.16 perf-profile.children.cycles-pp.syscall_slow_exit_work 0.32 ± 24% -0.2 0.17 ± 15% perf-profile.children.cycles-pp.ftrace_syscall_enter 0.28 ± 22% -0.2 0.12 ± 4% perf-profile.children.cycles-pp.ftrace_syscall_exit 0.17 ± 38% -0.1 0.03 ±100% perf-profile.children.cycles-pp.ring_buffer_lock_reserve 0.11 ± 29% -0.1 0.06 perf-profile.children.cycles-pp.irq_enter 0.07 ± 14% -0.0 0.03 ±100% perf-profile.children.cycles-pp.sched_clock_cpu 0.16 ± 15% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.free_pcppages_bulk 0.07 ± 13% -0.0 0.05 perf-profile.children.cycles-pp.__list_add_valid 0.14 ± 18% +0.0 0.17 ± 8% perf-profile.children.cycles-pp.vm_area_dup 0.15 ± 9% +0.0 0.18 ± 13% perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave 0.12 ± 18% +0.0 0.15 ± 13% perf-profile.children.cycles-pp.__put_user_4 0.15 ± 11% +0.0 0.19 ± 15% perf-profile.children.cycles-pp.free_uid 0.19 ± 14% +0.0 0.23 ± 11% perf-profile.children.cycles-pp.anon_vma_interval_tree_remove 0.10 ± 24% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.schedule_tail 0.25 ± 14% +0.0 0.30 ± 8% perf-profile.children.cycles-pp.wake_up_new_task 0.01 ±173% +0.0 0.06 perf-profile.children.cycles-pp.__put_task_struct 0.15 ± 11% +0.1 0.21 ± 12% perf-profile.children.cycles-pp.refcount_dec_not_one 0.04 ±103% +0.1 0.10 ± 5% perf-profile.children.cycles-pp.autoremove_wake_function 0.05 ± 74% +0.1 0.11 ± 4% perf-profile.children.cycles-pp.pipe_wait 0.14 ± 28% +0.1 0.21 ± 9% perf-profile.children.cycles-pp.__wake_up_common 0.24 ± 14% +0.1 0.31 ± 12% perf-profile.children.cycles-pp.put_cred_rcu 0.07 ± 70% +0.1 0.14 ± 3% perf-profile.children.cycles-pp.pipe_read 0.15 ± 29% +0.1 0.23 ± 6% perf-profile.children.cycles-pp.__wake_up_common_lock 0.04 ±104% +0.1 0.14 ± 3% perf-profile.children.cycles-pp.pipe_write 0.48 ± 10% +0.1 0.60 ± 15% perf-profile.children.cycles-pp.rcu_core 0.79 ± 9% +0.1 0.94 ± 4% perf-profile.children.cycles-pp.__softirqentry_text_start 0.74 ± 15% +0.1 0.89 ± 4% perf-profile.children.cycles-pp.unlink_anon_vmas 0.37 ± 19% +0.2 0.52 ± 7% perf-profile.children.cycles-pp.anon_vma_interval_tree_insert 0.74 ± 18% +0.2 0.90 ± 8% perf-profile.children.cycles-pp.do_wp_page 0.63 ± 20% +0.2 0.79 ± 6% perf-profile.children.cycles-pp.wp_page_copy 0.74 ± 17% +0.2 0.98 ± 2% perf-profile.children.cycles-pp.anon_vma_clone 0.99 ± 20% +0.3 1.33 perf-profile.children.cycles-pp.anon_vma_fork 2.20 ± 17% +0.4 2.62 ± 7% perf-profile.children.cycles-pp.filemap_map_pages 11.68 ± 25% +3.9 15.59 ± 12% perf-profile.children.cycles-pp.dup_mm 12.60 ± 23% +3.9 16.52 ± 11% perf-profile.children.cycles-pp.copy_process 17.92 ± 18% +3.9 21.86 ± 7% perf-profile.children.cycles-pp.exit_mmap 17.94 ± 18% +3.9 21.88 ± 7% perf-profile.children.cycles-pp.mmput 12.90 ± 22% +4.0 16.86 ± 11% perf-profile.children.cycles-pp._do_fork 12.90 ± 25% +4.6 17.49 ± 10% perf-profile.children.cycles-pp.do_exit 12.91 ± 25% +4.6 17.50 ± 10% perf-profile.children.cycles-pp.__x64_sys_exit_group 12.91 ± 25% +4.6 17.50 ± 10% perf-profile.children.cycles-pp.do_group_exit 50.20 ± 10% +5.2 55.40 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 50.16 ± 10% +5.2 55.37 ± 4% perf-profile.children.cycles-pp.do_syscall_64 3.00 ±100% -3.0 0.00 perf-profile.self.cycles-pp.sieve 0.17 ± 25% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state 0.13 ± 9% -0.0 0.09 ± 11% perf-profile.self.cycles-pp.ftrace_syscall_exit 0.06 ± 6% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.update_load_avg 0.07 ± 11% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.put_cred_rcu 0.13 ± 14% +0.0 0.16 perf-profile.self.cycles-pp.unlock_page 0.12 ± 13% +0.0 0.15 ± 6% perf-profile.self.cycles-pp.anon_vma_clone 0.18 ± 14% +0.0 0.21 ± 11% perf-profile.self.cycles-pp.anon_vma_interval_tree_remove 0.15 ± 11% +0.0 0.20 ± 10% perf-profile.self.cycles-pp.refcount_dec_not_one 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__anon_vma_interval_tree_augment_rotate 0.01 ±173% +0.1 0.08 ± 33% perf-profile.self.cycles-pp.update_blocked_averages 0.30 ± 24% +0.1 0.39 ± 10% perf-profile.self.cycles-pp.copy_page 0.37 ± 19% +0.1 0.52 ± 6% perf-profile.self.cycles-pp.anon_vma_interval_tree_insert 1.56 ± 20% +0.4 1.98 ± 8% perf-profile.self.cycles-pp.filemap_map_pages 315.75 ± 49% -36.2% 201.50 ± 3% interrupts.47:PCI-MSI.1572866-edge.eth0-TxRx-2 209.50 ± 7% +607.6% 1482 ± 85% interrupts.48:PCI-MSI.1572867-edge.eth0-TxRx-3 261.00 ± 40% +47.3% 384.50 ± 43% interrupts.54:PCI-MSI.1572873-edge.eth0-TxRx-9 262.75 ± 21% -20.5% 209.00 interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15 194.25 ± 8% +15.6% 224.50 ± 9% interrupts.70:PCI-MSI.1572889-edge.eth0-TxRx-25 194.50 ± 8% +20.3% 234.00 ± 13% interrupts.72:PCI-MSI.1572891-edge.eth0-TxRx-27 194.00 ± 8% +100.3% 388.50 ± 48% interrupts.76:PCI-MSI.1572895-edge.eth0-TxRx-31 16378 ± 4% -10.9% 14598 interrupts.CPU0.RES:Rescheduling_interrupts 611.75 ±131% +693.1% 4851 ± 6% interrupts.CPU10.NMI:Non-maskable_interrupts 611.75 ±131% +693.1% 4851 ± 6% interrupts.CPU10.PMI:Performance_monitoring_interrupts 2966 ± 65% -99.3% 22.00 ± 72% interrupts.CPU100.NMI:Non-maskable_interrupts 2966 ± 65% -99.3% 22.00 ± 72% interrupts.CPU100.PMI:Performance_monitoring_interrupts 12853 ± 5% -14.1% 11047 interrupts.CPU100.RES:Rescheduling_interrupts 4472 ± 7% -71.9% 1257 ± 97% interrupts.CPU101.NMI:Non-maskable_interrupts 4472 ± 7% -71.9% 1257 ± 97% interrupts.CPU101.PMI:Performance_monitoring_interrupts 9931 ± 10% -14.3% 8514 interrupts.CPU102.RES:Rescheduling_interrupts 9671 ± 8% -13.4% 8377 interrupts.CPU103.RES:Rescheduling_interrupts 1674 ±106% +184.5% 4762 ± 8% interrupts.CPU106.NMI:Non-maskable_interrupts 1674 ±106% +184.5% 4762 ± 8% interrupts.CPU106.PMI:Performance_monitoring_interrupts 12916 ± 6% -14.7% 11018 interrupts.CPU108.RES:Rescheduling_interrupts 12536 ± 9% -17.8% 10301 ± 4% interrupts.CPU109.RES:Rescheduling_interrupts 10666 ± 6% -17.7% 8781 interrupts.CPU11.RES:Rescheduling_interrupts 10791 ± 7% -13.4% 9347 interrupts.CPU111.RES:Rescheduling_interrupts 1536 ±121% +209.4% 4753 ± 7% interrupts.CPU112.NMI:Non-maskable_interrupts 1536 ±121% +209.4% 4753 ± 7% interrupts.CPU112.PMI:Performance_monitoring_interrupts 3978 ± 27% -99.9% 3.50 ± 14% interrupts.CPU118.NMI:Non-maskable_interrupts 3978 ± 27% -99.9% 3.50 ± 14% interrupts.CPU118.PMI:Performance_monitoring_interrupts 10560 ± 5% -18.5% 8611 ± 6% interrupts.CPU118.RES:Rescheduling_interrupts 12039 ± 7% -21.1% 9504 ± 6% interrupts.CPU12.RES:Rescheduling_interrupts 1123 ±169% +329.2% 4823 ± 8% interrupts.CPU122.NMI:Non-maskable_interrupts 1123 ±169% +329.2% 4823 ± 8% interrupts.CPU122.PMI:Performance_monitoring_interrupts 9473 ± 6% -13.5% 8193 ± 4% interrupts.CPU123.RES:Rescheduling_interrupts 1023 ±173% +124.4% 2297 ± 99% interrupts.CPU124.NMI:Non-maskable_interrupts 1023 ±173% +124.4% 2297 ± 99% interrupts.CPU124.PMI:Performance_monitoring_interrupts 1620 ±112% +120.1% 3565 ± 25% interrupts.CPU125.NMI:Non-maskable_interrupts 1620 ±112% +120.1% 3565 ± 25% interrupts.CPU125.PMI:Performance_monitoring_interrupts 1000 ±169% +183.7% 2836 ± 84% interrupts.CPU126.NMI:Non-maskable_interrupts 1000 ±169% +183.7% 2836 ± 84% interrupts.CPU126.PMI:Performance_monitoring_interrupts 12071 ± 6% -16.4% 10091 ± 6% interrupts.CPU126.RES:Rescheduling_interrupts 13431 ± 3% -14.6% 11466 ± 4% interrupts.CPU127.RES:Rescheduling_interrupts 10962 ± 6% -13.7% 9459 interrupts.CPU129.RES:Rescheduling_interrupts 11360 ± 5% -13.8% 9787 ± 5% interrupts.CPU13.RES:Rescheduling_interrupts 3423 ± 58% +41.9% 4857 ± 6% interrupts.CPU130.NMI:Non-maskable_interrupts 3423 ± 58% +41.9% 4857 ± 6% interrupts.CPU130.PMI:Performance_monitoring_interrupts 9669 ± 6% -13.0% 8411 ± 4% interrupts.CPU131.RES:Rescheduling_interrupts 12576 ± 7% -13.7% 10857 interrupts.CPU132.RES:Rescheduling_interrupts 3946 ± 27% -91.5% 334.50 ± 96% interrupts.CPU134.NMI:Non-maskable_interrupts 3946 ± 27% -91.5% 334.50 ± 96% interrupts.CPU134.PMI:Performance_monitoring_interrupts 10151 ± 8% -16.7% 8455 ± 4% interrupts.CPU134.RES:Rescheduling_interrupts 2942 ± 65% -99.3% 21.00 ±100% interrupts.CPU139.NMI:Non-maskable_interrupts 2942 ± 65% -99.3% 21.00 ±100% interrupts.CPU139.PMI:Performance_monitoring_interrupts 9758 ± 8% -16.7% 8125 ± 2% interrupts.CPU139.RES:Rescheduling_interrupts 1224 ±172% +203.8% 3719 ± 38% interrupts.CPU14.NMI:Non-maskable_interrupts 1224 ±172% +203.8% 3719 ± 38% interrupts.CPU14.PMI:Performance_monitoring_interrupts 10654 ± 10% -14.9% 9071 interrupts.CPU14.RES:Rescheduling_interrupts 11715 ± 7% -15.8% 9863 interrupts.CPU140.RES:Rescheduling_interrupts 10556 ± 5% -15.3% 8938 ± 3% interrupts.CPU141.RES:Rescheduling_interrupts 2317 ± 96% +113.1% 4939 ± 5% interrupts.CPU143.NMI:Non-maskable_interrupts 2317 ± 96% +113.1% 4939 ± 5% interrupts.CPU143.PMI:Performance_monitoring_interrupts 262.75 ± 21% -20.5% 209.00 interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15 9911 ± 5% -10.1% 8914 interrupts.CPU16.RES:Rescheduling_interrupts 15340 ± 2% -14.5% 13116 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 12851 ± 4% -11.4% 11384 ± 5% interrupts.CPU19.RES:Rescheduling_interrupts 315.75 ± 49% -36.2% 201.50 ± 3% interrupts.CPU2.47:PCI-MSI.1572866-edge.eth0-TxRx-2 0.75 ±173% +1.7e+05% 1252 ± 76% interrupts.CPU20.NMI:Non-maskable_interrupts 0.75 ±173% +1.7e+05% 1252 ± 76% interrupts.CPU20.PMI:Performance_monitoring_interrupts 12010 ± 8% -12.0% 10567 ± 2% interrupts.CPU20.RES:Rescheduling_interrupts 13261 ± 2% -19.3% 10706 interrupts.CPU21.RES:Rescheduling_interrupts 726.00 ±173% +541.1% 4654 ± 7% interrupts.CPU22.NMI:Non-maskable_interrupts 726.00 ±173% +541.1% 4654 ± 7% interrupts.CPU22.PMI:Performance_monitoring_interrupts 12065 ± 6% -15.6% 10178 ± 4% interrupts.CPU22.RES:Rescheduling_interrupts 194.25 ± 8% +15.6% 224.50 ± 9% interrupts.CPU25.70:PCI-MSI.1572889-edge.eth0-TxRx-25 10775 ± 7% -15.8% 9069 ± 3% interrupts.CPU25.RES:Rescheduling_interrupts 10707 ± 10% -13.3% 9279 interrupts.CPU26.RES:Rescheduling_interrupts 194.50 ± 8% +20.3% 234.00 ± 13% interrupts.CPU27.72:PCI-MSI.1572891-edge.eth0-TxRx-27 6.75 ±156% +16277.8% 1105 ± 98% interrupts.CPU27.NMI:Non-maskable_interrupts 6.75 ±156% +16277.8% 1105 ± 98% interrupts.CPU27.PMI:Performance_monitoring_interrupts 10577 ± 6% -11.9% 9316 ± 2% interrupts.CPU27.RES:Rescheduling_interrupts 1240 ±160% +277.1% 4676 ± 6% interrupts.CPU28.NMI:Non-maskable_interrupts 1240 ±160% +277.1% 4676 ± 6% interrupts.CPU28.PMI:Performance_monitoring_interrupts 16.00 ±129% +21987.5% 3534 ± 26% interrupts.CPU29.NMI:Non-maskable_interrupts 16.00 ±129% +21987.5% 3534 ± 26% interrupts.CPU29.PMI:Performance_monitoring_interrupts 209.50 ± 7% +607.6% 1482 ± 85% interrupts.CPU3.48:PCI-MSI.1572867-edge.eth0-TxRx-3 12626 ± 5% -12.0% 11109 ± 3% interrupts.CPU3.RES:Rescheduling_interrupts 194.00 ± 8% +100.3% 388.50 ± 48% interrupts.CPU31.76:PCI-MSI.1572895-edge.eth0-TxRx-31 1051 ±100% +356.8% 4802 ± 8% interrupts.CPU31.NMI:Non-maskable_interrupts 1051 ±100% +356.8% 4802 ± 8% interrupts.CPU31.PMI:Performance_monitoring_interrupts 5.50 ±152% +1700.0% 99.00 ± 69% interrupts.CPU32.NMI:Non-maskable_interrupts 5.50 ±152% +1700.0% 99.00 ± 69% interrupts.CPU32.PMI:Performance_monitoring_interrupts 10150 ± 7% -15.1% 8618 ± 6% interrupts.CPU32.RES:Rescheduling_interrupts 10730 ± 6% -14.0% 9231 ± 4% interrupts.CPU33.RES:Rescheduling_interrupts 10838 ± 5% -17.9% 8894 ± 2% interrupts.CPU34.RES:Rescheduling_interrupts 9877 ± 7% -12.8% 8611 ± 5% interrupts.CPU35.RES:Rescheduling_interrupts 1209 ± 90% -99.7% 3.50 ±100% interrupts.CPU36.NMI:Non-maskable_interrupts 1209 ± 90% -99.7% 3.50 ±100% interrupts.CPU36.PMI:Performance_monitoring_interrupts 14558 ± 2% -15.8% 12260 ± 4% interrupts.CPU36.RES:Rescheduling_interrupts 13040 ± 5% -18.1% 10680 ± 7% interrupts.CPU37.RES:Rescheduling_interrupts 12539 ± 5% -12.7% 10943 interrupts.CPU38.RES:Rescheduling_interrupts 639.75 ±173% +124.9% 1439 ± 80% interrupts.CPU39.NMI:Non-maskable_interrupts 639.75 ±173% +124.9% 1439 ± 80% interrupts.CPU39.PMI:Performance_monitoring_interrupts 12261 ± 6% -14.5% 10481 ± 6% interrupts.CPU39.RES:Rescheduling_interrupts 13154 ± 4% -18.0% 10789 ± 3% interrupts.CPU4.RES:Rescheduling_interrupts 11044 ± 3% -16.1% 9267 ± 5% interrupts.CPU40.RES:Rescheduling_interrupts 11192 ± 5% -18.5% 9124 ± 3% interrupts.CPU42.RES:Rescheduling_interrupts 11388 ± 5% -19.5% 9171 interrupts.CPU43.RES:Rescheduling_interrupts 12405 ± 4% -13.0% 10791 interrupts.CPU44.RES:Rescheduling_interrupts 12086 ± 7% -15.6% 10203 ± 7% interrupts.CPU45.RES:Rescheduling_interrupts 93.75 ±155% +5034.9% 4814 ± 6% interrupts.CPU46.NMI:Non-maskable_interrupts 93.75 ±155% +5034.9% 4814 ± 6% interrupts.CPU46.PMI:Performance_monitoring_interrupts 10908 ± 6% -14.3% 9346 ± 3% interrupts.CPU47.RES:Rescheduling_interrupts 10191 ± 7% -15.6% 8604 ± 6% interrupts.CPU48.RES:Rescheduling_interrupts 10421 ± 9% -14.5% 8913 interrupts.CPU49.RES:Rescheduling_interrupts 11994 ± 5% -19.2% 9690 ± 3% interrupts.CPU5.RES:Rescheduling_interrupts 3398 ± 58% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts 3398 ± 58% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts 10739 ± 4% -17.6% 8844 ± 4% interrupts.CPU50.RES:Rescheduling_interrupts 10198 ± 10% -16.1% 8552 ± 3% interrupts.CPU51.RES:Rescheduling_interrupts 11673 ± 6% -19.0% 9456 ± 7% interrupts.CPU52.RES:Rescheduling_interrupts 12088 ± 8% -18.9% 9808 ± 5% interrupts.CPU53.RES:Rescheduling_interrupts 15591 ± 3% -15.4% 13192 ± 2% interrupts.CPU54.RES:Rescheduling_interrupts 12738 ± 3% -18.7% 10357 ± 2% interrupts.CPU55.RES:Rescheduling_interrupts 11467 ± 7% -15.1% 9732 ± 2% interrupts.CPU56.RES:Rescheduling_interrupts 1174 ± 98% +313.2% 4852 ± 8% interrupts.CPU57.NMI:Non-maskable_interrupts 1174 ± 98% +313.2% 4852 ± 8% interrupts.CPU57.PMI:Performance_monitoring_interrupts 11791 ± 5% -8.2% 10822 interrupts.CPU57.RES:Rescheduling_interrupts 10778 ± 7% -15.6% 9094 interrupts.CPU58.RES:Rescheduling_interrupts 10887 ± 7% -15.7% 9183 interrupts.CPU59.RES:Rescheduling_interrupts 11517 ± 9% -18.4% 9395 ± 3% interrupts.CPU6.RES:Rescheduling_interrupts 12452 ± 4% -12.2% 10931 interrupts.CPU61.RES:Rescheduling_interrupts 76.75 ±173% +6202.3% 4837 ± 6% interrupts.CPU62.NMI:Non-maskable_interrupts 76.75 ±173% +6202.3% 4837 ± 6% interrupts.CPU62.PMI:Performance_monitoring_interrupts 9.50 ±173% +1194.7% 123.00 ± 86% interrupts.CPU63.NMI:Non-maskable_interrupts 9.50 ±173% +1194.7% 123.00 ± 86% interrupts.CPU63.PMI:Performance_monitoring_interrupts 10589 ± 8% -17.2% 8766 ± 3% interrupts.CPU63.RES:Rescheduling_interrupts 11311 ± 9% -14.8% 9641 ± 2% interrupts.CPU64.RES:Rescheduling_interrupts 1059 ±173% +111.6% 2242 ± 97% interrupts.CPU65.NMI:Non-maskable_interrupts 1059 ±173% +111.6% 2242 ± 97% interrupts.CPU65.PMI:Performance_monitoring_interrupts 9977 ± 6% -10.7% 8910 interrupts.CPU65.RES:Rescheduling_interrupts 52.75 ±156% -97.2% 1.50 ± 33% interrupts.CPU66.NMI:Non-maskable_interrupts 52.75 ±156% -97.2% 1.50 ± 33% interrupts.CPU66.PMI:Performance_monitoring_interrupts 10196 ± 6% -15.8% 8583 ± 4% interrupts.CPU66.RES:Rescheduling_interrupts 1030 ±166% +244.1% 3545 ± 29% interrupts.CPU67.NMI:Non-maskable_interrupts 1030 ±166% +244.1% 3545 ± 29% interrupts.CPU67.PMI:Performance_monitoring_interrupts 12609 ± 4% -17.6% 10389 ± 2% interrupts.CPU68.RES:Rescheduling_interrupts 11849 ± 7% -16.7% 9867 interrupts.CPU69.RES:Rescheduling_interrupts 12074 ± 6% -19.1% 9767 ± 3% interrupts.CPU7.RES:Rescheduling_interrupts 10885 ± 2% -16.4% 9097 ± 2% interrupts.CPU70.RES:Rescheduling_interrupts 13025 ± 9% -23.9% 9908 interrupts.CPU71.RES:Rescheduling_interrupts 13167 ± 4% -15.1% 11182 ± 3% interrupts.CPU72.RES:Rescheduling_interrupts 1786 ±112% +113.0% 3803 ± 37% interrupts.CPU73.NMI:Non-maskable_interrupts 1786 ±112% +113.0% 3803 ± 37% interrupts.CPU73.PMI:Performance_monitoring_interrupts 13026 ± 10% -17.2% 10791 ± 3% interrupts.CPU73.RES:Rescheduling_interrupts 11187 ± 5% -15.0% 9507 ± 5% interrupts.CPU74.RES:Rescheduling_interrupts 12505 ± 3% -14.3% 10719 ± 2% interrupts.CPU76.RES:Rescheduling_interrupts 10949 ± 6% -11.6% 9680 interrupts.CPU8.RES:Rescheduling_interrupts 10064 ± 8% -18.4% 8214 interrupts.CPU80.RES:Rescheduling_interrupts 2392 ± 73% -99.9% 3.00 ±100% interrupts.CPU82.NMI:Non-maskable_interrupts 2392 ± 73% -99.9% 3.00 ±100% interrupts.CPU82.PMI:Performance_monitoring_interrupts 10354 ± 8% -16.7% 8624 ± 4% interrupts.CPU82.RES:Rescheduling_interrupts 1082 ±160% +235.1% 3628 ± 28% interrupts.CPU84.NMI:Non-maskable_interrupts 1082 ±160% +235.1% 3628 ± 28% interrupts.CPU84.PMI:Performance_monitoring_interrupts 11224 ± 6% -17.3% 9280 interrupts.CPU85.RES:Rescheduling_interrupts 2319 ± 99% +109.4% 4857 ± 6% interrupts.CPU89.NMI:Non-maskable_interrupts 2319 ± 99% +109.4% 4857 ± 6% interrupts.CPU89.PMI:Performance_monitoring_interrupts 261.00 ± 40% +47.3% 384.50 ± 43% interrupts.CPU9.54:PCI-MSI.1572873-edge.eth0-TxRx-9 800.75 ±162% +258.5% 2870 ± 88% interrupts.CPU9.NMI:Non-maskable_interrupts 800.75 ±162% +258.5% 2870 ± 88% interrupts.CPU9.PMI:Performance_monitoring_interrupts 10918 ± 6% -16.9% 9070 interrupts.CPU9.RES:Rescheduling_interrupts 13417 ± 9% -19.6% 10786 ± 3% interrupts.CPU90.RES:Rescheduling_interrupts 2188 ± 67% +59.6% 3492 ± 25% interrupts.CPU91.NMI:Non-maskable_interrupts 2188 ± 67% +59.6% 3492 ± 25% interrupts.CPU91.PMI:Performance_monitoring_interrupts 11478 ± 4% -15.5% 9703 ± 7% interrupts.CPU92.RES:Rescheduling_interrupts 11186 ± 7% -13.3% 9702 ± 5% interrupts.CPU94.RES:Rescheduling_interrupts 10416 ± 9% -14.6% 8892 interrupts.CPU95.RES:Rescheduling_interrupts 10373 ± 7% -12.4% 9091 interrupts.CPU96.RES:Rescheduling_interrupts 4550 ± 6% -46.3% 2445 ± 7% interrupts.CPU97.NMI:Non-maskable_interrupts 4550 ± 6% -46.3% 2445 ± 7% interrupts.CPU97.PMI:Performance_monitoring_interrupts 10449 ± 8% -13.9% 8993 ± 4% interrupts.CPU97.RES:Rescheduling_interrupts 4478 ± 7% -71.7% 1268 ± 97% interrupts.CPU99.NMI:Non-maskable_interrupts 4478 ± 7% -71.7% 1268 ± 97% interrupts.CPU99.PMI:Performance_monitoring_interrupts 1602258 ± 6% -13.8% 1381405 ± 3% interrupts.RES:Rescheduling_interrupts *************************************************************************************************** lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory ========================================================================================= compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/2000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ex1/fork_test/aim7/0xb000036 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 50% 1:2 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :2 50% 1:2 dmesg.WARNING:stack_recursion *************************************************************************************************** lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory ========================================================================================= compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/8000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ex1/misc_rtns_1/aim7/0xb000036 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:4 -5% 0:2 perf-profile.children.cycles-pp.error_entry 0:4 -3% 0:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 2288 ± 2% +9.2% 2500 aim7.time.involuntary_context_switches 2890 +2.9% 2974 aim7.time.maximum_resident_set_size 0.00 ± 14% -0.0 0.00 ± 2% mpstat.cpu.all.iowait% 0.01 ± 30% +0.0 0.01 ± 6% mpstat.cpu.all.soft% 30.00 ±173% +383.3% 145.00 ± 9% proc-vmstat.nr_mlock 94893 ± 3% +6.4% 100935 ± 3% proc-vmstat.nr_page_table_pages 5058 +1.5% 5134 proc-vmstat.unevictable_pgs_culled 252673 +6.3% 268545 ± 5% numa-meminfo.node1.Unevictable 380351 ± 60% -73.7% 99869 ± 27% numa-meminfo.node2.AnonPages 176112 ± 88% -90.9% 16067 ± 19% numa-meminfo.node2.PageTables 4988 ± 2% +55.8% 7772 ± 31% numa-meminfo.node3.Mapped 4180 ± 9% +21.1% 5062 ± 7% slabinfo.eventpoll_pwq.active_objs 4180 ± 9% +21.1% 5062 ± 7% slabinfo.eventpoll_pwq.num_objs 541.75 ± 14% +25.5% 680.00 slabinfo.skbuff_fclone_cache.active_objs 541.75 ± 14% +25.5% 680.00 slabinfo.skbuff_fclone_cache.num_objs 6.20 ± 5% +0.4 6.57 ± 3% perf-stat.i.cache-miss-rate% 0.28 +0.0 0.29 perf-stat.i.dTLB-store-miss-rate% 12479108 -2.2% 12200011 perf-stat.i.node-load-misses 0.29 +0.0 0.29 perf-stat.overall.dTLB-store-miss-rate% 12329381 -2.2% 12053893 perf-stat.ps.node-load-misses 53.00 ± 3% +140.6% 127.50 ± 49% interrupts.139:PCI-MSI.1574921-edge.eth3-TxRx-9 1996 ± 10% -16.8% 1660 interrupts.CPU160.RES:Rescheduling_interrupts 1968 ± 9% -20.2% 1570 ± 4% interrupts.CPU163.RES:Rescheduling_interrupts 1979 ± 10% -20.4% 1575 ± 9% interrupts.CPU56.RES:Rescheduling_interrupts 1926 ± 8% -14.6% 1645 ± 2% interrupts.CPU66.RES:Rescheduling_interrupts 53.00 ± 3% +140.6% 127.50 ± 49% interrupts.CPU9.139:PCI-MSI.1574921-edge.eth3-TxRx-9 1235 +8.3% 1337 ± 6% numa-vmstat.node1.nr_mapped 63169 +6.3% 67134 ± 5% numa-vmstat.node1.nr_unevictable 63169 +6.3% 67134 ± 5% numa-vmstat.node1.nr_zone_unevictable 95077 ± 61% -73.7% 24959 ± 28% numa-vmstat.node2.nr_anon_pages 44094 ± 88% -90.9% 4014 ± 19% numa-vmstat.node2.nr_page_table_pages 1309 ± 10% +57.2% 2058 ± 24% numa-vmstat.node3.nr_mapped 376579 ± 7% +22.1% 459946 ± 12% numa-vmstat.node3.numa_hit 278936 ± 13% +24.4% 347071 ± 16% numa-vmstat.node3.numa_local 14.31 ± 54% +1356.5% 208.44 ± 91% sched_debug.cfs_rq:/.load_avg.avg 485.50 ± 10% +4220.5% 20976 ± 97% sched_debug.cfs_rq:/.load_avg.max 64.90 ± 34% +3027.9% 2030 ± 96% sched_debug.cfs_rq:/.load_avg.stddev 58047 ± 10% +23.4% 71641 ± 2% sched_debug.cfs_rq:/.min_vruntime.stddev 37.00 ± 4% -12.8% 32.25 sched_debug.cfs_rq:/.runnable_load_avg.max 58045 ± 10% +23.4% 71609 ± 2% sched_debug.cfs_rq:/.spread0.stddev 149.17 ± 15% +30.3% 194.33 ± 5% sched_debug.cfs_rq:/.util_avg.avg 8.45 ± 34% +53.5% 12.97 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.avg 42.87 ± 21% +24.7% 53.47 sched_debug.cfs_rq:/.util_est_enqueued.stddev 37.00 ± 4% -12.8% 32.25 sched_debug.cpu.cpu_load[0].max 0.85 ± 10% +22.9% 1.04 sched_debug.cpu.cpu_load[4].avg 70.12 ± 11% +48.0% 103.75 ± 22% sched_debug.cpu.cpu_load[4].max 5.40 ± 10% +43.2% 7.73 ± 19% sched_debug.cpu.cpu_load[4].stddev 22047 ± 3% +7.5% 23711 ± 7% sched_debug.cpu.nr_switches.max 13940 ± 4% +15.9% 16158 ± 6% sched_debug.cpu.ttwu_count.max 433.88 ± 11% +20.1% 521.25 ± 3% sched_debug.cpu.ttwu_local.max 37.71 ± 4% +10.2% 41.56 sched_debug.cpu.ttwu_local.stddev 0.49 ±104% +0.9 1.42 ± 32% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.59 ±103% +0.9 1.53 ± 23% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 0.61 ±103% +0.9 1.55 ± 21% perf-profile.calltrace.cycles-pp.page_fault 0.61 ±103% +0.9 1.55 ± 21% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 0.46 ±104% +1.9 2.33 ± 61% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 1.05 ± 77% -0.9 0.18 ±100% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.99 ± 87% -0.9 0.12 ±100% perf-profile.children.cycles-pp.load_balance 0.88 ±103% -0.8 0.09 ±100% perf-profile.children.cycles-pp.find_busiest_group 0.97 ± 87% -0.8 0.18 ±100% perf-profile.children.cycles-pp.native_write_msr 0.84 ±110% -0.8 0.09 ± 99% perf-profile.children.cycles-pp.update_sd_lb_stats 0.72 ±137% -0.7 0.06 ±100% perf-profile.children.cycles-pp.ksys_write 0.70 ±142% -0.7 0.04 ±100% perf-profile.children.cycles-pp.vfs_write 0.70 ±143% -0.7 0.04 ±100% perf-profile.children.cycles-pp.new_sync_write 0.71 ±141% -0.7 0.05 ±100% perf-profile.children.cycles-pp.idle_cpu 0.23 ± 60% +0.9 1.11 ± 69% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.09 ±102% +0.9 1.02 ± 84% perf-profile.children.cycles-pp.lockref_get_not_dead 0.10 ±101% +0.9 1.04 ± 81% perf-profile.children.cycles-pp.unlazy_walk 0.07 ±103% +1.0 1.02 ± 85% perf-profile.children.cycles-pp.filp_close 0.05 ±103% +1.0 1.00 ± 88% perf-profile.children.cycles-pp.__do_fault 0.01 ±173% +1.0 0.98 ± 92% perf-profile.children.cycles-pp.walk_component 0.56 ± 83% +1.8 2.38 ± 58% perf-profile.children.cycles-pp.handle_mm_fault 0.52 ± 83% +1.8 2.34 ± 61% perf-profile.children.cycles-pp.__handle_mm_fault 0.66 ± 86% +1.8 2.48 ± 52% perf-profile.children.cycles-pp.__do_page_fault 0.68 ± 86% +1.8 2.50 ± 50% perf-profile.children.cycles-pp.do_page_fault 0.10 ±100% +1.9 2.00 ± 88% perf-profile.children.cycles-pp.link_path_walk 0.97 ± 87% -0.8 0.18 ±100% perf-profile.self.cycles-pp.native_write_msr 0.71 ±141% -0.7 0.05 ±100% perf-profile.self.cycles-pp.idle_cpu 0.12 ± 62% +0.9 1.04 ± 80% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.09 ±102% +0.9 1.02 ± 84% perf-profile.self.cycles-pp.lockref_get_not_dead 40636 ± 10% +17.1% 47594 ± 6% softirqs.CPU0.TIMER 14166 +14.8% 16269 ± 2% softirqs.CPU1.SCHED 38044 ± 13% +29.5% 49279 ± 9% softirqs.CPU119.TIMER 35559 ± 11% +16.2% 41310 ± 4% softirqs.CPU120.TIMER 35455 ± 11% +17.0% 41488 ± 2% softirqs.CPU121.TIMER 35068 ± 10% +16.1% 40722 ± 5% softirqs.CPU122.TIMER 35803 ± 10% +13.0% 40466 ± 5% softirqs.CPU123.TIMER 35825 ± 10% +14.1% 40893 ± 3% softirqs.CPU124.TIMER 35520 ± 11% +13.7% 40369 ± 4% softirqs.CPU125.TIMER 35467 ± 12% +17.5% 41691 ± 8% softirqs.CPU126.TIMER 34958 ± 11% +15.1% 40232 ± 4% softirqs.CPU127.TIMER 35118 ± 11% +14.4% 40160 ± 4% softirqs.CPU128.TIMER 35132 ± 11% +16.9% 41075 ± 6% softirqs.CPU129.TIMER 35102 ± 11% +16.8% 40985 ± 3% softirqs.CPU130.TIMER 34985 ± 11% +14.6% 40090 ± 4% softirqs.CPU131.TIMER 34851 ± 11% +15.2% 40142 ± 4% softirqs.CPU132.TIMER 35377 ± 12% +13.3% 40081 ± 4% softirqs.CPU133.TIMER 35590 ± 11% +13.4% 40345 ± 4% softirqs.CPU134.TIMER 35790 ± 8% +12.2% 40158 ± 4% softirqs.CPU135.TIMER 35199 ± 10% +14.1% 40152 ± 4% softirqs.CPU136.TIMER 35621 ± 10% +12.7% 40149 ± 4% softirqs.CPU137.TIMER 34932 ± 11% +14.9% 40122 ± 4% softirqs.CPU138.TIMER 35099 ± 10% +14.2% 40084 ± 4% softirqs.CPU139.TIMER 35072 ± 11% +14.2% 40039 ± 4% softirqs.CPU140.TIMER 35675 ± 12% +13.5% 40489 ± 5% softirqs.CPU141.TIMER 35764 ± 12% +12.6% 40259 ± 4% softirqs.CPU142.TIMER 36980 ± 9% +14.4% 42313 ± 4% softirqs.CPU25.TIMER 36698 ± 11% +13.6% 41699 ± 4% softirqs.CPU26.TIMER 35995 ± 10% +15.4% 41530 ± 4% softirqs.CPU27.TIMER 36891 ± 9% +12.2% 41375 ± 4% softirqs.CPU28.TIMER 36396 ± 10% +13.4% 41291 ± 4% softirqs.CPU29.TIMER 36554 ± 11% +12.4% 41089 ± 4% softirqs.CPU30.TIMER 36128 ± 10% +14.0% 41170 ± 4% softirqs.CPU31.TIMER 36231 ± 10% +16.3% 42151 ± 6% softirqs.CPU32.TIMER 36349 ± 11% +15.7% 42052 ± 6% softirqs.CPU33.TIMER 36689 ± 9% +12.6% 41328 ± 4% softirqs.CPU34.TIMER 36109 ± 10% +14.4% 41318 ± 4% softirqs.CPU35.TIMER 36038 ± 10% +14.0% 41098 ± 3% softirqs.CPU36.TIMER 36379 ± 11% +13.0% 41123 ± 4% softirqs.CPU37.TIMER 36229 ± 11% +13.5% 41132 ± 4% softirqs.CPU38.TIMER 36601 ± 11% +12.2% 41074 ± 3% softirqs.CPU39.TIMER 36723 ± 9% +12.0% 41148 ± 3% softirqs.CPU40.TIMER 36789 ± 10% +11.5% 41010 ± 3% softirqs.CPU41.TIMER 36238 ± 8% +13.4% 41097 ± 3% softirqs.CPU42.TIMER 36335 ± 12% +13.1% 41079 ± 3% softirqs.CPU43.TIMER 36114 ± 9% +13.8% 41110 ± 4% softirqs.CPU44.TIMER 36427 ± 11% +13.5% 41361 ± 4% softirqs.CPU45.TIMER 36312 ± 11% +13.8% 41311 ± 4% softirqs.CPU46.TIMER 23726 ± 6% +12.0% 26584 ± 3% softirqs.CPU47.RCU 13762 ± 2% +12.5% 15485 ± 6% softirqs.CPU48.SCHED 38329 ± 11% +30.0% 49841 ± 16% softirqs.CPU48.TIMER *************************************************************************************************** lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-2019-05-14.cgz/lkp-knm01/unlink1/will-it-scale commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- %stddev %change %stddev \ | \ 1119 ± 2% +3.1% 1154 will-it-scale.time.involuntary_context_switches 318.75 ± 5% +9.2% 348.00 ± 3% will-it-scale.time.percent_of_cpu_this_job_got 928.56 ± 5% +9.6% 1017 ± 3% will-it-scale.time.system_time 1.07 ± 7% +0.2 1.26 ± 10% mpstat.cpu.all.sys% 275596 +1.2% 278949 proc-vmstat.nr_file_pages 94.05 +2.6% 96.48 ± 2% turbostat.PkgWatt 1821 ± 7% +14.4% 2084 ± 4% sched_debug.cfs_rq:/.exec_clock.avg 43331 ± 3% +10.0% 47671 ± 3% sched_debug.cfs_rq:/.load.stddev 272381 ± 8% -19.1% 220404 ± 5% sched_debug.cfs_rq:/.min_vruntime.avg 43257 ± 3% +10.2% 47663 ± 3% sched_debug.cfs_rq:/.runnable_weight.stddev 21.49 ± 4% -12.9% 18.71 ± 7% sched_debug.cpu.cpu_load[0].stddev 348.60 ± 4% -15.0% 296.25 ± 9% sched_debug.cpu.cpu_load[1].max 348.25 ± 3% -15.1% 295.83 ± 10% sched_debug.cpu.cpu_load[2].max -5.76 -30.5% -4.00 sched_debug.cpu.nr_uninterruptible.min 62873 ± 7% -20.3% 50079 sched_debug.cpu.sched_count.max 152.30 ± 6% +12.9% 171.99 ± 6% sched_debug.cpu.ttwu_local.avg 7.873e+08 ± 2% +4.0% 8.191e+08 perf-stat.i.branch-instructions 8.29 ± 7% +1.8 10.08 ± 11% perf-stat.i.branch-miss-rate% 65211385 ± 8% +26.6% 82541419 ± 12% perf-stat.i.branch-misses 12.91 ± 4% +24.6% 16.08 ± 18% perf-stat.i.cpi 4.73e+10 ± 5% +28.9% 6.098e+10 ± 19% perf-stat.i.cpu-cycles 1.80 +0.1 1.86 ± 3% perf-stat.i.iTLB-load-miss-rate% 67010767 ± 2% +7.4% 71977406 ± 3% perf-stat.i.iTLB-load-misses 3.694e+09 ± 2% +3.4% 3.819e+09 perf-stat.i.iTLB-loads 3.691e+09 ± 2% +3.4% 3.817e+09 perf-stat.i.instructions 2527 +0.5% 2539 perf-stat.i.minor-faults 2527 +0.5% 2540 perf-stat.i.page-faults 8.28 ± 7% +1.8 10.07 ± 11% perf-stat.overall.branch-miss-rate% 12.81 ± 4% +24.6% 15.96 ± 18% perf-stat.overall.cpi 1.78 +0.1 1.85 ± 3% perf-stat.overall.iTLB-load-miss-rate% 36179580 ± 3% +17.7% 42571444 ± 13% perf-stat.overall.path-length 7.846e+08 ± 2% +4.0% 8.163e+08 perf-stat.ps.branch-instructions 64994927 ± 8% +26.6% 82258003 ± 12% perf-stat.ps.branch-misses 4.714e+10 ± 5% +28.9% 6.076e+10 ± 19% perf-stat.ps.cpu-cycles 66779111 ± 2% +7.4% 71724036 ± 3% perf-stat.ps.iTLB-load-misses 3.681e+09 ± 2% +3.4% 3.806e+09 perf-stat.ps.iTLB-loads 3.678e+09 ± 2% +3.4% 3.804e+09 perf-stat.ps.instructions 2519 +0.5% 2531 perf-stat.ps.minor-faults 2519 +0.5% 2532 perf-stat.ps.page-faults 1.105e+12 ± 2% +3.8% 1.147e+12 perf-stat.total.instructions 110711 ± 4% +7.0% 118470 ± 5% softirqs.CPU109.TIMER 110627 ± 4% +6.2% 117444 ± 5% softirqs.CPU112.TIMER 112989 ± 7% +8.4% 122443 ± 5% softirqs.CPU123.TIMER 109980 ± 3% +5.1% 115558 ± 4% softirqs.CPU14.TIMER 109702 ± 4% +4.8% 114978 ± 4% softirqs.CPU17.TIMER 114458 ± 6% +7.4% 122929 ± 4% softirqs.CPU195.TIMER 113939 ± 3% +5.0% 119644 ± 5% softirqs.CPU236.TIMER 110976 ± 6% +4.9% 116424 ± 5% softirqs.CPU240.TIMER 108141 ± 5% +6.4% 115052 ± 7% softirqs.CPU253.TIMER 109346 ± 6% +8.2% 118301 ± 2% softirqs.CPU256.TIMER 109601 ± 5% +7.0% 117275 ± 4% softirqs.CPU266.TIMER 111080 ± 2% +6.8% 118592 ± 5% softirqs.CPU268.TIMER 110244 ± 5% +4.5% 115237 ± 6% softirqs.CPU274.TIMER 109345 ± 6% +8.0% 118148 ± 8% softirqs.CPU275.TIMER 105278 ± 2% +11.3% 117150 ± 8% softirqs.CPU287.TIMER 25916 ± 3% +3.4% 26788 ± 4% softirqs.CPU3.SCHED 109550 ± 6% +7.1% 117288 ± 7% softirqs.CPU51.TIMER 109353 ± 4% +7.6% 117652 ± 6% softirqs.CPU61.TIMER 109018 ± 3% +7.4% 117069 ± 6% softirqs.CPU66.TIMER 111093 ± 2% +8.6% 120616 ± 8% softirqs.CPU67.TIMER 108126 ± 3% +7.5% 116246 ± 6% softirqs.CPU69.TIMER 109627 ± 4% +8.4% 118799 ± 6% softirqs.CPU70.TIMER 106067 +11.0% 117778 ± 7% softirqs.CPU71.TIMER 109417 ± 4% +6.5% 116523 ± 6% softirqs.CPU77.TIMER 112795 +8.9% 122821 ± 7% softirqs.CPU83.TIMER 110905 ± 4% +5.5% 117001 ± 5% softirqs.CPU89.TIMER 26108 +6.3% 27763 ± 5% softirqs.CPU92.SCHED 1.41 ± 11% +0.2 1.58 perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 0.93 ± 18% +0.2 1.17 ± 3% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.61 ± 12% +0.3 1.89 ± 4% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 93.17 +0.6 93.78 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 93.17 +0.6 93.78 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 93.05 +0.6 93.66 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 7.79 ± 6% +0.7 8.52 ± 8% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 8.40 ± 7% +0.8 9.25 ± 9% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 3.12 -0.3 2.84 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu 1.03 ± 15% -0.2 0.84 ± 4% perf-profile.children.cycles-pp._raw_spin_lock 0.68 ± 11% -0.1 0.62 perf-profile.children.cycles-pp.update_sd_lb_stats 0.06 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.__strcspn_sse42 0.06 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.generic_start_main 0.06 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.main 0.06 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.run_builtin 0.13 ± 9% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.12 ± 13% +0.0 0.16 ± 9% perf-profile.children.cycles-pp.retint_kernel 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick 1.68 ± 12% +0.3 1.95 ± 5% perf-profile.children.cycles-pp.clockevents_program_event 93.17 +0.6 93.78 perf-profile.children.cycles-pp.start_secondary 7.99 ± 6% +0.7 8.64 ± 8% perf-profile.children.cycles-pp.update_process_times 8.59 ± 7% +0.8 9.38 ± 9% perf-profile.children.cycles-pp.tick_sched_handle 3.99 ± 8% -0.4 3.54 ± 4% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.99 ± 15% -0.2 0.81 ± 3% perf-profile.self.cycles-pp._raw_spin_lock 0.55 ± 3% -0.1 0.47 ± 5% perf-profile.self.cycles-pp.tick_nohz_next_event 0.09 ± 5% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.irq_work_run_list 0.16 ± 6% +0.0 0.19 perf-profile.self.cycles-pp.lapic_next_deadline 0.12 ± 15% +0.0 0.15 ± 6% perf-profile.self.cycles-pp.retint_kernel 1198 ± 11% -60.8% 470.50 ± 28% interrupts.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2 620.25 ± 12% -43.2% 352.00 ± 25% interrupts.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3 1083 ± 37% +71.1% 1853 ± 15% interrupts.CPU112.NMI:Non-maskable_interrupts 1083 ± 37% +71.1% 1853 ± 15% interrupts.CPU112.PMI:Performance_monitoring_interrupts 62.50 ± 82% -86.4% 8.50 ± 5% interrupts.CPU112.TLB:TLB_shootdowns 1137 ± 28% +59.1% 1809 ± 14% interrupts.CPU114.NMI:Non-maskable_interrupts 1137 ± 28% +59.1% 1809 ± 14% interrupts.CPU114.PMI:Performance_monitoring_interrupts 1137 ± 28% +63.1% 1855 ± 13% interrupts.CPU115.NMI:Non-maskable_interrupts 1137 ± 28% +63.1% 1855 ± 13% interrupts.CPU115.PMI:Performance_monitoring_interrupts 1196 ± 31% +57.8% 1888 ± 16% interrupts.CPU116.NMI:Non-maskable_interrupts 1196 ± 31% +57.8% 1888 ± 16% interrupts.CPU116.PMI:Performance_monitoring_interrupts 1198 ± 11% -60.8% 470.50 ± 28% interrupts.CPU12.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2 1375 ± 23% +43.4% 1972 ± 16% interrupts.CPU120.NMI:Non-maskable_interrupts 1375 ± 23% +43.4% 1972 ± 16% interrupts.CPU120.PMI:Performance_monitoring_interrupts 1377 ± 24% +45.2% 2000 ± 15% interrupts.CPU121.NMI:Non-maskable_interrupts 1377 ± 24% +45.2% 2000 ± 15% interrupts.CPU121.PMI:Performance_monitoring_interrupts 799.75 ± 9% +25.7% 1005 ± 6% interrupts.CPU123.RES:Rescheduling_interrupts 898.50 ± 17% -25.9% 665.50 ± 9% interrupts.CPU129.RES:Rescheduling_interrupts 620.25 ± 12% -43.2% 352.00 ± 25% interrupts.CPU13.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3 875.75 ± 7% -28.5% 626.50 ± 9% interrupts.CPU13.RES:Rescheduling_interrupts 1219 ± 40% +58.7% 1934 ± 12% interrupts.CPU131.NMI:Non-maskable_interrupts 1219 ± 40% +58.7% 1934 ± 12% interrupts.CPU131.PMI:Performance_monitoring_interrupts 1011 ± 45% +93.8% 1960 ± 12% interrupts.CPU132.NMI:Non-maskable_interrupts 1011 ± 45% +93.8% 1960 ± 12% interrupts.CPU132.PMI:Performance_monitoring_interrupts 1186 ± 34% +51.9% 1802 ± 11% interrupts.CPU133.NMI:Non-maskable_interrupts 1186 ± 34% +51.9% 1802 ± 11% interrupts.CPU133.PMI:Performance_monitoring_interrupts 1158 ± 33% +58.5% 1836 ± 18% interrupts.CPU134.NMI:Non-maskable_interrupts 1158 ± 33% +58.5% 1836 ± 18% interrupts.CPU134.PMI:Performance_monitoring_interrupts 1858 ± 64% -66.4% 625.00 ± 23% interrupts.CPU134.RES:Rescheduling_interrupts 1290 ± 27% +39.1% 1794 ± 15% interrupts.CPU135.NMI:Non-maskable_interrupts 1290 ± 27% +39.1% 1794 ± 15% interrupts.CPU135.PMI:Performance_monitoring_interrupts 1399 ± 71% -57.9% 589.50 ± 3% interrupts.CPU135.RES:Rescheduling_interrupts 1173 ± 35% +58.8% 1863 ± 16% interrupts.CPU136.NMI:Non-maskable_interrupts 1173 ± 35% +58.8% 1863 ± 16% interrupts.CPU136.PMI:Performance_monitoring_interrupts 1342 ± 26% +44.2% 1936 ± 17% interrupts.CPU137.NMI:Non-maskable_interrupts 1342 ± 26% +44.2% 1936 ± 17% interrupts.CPU137.PMI:Performance_monitoring_interrupts 885.50 ± 14% -30.3% 617.50 ± 9% interrupts.CPU137.RES:Rescheduling_interrupts 645.00 ± 12% +19.1% 768.00 ± 6% interrupts.CPU14.RES:Rescheduling_interrupts 861.75 ± 9% -31.5% 590.50 ± 4% interrupts.CPU140.RES:Rescheduling_interrupts 1142 ± 28% +66.7% 1904 ± 14% interrupts.CPU144.NMI:Non-maskable_interrupts 1142 ± 28% +66.7% 1904 ± 14% interrupts.CPU144.PMI:Performance_monitoring_interrupts 1386 ± 27% +39.3% 1931 ± 13% interrupts.CPU147.NMI:Non-maskable_interrupts 1386 ± 27% +39.3% 1931 ± 13% interrupts.CPU147.PMI:Performance_monitoring_interrupts 1392 ± 25% +42.2% 1981 ± 14% interrupts.CPU148.NMI:Non-maskable_interrupts 1392 ± 25% +42.2% 1981 ± 14% interrupts.CPU148.PMI:Performance_monitoring_interrupts 1383 ± 24% +34.2% 1856 ± 11% interrupts.CPU149.NMI:Non-maskable_interrupts 1383 ± 24% +34.2% 1856 ± 11% interrupts.CPU149.PMI:Performance_monitoring_interrupts 1365 ± 21% +36.8% 1868 ± 14% interrupts.CPU150.NMI:Non-maskable_interrupts 1365 ± 21% +36.8% 1868 ± 14% interrupts.CPU150.PMI:Performance_monitoring_interrupts 1386 ± 25% +29.3% 1792 ± 11% interrupts.CPU151.NMI:Non-maskable_interrupts 1386 ± 25% +29.3% 1792 ± 11% interrupts.CPU151.PMI:Performance_monitoring_interrupts 1381 ± 21% +36.9% 1891 ± 13% interrupts.CPU152.NMI:Non-maskable_interrupts 1381 ± 21% +36.9% 1891 ± 13% interrupts.CPU152.PMI:Performance_monitoring_interrupts 1348 ± 24% +38.9% 1872 ± 14% interrupts.CPU153.NMI:Non-maskable_interrupts 1348 ± 24% +38.9% 1872 ± 14% interrupts.CPU153.PMI:Performance_monitoring_interrupts 1435 ± 27% +39.5% 2003 ± 13% interrupts.CPU157.NMI:Non-maskable_interrupts 1435 ± 27% +39.5% 2003 ± 13% interrupts.CPU157.PMI:Performance_monitoring_interrupts 32.75 ± 64% +193.1% 96.00 ± 23% interrupts.CPU158.TLB:TLB_shootdowns 1297 ± 26% +36.2% 1766 ± 8% interrupts.CPU159.NMI:Non-maskable_interrupts 1297 ± 26% +36.2% 1766 ± 8% interrupts.CPU159.PMI:Performance_monitoring_interrupts 920.00 ± 8% -36.1% 587.50 ± 17% interrupts.CPU159.RES:Rescheduling_interrupts 1329 ± 25% +39.4% 1853 ± 18% interrupts.CPU160.NMI:Non-maskable_interrupts 1329 ± 25% +39.4% 1853 ± 18% interrupts.CPU160.PMI:Performance_monitoring_interrupts 1377 ± 26% +37.2% 1889 ± 14% interrupts.CPU161.NMI:Non-maskable_interrupts 1377 ± 26% +37.2% 1889 ± 14% interrupts.CPU161.PMI:Performance_monitoring_interrupts 1397 ± 20% +36.8% 1910 ± 13% interrupts.CPU162.NMI:Non-maskable_interrupts 1397 ± 20% +36.8% 1910 ± 13% interrupts.CPU162.PMI:Performance_monitoring_interrupts 1350 ± 23% +33.0% 1796 ± 11% interrupts.CPU163.NMI:Non-maskable_interrupts 1350 ± 23% +33.0% 1796 ± 11% interrupts.CPU163.PMI:Performance_monitoring_interrupts 1408 ± 26% +43.0% 2013 ± 18% interrupts.CPU164.NMI:Non-maskable_interrupts 1408 ± 26% +43.0% 2013 ± 18% interrupts.CPU164.PMI:Performance_monitoring_interrupts 695.25 ± 21% +21.2% 842.50 ± 9% interrupts.CPU17.RES:Rescheduling_interrupts 1396 ± 25% +30.5% 1823 ± 7% interrupts.CPU184.NMI:Non-maskable_interrupts 1396 ± 25% +30.5% 1823 ± 7% interrupts.CPU184.PMI:Performance_monitoring_interrupts 1354 ± 25% +36.8% 1852 ± 8% interrupts.CPU185.NMI:Non-maskable_interrupts 1354 ± 25% +36.8% 1852 ± 8% interrupts.CPU185.PMI:Performance_monitoring_interrupts 1169 ± 27% +62.3% 1897 ± 17% interrupts.CPU190.NMI:Non-maskable_interrupts 1169 ± 27% +62.3% 1897 ± 17% interrupts.CPU190.PMI:Performance_monitoring_interrupts 1059 ± 21% -27.3% 770.50 interrupts.CPU190.RES:Rescheduling_interrupts 1124 ± 29% +64.0% 1843 ± 17% interrupts.CPU191.NMI:Non-maskable_interrupts 1124 ± 29% +64.0% 1843 ± 17% interrupts.CPU191.PMI:Performance_monitoring_interrupts 1159 ± 29% +71.5% 1988 ± 17% interrupts.CPU192.NMI:Non-maskable_interrupts 1159 ± 29% +71.5% 1988 ± 17% interrupts.CPU192.PMI:Performance_monitoring_interrupts 1223 ± 30% +57.9% 1931 ± 12% interrupts.CPU194.NMI:Non-maskable_interrupts 1223 ± 30% +57.9% 1931 ± 12% interrupts.CPU194.PMI:Performance_monitoring_interrupts 1234 ± 32% +53.0% 1888 ± 11% interrupts.CPU195.NMI:Non-maskable_interrupts 1234 ± 32% +53.0% 1888 ± 11% interrupts.CPU195.PMI:Performance_monitoring_interrupts 1173 ± 25% +68.7% 1980 ± 11% interrupts.CPU196.NMI:Non-maskable_interrupts 1173 ± 25% +68.7% 1980 ± 11% interrupts.CPU196.PMI:Performance_monitoring_interrupts 685.25 ± 10% +19.0% 815.50 ± 16% interrupts.CPU2.NMI:Non-maskable_interrupts 685.25 ± 10% +19.0% 815.50 ± 16% interrupts.CPU2.PMI:Performance_monitoring_interrupts 79.75 ± 56% -67.4% 26.00 ± 7% interrupts.CPU208.TLB:TLB_shootdowns 786.00 ± 9% +90.1% 1494 ± 47% interrupts.CPU220.NMI:Non-maskable_interrupts 786.00 ± 9% +90.1% 1494 ± 47% interrupts.CPU220.PMI:Performance_monitoring_interrupts 749.25 ± 4% +100.5% 1502 ± 45% interrupts.CPU221.NMI:Non-maskable_interrupts 749.25 ± 4% +100.5% 1502 ± 45% interrupts.CPU221.PMI:Performance_monitoring_interrupts 749.75 ± 3% +149.4% 1870 ± 18% interrupts.CPU223.NMI:Non-maskable_interrupts 749.75 ± 3% +149.4% 1870 ± 18% interrupts.CPU223.PMI:Performance_monitoring_interrupts 706.25 ± 3% +156.1% 1809 ± 19% interrupts.CPU224.NMI:Non-maskable_interrupts 706.25 ± 3% +156.1% 1809 ± 19% interrupts.CPU224.PMI:Performance_monitoring_interrupts 22.00 ± 68% +256.8% 78.50 ± 40% interrupts.CPU227.TLB:TLB_shootdowns 1002 ± 37% +45.5% 1458 ± 43% interrupts.CPU231.NMI:Non-maskable_interrupts 1002 ± 37% +45.5% 1458 ± 43% interrupts.CPU231.PMI:Performance_monitoring_interrupts 977.50 ± 35% +55.0% 1515 ± 46% interrupts.CPU233.NMI:Non-maskable_interrupts 977.50 ± 35% +55.0% 1515 ± 46% interrupts.CPU233.PMI:Performance_monitoring_interrupts 970.00 ± 34% +57.6% 1529 ± 45% interrupts.CPU239.NMI:Non-maskable_interrupts 970.00 ± 34% +57.6% 1529 ± 45% interrupts.CPU239.PMI:Performance_monitoring_interrupts 1999 ±100% -67.0% 659.00 interrupts.CPU239.RES:Rescheduling_interrupts 979.75 ± 35% +53.1% 1500 ± 43% interrupts.CPU240.NMI:Non-maskable_interrupts 979.75 ± 35% +53.1% 1500 ± 43% interrupts.CPU240.PMI:Performance_monitoring_interrupts 1019 ± 38% +46.7% 1495 ± 43% interrupts.CPU241.NMI:Non-maskable_interrupts 1019 ± 38% +46.7% 1495 ± 43% interrupts.CPU241.PMI:Performance_monitoring_interrupts 687.75 ± 19% +51.1% 1039 ± 3% interrupts.CPU241.RES:Rescheduling_interrupts 872.00 ± 11% +61.3% 1406 ± 19% interrupts.CPU243.RES:Rescheduling_interrupts 763.00 ± 7% -12.1% 670.50 interrupts.CPU25.RES:Rescheduling_interrupts 933.25 ± 33% +53.0% 1428 ± 43% interrupts.CPU253.NMI:Non-maskable_interrupts 933.25 ± 33% +53.0% 1428 ± 43% interrupts.CPU253.PMI:Performance_monitoring_interrupts 924.75 ± 35% +57.1% 1453 ± 46% interrupts.CPU256.NMI:Non-maskable_interrupts 924.75 ± 35% +57.1% 1453 ± 46% interrupts.CPU256.PMI:Performance_monitoring_interrupts 938.50 ± 36% +54.5% 1450 ± 44% interrupts.CPU257.NMI:Non-maskable_interrupts 938.50 ± 36% +54.5% 1450 ± 44% interrupts.CPU257.PMI:Performance_monitoring_interrupts 1122 ± 29% -43.5% 634.00 ± 13% interrupts.CPU258.RES:Rescheduling_interrupts 1225 ± 27% -40.6% 727.50 ± 5% interrupts.CPU259.RES:Rescheduling_interrupts 41.75 ± 27% +79.6% 75.00 ± 33% interrupts.CPU261.TLB:TLB_shootdowns 725.50 ± 8% -24.5% 548.00 ± 10% interrupts.CPU264.RES:Rescheduling_interrupts 741.25 ± 7% +103.8% 1511 ± 47% interrupts.CPU268.NMI:Non-maskable_interrupts 741.25 ± 7% +103.8% 1511 ± 47% interrupts.CPU268.PMI:Performance_monitoring_interrupts 683.25 ± 7% +113.6% 1459 ± 44% interrupts.CPU269.NMI:Non-maskable_interrupts 683.25 ± 7% +113.6% 1459 ± 44% interrupts.CPU269.PMI:Performance_monitoring_interrupts 793.00 ± 17% +56.6% 1242 ± 26% interrupts.CPU271.RES:Rescheduling_interrupts 690.50 ± 8% +97.6% 1364 ± 44% interrupts.CPU274.NMI:Non-maskable_interrupts 690.50 ± 8% +97.6% 1364 ± 44% interrupts.CPU274.PMI:Performance_monitoring_interrupts 690.50 ± 9% +95.7% 1351 ± 46% interrupts.CPU276.NMI:Non-maskable_interrupts 690.50 ± 9% +95.7% 1351 ± 46% interrupts.CPU276.PMI:Performance_monitoring_interrupts 888.75 ± 22% -26.6% 652.00 ± 2% interrupts.CPU278.RES:Rescheduling_interrupts 944.25 ± 43% +44.2% 1362 ± 45% interrupts.CPU279.NMI:Non-maskable_interrupts 944.25 ± 43% +44.2% 1362 ± 45% interrupts.CPU279.PMI:Performance_monitoring_interrupts 594.00 ± 13% +18.5% 704.00 ± 3% interrupts.CPU28.RES:Rescheduling_interrupts 812.75 ± 9% -34.2% 534.50 ± 25% interrupts.CPU280.RES:Rescheduling_interrupts 724.75 ± 7% -16.3% 606.50 ± 4% interrupts.CPU281.RES:Rescheduling_interrupts 660.50 ± 8% +29.6% 856.00 ± 17% interrupts.CPU3.NMI:Non-maskable_interrupts 660.50 ± 8% +29.6% 856.00 ± 17% interrupts.CPU3.PMI:Performance_monitoring_interrupts 697.75 ± 15% +64.2% 1145 ± 12% interrupts.CPU30.NMI:Non-maskable_interrupts 697.75 ± 15% +64.2% 1145 ± 12% interrupts.CPU30.PMI:Performance_monitoring_interrupts 694.25 ± 8% +72.5% 1197 ± 17% interrupts.CPU31.NMI:Non-maskable_interrupts 694.25 ± 8% +72.5% 1197 ± 17% interrupts.CPU31.PMI:Performance_monitoring_interrupts 643.25 ± 14% +33.2% 857.00 ± 19% interrupts.CPU32.NMI:Non-maskable_interrupts 643.25 ± 14% +33.2% 857.00 ± 19% interrupts.CPU32.PMI:Performance_monitoring_interrupts 861.75 ± 7% -17.3% 712.50 interrupts.CPU32.RES:Rescheduling_interrupts 633.75 ± 14% +37.3% 870.00 ± 19% interrupts.CPU33.NMI:Non-maskable_interrupts 633.75 ± 14% +37.3% 870.00 ± 19% interrupts.CPU33.PMI:Performance_monitoring_interrupts 642.75 ± 9% +34.3% 863.00 ± 18% interrupts.CPU34.NMI:Non-maskable_interrupts 642.75 ± 9% +34.3% 863.00 ± 18% interrupts.CPU34.PMI:Performance_monitoring_interrupts 649.00 ± 5% +31.9% 856.00 ± 20% interrupts.CPU35.NMI:Non-maskable_interrupts 649.00 ± 5% +31.9% 856.00 ± 20% interrupts.CPU35.PMI:Performance_monitoring_interrupts 644.75 ± 7% +37.4% 886.00 ± 19% interrupts.CPU38.NMI:Non-maskable_interrupts 644.75 ± 7% +37.4% 886.00 ± 19% interrupts.CPU38.PMI:Performance_monitoring_interrupts 667.25 ± 6% +30.5% 870.50 ± 21% interrupts.CPU39.NMI:Non-maskable_interrupts 667.25 ± 6% +30.5% 870.50 ± 21% interrupts.CPU39.PMI:Performance_monitoring_interrupts 673.75 +26.3% 851.00 ± 15% interrupts.CPU4.NMI:Non-maskable_interrupts 673.75 +26.3% 851.00 ± 15% interrupts.CPU4.PMI:Performance_monitoring_interrupts 664.50 ± 10% +28.7% 855.00 ± 17% interrupts.CPU41.NMI:Non-maskable_interrupts 664.50 ± 10% +28.7% 855.00 ± 17% interrupts.CPU41.PMI:Performance_monitoring_interrupts 697.75 ± 16% +31.4% 917.00 ± 12% interrupts.CPU42.RES:Rescheduling_interrupts 775.00 ± 5% -16.1% 650.50 ± 2% interrupts.CPU45.RES:Rescheduling_interrupts 690.75 ± 4% +34.4% 928.50 ± 20% interrupts.CPU48.NMI:Non-maskable_interrupts 690.75 ± 4% +34.4% 928.50 ± 20% interrupts.CPU48.PMI:Performance_monitoring_interrupts 761.50 ± 5% +27.1% 967.50 ± 20% interrupts.CPU49.NMI:Non-maskable_interrupts 761.50 ± 5% +27.1% 967.50 ± 20% interrupts.CPU49.PMI:Performance_monitoring_interrupts 643.50 ± 3% +32.6% 853.00 ± 22% interrupts.CPU5.NMI:Non-maskable_interrupts 643.50 ± 3% +32.6% 853.00 ± 22% interrupts.CPU5.PMI:Performance_monitoring_interrupts 835.75 ± 14% +310.5% 3430 ± 67% interrupts.CPU55.RES:Rescheduling_interrupts 636.75 ± 5% +34.1% 854.00 ± 19% interrupts.CPU6.NMI:Non-maskable_interrupts 636.75 ± 5% +34.1% 854.00 ± 19% interrupts.CPU6.PMI:Performance_monitoring_interrupts 857.75 ± 30% +45.2% 1245 ± 15% interrupts.CPU61.NMI:Non-maskable_interrupts 857.75 ± 30% +45.2% 1245 ± 15% interrupts.CPU61.PMI:Performance_monitoring_interrupts 1333 ± 49% -46.3% 716.50 interrupts.CPU62.RES:Rescheduling_interrupts 836.50 ± 32% +43.5% 1200 ± 8% interrupts.CPU63.NMI:Non-maskable_interrupts 836.50 ± 32% +43.5% 1200 ± 8% interrupts.CPU63.PMI:Performance_monitoring_interrupts 1363 ± 39% -47.1% 721.00 ± 6% interrupts.CPU63.RES:Rescheduling_interrupts 842.00 ± 34% +46.6% 1234 ± 10% interrupts.CPU65.NMI:Non-maskable_interrupts 842.00 ± 34% +46.6% 1234 ± 10% interrupts.CPU65.PMI:Performance_monitoring_interrupts 813.75 ± 14% -22.6% 629.50 interrupts.CPU66.RES:Rescheduling_interrupts 65.25 ± 71% -72.4% 18.00 ± 38% interrupts.CPU66.TLB:TLB_shootdowns 646.25 ± 4% +28.7% 831.50 ± 17% interrupts.CPU7.NMI:Non-maskable_interrupts 646.25 ± 4% +28.7% 831.50 ± 17% interrupts.CPU7.PMI:Performance_monitoring_interrupts 816.25 ± 13% -23.6% 623.50 ± 4% interrupts.CPU72.RES:Rescheduling_interrupts 963.25 ± 28% -33.1% 644.00 ± 9% interrupts.CPU76.RES:Rescheduling_interrupts 59.50 ± 53% -65.5% 20.50 ± 31% interrupts.CPU77.TLB:TLB_shootdowns 856.00 ± 10% -24.7% 644.50 ± 9% interrupts.CPU78.RES:Rescheduling_interrupts 639.50 ± 5% +31.4% 840.50 ± 19% interrupts.CPU8.NMI:Non-maskable_interrupts 639.50 ± 5% +31.4% 840.50 ± 19% interrupts.CPU8.PMI:Performance_monitoring_interrupts 771.00 ± 4% +91.1% 1473 ± 43% interrupts.CPU80.NMI:Non-maskable_interrupts 771.00 ± 4% +91.1% 1473 ± 43% interrupts.CPU80.PMI:Performance_monitoring_interrupts 767.50 ± 6% +20.7% 926.50 ± 11% interrupts.CPU81.NMI:Non-maskable_interrupts 767.50 ± 6% +20.7% 926.50 ± 11% interrupts.CPU81.PMI:Performance_monitoring_interrupts 779.25 ± 4% +14.9% 895.00 ± 12% interrupts.CPU83.NMI:Non-maskable_interrupts 779.25 ± 4% +14.9% 895.00 ± 12% interrupts.CPU83.PMI:Performance_monitoring_interrupts 114.25 ± 96% -83.8% 18.50 ± 2% interrupts.CPU84.TLB:TLB_shootdowns 915.75 ± 24% +48.9% 1363 ± 21% interrupts.CPU87.NMI:Non-maskable_interrupts 915.75 ± 24% +48.9% 1363 ± 21% interrupts.CPU87.PMI:Performance_monitoring_interrupts 720.50 ± 19% +17.8% 849.00 ± 4% interrupts.CPU87.RES:Rescheduling_interrupts 952.25 ± 30% +30.7% 1245 ± 22% interrupts.CPU88.NMI:Non-maskable_interrupts 952.25 ± 30% +30.7% 1245 ± 22% interrupts.CPU88.PMI:Performance_monitoring_interrupts 947.25 ± 28% +33.5% 1264 ± 20% interrupts.CPU89.NMI:Non-maskable_interrupts 947.25 ± 28% +33.5% 1264 ± 20% interrupts.CPU89.PMI:Performance_monitoring_interrupts 647.75 ± 7% +28.5% 832.50 ± 19% interrupts.CPU9.NMI:Non-maskable_interrupts 647.75 ± 7% +28.5% 832.50 ± 19% interrupts.CPU9.PMI:Performance_monitoring_interrupts 948.50 ± 28% +37.0% 1299 ± 21% interrupts.CPU90.NMI:Non-maskable_interrupts 948.50 ± 28% +37.0% 1299 ± 21% interrupts.CPU90.PMI:Performance_monitoring_interrupts 933.00 ± 31% +39.4% 1301 ± 22% interrupts.CPU91.NMI:Non-maskable_interrupts 933.00 ± 31% +39.4% 1301 ± 22% interrupts.CPU91.PMI:Performance_monitoring_interrupts 914.75 ± 26% +103.3% 1859 ± 21% interrupts.CPU92.NMI:Non-maskable_interrupts 914.75 ± 26% +103.3% 1859 ± 21% interrupts.CPU92.PMI:Performance_monitoring_interrupts 927.00 ± 30% +105.9% 1908 ± 21% interrupts.CPU93.NMI:Non-maskable_interrupts 927.00 ± 30% +105.9% 1908 ± 21% interrupts.CPU93.PMI:Performance_monitoring_interrupts 1009 ± 29% +91.5% 1933 ± 21% interrupts.CPU94.NMI:Non-maskable_interrupts 1009 ± 29% +91.5% 1933 ± 21% interrupts.CPU94.PMI:Performance_monitoring_interrupts 1114 ± 30% +68.9% 1882 ± 18% interrupts.CPU95.NMI:Non-maskable_interrupts 1114 ± 30% +68.9% 1882 ± 18% interrupts.CPU95.PMI:Performance_monitoring_interrupts 1124 ± 31% +70.6% 1918 ± 16% interrupts.CPU96.NMI:Non-maskable_interrupts 1124 ± 31% +70.6% 1918 ± 16% interrupts.CPU96.PMI:Performance_monitoring_interrupts 1133 ± 36% +66.8% 1889 ± 13% interrupts.CPU97.NMI:Non-maskable_interrupts 1133 ± 36% +66.8% 1889 ± 13% interrupts.CPU97.PMI:Performance_monitoring_interrupts 3962 ±138% -85.8% 561.50 ± 24% interrupts.CPU99.RES:Rescheduling_interrupts *************************************************************************************************** lkp-hsw-ep2: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/300s/lkp-hsw-ep2/alltests/reaim/0x43 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:2 dmesg.WARNING:at_ip__slab_free/0x 2:4 -37% 0:2 perf-profile.children.cycles-pp.error_entry 1:4 -30% 0:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 160.75 ± 7% +23.3% 198.29 ± 16% reaim.child_systime 506927 -1.7% 498446 reaim.max_jobs_per_min 7931060 -3.2% 7678799 reaim.time.involuntary_context_switches 3107 ± 5% +18.1% 3668 ± 13% reaim.time.system_time 7217664 ± 6% +11.2% 8027136 meminfo.DirectMap2M 16.52 ± 5% +2.6 19.11 ± 10% mpstat.cpu.all.sys% 14534 ± 4% -6.2% 13628 proc-vmstat.nr_shmem 2809 ± 11% +35.4% 3805 slabinfo.eventpoll_pwq.active_objs 2809 ± 11% +35.4% 3805 slabinfo.eventpoll_pwq.num_objs 39477 ± 4% -9.9% 35575 softirqs.CPU21.NET_RX 39035 ± 2% -9.3% 35393 softirqs.CPU29.NET_RX 41045 ± 7% -11.1% 36472 softirqs.CPU4.NET_RX 38754 ± 3% -9.8% 34966 softirqs.CPU64.NET_RX 1.498e+10 +1.7% 1.523e+10 perf-stat.i.branch-instructions 2.23 ± 2% +0.1 2.34 ± 2% perf-stat.i.cache-miss-rate% 36.48 +0.8 37.33 perf-stat.i.node-store-miss-rate% 35.22 +0.5 35.71 perf-stat.overall.node-store-miss-rate% 16774806 +2.7% 17222201 ± 2% perf-stat.overall.path-length 1.493e+10 +1.7% 1.518e+10 perf-stat.ps.branch-instructions 1777 ± 19% +33.1% 2366 ± 5% sched_debug.cfs_rq:/.load.min 0.43 ± 16% +53.8% 0.67 sched_debug.cfs_rq:/.nr_running.min 0.17 ± 6% -32.9% 0.12 ± 12% sched_debug.cfs_rq:/.nr_running.stddev 3.35 ± 27% -85.7% 0.48 ± 50% sched_debug.cfs_rq:/.removed.util_avg.avg 125.62 ± 10% -81.3% 23.50 ± 46% sched_debug.cfs_rq:/.removed.util_avg.max 18.92 ± 19% -84.7% 2.89 ± 46% sched_debug.cfs_rq:/.removed.util_avg.stddev 0.58 ± 46% +59.4% 0.92 ± 9% sched_debug.cfs_rq:/.runnable_load_avg.min 1088 ± 21% +50.8% 1641 ± 11% sched_debug.cfs_rq:/.runnable_weight.min -92558 -305.9% 190556 ± 5% sched_debug.cfs_rq:/.spread0.avg 511049 ± 44% +69.5% 866184 ± 19% sched_debug.cfs_rq:/.spread0.max 600188 ± 6% +13.4% 680581 ± 6% sched_debug.cpu.avg_idle.avg 28.76 ± 20% +29.4% 37.21 ± 12% sched_debug.cpu.clock.stddev 28.76 ± 20% +29.4% 37.21 ± 12% sched_debug.cpu.clock_task.stddev 2705 ±100% +314.4% 11209 ± 39% sched_debug.cpu.curr->pid.min 9054 ± 9% -21.8% 7084 ± 8% sched_debug.cpu.curr->pid.stddev 19617 ± 14% -25.3% 14660 ± 13% sched_debug.cpu.load.avg 534995 ± 35% -62.1% 202918 ± 77% sched_debug.cpu.load.max 1777 ± 19% +33.1% 2366 ± 5% sched_debug.cpu.load.min 65254 ± 31% -59.2% 26605 ± 73% sched_debug.cpu.load.stddev 0.43 ± 16% +53.8% 0.67 sched_debug.cpu.nr_running.min 278.68 ± 3% +13.9% 317.50 sched_debug.cpu.nr_uninterruptible.max 3.18 ± 18% -1.6 1.56 ± 20% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit 1.92 ± 49% -1.3 0.58 ± 11% perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput 2.04 ± 34% -1.3 0.76 ± 3% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit 5.01 ± 21% -2.5 2.53 ± 6% perf-profile.children.cycles-pp.down_write 4.24 ± 28% -2.3 1.96 ± 5% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 4.70 ± 14% -2.2 2.46 ± 23% perf-profile.children.cycles-pp.free_pgtables 3.58 ± 34% -2.0 1.55 ± 12% perf-profile.children.cycles-pp.osq_lock 2.96 ± 27% -1.8 1.20 ± 4% perf-profile.children.cycles-pp.unlink_file_vma 0.78 ± 12% -0.3 0.45 ± 25% perf-profile.children.cycles-pp.vma_link 0.18 ± 13% -0.1 0.11 ± 27% perf-profile.children.cycles-pp.rwsem_wake 0.17 ± 15% -0.1 0.10 ± 15% perf-profile.children.cycles-pp.queued_read_lock_slowpath 0.08 ± 23% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__alloc_fd 0.07 ± 11% -0.0 0.03 ±100% perf-profile.children.cycles-pp.sched_clock 0.14 ± 8% -0.0 0.09 ± 22% perf-profile.children.cycles-pp.wake_up_q 0.09 ± 14% -0.0 0.05 perf-profile.children.cycles-pp.sched_clock_cpu 0.01 ±173% +0.1 0.08 ± 41% perf-profile.children.cycles-pp.memcpy_toio 0.01 ±173% +0.1 0.08 ± 41% perf-profile.children.cycles-pp.fb_flashcursor 0.01 ±173% +0.1 0.08 ± 41% perf-profile.children.cycles-pp.bit_cursor 0.01 ±173% +0.1 0.08 ± 41% perf-profile.children.cycles-pp.soft_cursor 0.01 ±173% +0.1 0.08 ± 41% perf-profile.children.cycles-pp.mga_dirty_update 0.04 ±173% +0.2 0.28 ± 7% perf-profile.children.cycles-pp.start_kernel 3.54 ± 34% -2.0 1.54 ± 12% perf-profile.self.cycles-pp.osq_lock 0.27 ± 16% -0.1 0.16 ± 31% perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.07 ± 5% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.vfprintf 0.01 ±173% +0.1 0.08 ± 41% perf-profile.self.cycles-pp.memcpy_toio 178.25 ± 14% +21.7% 217.00 ± 15% interrupts.49:PCI-MSI.1572878-edge.eth0-TxRx-14 151.50 +7.9% 163.50 ± 2% interrupts.81:PCI-MSI.1572910-edge.eth0-TxRx-46 824.75 ±162% +603.2% 5799 ± 3% interrupts.CPU0.NMI:Non-maskable_interrupts 824.75 ±162% +603.2% 5799 ± 3% interrupts.CPU0.PMI:Performance_monitoring_interrupts 38973 ± 4% -6.1% 36586 interrupts.CPU10.RES:Rescheduling_interrupts 178.25 ± 14% +21.7% 217.00 ± 15% interrupts.CPU14.49:PCI-MSI.1572878-edge.eth0-TxRx-14 4727 ± 30% -70.4% 1397 ± 99% interrupts.CPU17.NMI:Non-maskable_interrupts 4727 ± 30% -70.4% 1397 ± 99% interrupts.CPU17.PMI:Performance_monitoring_interrupts 163.50 ± 14% +28.7% 210.50 ± 6% interrupts.CPU21.TLB:TLB_shootdowns 1151 ±100% +276.3% 4331 ± 28% interrupts.CPU31.NMI:Non-maskable_interrupts 1151 ±100% +276.3% 4331 ± 28% interrupts.CPU31.PMI:Performance_monitoring_interrupts 4371 ± 25% -99.3% 29.00 ± 96% interrupts.CPU32.NMI:Non-maskable_interrupts 4371 ± 25% -99.3% 29.00 ± 96% interrupts.CPU32.PMI:Performance_monitoring_interrupts 2828 ± 5% +9.9% 3110 ± 2% interrupts.CPU33.CAL:Function_call_interrupts 173.00 ± 8% -22.5% 134.00 ± 2% interrupts.CPU33.TLB:TLB_shootdowns 5807 ± 16% -76.3% 1378 ± 99% interrupts.CPU38.NMI:Non-maskable_interrupts 5807 ± 16% -76.3% 1378 ± 99% interrupts.CPU38.PMI:Performance_monitoring_interrupts 206.25 ± 8% -19.5% 166.00 ± 7% interrupts.CPU40.TLB:TLB_shootdowns 151.50 +7.9% 163.50 ± 2% interrupts.CPU46.81:PCI-MSI.1572910-edge.eth0-TxRx-46 166.50 ± 5% +18.3% 197.00 ± 6% interrupts.CPU5.TLB:TLB_shootdowns 5546 ± 25% -47.7% 2902 ± 4% interrupts.CPU50.NMI:Non-maskable_interrupts 5546 ± 25% -47.7% 2902 ± 4% interrupts.CPU50.PMI:Performance_monitoring_interrupts 2270 ± 53% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 2270 ± 53% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 142.00 ± 13% +33.8% 190.00 ± 11% interrupts.CPU56.TLB:TLB_shootdowns 38592 ± 2% +5.0% 40532 ± 3% interrupts.CPU58.RES:Rescheduling_interrupts 212.75 ± 26% -31.6% 145.50 ± 7% interrupts.CPU59.TLB:TLB_shootdowns 2086 ± 60% +180.0% 5841 ± 4% interrupts.CPU60.NMI:Non-maskable_interrupts 2086 ± 60% +180.0% 5841 ± 4% interrupts.CPU60.PMI:Performance_monitoring_interrupts 26.25 ±128% +6105.7% 1629 ± 89% interrupts.CPU61.NMI:Non-maskable_interrupts 26.25 ±128% +6105.7% 1629 ± 89% interrupts.CPU61.PMI:Performance_monitoring_interrupts 4792 ± 35% -39.1% 2918 ± 5% interrupts.CPU63.NMI:Non-maskable_interrupts 4792 ± 35% -39.1% 2918 ± 5% interrupts.CPU63.PMI:Performance_monitoring_interrupts 168.00 ± 16% -23.8% 128.00 ± 5% interrupts.CPU66.TLB:TLB_shootdowns 3936 ± 32% -83.3% 655.50 ±100% interrupts.CPU67.NMI:Non-maskable_interrupts 3936 ± 32% -83.3% 655.50 ±100% interrupts.CPU67.PMI:Performance_monitoring_interrupts 2560 ± 85% +124.6% 5751 ± 5% interrupts.CPU68.NMI:Non-maskable_interrupts 2560 ± 85% +124.6% 5751 ± 5% interrupts.CPU68.PMI:Performance_monitoring_interrupts 3693 ± 21% +58.6% 5857 ± 4% interrupts.CPU70.NMI:Non-maskable_interrupts 3693 ± 21% +58.6% 5857 ± 4% interrupts.CPU70.PMI:Performance_monitoring_interrupts 112.00 ± 7% -8.9% 102.00 interrupts.IWI:IRQ_work_interrupts *************************************************************************************************** lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/900t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/disk_cp/reaim/0x42e commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -15% 0:2 perf-profile.children.cycles-pp.error_entry 0:4 -11% 0:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 1116 -2.7% 1087 reaim.child_systime 173.74 +1.0% 175.47 reaim.child_utime 2468 -1.7% 2425 reaim.time.percent_of_cpu_this_job_got 6702 -2.7% 6524 reaim.time.system_time 1042 +1.0% 1052 reaim.time.user_time 17750445 +1.1% 17946505 reaim.time.voluntary_context_switches 3128 ± 16% +25.7% 3930 ± 2% numa-vmstat.node1.nr_mapped 108987 ± 3% +29.1% 140714 ± 21% softirqs.CPU28.TIMER 47.75 +2.6% 49.00 vmstat.cpu.id 8.151e+08 ± 27% +32.2% 1.078e+09 cpuidle.C1E.time 5709370 ± 25% +25.1% 7144757 cpuidle.C1E.usage 1294 ± 5% +13.7% 1471 ± 8% slabinfo.pool_workqueue.active_objs 1308 ± 4% +12.5% 1472 ± 8% slabinfo.pool_workqueue.num_objs 1586 -1.8% 1558 turbostat.Avg_MHz 5709242 ± 25% +25.1% 7144385 turbostat.C1E 5.38 ± 27% +1.8 7.13 turbostat.C1E% 0.01 ±110% +233.3% 0.03 ± 19% turbostat.Pkg%pc3 5342 ±173% +595.0% 37130 ± 28% sched_debug.cfs_rq:/.MIN_vruntime.avg 256451 ±173% +595.0% 1782246 ± 28% sched_debug.cfs_rq:/.MIN_vruntime.max 36628 ±173% +595.0% 254551 ± 28% sched_debug.cfs_rq:/.MIN_vruntime.stddev 477.71 ± 29% +214.6% 1503 ± 57% sched_debug.cfs_rq:/.load_avg.max 89.14 ± 27% +160.7% 232.44 ± 54% sched_debug.cfs_rq:/.load_avg.stddev 5342 ±173% +595.0% 37130 ± 28% sched_debug.cfs_rq:/.max_vruntime.avg 256451 ±173% +595.0% 1782246 ± 28% sched_debug.cfs_rq:/.max_vruntime.max 36628 ±173% +595.0% 254551 ± 28% sched_debug.cfs_rq:/.max_vruntime.stddev 1.00 +16.7% 1.17 sched_debug.cfs_rq:/.nr_running.max 33.74 ± 6% -16.1% 28.32 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.avg 11.40 +4.7% 11.93 perf-stat.i.MPKI 7.907e+09 -2.3% 7.726e+09 perf-stat.i.branch-instructions 1.74 +0.1 1.83 perf-stat.i.branch-miss-rate% 4.285e+08 +1.2% 4.335e+08 perf-stat.i.cache-references 139218 +1.1% 140680 perf-stat.i.context-switches 7.603e+10 -1.9% 7.457e+10 perf-stat.i.cpu-cycles 925.05 -2.7% 900.07 perf-stat.i.cycles-between-cache-misses 1.161e+10 -2.5% 1.132e+10 perf-stat.i.dTLB-loads 10864114 -1.1% 10743610 perf-stat.i.iTLB-load-misses 1201431 +4.7% 1258341 perf-stat.i.iTLB-loads 3.926e+10 -2.4% 3.832e+10 perf-stat.i.instructions 10.91 +3.6% 11.31 perf-stat.overall.MPKI 1.63 +0.1 1.68 perf-stat.overall.branch-miss-rate% 921.98 -2.6% 898.33 perf-stat.overall.cycles-between-cache-misses 22791180 -2.5% 22226281 perf-stat.overall.path-length 7.882e+09 -2.3% 7.701e+09 perf-stat.ps.branch-instructions 4.271e+08 +1.2% 4.321e+08 perf-stat.ps.cache-references 138768 +1.1% 140229 perf-stat.ps.context-switches 7.578e+10 -1.9% 7.434e+10 perf-stat.ps.cpu-cycles 1.157e+10 -2.5% 1.129e+10 perf-stat.ps.dTLB-loads 10829073 -1.1% 10709146 perf-stat.ps.iTLB-load-misses 1197556 +4.7% 1254304 perf-stat.ps.iTLB-loads 3.914e+10 -2.4% 3.82e+10 perf-stat.ps.instructions 1.231e+13 -2.5% 1.2e+13 perf-stat.total.instructions 925085 ± 3% +5.5% 976146 interrupts.CAL:Function_call_interrupts 11296 ± 4% +9.2% 12340 ± 3% interrupts.CPU0.RES:Rescheduling_interrupts 3692 ± 32% -100.0% 0.50 ±100% interrupts.CPU10.NMI:Non-maskable_interrupts 3692 ± 32% -100.0% 0.50 ±100% interrupts.CPU10.PMI:Performance_monitoring_interrupts 3699 ± 32% -96.4% 132.00 ± 64% interrupts.CPU20.NMI:Non-maskable_interrupts 3699 ± 32% -96.4% 132.00 ± 64% interrupts.CPU20.PMI:Performance_monitoring_interrupts 635.25 ±163% -99.9% 0.50 ±100% interrupts.CPU22.NMI:Non-maskable_interrupts 635.25 ±163% -99.9% 0.50 ±100% interrupts.CPU22.PMI:Performance_monitoring_interrupts 19144 ± 2% +9.9% 21036 interrupts.CPU23.CAL:Function_call_interrupts 706.75 ±143% +420.9% 3681 ± 31% interrupts.CPU23.NMI:Non-maskable_interrupts 706.75 ±143% +420.9% 3681 ± 31% interrupts.CPU23.PMI:Performance_monitoring_interrupts 16166 +10.2% 17809 interrupts.CPU23.TLB:TLB_shootdowns 16484 ± 5% +5.7% 17432 ± 4% interrupts.CPU25.TLB:TLB_shootdowns 654.75 ±157% +464.7% 3697 ± 30% interrupts.CPU27.NMI:Non-maskable_interrupts 654.75 ±157% +464.7% 3697 ± 30% interrupts.CPU27.PMI:Performance_monitoring_interrupts 15719 ± 4% +7.8% 16941 ± 5% interrupts.CPU29.TLB:TLB_shootdowns 191.75 ±152% +1195.2% 2483 ± 4% interrupts.CPU30.NMI:Non-maskable_interrupts 191.75 ±152% +1195.2% 2483 ± 4% interrupts.CPU30.PMI:Performance_monitoring_interrupts 2.50 ± 82% +1.5e+05% 3665 ± 30% interrupts.CPU34.NMI:Non-maskable_interrupts 2.50 ± 82% +1.5e+05% 3665 ± 30% interrupts.CPU34.PMI:Performance_monitoring_interrupts 10577 ± 2% +5.7% 11181 ± 5% interrupts.CPU34.RES:Rescheduling_interrupts 671.50 ±151% +449.5% 3690 ± 30% interrupts.CPU44.NMI:Non-maskable_interrupts 671.50 ±151% +449.5% 3690 ± 30% interrupts.CPU44.PMI:Performance_monitoring_interrupts 10379 ± 3% +8.5% 11258 ± 3% interrupts.CPU44.RES:Rescheduling_interrupts 16081 ± 2% +11.2% 17882 ± 2% interrupts.CPU47.TLB:TLB_shootdowns 18858 ± 5% +7.0% 20175 ± 4% interrupts.CPU5.CAL:Function_call_interrupts 15595 ± 6% +7.9% 16826 ± 5% interrupts.CPU5.TLB:TLB_shootdowns 20.85 -1.1 19.71 ± 2% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables 14.50 -0.9 13.56 ± 2% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64 14.52 -0.9 13.59 ± 2% perf-profile.calltrace.cycles-pp.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.03 -0.8 11.20 perf-profile.calltrace.cycles-pp.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler 12.13 -0.8 11.31 perf-profile.calltrace.cycles-pp.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve 12.04 -0.8 11.22 perf-profile.calltrace.cycles-pp.mmput.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file 10.75 -0.7 10.01 perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary 9.92 -0.7 9.21 perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec 24.45 -1.2 23.26 ± 3% perf-profile.children.cycles-pp.free_pgtables 22.85 -1.1 21.73 ± 2% perf-profile.children.cycles-pp.unlink_file_vma 15.96 -1.0 15.00 ± 2% perf-profile.children.cycles-pp.__do_execve_file 16.03 -0.9 15.09 ± 2% perf-profile.children.cycles-pp.__x64_sys_execve 14.52 -0.9 13.58 ± 2% perf-profile.children.cycles-pp.search_binary_handler 14.50 -0.9 13.56 ± 2% perf-profile.children.cycles-pp.load_elf_binary 12.14 -0.8 11.31 perf-profile.children.cycles-pp.flush_old_exec 0.30 ± 2% -0.0 0.27 ± 3% perf-profile.children.cycles-pp.perf_iterate_sb 0.17 ± 5% -0.0 0.15 ± 3% perf-profile.children.cycles-pp.unwind_next_frame 0.10 -0.0 0.08 ± 6% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.14 ± 6% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.perf_event_mmap_output 0.08 ± 10% -0.0 0.06 perf-profile.children.cycles-pp.move_queued_task 0.05 ± 9% +0.0 0.07 perf-profile.children.cycles-pp.__x64_sys_close 0.06 ± 7% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.fput_many 0.04 ± 57% +0.0 0.06 perf-profile.children.cycles-pp.lapic_next_deadline 0.54 ± 4% +0.0 0.56 ± 2% perf-profile.children.cycles-pp.alloc_set_pte 0.15 ± 8% +0.0 0.18 ± 5% perf-profile.children.cycles-pp.__d_lookup_rcu 0.20 ± 10% +0.0 0.23 ± 2% perf-profile.children.cycles-pp.wait_consider_task 0.10 ± 21% +0.0 0.14 ± 14% perf-profile.children.cycles-pp.ksys_write 0.09 ± 23% +0.0 0.13 ± 15% perf-profile.children.cycles-pp.new_sync_write 0.06 ± 68% +0.0 0.11 ± 23% perf-profile.children.cycles-pp.__waitpid 0.24 ± 6% +0.0 0.28 ± 3% perf-profile.children.cycles-pp.lookup_fast 0.57 ± 6% +0.0 0.61 ± 3% perf-profile.children.cycles-pp.anon_vma_fork 0.00 +0.1 0.05 perf-profile.children.cycles-pp.cp_new_stat 0.10 ± 4% +0.1 0.16 ± 25% perf-profile.children.cycles-pp.ktime_get 0.10 ± 5% +0.1 0.16 ± 31% perf-profile.children.cycles-pp.clockevents_program_event 0.68 ± 4% +0.1 0.74 ± 2% perf-profile.children.cycles-pp.schedule 1.11 ± 6% +0.1 1.18 ± 2% perf-profile.children.cycles-pp.__schedule 0.32 ± 7% +0.1 0.39 ± 12% perf-profile.children.cycles-pp.__slab_free 0.09 ± 13% -0.0 0.07 perf-profile.self.cycles-pp.cpuidle_enter_state 0.07 ± 6% +0.0 0.08 perf-profile.self.cycles-pp.___slab_alloc 0.09 ± 4% +0.0 0.10 perf-profile.self.cycles-pp.avc_has_perm_noaudit 0.17 ± 4% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.alloc_set_pte 0.05 +0.0 0.07 ± 7% perf-profile.self.cycles-pp._vm_normal_page 0.21 ± 6% +0.0 0.24 ± 4% perf-profile.self.cycles-pp.anon_vma_interval_tree_insert 0.34 ± 6% +0.0 0.38 perf-profile.self.cycles-pp.__handle_mm_fault 0.15 ± 7% +0.0 0.18 ± 5% perf-profile.self.cycles-pp.__d_lookup_rcu 0.00 +0.1 0.05 perf-profile.self.cycles-pp.path_openat 0.00 +0.1 0.05 perf-profile.self.cycles-pp.enqueue_task_fair *************************************************************************************************** lkp-hsw-ep2: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/300s/lkp-hsw-ep2/new_dbase/reaim/0x43 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :2 dmesg.WARNING:at_ip_perf_event_mmap_output/0x 1:4 -17% 0:2 perf-profile.children.cycles-pp.error_entry %stddev %change %stddev \ | \ 10584147 +1.9% 10785891 perf-stat.i.node-load-misses 10542342 +1.9% 10743011 perf-stat.ps.node-load-misses 1393 ± 7% -19.5% 1122 ± 2% slabinfo.task_group.active_objs 1393 ± 7% -19.5% 1122 ± 2% slabinfo.task_group.num_objs 34065 +0.9% 34375 proc-vmstat.nr_kernel_stack 13974 +2.7% 14349 proc-vmstat.nr_page_table_pages 3127 ± 92% +91.2% 5980 ± 34% proc-vmstat.numa_hint_faults 122943 ± 12% +12.9% 138853 ± 12% numa-vmstat.node0.nr_active_anon 13963 ± 48% +34.6% 18801 ± 41% numa-vmstat.node0.nr_kernel_stack 4857 ± 76% +56.6% 7609 ± 57% numa-vmstat.node0.nr_page_table_pages 122943 ± 12% +12.9% 138852 ± 12% numa-vmstat.node0.nr_zone_active_anon 494560 ± 12% +12.6% 557012 ± 12% numa-meminfo.node0.Active 494224 ± 12% +12.6% 556676 ± 12% numa-meminfo.node0.Active(anon) 474941 ± 14% +13.0% 536795 ± 14% numa-meminfo.node0.AnonPages 13949 ± 49% +32.9% 18544 ± 40% numa-meminfo.node0.KernelStack 1479782 ± 9% +9.4% 1618507 ± 8% numa-meminfo.node0.MemUsed 19602 ± 76% +52.1% 29814 ± 56% numa-meminfo.node0.PageTables 117010 +24.4% 145600 ± 14% softirqs.CPU10.TIMER 13137 ± 2% -12.0% 11564 softirqs.CPU18.SCHED 82501 ± 5% +8.2% 89260 ± 8% softirqs.CPU29.RCU 117180 +12.4% 131653 ± 5% softirqs.CPU35.TIMER 12029 ± 5% -6.1% 11294 softirqs.CPU37.SCHED 117019 +10.1% 128814 ± 5% softirqs.CPU46.TIMER 74749 ± 6% +9.2% 81658 ± 6% softirqs.CPU60.RCU 72124 ± 6% +10.9% 80008 ± 9% softirqs.CPU65.RCU 117036 ± 2% +20.5% 141060 ± 13% softirqs.CPU71.TIMER 0.10 ± 16% +37.6% 0.14 sched_debug.cfs_rq:/.nr_running.stddev 11.66 ± 6% -17.7% 9.60 ± 8% sched_debug.cfs_rq:/.runnable_load_avg.avg 90.96 ± 19% +18.5% 107.82 ± 9% sched_debug.cpu.clock.stddev 90.96 ± 19% +18.5% 107.82 ± 9% sched_debug.cpu.clock_task.stddev 16220 -13.9% 13969 sched_debug.cpu.nr_switches.stddev 321.04 ± 4% +39.0% 446.25 ± 13% sched_debug.cpu.nr_uninterruptible.max 142.97 +20.7% 172.63 ± 8% sched_debug.cpu.nr_uninterruptible.stddev 17028 ± 2% -9.8% 15355 sched_debug.cpu.sched_count.stddev 21177 ± 3% +12.2% 23770 sched_debug.cpu.ttwu_count.min 10062 ± 2% -11.7% 8882 sched_debug.cpu.ttwu_count.stddev 12442 +12.0% 13935 sched_debug.cpu.ttwu_local.min 5892 ± 3% -12.1% 5178 sched_debug.cpu.ttwu_local.stddev 169.25 ± 8% +344.9% 753.00 ± 75% interrupts.38:PCI-MSI.1572867-edge.eth0-TxRx-3 491.50 ± 72% -64.8% 173.00 interrupts.39:PCI-MSI.1572868-edge.eth0-TxRx-4 299.25 ± 58% -42.5% 172.00 interrupts.45:PCI-MSI.1572874-edge.eth0-TxRx-10 178.50 ± 7% +97.8% 353.00 ± 9% interrupts.48:PCI-MSI.1572877-edge.eth0-TxRx-13 210.75 ± 27% +72.0% 362.50 ± 31% interrupts.50:PCI-MSI.1572879-edge.eth0-TxRx-15 150.00 +7.3% 161.00 ± 6% interrupts.73:PCI-MSI.1572902-edge.eth0-TxRx-38 150.00 +6.7% 160.00 ± 5% interrupts.96:PCI-MSI.1572925-edge.eth0-TxRx-61 299.25 ± 58% -42.5% 172.00 interrupts.CPU10.45:PCI-MSI.1572874-edge.eth0-TxRx-10 178.50 ± 7% +97.8% 353.00 ± 9% interrupts.CPU13.48:PCI-MSI.1572877-edge.eth0-TxRx-13 2731 ± 8% +11.9% 3055 ± 2% interrupts.CPU13.CAL:Function_call_interrupts 210.75 ± 27% +72.0% 362.50 ± 31% interrupts.CPU15.50:PCI-MSI.1572879-edge.eth0-TxRx-15 2401 ±100% -100.0% 1.00 ±100% interrupts.CPU15.NMI:Non-maskable_interrupts 2401 ±100% -100.0% 1.00 ±100% interrupts.CPU15.PMI:Performance_monitoring_interrupts 138.50 ± 25% +26.4% 175.00 ± 6% interrupts.CPU17.TLB:TLB_shootdowns 4544 ± 57% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts 4544 ± 57% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts 169.25 ± 8% +344.9% 753.00 ± 75% interrupts.CPU3.38:PCI-MSI.1572867-edge.eth0-TxRx-3 825.50 ±152% +450.6% 4545 ± 32% interrupts.CPU36.NMI:Non-maskable_interrupts 825.50 ±152% +450.6% 4545 ± 32% interrupts.CPU36.PMI:Performance_monitoring_interrupts 150.00 +7.3% 161.00 ± 6% interrupts.CPU38.73:PCI-MSI.1572902-edge.eth0-TxRx-38 491.50 ± 72% -64.8% 173.00 interrupts.CPU4.39:PCI-MSI.1572868-edge.eth0-TxRx-4 176.50 ± 11% -24.9% 132.50 ± 16% interrupts.CPU4.TLB:TLB_shootdowns 4580 ± 33% -66.0% 1558 ± 95% interrupts.CPU44.NMI:Non-maskable_interrupts 4580 ± 33% -66.0% 1558 ± 95% interrupts.CPU44.PMI:Performance_monitoring_interrupts 2732 ± 80% -95.9% 111.50 ± 9% interrupts.CPU47.NMI:Non-maskable_interrupts 2732 ± 80% -95.9% 111.50 ± 9% interrupts.CPU47.PMI:Performance_monitoring_interrupts 144.75 ± 5% +10.5% 160.00 ± 4% interrupts.CPU56.TLB:TLB_shootdowns 1725 ±145% +188.6% 4981 ± 22% interrupts.CPU58.NMI:Non-maskable_interrupts 1725 ±145% +188.6% 4981 ± 22% interrupts.CPU58.PMI:Performance_monitoring_interrupts 145.25 ± 16% +30.8% 190.00 interrupts.CPU58.TLB:TLB_shootdowns 1515 ±173% +201.8% 4573 ± 33% interrupts.CPU6.NMI:Non-maskable_interrupts 1515 ±173% +201.8% 4573 ± 33% interrupts.CPU6.PMI:Performance_monitoring_interrupts 150.00 +6.7% 160.00 ± 5% interrupts.CPU61.96:PCI-MSI.1572925-edge.eth0-TxRx-61 2881 ± 4% +9.1% 3143 interrupts.CPU62.CAL:Function_call_interrupts 1622 ±159% +183.6% 4602 ± 33% interrupts.CPU62.NMI:Non-maskable_interrupts 1622 ±159% +183.6% 4602 ± 33% interrupts.CPU62.PMI:Performance_monitoring_interrupts 2449 ± 97% +150.0% 6124 interrupts.CPU68.NMI:Non-maskable_interrupts 2449 ± 97% +150.0% 6124 interrupts.CPU68.PMI:Performance_monitoring_interrupts 2846 ± 5% +10.1% 3133 interrupts.CPU7.CAL:Function_call_interrupts 3055 ± 70% +50.6% 4601 ± 32% interrupts.CPU7.NMI:Non-maskable_interrupts 3055 ± 70% +50.6% 4601 ± 32% interrupts.CPU7.PMI:Performance_monitoring_interrupts 33.94 ± 6% -3.0 30.98 perf-profile.calltrace.cycles-pp.sieve 9.37 ± 6% -0.9 8.44 perf-profile.calltrace.cycles-pp.string_rtns_1 3.44 ± 6% -0.3 3.10 perf-profile.calltrace.cycles-pp.brk 2.53 ± 8% -0.3 2.26 ± 3% perf-profile.calltrace.cycles-pp.page_test 2.72 ± 6% -0.3 2.46 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.brk 2.71 ± 6% -0.3 2.45 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 2.25 ± 6% -0.2 2.04 perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 1.55 ± 7% -0.2 1.38 ± 3% perf-profile.calltrace.cycles-pp.page_fault.page_test 1.55 ± 7% -0.2 1.38 ± 3% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.page_test 1.48 ± 7% -0.2 1.31 ± 4% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault.page_test 1.25 ± 7% -0.1 1.10 ± 4% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.page_test 0.89 ± 8% -0.1 0.79 perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 1.16 ± 5% -0.1 1.07 perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 0.77 ± 6% -0.1 0.69 perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk 0.78 ± 6% -0.1 0.70 perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 0.41 ± 57% +0.2 0.59 ± 2% perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 0.55 ± 57% +0.3 0.82 ± 2% perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode 3.17 ± 8% +0.3 3.45 ± 2% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.85 ± 21% +0.3 1.14 ± 9% perf-profile.calltrace.cycles-pp.__libc_fork 1.31 ± 18% +0.3 1.60 ± 4% perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.59 ± 59% +0.4 0.97 ± 11% perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 0.63 ± 59% +0.4 1.01 ± 11% perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 0.30 ±100% +0.4 0.69 ± 10% perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.63 ± 59% +0.4 1.02 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork 0.63 ± 59% +0.4 1.02 ± 11% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 0.13 ±173% +0.4 0.53 ± 4% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault 2.11 ± 17% +0.4 2.52 perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 2.44 ± 17% +0.5 2.91 ± 2% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 2.58 ± 16% +0.5 3.05 ± 2% perf-profile.calltrace.cycles-pp.page_fault 2.53 ± 17% +0.5 3.00 ± 2% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 1.58 ± 25% +0.5 2.09 ± 12% perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.66 ± 24% +0.5 2.17 ± 12% perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.33 ±105% +0.5 0.86 ± 2% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.vma_link.mmap_region.do_mmap 0.33 ±105% +0.5 0.87 ± 2% perf-profile.calltrace.cycles-pp.down_write.vma_link.mmap_region.do_mmap.vm_mmap_pgoff 1.23 ± 39% +0.6 1.79 ± 15% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit 0.35 ±104% +0.6 0.93 ± 2% perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 1.33 ± 37% +0.6 1.90 ± 14% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit 0.77 ± 62% +0.6 1.38 perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 0.82 ± 62% +0.6 1.43 ± 2% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.79 ± 62% +0.6 1.41 ± 2% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.61 ± 2% perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.81 ± 62% +0.6 1.43 ± 2% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.81 ± 60% +0.6 1.44 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.20 ±173% +0.6 0.82 perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.vma_link.mmap_region 1.14 ± 58% +0.7 1.79 perf-profile.calltrace.cycles-pp.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.15 ± 58% +0.7 1.80 perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.12 ± 27% +0.7 2.83 ± 8% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 2.12 ± 27% +0.7 2.83 ± 8% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 3.23 ± 29% +0.7 3.94 ± 5% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64 0.90 ± 74% +0.7 1.62 ± 37% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.dup_mm.copy_process 3.24 ± 29% +0.7 3.96 ± 5% perf-profile.calltrace.cycles-pp.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.38 ± 25% +0.7 3.12 ± 8% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.38 ± 25% +0.7 3.12 ± 7% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.38 ± 25% +0.7 3.12 ± 7% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.92 ± 74% +0.8 1.68 ± 36% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.dup_mm.copy_process._do_fork 0.93 ± 74% +0.8 1.69 ± 36% perf-profile.calltrace.cycles-pp.down_write.dup_mm.copy_process._do_fork.do_syscall_64 3.83 ± 7% +0.8 4.60 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64 3.02 ± 11% +0.8 3.79 ± 3% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 3.50 ± 9% +0.8 4.32 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 3.53 ± 9% +0.8 4.36 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 3.76 ± 8% +0.8 4.60 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 3.77 ± 8% +0.8 4.60 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 3.77 ± 8% +0.8 4.60 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 1.91 ± 35% +0.8 2.75 ± 13% perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.28 ± 60% +1.2 3.53 ± 13% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables 2.39 ± 60% +1.3 3.67 ± 13% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.exit_mmap 2.42 ± 60% +1.3 3.71 ± 13% perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput 8.91 ± 28% +3.3 12.18 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.93 ± 28% +3.3 12.20 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 33.97 ± 6% -3.0 31.00 perf-profile.children.cycles-pp.sieve 9.44 ± 6% -0.9 8.49 perf-profile.children.cycles-pp.string_rtns_1 3.47 ± 6% -0.3 3.12 perf-profile.children.cycles-pp.brk 2.87 ± 8% -0.3 2.57 ± 2% perf-profile.children.cycles-pp.page_test 0.16 ± 8% -0.0 0.13 perf-profile.children.cycles-pp.__libc_calloc 0.17 ± 2% -0.0 0.16 perf-profile.children.cycles-pp.free 0.07 ± 6% +0.0 0.08 perf-profile.children.cycles-pp.wait_consider_task 0.05 ± 8% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.dequeue_task_fair 0.06 ± 7% +0.0 0.07 perf-profile.children.cycles-pp.finish_task_switch 0.05 +0.0 0.07 ± 7% perf-profile.children.cycles-pp.up_read 0.10 ± 11% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.__slab_free 0.09 ± 9% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.__x64_sys_munmap 0.08 ± 14% +0.0 0.10 ± 5% perf-profile.children.cycles-pp.enqueue_entity 0.05 ± 8% +0.0 0.07 perf-profile.children.cycles-pp.do_dentry_open 0.12 ± 9% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.vmacache_find 0.19 ± 5% +0.0 0.21 perf-profile.children.cycles-pp.exit_to_usermode_loop 0.08 ± 11% +0.0 0.10 ± 5% perf-profile.children.cycles-pp.load_balance 0.07 ± 13% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.ttwu_do_activate 0.06 ± 11% +0.0 0.08 ± 12% perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate 0.32 ± 4% +0.0 0.34 perf-profile.children.cycles-pp.copy_p4d_range 0.09 ± 12% +0.0 0.11 ± 9% perf-profile.children.cycles-pp.wake_up_q 0.06 ± 7% +0.0 0.08 ± 12% perf-profile.children.cycles-pp.__tlb_remove_page_size 0.11 ± 13% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.wait4 0.21 ± 5% +0.0 0.24 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc 0.08 ± 10% +0.0 0.11 perf-profile.children.cycles-pp.activate_task 0.04 ± 58% +0.0 0.07 perf-profile.children.cycles-pp.__list_add_valid 0.27 ± 8% +0.0 0.30 ± 3% perf-profile.children.cycles-pp.page_remove_rmap 0.28 ± 5% +0.0 0.31 ± 6% perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay 0.15 ± 11% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.do_wait 0.30 ± 4% +0.0 0.33 perf-profile.children.cycles-pp.perf_iterate_sb 0.24 ± 11% +0.0 0.27 ± 3% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.11 ± 7% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 0.21 ± 10% +0.0 0.25 perf-profile.children.cycles-pp.anon_vma_fork 0.08 ± 10% +0.0 0.12 ± 25% perf-profile.children.cycles-pp.memcpy_erms 0.24 ± 7% +0.0 0.28 ± 5% perf-profile.children.cycles-pp.schedule 0.23 ± 13% +0.0 0.27 ± 5% perf-profile.children.cycles-pp.lru_add_drain 0.27 ± 8% +0.0 0.30 perf-profile.children.cycles-pp.scheduler_tick 0.19 ± 9% +0.0 0.23 ± 4% perf-profile.children.cycles-pp.anon_vma_clone 0.33 ± 7% +0.0 0.36 ± 4% perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.16 ± 13% +0.0 0.20 ± 4% perf-profile.children.cycles-pp.kernel_wait4 0.41 ± 7% +0.0 0.46 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.interrupt_entry 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.find_busiest_group 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.update_load_avg 0.17 ± 6% +0.0 0.21 ± 4% perf-profile.children.cycles-pp.try_to_wake_up 0.16 ± 13% +0.0 0.21 ± 7% perf-profile.children.cycles-pp.__do_sys_wait4 0.11 ± 29% +0.0 0.16 ± 9% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.00 +0.1 0.05 perf-profile.children.cycles-pp.update_sd_lb_stats 0.40 ± 6% +0.1 0.45 ± 3% perf-profile.children.cycles-pp.update_process_times 0.42 ± 6% +0.1 0.47 ± 4% perf-profile.children.cycles-pp.tick_sched_handle 0.51 ± 6% +0.1 0.56 ± 3% perf-profile.children.cycles-pp.do_sys_open 0.52 ± 6% +0.1 0.57 ± 2% perf-profile.children.cycles-pp.path_openat 0.52 ± 6% +0.1 0.58 ± 3% perf-profile.children.cycles-pp.do_filp_open 0.45 ± 5% +0.1 0.52 ± 3% perf-profile.children.cycles-pp.tick_sched_timer 0.55 ± 5% +0.1 0.62 ± 4% perf-profile.children.cycles-pp.perf_event_mmap 0.33 ± 7% +0.1 0.40 ± 4% perf-profile.children.cycles-pp.__schedule 0.32 ± 16% +0.1 0.40 ± 13% perf-profile.children.cycles-pp._raw_spin_lock 0.40 ± 15% +0.1 0.48 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.62 ± 3% +0.1 0.70 perf-profile.children.cycles-pp.__hrtimer_run_queues 0.59 ± 10% +0.1 0.67 ± 5% perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.80 ± 3% +0.1 0.91 perf-profile.children.cycles-pp.hrtimer_interrupt 0.42 ± 18% +0.1 0.53 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.42 ± 7% +0.1 1.53 perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.91 ± 10% +0.1 1.02 ± 3% perf-profile.children.cycles-pp.release_pages 0.82 ± 9% +0.1 0.95 ± 4% perf-profile.children.cycles-pp.filemap_map_pages 1.01 ± 8% +0.1 1.14 ± 4% perf-profile.children.cycles-pp.unmap_page_range 1.07 ± 8% +0.1 1.21 ± 3% perf-profile.children.cycles-pp.unmap_vmas 1.27 ± 7% +0.2 1.42 perf-profile.children.cycles-pp.do_brk_flags 1.57 ± 8% +0.2 1.73 ± 2% perf-profile.children.cycles-pp.tlb_flush_mmu 1.59 ± 8% +0.2 1.77 ± 2% perf-profile.children.cycles-pp.tlb_finish_mmu 1.19 ± 5% +0.2 1.37 ± 3% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 1.32 ± 3% +0.2 1.50 ± 2% perf-profile.children.cycles-pp.apic_timer_interrupt 1.77 ± 9% +0.2 1.99 ± 4% perf-profile.children.cycles-pp.unmap_region 2.11 ± 10% +0.3 2.38 ± 4% perf-profile.children.cycles-pp.__do_munmap 0.90 ± 21% +0.3 1.18 ± 10% perf-profile.children.cycles-pp.__libc_fork 3.90 ± 8% +0.3 4.23 perf-profile.children.cycles-pp.__handle_mm_fault 4.12 ± 7% +0.3 4.46 perf-profile.children.cycles-pp.handle_mm_fault 3.14 ± 8% +0.4 3.49 ± 3% perf-profile.children.cycles-pp.__x64_sys_brk 0.74 ± 36% +0.5 1.20 perf-profile.children.cycles-pp.vma_link 1.12 ± 29% +0.5 1.62 ± 2% perf-profile.children.cycles-pp.ksys_mmap_pgoff 1.22 ± 29% +0.6 1.79 perf-profile.children.cycles-pp.mmap_region 1.31 ± 28% +0.6 1.88 perf-profile.children.cycles-pp.do_mmap 1.36 ± 27% +0.6 1.93 perf-profile.children.cycles-pp.vm_mmap_pgoff 3.32 ± 24% +0.6 3.96 ± 5% perf-profile.children.cycles-pp.search_binary_handler 3.98 ± 21% +0.6 4.62 ± 4% perf-profile.children.cycles-pp.__do_execve_file 4.00 ± 21% +0.6 4.65 ± 4% perf-profile.children.cycles-pp.__x64_sys_execve 3.07 ± 11% +0.7 3.79 ± 3% perf-profile.children.cycles-pp.intel_idle 2.40 ± 25% +0.7 3.15 ± 7% perf-profile.children.cycles-pp.__x64_sys_exit_group 2.40 ± 25% +0.7 3.15 ± 7% perf-profile.children.cycles-pp.do_group_exit 2.40 ± 25% +0.8 3.15 ± 7% perf-profile.children.cycles-pp.do_exit 3.60 ± 8% +0.8 4.36 ± 2% perf-profile.children.cycles-pp.cpuidle_enter 3.60 ± 8% +0.8 4.36 ± 2% perf-profile.children.cycles-pp.cpuidle_enter_state 2.01 ± 28% +0.8 2.78 ± 13% perf-profile.children.cycles-pp.dup_mm 3.83 ± 7% +0.8 4.60 ± 3% perf-profile.children.cycles-pp.do_idle 3.83 ± 7% +0.8 4.60 ± 3% perf-profile.children.cycles-pp.secondary_startup_64 3.83 ± 7% +0.8 4.60 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 2.31 ± 25% +0.8 3.09 ± 12% perf-profile.children.cycles-pp.copy_process 2.43 ± 24% +0.8 3.22 ± 12% perf-profile.children.cycles-pp._do_fork 3.77 ± 8% +0.8 4.60 ± 3% perf-profile.children.cycles-pp.start_secondary 2.89 ± 39% +1.1 4.01 ± 11% perf-profile.children.cycles-pp.unlink_file_vma 3.24 ± 36% +1.1 4.38 ± 11% perf-profile.children.cycles-pp.free_pgtables 4.46 ± 28% +1.3 5.72 ± 8% perf-profile.children.cycles-pp.exit_mmap 4.46 ± 28% +1.3 5.73 ± 8% perf-profile.children.cycles-pp.mmput 5.02 ± 40% +2.2 7.23 ± 10% perf-profile.children.cycles-pp.osq_lock 5.33 ± 40% +2.4 7.70 ± 11% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 5.50 ± 39% +2.4 7.88 ± 10% perf-profile.children.cycles-pp.down_write 17.38 ± 16% +3.4 20.80 ± 3% perf-profile.children.cycles-pp.do_syscall_64 17.43 ± 16% +3.4 20.86 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 33.64 ± 6% -2.9 30.71 perf-profile.self.cycles-pp.sieve 9.29 ± 6% -0.9 8.35 perf-profile.self.cycles-pp.string_rtns_1 0.43 ± 9% -0.0 0.40 perf-profile.self.cycles-pp.page_test 0.14 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.free 0.09 ± 9% -0.0 0.07 perf-profile.self.cycles-pp.memcpy 0.08 +0.0 0.09 perf-profile.self.cycles-pp.alloc_set_pte 0.05 ± 9% +0.0 0.07 perf-profile.self.cycles-pp.native_write_msr 0.12 ± 9% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.vmacache_find 0.10 ± 11% +0.0 0.11 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.16 ± 11% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.copy_p4d_range 0.15 ± 5% +0.0 0.17 perf-profile.self.cycles-pp.perf_iterate_sb 0.04 ± 58% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate 0.23 ± 8% +0.0 0.26 perf-profile.self.cycles-pp.page_remove_rmap 0.18 ± 8% +0.0 0.21 ± 2% perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.04 ± 58% +0.0 0.07 perf-profile.self.cycles-pp.__list_add_valid 0.08 ± 26% +0.0 0.12 ± 13% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.22 ± 7% +0.0 0.26 ± 9% perf-profile.self.cycles-pp._raw_spin_lock 0.03 ±100% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.cpuidle_enter_state 0.41 ± 7% +0.0 0.46 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.32 ± 8% +0.0 0.36 perf-profile.self.cycles-pp.release_pages 0.35 ± 7% +0.0 0.39 ± 2% perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.interrupt_entry 0.56 ± 8% +0.1 0.62 perf-profile.self.cycles-pp.unmap_page_range 0.19 ± 36% +0.1 0.29 ± 12% perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.42 ± 18% +0.1 0.53 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 3.07 ± 11% +0.7 3.79 ± 3% perf-profile.self.cycles-pp.intel_idle 4.97 ± 41% +2.2 7.17 ± 11% perf-profile.self.cycles-pp.osq_lock *************************************************************************************************** lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/300t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/disk_wrt/reaim/0x42e commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:4 -4% 0:2 perf-profile.children.cycles-pp.error_entry 0:4 -3% 0:2 perf-profile.self.cycles-pp.error_entry *************************************************************************************************** lkp-bdw-ex2: 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/lkp-bdw-ex2/small-allocs/vm-scalability/0xb000036 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 2:4 -27% 1:2 perf-profile.calltrace.cycles-pp.error_entry.do_access 2:4 -32% 1:2 perf-profile.children.cycles-pp.error_entry 2:4 -26% 1:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 3852 -4.9% 3664 vm-scalability.time.maximum_resident_set_size 0.02 ±109% +0.0 0.05 ± 63% mpstat.cpu.all.soft% 57.25 -2.2% 56.00 turbostat.PkgTmp 53.13 +2.0% 54.21 boot-time.boot 8844 ± 2% +3.5% 9154 boot-time.idle 117451 ± 5% +14.1% 134069 ± 6% softirqs.CPU191.TIMER 20234 ± 8% +6.3% 21507 ± 7% softirqs.CPU88.RCU 615729 -1.5% 606471 proc-vmstat.nr_page_table_pages 24641 +1.4% 24974 proc-vmstat.nr_slab_reclaimable 1632391 -1.5% 1607656 proc-vmstat.nr_slab_unreclaimable 717417 ± 10% +64.0% 1176379 ± 3% numa-numastat.node2.local_node 743836 ± 9% +58.2% 1176412 ± 3% numa-numastat.node2.numa_hit 26418 ± 30% -99.9% 37.50 ± 44% numa-numastat.node2.other_node 15524 ± 72% +100.3% 31101 numa-numastat.node3.other_node 6.28 ± 8% +9.1% 6.86 ± 8% perf-stat.i.MPKI 0.45 ± 5% +0.0 0.50 ± 3% perf-stat.i.branch-miss-rate% 47984336 ± 4% +9.8% 52679218 ± 2% perf-stat.i.branch-misses 2.18e+08 ± 8% +9.3% 2.383e+08 ± 8% perf-stat.i.cache-references 4924680 ± 6% +7.9% 5312753 ± 7% perf-stat.i.iTLB-load-misses 491869 ± 5% +5.7% 520141 ± 4% perf-stat.i.node-loads 3.03 ± 5% +13.8% 3.45 ± 9% perf-stat.overall.MPKI 0.29 +0.0 0.32 perf-stat.overall.branch-miss-rate% 47906630 ± 4% +9.8% 52585388 ± 2% perf-stat.ps.branch-misses 2.176e+08 ± 8% +9.3% 2.379e+08 ± 8% perf-stat.ps.cache-references 4917994 ± 6% +7.8% 5303407 ± 7% perf-stat.ps.iTLB-load-misses 489770 ± 5% +5.8% 518214 ± 4% perf-stat.ps.node-loads 17149 ± 28% -52.0% 8235 ± 3% sched_debug.cfs_rq:/.exec_clock.max 1378506 ± 52% -92.1% 108984 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg 1439941 ± 51% -89.9% 144995 ± 3% sched_debug.cfs_rq:/.min_vruntime.max 1173447 ± 53% -94.3% 67349 ± 10% sched_debug.cfs_rq:/.min_vruntime.min 0.15 ± 35% -59.2% 0.06 ± 12% sched_debug.cfs_rq:/.nr_running.avg 0.15 ± 33% -57.3% 0.07 sched_debug.cfs_rq:/.nr_spread_over.avg 1.89 ± 34% -54.4% 0.86 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.avg 176.85 ± 27% -43.4% 100.17 ± 3% sched_debug.cfs_rq:/.util_avg.avg 94.34 ± 59% -92.3% 7.23 ± 11% sched_debug.cfs_rq:/.util_est_enqueued.avg 1.21 ± 20% -40.7% 0.71 sched_debug.cpu.cpu_load[0].avg 450.75 ± 41% -61.6% 173.13 ± 6% sched_debug.cpu.curr->pid.avg 8160 ± 9% -19.4% 6580 ± 2% sched_debug.cpu.curr->pid.max 914.49 ± 6% -13.8% 788.17 ± 3% sched_debug.cpu.curr->pid.stddev 72735 ± 4% +144.1% 177555 ± 56% sched_debug.cpu.load.max 1077 ± 5% +29.6% 1396 ± 22% sched_debug.cpu.nr_load_updates.stddev 0.14 ± 43% -61.4% 0.05 ± 7% sched_debug.cpu.nr_running.avg 0.70 ± 5% +9.7% 0.77 sched_debug.cpu.nr_uninterruptible.avg 89157 ± 40% -81.2% 16779 ± 92% numa-meminfo.node0.Active 89092 ± 40% -81.2% 16779 ± 92% numa-meminfo.node0.Active(anon) 60805 ± 41% -84.3% 9523 ±100% numa-meminfo.node0.AnonHugePages 83875 ± 40% -84.5% 13028 ± 91% numa-meminfo.node0.AnonPages 255728 ± 4% +108.4% 532879 numa-meminfo.node2.FilePages 1763 ± 11% +13493.2% 239750 numa-meminfo.node2.Inactive 1188 ± 46% +20035.5% 239361 numa-meminfo.node2.Inactive(anon) 15779 ± 28% +125.9% 35646 ± 6% numa-meminfo.node2.KReclaimable 5937 ± 10% +68.2% 9985 numa-meminfo.node2.KernelStack 15779 ± 28% +125.9% 35646 ± 6% numa-meminfo.node2.SReclaimable 2215 ± 67% +11124.7% 248655 numa-meminfo.node2.Shmem 252853 ± 3% +12.2% 283657 numa-meminfo.node2.Unevictable 83953 ± 11% +59.0% 133447 ± 20% numa-meminfo.node3.Active 83842 ± 11% +59.1% 133371 ± 20% numa-meminfo.node3.Active(anon) 54812 ± 18% +77.5% 97277 ± 21% numa-meminfo.node3.AnonHugePages 78034 ± 11% +69.6% 132378 ± 20% numa-meminfo.node3.AnonPages 513368 ± 3% -51.0% 251319 numa-meminfo.node3.FilePages 239267 -99.3% 1632 ± 2% numa-meminfo.node3.Inactive 238889 -99.7% 667.50 ± 6% numa-meminfo.node3.Inactive(anon) 9016 ± 18% -33.3% 6017 ± 4% numa-meminfo.node3.KernelStack 2964209 ± 4% -9.6% 2678216 numa-meminfo.node3.MemUsed 243897 -99.6% 862.00 ± 10% numa-meminfo.node3.Shmem 22271 ± 40% -81.2% 4197 ± 92% numa-vmstat.node0.nr_active_anon 20968 ± 40% -84.4% 3263 ± 91% numa-vmstat.node0.nr_anon_pages 22271 ± 40% -81.2% 4197 ± 92% numa-vmstat.node0.nr_zone_active_anon 63932 ± 4% +108.4% 133212 numa-vmstat.node2.nr_file_pages 296.50 ± 46% +20081.3% 59837 numa-vmstat.node2.nr_inactive_anon 5938 ± 10% +68.3% 9996 numa-vmstat.node2.nr_kernel_stack 1234 +12.3% 1386 numa-vmstat.node2.nr_mapped 553.25 ± 67% +11134.9% 62157 numa-vmstat.node2.nr_shmem 3944 ± 28% +125.9% 8910 ± 6% numa-vmstat.node2.nr_slab_reclaimable 63213 ± 3% +12.2% 70914 numa-vmstat.node2.nr_unevictable 296.50 ± 46% +20081.3% 59837 numa-vmstat.node2.nr_zone_inactive_anon 63213 ± 3% +12.2% 70914 numa-vmstat.node2.nr_zone_unevictable 674596 ± 10% +55.5% 1049184 ± 5% numa-vmstat.node2.numa_hit 565285 ± 13% +70.8% 965693 ± 5% numa-vmstat.node2.numa_local 109311 ± 7% -23.6% 83491 numa-vmstat.node2.numa_other 20964 ± 11% +59.0% 33339 ± 20% numa-vmstat.node3.nr_active_anon 19515 ± 11% +69.6% 33089 ± 20% numa-vmstat.node3.nr_anon_pages 128296 ± 3% -51.0% 62829 numa-vmstat.node3.nr_file_pages 59679 -99.7% 166.50 ± 6% numa-vmstat.node3.nr_inactive_anon 9013 ± 18% -33.3% 6014 ± 4% numa-vmstat.node3.nr_kernel_stack 60929 -99.6% 215.00 ± 10% numa-vmstat.node3.nr_shmem 20964 ± 11% +59.0% 33339 ± 20% numa-vmstat.node3.nr_zone_active_anon 59679 -99.7% 166.50 ± 6% numa-vmstat.node3.nr_zone_inactive_anon 98042 ± 11% +16.1% 113819 numa-vmstat.node3.numa_other 2.35 ± 7% -0.4 1.94 perf-profile.calltrace.cycles-pp.mmap64 2.29 ± 8% -0.4 1.89 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 2.29 ± 8% -0.4 1.89 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mmap64 2.21 ± 7% -0.4 1.83 perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 2.14 ± 8% -0.4 1.78 perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 0.66 ± 10% +0.1 0.77 ± 7% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.40 ± 57% +0.2 0.56 ± 3% perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.vma_link.mmap_region.do_mmap 0.72 ± 22% +0.2 0.90 ± 11% perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 7.35 +0.2 7.56 perf-profile.calltrace.cycles-pp.do_rw_once 1.32 ± 6% +0.2 1.55 perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 1.69 ± 6% +0.2 1.94 ± 2% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 0.49 ± 61% +0.3 0.74 ± 10% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 1.39 ± 20% +0.3 1.72 ± 11% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.53 ± 20% +0.4 1.89 ± 11% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.14 ±173% +0.4 0.54 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.vma_link.mmap_region 1.74 ± 18% +0.4 2.15 ± 10% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 4.31 ± 9% +0.6 4.92 ± 8% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 8.64 ± 6% +0.7 9.36 ± 2% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 61.25 ± 2% +1.3 62.53 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 61.75 ± 2% +1.4 63.12 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 67.87 ± 2% +1.6 69.50 perf-profile.calltrace.cycles-pp.secondary_startup_64 67.47 ± 2% +1.7 69.17 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 67.47 ± 2% +1.7 69.17 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 67.41 ± 2% +1.7 69.12 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 2.36 ± 7% -0.4 1.96 perf-profile.children.cycles-pp.mmap64 0.12 ± 13% +0.0 0.15 ± 10% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.13 ± 5% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.rcu_idle_exit 0.09 ± 7% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.cpu_load_update 0.31 ± 6% +0.0 0.34 perf-profile.children.cycles-pp._raw_spin_lock 0.23 ± 3% +0.0 0.26 ± 5% perf-profile.children.cycles-pp.sched_clock 0.08 ± 15% +0.0 0.11 ± 18% perf-profile.children.cycles-pp.rcu_irq_exit 0.10 ± 21% +0.0 0.14 ± 18% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit 0.04 ± 57% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.hrtimer_forward 0.10 ± 26% +0.0 0.14 ± 18% perf-profile.children.cycles-pp.rcu_irq_enter 0.10 ± 24% +0.0 0.14 ± 11% perf-profile.children.cycles-pp.perf_event_task_tick 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.irq_work_tick 0.09 ± 22% +0.0 0.13 ± 7% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.14 ± 22% +0.0 0.18 ± 16% perf-profile.children.cycles-pp.interrupt_entry 0.30 ± 5% +0.0 0.34 perf-profile.children.cycles-pp.sched_clock_cpu 0.05 ± 64% +0.1 0.11 ± 23% perf-profile.children.cycles-pp.rb_insert_color 0.01 ±173% +0.1 0.07 ± 14% perf-profile.children.cycles-pp.tick_sched_do_timer 0.30 ± 8% +0.1 0.35 ± 7% perf-profile.children.cycles-pp.find_next_bit 0.49 ± 9% +0.1 0.56 ± 11% perf-profile.children.cycles-pp.__next_timer_interrupt 0.24 ± 25% +0.1 0.32 ± 14% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.20 ± 32% +0.1 0.28 ± 17% perf-profile.children.cycles-pp.timerqueue_add 0.30 ± 10% +0.1 0.39 ± 10% perf-profile.children.cycles-pp.read_tsc 0.23 ± 30% +0.1 0.32 ± 15% perf-profile.children.cycles-pp.enqueue_hrtimer 0.61 ± 22% +0.1 0.76 ± 9% perf-profile.children.cycles-pp.tick_irq_enter 0.28 ± 17% +0.2 0.46 ± 16% perf-profile.children.cycles-pp.lapic_next_deadline 0.75 ± 21% +0.2 0.93 ± 9% perf-profile.children.cycles-pp.irq_enter 1.36 ± 6% +0.2 1.58 perf-profile.children.cycles-pp.tick_nohz_next_event 1.71 ± 5% +0.2 1.95 ± 2% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 1.59 ± 14% +0.3 1.86 ± 11% perf-profile.children.cycles-pp.update_process_times 1.73 ± 14% +0.3 2.03 ± 11% perf-profile.children.cycles-pp.tick_sched_handle 1.96 ± 13% +0.3 2.29 ± 10% perf-profile.children.cycles-pp.tick_sched_timer 3.23 ± 7% +0.3 3.56 ± 8% perf-profile.children.cycles-pp.menu_select 3.37 ± 13% +0.4 3.76 ± 10% perf-profile.children.cycles-pp.__hrtimer_run_queues 4.67 ± 7% +0.6 5.30 ± 10% perf-profile.children.cycles-pp.hrtimer_interrupt 62.12 ± 2% +1.3 63.42 perf-profile.children.cycles-pp.cpuidle_enter 62.10 ± 2% +1.3 63.41 perf-profile.children.cycles-pp.cpuidle_enter_state 67.88 ± 2% +1.6 69.51 perf-profile.children.cycles-pp.do_idle 67.87 ± 2% +1.6 69.50 perf-profile.children.cycles-pp.secondary_startup_64 67.87 ± 2% +1.6 69.50 perf-profile.children.cycles-pp.cpu_startup_entry 67.47 ± 2% +1.7 69.17 perf-profile.children.cycles-pp.start_secondary 0.29 -0.0 0.27 perf-profile.self.cycles-pp.do_syscall_64 0.06 ± 6% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.__perf_sw_event 0.12 ± 13% +0.0 0.15 ± 10% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.09 ± 7% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.cpu_load_update 0.29 ± 6% +0.0 0.32 perf-profile.self.cycles-pp._raw_spin_lock 0.20 ± 2% +0.0 0.23 ± 4% perf-profile.self.cycles-pp.native_sched_clock 0.10 ± 21% +0.0 0.13 ± 15% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.08 ± 15% +0.0 0.11 ± 18% perf-profile.self.cycles-pp.rcu_irq_exit 0.04 ± 59% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.__hrtimer_get_next_event 0.04 ± 57% +0.0 0.07 ± 14% perf-profile.self.cycles-pp.hrtimer_forward 0.03 ±100% +0.0 0.06 perf-profile.self.cycles-pp.rcu_idle_exit 0.14 ± 19% +0.0 0.17 ± 17% perf-profile.self.cycles-pp.interrupt_entry 0.10 ± 24% +0.0 0.14 ± 11% perf-profile.self.cycles-pp.perf_event_task_tick 0.26 ± 8% +0.0 0.30 ± 6% perf-profile.self.cycles-pp.find_next_bit 0.06 ± 26% +0.0 0.11 perf-profile.self.cycles-pp.lapic_next_deadline 0.13 ± 13% +0.1 0.18 ± 2% perf-profile.self.cycles-pp.hrtimer_interrupt 0.21 ± 17% +0.1 0.27 ± 13% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.04 ±103% +0.1 0.11 ± 23% perf-profile.self.cycles-pp.rb_insert_color 0.28 ± 10% +0.1 0.37 ± 10% perf-profile.self.cycles-pp.read_tsc 0.85 ± 10% +0.1 0.98 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state 0.30 ± 35% +0.2 0.47 ± 7% perf-profile.self.cycles-pp.tick_nohz_next_event 519.50 ± 25% +24.0% 644.00 ± 17% interrupts.110:PCI-MSI.1574912-edge.eth1-TxRx-0 196.25 ± 9% +57.7% 309.50 ± 20% interrupts.116:PCI-MSI.1574918-edge.eth1-TxRx-6 310.50 ± 53% -43.2% 176.50 interrupts.122:PCI-MSI.1574924-edge.eth1-TxRx-12 183.25 +15.4% 211.50 ± 11% interrupts.161:PCI-MSI.1574963-edge.eth1-TxRx-51 180.25 ± 2% +55.1% 279.50 ± 32% interrupts.164:PCI-MSI.1574966-edge.eth1-TxRx-54 519.50 ± 25% +24.0% 644.00 ± 17% interrupts.CPU0.110:PCI-MSI.1574912-edge.eth1-TxRx-0 1203 ± 6% -28.0% 866.50 ± 20% interrupts.CPU1.NMI:Non-maskable_interrupts 1203 ± 6% -28.0% 866.50 ± 20% interrupts.CPU1.PMI:Performance_monitoring_interrupts 5359 ± 39% +60.5% 8604 ± 6% interrupts.CPU10.RES:Rescheduling_interrupts 5784 ± 37% +58.5% 9168 ± 15% interrupts.CPU11.RES:Rescheduling_interrupts 909.00 ± 26% +39.5% 1268 ± 4% interrupts.CPU110.NMI:Non-maskable_interrupts 909.00 ± 26% +39.5% 1268 ± 4% interrupts.CPU110.PMI:Performance_monitoring_interrupts 922.25 ± 25% +34.0% 1236 interrupts.CPU111.NMI:Non-maskable_interrupts 922.25 ± 25% +34.0% 1236 interrupts.CPU111.PMI:Performance_monitoring_interrupts 922.00 ± 30% +34.8% 1243 ± 4% interrupts.CPU114.NMI:Non-maskable_interrupts 922.00 ± 30% +34.8% 1243 ± 4% interrupts.CPU114.PMI:Performance_monitoring_interrupts 1033 ± 24% +19.0% 1229 ± 4% interrupts.CPU116.NMI:Non-maskable_interrupts 1033 ± 24% +19.0% 1229 ± 4% interrupts.CPU116.PMI:Performance_monitoring_interrupts 867.50 ± 30% +33.7% 1160 ± 7% interrupts.CPU118.NMI:Non-maskable_interrupts 867.50 ± 30% +33.7% 1160 ± 7% interrupts.CPU118.PMI:Performance_monitoring_interrupts 6329 ± 35% +30.5% 8260 ± 23% interrupts.CPU119.RES:Rescheduling_interrupts 310.50 ± 53% -43.2% 176.50 interrupts.CPU12.122:PCI-MSI.1574924-edge.eth1-TxRx-12 6510 ± 26% +25.0% 8140 ± 22% interrupts.CPU120.RES:Rescheduling_interrupts 6814 ± 26% +41.7% 9655 interrupts.CPU122.RES:Rescheduling_interrupts 6239 ± 29% +24.6% 7774 ± 23% interrupts.CPU125.RES:Rescheduling_interrupts 6504 ± 28% +25.1% 8138 ± 24% interrupts.CPU126.RES:Rescheduling_interrupts 6206 ± 30% +28.5% 7974 ± 22% interrupts.CPU128.RES:Rescheduling_interrupts 6156 ± 30% +23.8% 7620 ± 25% interrupts.CPU129.RES:Rescheduling_interrupts 5867 ± 38% +57.2% 9223 ± 22% interrupts.CPU13.RES:Rescheduling_interrupts 6343 ± 29% +22.4% 7765 ± 28% interrupts.CPU132.RES:Rescheduling_interrupts 6263 ± 32% +26.6% 7926 ± 27% interrupts.CPU138.RES:Rescheduling_interrupts 1071 ± 9% +20.7% 1294 interrupts.CPU14.NMI:Non-maskable_interrupts 1071 ± 9% +20.7% 1294 interrupts.CPU14.PMI:Performance_monitoring_interrupts 6903 ± 25% +40.6% 9702 ± 8% interrupts.CPU142.RES:Rescheduling_interrupts 5677 ± 40% +53.7% 8724 ± 30% interrupts.CPU16.RES:Rescheduling_interrupts 721.25 ± 24% +39.3% 1004 ± 32% interrupts.CPU160.NMI:Non-maskable_interrupts 721.25 ± 24% +39.3% 1004 ± 32% interrupts.CPU160.PMI:Performance_monitoring_interrupts 785.75 ± 29% +27.8% 1004 ± 29% interrupts.CPU161.NMI:Non-maskable_interrupts 785.75 ± 29% +27.8% 1004 ± 29% interrupts.CPU161.PMI:Performance_monitoring_interrupts 778.50 ± 34% +49.2% 1161 ± 8% interrupts.CPU162.NMI:Non-maskable_interrupts 778.50 ± 34% +49.2% 1161 ± 8% interrupts.CPU162.PMI:Performance_monitoring_interrupts 4841 ± 22% +59.2% 7707 ± 9% interrupts.CPU170.RES:Rescheduling_interrupts 5010 ± 26% +41.1% 7067 ± 17% interrupts.CPU175.RES:Rescheduling_interrupts 5333 ± 30% +50.5% 8025 ± 11% interrupts.CPU176.RES:Rescheduling_interrupts 4825 ± 19% +52.4% 7354 ± 17% interrupts.CPU179.RES:Rescheduling_interrupts 827.00 ± 36% +38.0% 1141 ± 14% interrupts.CPU180.NMI:Non-maskable_interrupts 827.00 ± 36% +38.0% 1141 ± 14% interrupts.CPU180.PMI:Performance_monitoring_interrupts 1266 ± 3% -30.5% 880.00 ± 28% interrupts.CPU2.NMI:Non-maskable_interrupts 1266 ± 3% -30.5% 880.00 ± 28% interrupts.CPU2.PMI:Performance_monitoring_interrupts 6759 ± 31% +36.8% 9246 ± 5% interrupts.CPU20.RES:Rescheduling_interrupts 5976 ± 35% +41.2% 8438 ± 14% interrupts.CPU21.RES:Rescheduling_interrupts 5601 ± 39% +72.7% 9674 ± 14% interrupts.CPU22.RES:Rescheduling_interrupts 6356 ± 30% +34.8% 8568 ± 14% interrupts.CPU25.RES:Rescheduling_interrupts 6356 ± 29% +24.6% 7922 ± 22% interrupts.CPU26.RES:Rescheduling_interrupts 1287 ± 4% -5.7% 1213 interrupts.CPU27.NMI:Non-maskable_interrupts 1287 ± 4% -5.7% 1213 interrupts.CPU27.PMI:Performance_monitoring_interrupts 6381 ± 29% +42.2% 9075 ± 10% interrupts.CPU27.RES:Rescheduling_interrupts 6168 ± 28% +27.0% 7833 ± 23% interrupts.CPU28.RES:Rescheduling_interrupts 6346 ± 29% +25.3% 7955 ± 24% interrupts.CPU29.RES:Rescheduling_interrupts 6413 ± 26% +26.7% 8128 ± 23% interrupts.CPU30.RES:Rescheduling_interrupts 6115 ± 33% +70.4% 10417 ± 7% interrupts.CPU31.RES:Rescheduling_interrupts 6226 ± 28% +26.1% 7853 ± 25% interrupts.CPU32.RES:Rescheduling_interrupts 5697 ± 33% +69.0% 9630 ± 11% interrupts.CPU4.RES:Rescheduling_interrupts 6328 ± 27% +38.1% 8740 ± 8% interrupts.CPU41.RES:Rescheduling_interrupts 6079 ± 31% +22.8% 7465 ± 28% interrupts.CPU42.RES:Rescheduling_interrupts 6296 ± 26% +20.2% 7565 ± 25% interrupts.CPU44.RES:Rescheduling_interrupts 6309 ± 32% +25.8% 7939 ± 27% interrupts.CPU47.RES:Rescheduling_interrupts 927.75 ± 33% +41.3% 1310 ± 3% interrupts.CPU48.NMI:Non-maskable_interrupts 927.75 ± 33% +41.3% 1310 ± 3% interrupts.CPU48.PMI:Performance_monitoring_interrupts 701.75 ± 18% +69.4% 1188 ± 7% interrupts.CPU50.NMI:Non-maskable_interrupts 701.75 ± 18% +69.4% 1188 ± 7% interrupts.CPU50.PMI:Performance_monitoring_interrupts 183.25 +15.4% 211.50 ± 11% interrupts.CPU51.161:PCI-MSI.1574963-edge.eth1-TxRx-51 180.25 ± 2% +55.1% 279.50 ± 32% interrupts.CPU54.164:PCI-MSI.1574966-edge.eth1-TxRx-54 1273 -23.5% 974.00 ± 13% interrupts.CPU54.NMI:Non-maskable_interrupts 1273 -23.5% 974.00 ± 13% interrupts.CPU54.PMI:Performance_monitoring_interrupts 1250 ± 2% -27.2% 910.50 ± 3% interrupts.CPU55.NMI:Non-maskable_interrupts 1250 ± 2% -27.2% 910.50 ± 3% interrupts.CPU55.PMI:Performance_monitoring_interrupts 6200 ± 26% +43.9% 8925 ± 22% interrupts.CPU55.RES:Rescheduling_interrupts 6497 ± 23% +34.0% 8708 ± 16% interrupts.CPU58.RES:Rescheduling_interrupts 196.25 ± 9% +57.7% 309.50 ± 20% interrupts.CPU6.116:PCI-MSI.1574918-edge.eth1-TxRx-6 1197 ± 10% -43.8% 673.00 ± 3% interrupts.CPU64.NMI:Non-maskable_interrupts 1197 ± 10% -43.8% 673.00 ± 3% interrupts.CPU64.PMI:Performance_monitoring_interrupts 1286 ± 4% -48.6% 661.50 interrupts.CPU65.NMI:Non-maskable_interrupts 1286 ± 4% -48.6% 661.50 interrupts.CPU65.PMI:Performance_monitoring_interrupts 1059 ± 27% -51.1% 518.50 ± 8% interrupts.CPU68.NMI:Non-maskable_interrupts 1059 ± 27% -51.1% 518.50 ± 8% interrupts.CPU68.PMI:Performance_monitoring_interrupts 1076 ± 25% -50.0% 537.50 interrupts.CPU69.NMI:Non-maskable_interrupts 1076 ± 25% -50.0% 537.50 interrupts.CPU69.PMI:Performance_monitoring_interrupts 5701 ± 21% +40.9% 8032 ± 20% interrupts.CPU72.RES:Rescheduling_interrupts 5004 ± 30% +51.4% 7576 ± 18% interrupts.CPU73.RES:Rescheduling_interrupts 5079 ± 34% +53.5% 7796 ± 16% interrupts.CPU74.RES:Rescheduling_interrupts 4924 ± 33% +64.4% 8098 ± 2% interrupts.CPU75.RES:Rescheduling_interrupts 1056 ± 24% -44.4% 587.00 ± 2% interrupts.CPU77.NMI:Non-maskable_interrupts 1056 ± 24% -44.4% 587.00 ± 2% interrupts.CPU77.PMI:Performance_monitoring_interrupts 5362 ± 25% +44.7% 7759 ± 13% interrupts.CPU77.RES:Rescheduling_interrupts 5660 ± 37% +37.9% 7807 ± 20% interrupts.CPU8.RES:Rescheduling_interrupts 4760 ± 26% +58.6% 7549 ± 14% interrupts.CPU81.RES:Rescheduling_interrupts 786.50 ± 34% +41.9% 1116 ± 16% interrupts.CPU83.NMI:Non-maskable_interrupts 786.50 ± 34% +41.9% 1116 ± 16% interrupts.CPU83.PMI:Performance_monitoring_interrupts 5101 ± 29% +58.7% 8095 ± 8% interrupts.CPU83.RES:Rescheduling_interrupts 5037 ± 25% +56.7% 7895 ± 4% interrupts.CPU86.RES:Rescheduling_interrupts 5031 ± 24% +33.8% 6731 ± 8% interrupts.CPU89.RES:Rescheduling_interrupts 5011 ± 20% +35.5% 6789 ± 5% interrupts.CPU90.RES:Rescheduling_interrupts 5051 ± 21% +43.0% 7224 ± 6% interrupts.CPU91.RES:Rescheduling_interrupts 5231 ± 17% +41.0% 7373 ± 12% interrupts.CPU93.RES:Rescheduling_interrupts 1156218 ± 26% +23.3% 1425107 ± 18% interrupts.RES:Rescheduling_interrupts *************************************************************************************************** lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory ========================================================================================= compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/30/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/dbase/reaim/0x14 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 14% 0:2 perf-profile.calltrace.cycles-pp.error_entry 1:4 -8% 1:2 perf-profile.children.cycles-pp.error_entry 1:4 -7% 0:2 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 442.37 ± 3% -5.1% 419.99 ± 2% reaim.child_systime 13286 ± 3% -5.1% 12615 ± 2% reaim.time.system_time 111693 ± 6% +10.8% 123765 meminfo.Shmem 26070 ± 22% +39.0% 36245 numa-numastat.node3.other_node 38.05 ± 2% +3.3% 39.30 boot-time.boot 4698 ± 2% +4.2% 4897 boot-time.idle 87.32 ± 8% -12.7% 76.27 ± 2% sched_debug.cpu.clock.stddev 87.32 ± 8% -12.7% 76.27 ± 2% sched_debug.cpu.clock_task.stddev 28159 ± 19% -28.9% 20019 numa-vmstat.node1.nr_slab_unreclaimable 98486 ± 11% +17.2% 115461 ± 4% numa-vmstat.node2.nr_anon_pages 19281 ± 3% +15.5% 22264 ± 10% numa-vmstat.node3.nr_kernel_stack 23454 +16.2% 27253 ± 9% numa-vmstat.node3.nr_slab_unreclaimable 107391 ± 5% +9.7% 117758 numa-vmstat.node3.numa_other 150.50 ± 3% +14.6% 172.50 ± 3% proc-vmstat.nr_dirtied 283397 +1.5% 287623 proc-vmstat.nr_file_pages 27944 ± 6% +10.5% 30887 proc-vmstat.nr_shmem 150.50 ± 3% +14.3% 172.00 ± 3% proc-vmstat.nr_written 49856 ± 4% +4.2% 51959 proc-vmstat.pgactivate 5655 ± 11% +12.7% 6376 ± 10% numa-meminfo.node1.Mapped 112281 ± 19% -28.6% 80154 numa-meminfo.node1.SUnreclaim 28102 ±117% +196.1% 83206 ± 4% numa-meminfo.node2.AnonHugePages 393685 ± 11% +17.6% 463066 ± 4% numa-meminfo.node2.AnonPages 19251 ± 4% +17.0% 22515 ± 10% numa-meminfo.node3.KernelStack 93821 +16.6% 109402 ± 9% numa-meminfo.node3.SUnreclaim 90973 ± 2% +2.5% 93275 perf-stat.i.context-switches 1.61 ± 2% -5.8% 1.52 perf-stat.i.cpi 26562 +1.4% 26929 perf-stat.i.cpu-migrations 22500284 +2.5% 23068436 perf-stat.i.dTLB-store-misses 13732559 ± 5% -8.0% 12628121 perf-stat.i.iTLB-load-misses 15609 ± 3% +8.3% 16896 perf-stat.i.instructions-per-iTLB-miss 0.82 +1.5% 0.84 perf-stat.i.ipc 35101206 +1.1% 35493977 perf-stat.i.node-load-misses 0.12 +0.0 0.12 perf-stat.overall.dTLB-store-miss-rate% 16572 ± 3% +8.1% 17908 perf-stat.overall.instructions-per-iTLB-miss 26438 +1.3% 26785 perf-stat.ps.cpu-migrations 22411063 +2.5% 22965030 perf-stat.ps.dTLB-store-misses 13678545 ± 5% -8.1% 12571691 perf-stat.ps.iTLB-load-misses 34938255 +1.0% 35300180 perf-stat.ps.node-load-misses 844.25 ± 10% +31.1% 1106 ± 3% slabinfo.biovec-128.active_objs 844.25 ± 10% +31.1% 1106 ± 3% slabinfo.biovec-128.num_objs 1555 ± 10% +22.4% 1904 ± 3% slabinfo.dmaengine-unmap-16.active_objs 1555 ± 10% +22.4% 1904 ± 3% slabinfo.dmaengine-unmap-16.num_objs 3626 ± 8% +19.3% 4325 ± 8% slabinfo.eventpoll_pwq.active_objs 3626 ± 8% +19.3% 4325 ± 8% slabinfo.eventpoll_pwq.num_objs 697.00 ± 7% -11.2% 619.00 slabinfo.file_lock_cache.active_objs 697.00 ± 7% -11.2% 619.00 slabinfo.file_lock_cache.num_objs 4278 ± 6% +26.2% 5399 ± 8% slabinfo.kmalloc-rcl-64.active_objs 4278 ± 6% +26.2% 5399 ± 8% slabinfo.kmalloc-rcl-64.num_objs 88376 ± 17% -40.0% 53057 slabinfo.lsm_file_cache.active_objs 344.75 ± 17% -40.0% 207.00 slabinfo.lsm_file_cache.active_slabs 88376 ± 17% -40.0% 53057 slabinfo.lsm_file_cache.num_objs 344.75 ± 17% -40.0% 207.00 slabinfo.lsm_file_cache.num_slabs 1391 ± 13% +23.5% 1717 slabinfo.nfs_read_data.active_objs 1391 ± 13% +23.5% 1717 slabinfo.nfs_read_data.num_objs 1663 ± 11% +40.6% 2340 ± 8% slabinfo.skbuff_ext_cache.active_objs 1663 ± 11% +40.6% 2340 ± 8% slabinfo.skbuff_ext_cache.num_objs 1808 ± 18% +17.6% 2126 ± 6% slabinfo.skbuff_fclone_cache.active_objs 1808 ± 18% +17.6% 2126 ± 6% slabinfo.skbuff_fclone_cache.num_objs 937.75 ± 11% +28.2% 1202 slabinfo.xfrm_state.active_objs 937.75 ± 11% +28.2% 1202 slabinfo.xfrm_state.num_objs 356.25 ± 34% +209.1% 1101 ± 18% interrupts.47:PCI-MSI.1572866-edge.eth0-TxRx-2 514.25 ± 21% -46.3% 276.00 interrupts.48:PCI-MSI.1572867-edge.eth0-TxRx-3 349.50 ± 45% +124.2% 783.50 ± 6% interrupts.52:PCI-MSI.1572871-edge.eth0-TxRx-7 388.25 ± 44% -34.4% 254.50 ± 2% interrupts.57:PCI-MSI.1572876-edge.eth0-TxRx-12 259.50 ± 3% +105.8% 534.00 ± 50% interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15 4201 ± 67% -100.0% 1.50 ± 33% interrupts.CPU102.NMI:Non-maskable_interrupts 4201 ± 67% -100.0% 1.50 ± 33% interrupts.CPU102.PMI:Performance_monitoring_interrupts 6610 ± 6% +6.0% 7008 ± 5% interrupts.CPU104.CAL:Function_call_interrupts 2142 ± 9% +8.7% 2328 ± 7% interrupts.CPU107.TLB:TLB_shootdowns 3156 ±100% +110.9% 6658 interrupts.CPU108.NMI:Non-maskable_interrupts 3156 ±100% +110.9% 6658 interrupts.CPU108.PMI:Performance_monitoring_interrupts 2160 ± 10% +11.1% 2399 ± 5% interrupts.CPU112.TLB:TLB_shootdowns 1430 ±150% +245.3% 4938 ± 33% interrupts.CPU119.NMI:Non-maskable_interrupts 1430 ±150% +245.3% 4938 ± 33% interrupts.CPU119.PMI:Performance_monitoring_interrupts 388.25 ± 44% -34.4% 254.50 ± 2% interrupts.CPU12.57:PCI-MSI.1572876-edge.eth0-TxRx-12 1678 ±160% +304.1% 6781 ± 3% interrupts.CPU13.NMI:Non-maskable_interrupts 1678 ±160% +304.1% 6781 ± 3% interrupts.CPU13.PMI:Performance_monitoring_interrupts 2866 ±101% -100.0% 0.50 ±100% interrupts.CPU131.NMI:Non-maskable_interrupts 2866 ±101% -100.0% 0.50 ±100% interrupts.CPU131.PMI:Performance_monitoring_interrupts 1572 ±170% +314.2% 6515 interrupts.CPU137.NMI:Non-maskable_interrupts 1572 ±170% +314.2% 6515 interrupts.CPU137.PMI:Performance_monitoring_interrupts 3900 ± 65% +67.4% 6529 ± 2% interrupts.CPU141.NMI:Non-maskable_interrupts 3900 ± 65% +67.4% 6529 ± 2% interrupts.CPU141.PMI:Performance_monitoring_interrupts 259.50 ± 3% +105.8% 534.00 ± 50% interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15 10701 ± 4% -9.7% 9658 interrupts.CPU15.RES:Rescheduling_interrupts 356.25 ± 34% +209.1% 1101 ± 18% interrupts.CPU2.47:PCI-MSI.1572866-edge.eth0-TxRx-2 4498 ± 49% -100.0% 2.00 interrupts.CPU22.NMI:Non-maskable_interrupts 4498 ± 49% -100.0% 2.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts 4182 ± 63% -98.2% 75.50 ± 90% interrupts.CPU23.NMI:Non-maskable_interrupts 4182 ± 63% -98.2% 75.50 ± 90% interrupts.CPU23.PMI:Performance_monitoring_interrupts 2150 ± 12% +30.9% 2814 ± 23% interrupts.CPU26.TLB:TLB_shootdowns 514.25 ± 21% -46.3% 276.00 interrupts.CPU3.48:PCI-MSI.1572867-edge.eth0-TxRx-3 2126 ± 8% +9.1% 2319 ± 5% interrupts.CPU37.TLB:TLB_shootdowns 3959 ± 57% +69.3% 6704 interrupts.CPU42.NMI:Non-maskable_interrupts 3959 ± 57% +69.3% 6704 interrupts.CPU42.PMI:Performance_monitoring_interrupts 4068 ± 34% -85.2% 603.50 ± 99% interrupts.CPU47.NMI:Non-maskable_interrupts 4068 ± 34% -85.2% 603.50 ± 99% interrupts.CPU47.PMI:Performance_monitoring_interrupts 6374 ± 4% +6.3% 6776 ± 5% interrupts.CPU48.CAL:Function_call_interrupts 2782 ± 76% -100.0% 0.50 ±100% interrupts.CPU58.NMI:Non-maskable_interrupts 2782 ± 76% -100.0% 0.50 ±100% interrupts.CPU58.PMI:Performance_monitoring_interrupts 3930 ± 68% +70.1% 6684 interrupts.CPU60.NMI:Non-maskable_interrupts 3930 ± 68% +70.1% 6684 interrupts.CPU60.PMI:Performance_monitoring_interrupts 2355 ±108% +110.9% 4966 ± 31% interrupts.CPU62.NMI:Non-maskable_interrupts 2355 ±108% +110.9% 4966 ± 31% interrupts.CPU62.PMI:Performance_monitoring_interrupts 3140 ± 70% +110.4% 6608 interrupts.CPU63.NMI:Non-maskable_interrupts 3140 ± 70% +110.4% 6608 interrupts.CPU63.PMI:Performance_monitoring_interrupts 3356 ± 71% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts 3356 ± 71% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts 2113 ± 11% +11.1% 2348 ± 6% interrupts.CPU66.TLB:TLB_shootdowns 349.50 ± 45% +124.2% 783.50 ± 6% interrupts.CPU7.52:PCI-MSI.1572871-edge.eth0-TxRx-7 6486 ± 3% +9.3% 7091 interrupts.CPU7.CAL:Function_call_interrupts 1.25 ±173% +5.3e+05% 6593 interrupts.CPU7.NMI:Non-maskable_interrupts 1.25 ±173% +5.3e+05% 6593 interrupts.CPU7.PMI:Performance_monitoring_interrupts 2089 ± 9% +11.5% 2330 ± 11% interrupts.CPU74.TLB:TLB_shootdowns 5708 ± 27% -99.5% 26.00 ± 76% interrupts.CPU79.NMI:Non-maskable_interrupts 5708 ± 27% -99.5% 26.00 ± 76% interrupts.CPU79.PMI:Performance_monitoring_interrupts 846.25 ±168% +316.0% 3520 ± 89% interrupts.CPU83.NMI:Non-maskable_interrupts 846.25 ±168% +316.0% 3520 ± 89% interrupts.CPU83.PMI:Performance_monitoring_interrupts 2117 ± 10% +10.4% 2337 ± 6% interrupts.CPU84.TLB:TLB_shootdowns 3002 ± 73% +122.1% 6667 interrupts.CPU9.NMI:Non-maskable_interrupts 3002 ± 73% +122.1% 6667 interrupts.CPU9.PMI:Performance_monitoring_interrupts 2132 ± 8% +8.8% 2320 ± 6% interrupts.CPU93.TLB:TLB_shootdowns 2002 ±110% +144.9% 4904 ± 31% interrupts.CPU94.NMI:Non-maskable_interrupts 2002 ±110% +144.9% 4904 ± 31% interrupts.CPU94.PMI:Performance_monitoring_interrupts 2099 ± 7% +13.1% 2375 ± 3% interrupts.CPU96.TLB:TLB_shootdowns 1896 ±144% +257.9% 6786 interrupts.CPU99.NMI:Non-maskable_interrupts 1896 ±144% +257.9% 6786 interrupts.CPU99.PMI:Performance_monitoring_interrupts 17.22 ± 68% -17.2 0.00 perf-profile.calltrace.cycles-pp.sieve 4.71 ± 68% -4.7 0.00 perf-profile.calltrace.cycles-pp.string_rtns_1 0.13 ±173% +0.4 0.52 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.setlocale 0.18 ±173% +0.5 0.67 ± 9% perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 0.35 ±108% +0.5 0.85 ± 11% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.19 ±173% +0.5 0.70 ± 15% perf-profile.calltrace.cycles-pp.clear_page_erms.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma 0.36 ±107% +0.5 0.88 ± 11% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.20 ±173% +0.5 0.74 ± 14% perf-profile.calltrace.cycles-pp.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page 0.87 ± 38% +0.6 1.44 ± 15% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk 0.20 ±173% +0.6 0.77 ± 10% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.88 ± 38% +0.6 1.46 ± 15% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 0.41 ±107% +0.6 0.99 ± 11% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.45 ±106% +0.6 1.04 ± 13% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault 0.20 ±173% +0.6 0.79 ± 13% perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu 0.20 ±173% +0.6 0.80 ± 13% perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region 0.40 ±107% +0.6 1.00 ± 4% perf-profile.calltrace.cycles-pp.setlocale 0.23 ±173% +0.6 0.84 ± 11% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.46 ±107% +0.7 1.11 ± 11% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.20 ±173% +0.7 0.86 ± 17% perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap 0.38 ±102% +0.7 1.06 ± 7% perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.55 ±107% +0.7 1.27 ± 13% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault 1.16 ± 39% +0.7 1.89 ± 14% perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.59 ±107% +0.8 1.39 ± 13% perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault 0.58 ±108% +0.8 1.39 ± 11% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt 0.54 ±107% +0.8 1.35 ± 13% perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 0.59 ±108% +0.8 1.42 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt 0.64 ±110% +1.0 1.65 ± 12% perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.73 ±107% +1.1 1.82 ± 13% perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode 1.57 ± 51% +1.1 2.69 ± 13% perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.80 ±110% +1.3 2.06 ± 14% perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.92 ± 37% +2.2 5.11 ± 17% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 1.59 ±110% +2.5 4.06 ± 13% perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.56 ± 62% +2.8 5.37 ± 17% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 2.96 ± 63% +3.2 6.16 ± 16% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 3.06 ± 63% +3.3 6.33 ± 16% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 3.10 ± 63% +3.3 6.41 ± 16% perf-profile.calltrace.cycles-pp.page_fault 15.46 ± 36% +16.9 32.39 ± 13% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 16.01 ± 35% +17.2 33.22 ± 13% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 16.04 ± 35% +17.2 33.26 ± 13% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 16.41 ± 34% +17.4 33.85 ± 13% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 16.41 ± 34% +17.4 33.85 ± 13% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 16.41 ± 34% +17.4 33.85 ± 13% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 16.53 ± 34% +17.5 34.08 ± 12% perf-profile.calltrace.cycles-pp.secondary_startup_64 17.23 ± 68% -17.2 0.00 perf-profile.children.cycles-pp.sieve 4.75 ± 68% -4.7 0.00 perf-profile.children.cycles-pp.string_rtns_1 0.34 ± 24% -0.1 0.22 perf-profile.children.cycles-pp.__xstat64 0.08 ± 10% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.07 ± 34% +0.0 0.10 ± 14% perf-profile.children.cycles-pp.mem_cgroup_uncharge_list 0.11 ± 27% +0.0 0.14 ± 14% perf-profile.children.cycles-pp.ksys_read 0.05 ± 63% +0.0 0.09 ± 11% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics 0.09 ± 33% +0.0 0.13 ± 15% perf-profile.children.cycles-pp.__mod_lruvec_state 0.09 ± 26% +0.0 0.13 ± 7% perf-profile.children.cycles-pp.new_sync_read 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.__update_load_avg_se 0.11 ± 30% +0.0 0.15 ± 16% perf-profile.children.cycles-pp.trailing_symlink 0.03 ±102% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.page_get_link 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.schedule_tail 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.arch_dup_task_struct 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.migration_cpu_stop 0.10 ± 18% +0.0 0.14 ± 7% perf-profile.children.cycles-pp.clockevents_program_event 0.03 ±100% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.free_pgd_range 0.05 ± 64% +0.0 0.10 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.04 ±102% +0.0 0.08 ± 12% perf-profile.children.cycles-pp.up_read 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.cap_mmap_addr 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.d_alloc 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.shmem_getpage_gfp 0.03 ±102% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.avc_has_perm 0.11 ± 28% +0.0 0.15 ± 16% perf-profile.children.cycles-pp.new_sync_write 0.08 ± 44% +0.0 0.13 ± 15% perf-profile.children.cycles-pp.vma_compute_subtree_gap 0.05 ± 67% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.strlcpy 0.02 ±173% +0.0 0.07 ± 23% perf-profile.children.cycles-pp.inode_permission 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp._vm_normal_page 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.__getpid 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.security_mmap_file 0.01 ±173% +0.0 0.06 ± 16% perf-profile.children.cycles-pp.d_path 0.01 ±173% +0.0 0.06 perf-profile.children.cycles-pp.unmap_single_vma 0.04 ±103% +0.0 0.09 ± 17% perf-profile.children.cycles-pp.pgd_alloc 0.03 ±100% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.__vsnprintf_chk 0.11 ± 30% +0.0 0.16 ± 9% perf-profile.children.cycles-pp.mem_cgroup_commit_charge 0.12 ± 26% +0.0 0.17 ± 14% perf-profile.children.cycles-pp.vfs_write 0.06 ± 67% +0.1 0.11 ± 18% perf-profile.children.cycles-pp.sync_regs 0.03 ±102% +0.1 0.08 ± 12% perf-profile.children.cycles-pp.__wake_up_common 0.01 ±173% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.wake_q_add 0.01 ±173% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.cpumask_next_and 0.00 +0.1 0.05 perf-profile.children.cycles-pp.vma_interval_tree_augment_rotate 0.00 +0.1 0.05 perf-profile.children.cycles-pp.find_next_and_bit 0.00 +0.1 0.05 perf-profile.children.cycles-pp.fopen 0.13 ± 29% +0.1 0.18 ± 11% perf-profile.children.cycles-pp.ksys_write 0.07 ± 39% +0.1 0.12 ± 4% perf-profile.children.cycles-pp.vm_area_dup 0.07 ± 64% +0.1 0.12 ± 8% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.03 ±105% +0.1 0.08 ± 5% perf-profile.children.cycles-pp.d_alloc_parallel 0.03 ±100% +0.1 0.08 ± 5% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.04 ±103% +0.1 0.09 ± 11% perf-profile.children.cycles-pp.__tlb_remove_page_size 0.08 ± 34% +0.1 0.13 ± 30% perf-profile.children.cycles-pp.__count_memcg_events 0.08 ± 42% +0.1 0.13 ± 7% perf-profile.children.cycles-pp.create_elf_tables 0.08 ± 42% +0.1 0.13 ± 7% perf-profile.children.cycles-pp.copy_strings_kernel 0.02 ±173% +0.1 0.07 ± 14% perf-profile.children.cycles-pp.__anon_vma_prepare 0.01 ±173% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.generic_file_read_iter 0.04 ±102% +0.1 0.09 ± 11% perf-profile.children.cycles-pp.__list_add_valid 0.03 ±102% +0.1 0.08 ± 5% perf-profile.children.cycles-pp.osq_unlock 0.06 ± 71% +0.1 0.11 ± 9% perf-profile.children.cycles-pp.__pmd_alloc 0.04 ±101% +0.1 0.10 ± 5% perf-profile.children.cycles-pp.__check_object_size 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.__cxa_atexit 0.01 ±173% +0.1 0.07 ± 14% perf-profile.children.cycles-pp.kernel_read 0.04 ± 59% +0.1 0.10 ± 19% perf-profile.children.cycles-pp.__vmalloc_node_range 0.04 ± 63% +0.1 0.10 ± 30% perf-profile.children.cycles-pp.generic_perform_write 0.05 ±102% +0.1 0.10 ± 14% perf-profile.children.cycles-pp.mm_init 0.03 ±100% +0.1 0.09 ± 11% perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate 0.08 ± 37% +0.1 0.14 perf-profile.children.cycles-pp.do_open_execat 0.02 ±173% +0.1 0.08 ± 6% perf-profile.children.cycles-pp._IO_file_open 0.01 ±173% +0.1 0.07 ± 14% perf-profile.children.cycles-pp.__ctype_init 0.10 ± 39% +0.1 0.16 ± 9% perf-profile.children.cycles-pp.__might_sleep 0.09 ± 37% +0.1 0.15 ± 10% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.06 ± 60% +0.1 0.11 ± 30% perf-profile.children.cycles-pp.generic_file_write_iter 0.04 ±107% +0.1 0.10 ± 15% perf-profile.children.cycles-pp.do_dentry_open 0.05 ± 64% +0.1 0.11 ± 36% perf-profile.children.cycles-pp.worker_thread 0.03 ±100% +0.1 0.09 ± 17% perf-profile.children.cycles-pp.xas_find 0.07 ± 39% +0.1 0.14 ± 3% perf-profile.children.cycles-pp.__alloc_file 0.06 ± 70% +0.1 0.12 ± 4% perf-profile.children.cycles-pp.setup_arg_pages 0.04 ±108% +0.1 0.10 ± 14% perf-profile.children.cycles-pp.lookup_slow 0.03 ±105% +0.1 0.10 ± 15% perf-profile.children.cycles-pp.unwind_next_frame 0.03 ±100% +0.1 0.09 ± 11% perf-profile.children.cycles-pp.xas_load 0.02 ±173% +0.1 0.08 perf-profile.children.cycles-pp._copy_from_user 0.08 ± 34% +0.1 0.14 ± 7% perf-profile.children.cycles-pp.__rb_insert_augmented 0.05 ± 62% +0.1 0.11 ± 27% perf-profile.children.cycles-pp.__generic_file_write_iter 0.04 ±106% +0.1 0.11 ± 4% perf-profile.children.cycles-pp.change_protection 0.08 ± 45% +0.1 0.15 ± 3% perf-profile.children.cycles-pp.alloc_empty_file 0.07 ± 30% +0.1 0.14 ± 7% perf-profile.children.cycles-pp.update_rq_clock 0.10 ± 42% +0.1 0.16 ± 9% perf-profile.children.cycles-pp.anon_vma_interval_tree_remove 0.12 ± 40% +0.1 0.19 ± 10% perf-profile.children.cycles-pp.selinux_inode_permission 0.12 ± 40% +0.1 0.18 ± 13% perf-profile.children.cycles-pp.__get_free_pages 0.12 ± 43% +0.1 0.18 ± 13% perf-profile.children.cycles-pp.__pte_alloc 0.13 ± 32% +0.1 0.20 ± 15% perf-profile.children.cycles-pp.vfs_read 0.12 ± 30% +0.1 0.18 ± 2% perf-profile.children.cycles-pp.copy_user_highpage 0.09 ± 39% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.dequeue_task_fair 0.09 ± 33% +0.1 0.16 perf-profile.children.cycles-pp.time 0.06 ± 70% +0.1 0.13 perf-profile.children.cycles-pp.dequeue_entity 0.04 ±108% +0.1 0.11 ± 9% perf-profile.children.cycles-pp.release_task 0.12 ± 33% +0.1 0.18 ± 2% perf-profile.children.cycles-pp.copy_page 0.12 ± 38% +0.1 0.19 ± 21% perf-profile.children.cycles-pp.avc_has_perm_noaudit 0.04 ±102% +0.1 0.10 ± 14% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.04 ±104% +0.1 0.11 ± 4% perf-profile.children.cycles-pp.unlock_page 0.09 ± 40% +0.1 0.16 ± 12% perf-profile.children.cycles-pp.__d_lookup_rcu 0.04 ±106% +0.1 0.11 ± 9% perf-profile.children.cycles-pp.rwsem_mark_wake 0.04 ±107% +0.1 0.11 ± 4% perf-profile.children.cycles-pp.__strcasecmp 0.13 ± 36% +0.1 0.21 ± 12% perf-profile.children.cycles-pp.security_inode_permission 0.11 ± 38% +0.1 0.18 ± 8% perf-profile.children.cycles-pp.do_faccessat 0.10 ± 47% +0.1 0.17 ± 8% perf-profile.children.cycles-pp.get_user_pages_remote 0.04 ±100% +0.1 0.11 ± 4% perf-profile.children.cycles-pp.update_load_avg 0.12 ± 35% +0.1 0.20 ± 7% perf-profile.children.cycles-pp.update_curr 0.12 ± 40% +0.1 0.20 ± 2% perf-profile.children.cycles-pp.__slab_free 0.09 ± 40% +0.1 0.17 ± 3% perf-profile.children.cycles-pp.do_task_dead 0.13 ± 44% +0.1 0.21 ± 14% perf-profile.children.cycles-pp.wake_up_new_task 0.04 ±108% +0.1 0.12 perf-profile.children.cycles-pp.finish_task_switch 0.09 ± 37% +0.1 0.17 ± 15% perf-profile.children.cycles-pp.page_add_file_rmap 0.07 ± 39% +0.1 0.15 ± 13% perf-profile.children.cycles-pp.stack_trace_save_tsk 0.16 ± 34% +0.1 0.24 ± 12% perf-profile.children.cycles-pp.mem_cgroup_try_charge 0.12 ± 37% +0.1 0.21 ± 21% perf-profile.children.cycles-pp.cred_has_capability 0.14 ± 38% +0.1 0.22 ± 6% perf-profile.children.cycles-pp.anon_vma_interval_tree_insert 0.06 ± 68% +0.1 0.14 ± 14% perf-profile.children.cycles-pp.arch_stack_walk 0.10 ± 44% +0.1 0.19 ± 5% perf-profile.children.cycles-pp.remove_vma 0.09 ± 41% +0.1 0.17 ± 14% perf-profile.children.cycles-pp.__account_scheduler_latency 0.06 ± 77% +0.1 0.15 ± 13% perf-profile.children.cycles-pp.refcount_dec_not_one 0.06 ± 77% +0.1 0.15 ± 13% perf-profile.children.cycles-pp.free_uid 0.06 ± 77% +0.1 0.15 ± 13% perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave 0.05 ±108% +0.1 0.15 ± 10% perf-profile.children.cycles-pp.wait_consider_task 0.15 ± 36% +0.1 0.25 ± 24% perf-profile.children.cycles-pp.selinux_vm_enough_memory 0.15 ± 38% +0.1 0.24 ± 10% perf-profile.children.cycles-pp.lookup_fast 0.06 ± 70% +0.1 0.16 ± 12% perf-profile.children.cycles-pp.sched_ttwu_pending 0.05 ±106% +0.1 0.15 ± 10% perf-profile.children.cycles-pp.queued_write_lock_slowpath 0.15 ± 42% +0.1 0.26 ± 7% perf-profile.children.cycles-pp.___might_sleep 0.04 ±103% +0.1 0.15 ± 6% perf-profile.children.cycles-pp.idle_cpu 0.19 ± 41% +0.1 0.30 ± 6% perf-profile.children.cycles-pp.kmem_cache_alloc 0.15 ± 40% +0.1 0.26 ± 3% perf-profile.children.cycles-pp.free_pages_and_swap_cache 0.17 ± 37% +0.1 0.28 ± 7% perf-profile.children.cycles-pp.up_write 0.23 ± 36% +0.1 0.34 ± 7% perf-profile.children.cycles-pp.get_unmapped_area 0.22 ± 34% +0.1 0.34 ± 13% perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay 0.20 ± 38% +0.1 0.32 ± 9% perf-profile.children.cycles-pp.alloc_set_pte 0.14 ± 44% +0.1 0.27 ± 3% perf-profile.children.cycles-pp.wait4 0.29 ± 35% +0.1 0.42 ± 8% perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.18 ± 38% +0.1 0.32 ± 7% perf-profile.children.cycles-pp.kmem_cache_free 0.06 ± 65% +0.1 0.19 perf-profile.children.cycles-pp.schedule_idle 0.24 ± 37% +0.1 0.38 ± 14% perf-profile.children.cycles-pp.security_vm_enough_memory_mm 0.21 ± 38% +0.1 0.35 ± 9% perf-profile.children.cycles-pp.walk_component 0.25 ± 34% +0.1 0.39 ± 5% perf-profile.children.cycles-pp.wp_page_copy 0.13 ± 46% +0.1 0.27 ± 11% perf-profile.children.cycles-pp.put_cred_rcu 0.27 ± 40% +0.1 0.41 ± 9% perf-profile.children.cycles-pp.find_vma 0.27 ± 40% +0.1 0.41 ± 9% perf-profile.children.cycles-pp.unlink_anon_vmas 0.14 ± 39% +0.1 0.29 ± 12% perf-profile.children.cycles-pp.ttwu_do_activate 0.26 ± 43% +0.2 0.41 ± 11% perf-profile.children.cycles-pp.select_task_rq_fair 0.15 ± 40% +0.2 0.31 ± 8% perf-profile.children.cycles-pp.enqueue_entity 0.32 ± 36% +0.2 0.47 ± 10% perf-profile.children.cycles-pp.vma_merge 0.23 ± 38% +0.2 0.38 ± 5% perf-profile.children.cycles-pp.kthread 0.35 ± 35% +0.2 0.51 ± 10% perf-profile.children.cycles-pp.__lru_cache_add 0.26 ± 44% +0.2 0.42 ± 8% perf-profile.children.cycles-pp.perf_iterate_sb 0.17 ± 39% +0.2 0.33 ± 9% perf-profile.children.cycles-pp.activate_task 0.30 ± 40% +0.2 0.46 ± 11% perf-profile.children.cycles-pp.link_path_walk 0.18 ± 38% +0.2 0.35 ± 9% perf-profile.children.cycles-pp.enqueue_task_fair 0.26 ± 44% +0.2 0.44 ± 6% perf-profile.children.cycles-pp.copy_strings 0.26 ± 44% +0.2 0.44 ± 9% perf-profile.children.cycles-pp._dl_addr 0.26 ± 39% +0.2 0.45 ± 2% perf-profile.children.cycles-pp.ret_from_fork 0.23 ± 49% +0.2 0.43 perf-profile.children.cycles-pp.do_wait 0.38 ± 32% +0.2 0.58 ± 9% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.46 ± 38% +0.2 0.67 ± 14% perf-profile.children.cycles-pp.path_lookupat 0.23 ± 51% +0.2 0.45 perf-profile.children.cycles-pp.kernel_wait4 0.47 ± 37% +0.2 0.69 ± 15% perf-profile.children.cycles-pp.filename_lookup 0.26 ± 41% +0.2 0.47 ± 7% perf-profile.children.cycles-pp.rcu_core 0.23 ± 51% +0.2 0.45 ± 2% perf-profile.children.cycles-pp.__do_sys_wait4 0.47 ± 36% +0.2 0.69 ± 9% perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.29 ± 43% +0.2 0.52 ± 4% perf-profile.children.cycles-pp.vma_interval_tree_insert 0.39 ± 38% +0.2 0.62 ± 5% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.43 ± 35% +0.3 0.71 ± 12% perf-profile.children.cycles-pp.native_irq_return_iret 0.45 ± 31% +0.3 0.73 ± 10% perf-profile.children.cycles-pp.do_sys_open 0.33 ± 40% +0.3 0.61 ± 8% perf-profile.children.cycles-pp.copy_p4d_range 0.46 ± 31% +0.3 0.75 ± 10% perf-profile.children.cycles-pp.do_filp_open 0.46 ± 31% +0.3 0.74 ± 11% perf-profile.children.cycles-pp.path_openat 0.48 ± 40% +0.3 0.76 ± 8% perf-profile.children.cycles-pp.perf_event_mmap 0.34 ± 40% +0.3 0.62 ± 8% perf-profile.children.cycles-pp.copy_page_range 0.41 ± 25% +0.3 0.71 ± 10% perf-profile.children.cycles-pp.__softirqentry_text_start 0.72 ± 36% +0.3 1.02 ± 12% perf-profile.children.cycles-pp.clear_page_erms 0.44 ± 23% +0.3 0.76 ± 9% perf-profile.children.cycles-pp.irq_exit 0.57 ± 36% +0.3 0.89 ± 12% perf-profile.children.cycles-pp.native_flush_tlb_one_user 0.46 ± 42% +0.3 0.79 ± 3% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.59 ± 35% +0.3 0.92 ± 13% perf-profile.children.cycles-pp.flush_tlb_func_common 0.75 ± 35% +0.3 1.08 ± 12% perf-profile.children.cycles-pp.prep_new_page 0.35 ± 39% +0.4 0.71 perf-profile.children.cycles-pp.update_sd_lb_stats 0.84 ± 33% +0.4 1.21 ± 6% perf-profile.children.cycles-pp.scheduler_tick 0.36 ± 40% +0.4 0.73 perf-profile.children.cycles-pp.find_busiest_group 0.64 ± 36% +0.4 1.02 ± 16% perf-profile.children.cycles-pp.flush_tlb_mm_range 0.58 ± 45% +0.4 1.00 ± 4% perf-profile.children.cycles-pp.setlocale 0.97 ± 30% +0.4 1.40 ± 6% perf-profile.children.cycles-pp.tick_sched_handle 0.96 ± 31% +0.4 1.39 ± 6% perf-profile.children.cycles-pp.update_process_times 0.46 ± 39% +0.4 0.89 perf-profile.children.cycles-pp.load_balance 1.01 ± 30% +0.4 1.46 ± 6% perf-profile.children.cycles-pp.tick_sched_timer 1.07 ± 35% +0.5 1.53 ± 12% perf-profile.children.cycles-pp.alloc_pages_vma 0.49 ± 42% +0.5 0.96 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 1.21 ± 26% +0.5 1.70 ± 6% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.55 ± 40% +0.5 1.04 ± 5% perf-profile.children.cycles-pp.schedule 1.11 ± 37% +0.5 1.60 ± 11% perf-profile.children.cycles-pp.get_page_from_freelist 0.93 ± 35% +0.5 1.43 ± 13% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.32 ± 34% +0.5 0.84 ± 46% perf-profile.children.cycles-pp.do_wp_page 1.42 ± 25% +0.5 1.96 ± 4% perf-profile.children.cycles-pp.hrtimer_interrupt 1.31 ± 37% +0.6 1.90 ± 11% perf-profile.children.cycles-pp.__alloc_pages_nodemask 1.11 ± 38% +0.6 1.71 ± 11% perf-profile.children.cycles-pp.do_brk_flags 0.61 ± 43% +0.6 1.23 perf-profile.children.cycles-pp.rwsem_wake 0.55 ± 45% +0.6 1.20 perf-profile.children.cycles-pp.wake_up_q 1.00 ± 44% +0.7 1.66 ± 8% perf-profile.children.cycles-pp.unmap_page_range 0.61 ± 44% +0.7 1.28 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 1.26 ± 36% +0.7 1.93 ± 13% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 1.06 ± 44% +0.7 1.75 ± 8% perf-profile.children.cycles-pp.unmap_vmas 0.73 ± 40% +0.7 1.43 ± 4% perf-profile.children.cycles-pp.__schedule 0.63 ± 41% +0.8 1.45 ± 25% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.76 ± 50% +0.8 1.59 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.95 ± 23% +0.9 2.80 perf-profile.children.cycles-pp.smp_apic_timer_interrupt 2.04 ± 22% +0.9 2.90 perf-profile.children.cycles-pp.apic_timer_interrupt 1.94 ± 35% +0.9 2.81 ± 12% perf-profile.children.cycles-pp.do_anonymous_page 1.58 ± 40% +0.9 2.45 ± 10% perf-profile.children.cycles-pp.tlb_flush_mmu 1.05 ± 34% +0.9 1.93 ± 7% perf-profile.children.cycles-pp.filemap_map_pages 1.59 ± 41% +0.9 2.49 ± 10% perf-profile.children.cycles-pp.tlb_finish_mmu 2.75 ± 36% +1.4 4.12 ± 13% perf-profile.children.cycles-pp.__x64_sys_brk 3.85 ± 35% +2.6 6.48 ± 14% perf-profile.children.cycles-pp.__handle_mm_fault 4.04 ± 35% +2.7 6.79 ± 15% perf-profile.children.cycles-pp.handle_mm_fault 4.97 ± 31% +2.9 7.88 ± 14% perf-profile.children.cycles-pp.page_fault 4.55 ± 35% +3.0 7.55 ± 14% perf-profile.children.cycles-pp.__do_page_fault 4.70 ± 35% +3.0 7.74 ± 14% perf-profile.children.cycles-pp.do_page_fault 15.57 ± 37% +17.0 32.62 ± 12% perf-profile.children.cycles-pp.intel_idle 16.16 ± 35% +17.3 33.48 ± 12% perf-profile.children.cycles-pp.cpuidle_enter_state 16.16 ± 35% +17.3 33.48 ± 12% perf-profile.children.cycles-pp.cpuidle_enter 16.41 ± 34% +17.4 33.85 ± 13% perf-profile.children.cycles-pp.start_secondary 16.53 ± 34% +17.5 34.08 ± 12% perf-profile.children.cycles-pp.do_idle 16.53 ± 34% +17.5 34.08 ± 12% perf-profile.children.cycles-pp.secondary_startup_64 16.53 ± 34% +17.5 34.08 ± 12% perf-profile.children.cycles-pp.cpu_startup_entry 17.02 ± 68% -17.0 0.00 perf-profile.self.cycles-pp.sieve 4.65 ± 68% -4.7 0.00 perf-profile.self.cycles-pp.string_rtns_1 0.07 ± 34% +0.0 0.10 ± 14% perf-profile.self.cycles-pp.do_brk_flags 0.07 ± 34% +0.0 0.10 ± 14% perf-profile.self.cycles-pp.selinux_inode_permission 0.03 ±102% +0.0 0.07 ± 14% perf-profile.self.cycles-pp.__mod_node_page_state 0.08 ± 34% +0.0 0.12 ± 8% perf-profile.self.cycles-pp.update_curr 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.alloc_pages_vma 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.prep_new_page 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.anon_vma_fork 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.unwind_next_frame 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.cap_mmap_addr 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.unmap_single_vma 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.queued_write_lock_slowpath 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.cap_vm_enough_memory 0.03 ±100% +0.0 0.07 ± 14% perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate 0.07 ± 34% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.perf_event_mmap_output 0.01 ±173% +0.0 0.06 ± 16% perf-profile.self.cycles-pp.do_wp_page 0.01 ±173% +0.0 0.06 ± 16% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.08 ± 34% +0.0 0.12 ± 12% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.03 ±105% +0.0 0.08 ± 12% perf-profile.self.cycles-pp.__list_add_valid 0.07 ± 26% +0.0 0.12 ± 8% perf-profile.self.cycles-pp.update_rq_clock 0.06 ± 66% +0.0 0.10 ± 14% perf-profile.self.cycles-pp.sync_regs 0.03 ±102% +0.1 0.08 ± 12% perf-profile.self.cycles-pp.mem_cgroup_try_charge 0.01 ±173% +0.1 0.07 ± 7% perf-profile.self.cycles-pp.wake_q_add 0.00 +0.1 0.05 perf-profile.self.cycles-pp.wake_up_q 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__lru_cache_add 0.00 +0.1 0.05 perf-profile.self.cycles-pp.vma_interval_tree_augment_rotate 0.03 ±105% +0.1 0.09 ± 17% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.03 ±100% +0.1 0.08 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.04 ±102% +0.1 0.09 ± 11% perf-profile.self.cycles-pp.syscall_slow_exit_work 0.03 ±102% +0.1 0.08 ± 5% perf-profile.self.cycles-pp.__schedule 0.04 ±101% +0.1 0.10 ± 15% perf-profile.self.cycles-pp.anon_vma_clone 0.09 ± 41% +0.1 0.14 ± 7% perf-profile.self.cycles-pp.__might_sleep 0.07 ± 72% +0.1 0.12 ± 16% perf-profile.self.cycles-pp.vma_compute_subtree_gap 0.01 ±173% +0.1 0.07 ± 14% perf-profile.self.cycles-pp._cond_resched 0.01 ±173% +0.1 0.07 perf-profile.self.cycles-pp.dup_mm 0.03 ±100% +0.1 0.09 perf-profile.self.cycles-pp.vma_merge 0.03 ±100% +0.1 0.08 ± 5% perf-profile.self.cycles-pp.osq_unlock 0.07 ± 34% +0.1 0.13 ± 30% perf-profile.self.cycles-pp.__count_memcg_events 0.01 ±173% +0.1 0.07 ± 14% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.11 ± 35% +0.1 0.16 ± 9% perf-profile.self.cycles-pp.__vma_adjust 0.00 +0.1 0.06 perf-profile.self.cycles-pp.load_balance 0.11 ± 43% +0.1 0.17 ± 17% perf-profile.self.cycles-pp.handle_mm_fault 0.09 ± 43% +0.1 0.15 ± 13% perf-profile.self.cycles-pp.anon_vma_interval_tree_remove 0.03 ±105% +0.1 0.10 ± 5% perf-profile.self.cycles-pp.put_cred_rcu 0.02 ±173% +0.1 0.08 perf-profile.self.cycles-pp._raw_spin_lock_irq 0.04 ±106% +0.1 0.11 ± 4% perf-profile.self.cycles-pp.unlock_page 0.01 ±173% +0.1 0.08 perf-profile.self.cycles-pp.__alloc_file 0.09 ± 40% +0.1 0.16 ± 9% perf-profile.self.cycles-pp.__d_lookup_rcu 0.12 ± 33% +0.1 0.18 ± 2% perf-profile.self.cycles-pp.copy_page 0.12 ± 40% +0.1 0.19 ± 5% perf-profile.self.cycles-pp.__slab_free 0.07 ± 34% +0.1 0.14 ± 14% perf-profile.self.cycles-pp.page_add_file_rmap 0.06 ± 67% +0.1 0.13 ± 7% perf-profile.self.cycles-pp.__rb_insert_augmented 0.11 ± 38% +0.1 0.18 ± 18% perf-profile.self.cycles-pp.avc_has_perm_noaudit 0.05 ± 67% +0.1 0.12 ± 12% perf-profile.self.cycles-pp.kmem_cache_free 0.17 ± 35% +0.1 0.24 ± 10% perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.12 ± 42% +0.1 0.21 ± 12% perf-profile.self.cycles-pp.perf_iterate_sb 0.13 ± 40% +0.1 0.21 ± 14% perf-profile.self.cycles-pp.find_vma 0.14 ± 38% +0.1 0.22 ± 6% perf-profile.self.cycles-pp.anon_vma_interval_tree_insert 0.11 ± 41% +0.1 0.19 ± 5% perf-profile.self.cycles-pp.vma_interval_tree_remove 0.15 ± 38% +0.1 0.24 ± 14% perf-profile.self.cycles-pp.__do_page_fault 0.06 ± 77% +0.1 0.15 ± 13% perf-profile.self.cycles-pp.refcount_dec_not_one 0.15 ± 40% +0.1 0.24 ± 10% perf-profile.self.cycles-pp.___might_sleep 0.18 ± 42% +0.1 0.28 ± 12% perf-profile.self.cycles-pp.select_task_rq_fair 0.14 ± 41% +0.1 0.24 ± 4% perf-profile.self.cycles-pp.free_pages_and_swap_cache 0.04 ±101% +0.1 0.15 ± 3% perf-profile.self.cycles-pp.idle_cpu 0.14 ± 34% +0.1 0.24 ± 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.16 ± 35% +0.1 0.27 ± 7% perf-profile.self.cycles-pp.up_write 0.15 ± 36% +0.1 0.28 ± 3% perf-profile.self.cycles-pp.down_write 0.11 ± 42% +0.1 0.23 ± 2% perf-profile.self.cycles-pp.try_to_wake_up 0.32 ± 36% +0.2 0.49 ± 13% perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.24 ± 34% +0.2 0.42 ± 10% perf-profile.self.cycles-pp.__handle_mm_fault 0.25 ± 45% +0.2 0.43 ± 9% perf-profile.self.cycles-pp._dl_addr 0.17 ± 41% +0.2 0.35 ± 5% perf-profile.self.cycles-pp.copy_p4d_range 0.31 ± 36% +0.2 0.52 perf-profile.self.cycles-pp._raw_spin_lock 0.35 ± 37% +0.2 0.57 ± 9% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.39 ± 38% +0.2 0.62 ± 6% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.28 ± 44% +0.2 0.51 ± 3% perf-profile.self.cycles-pp.vma_interval_tree_insert 0.27 ± 42% +0.3 0.54 perf-profile.self.cycles-pp.update_sd_lb_stats 0.43 ± 34% +0.3 0.70 ± 11% perf-profile.self.cycles-pp.native_irq_return_iret 0.36 ± 42% +0.3 0.63 ± 3% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.70 ± 36% +0.3 1.01 ± 12% perf-profile.self.cycles-pp.clear_page_erms 0.57 ± 35% +0.3 0.89 ± 12% perf-profile.self.cycles-pp.native_flush_tlb_one_user 0.62 ± 35% +0.4 0.98 ± 5% perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.56 ± 45% +0.4 0.94 ± 8% perf-profile.self.cycles-pp.unmap_page_range 0.89 ± 36% +0.5 1.38 ± 12% perf-profile.self.cycles-pp.prepare_exit_to_usermode 1.07 ± 39% +0.6 1.72 ± 10% perf-profile.self.cycles-pp.do_syscall_64 0.74 ± 33% +0.7 1.40 ± 6% perf-profile.self.cycles-pp.filemap_map_pages 0.75 ± 50% +0.8 1.59 ± 19% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 15.56 ± 37% +17.0 32.59 ± 12% perf-profile.self.cycles-pp.intel_idle *************************************************************************************************** lkp-bdw-ep2: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/2000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ep2/fork_test/aim7/0xb000036 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:4 -7% 0:2 perf-profile.children.cycles-pp.error_entry %stddev %change %stddev \ | \ 330032 ± 5% +8.1% 356656 ± 5% meminfo.DirectMap4k 0.00 ±173% +5700.0% 0.15 ± 51% turbostat.Pkg%pc6 20.03 +1.4% 20.32 turbostat.RAMWatt 1552 ±106% +185.7% 4436 numa-vmstat.node0.nr_inactive_anon 1552 ±106% +185.7% 4436 numa-vmstat.node0.nr_zone_inactive_anon 15713 ± 57% -60.7% 6171 numa-vmstat.node1.nr_page_table_pages 1703 ± 8% +9.7% 1869 ± 5% slabinfo.dmaengine-unmap-16.active_objs 1703 ± 8% +9.7% 1869 ± 5% slabinfo.dmaengine-unmap-16.num_objs 7189 +8.8% 7820 ± 6% slabinfo.files_cache.active_objs 7189 +8.8% 7820 ± 6% slabinfo.files_cache.num_objs 0.51 ± 7% +0.1 0.56 perf-stat.i.branch-miss-rate% 15.18 +0.3 15.49 perf-stat.i.cache-miss-rate% 36584641 ± 7% +5.9% 38751965 perf-stat.i.dTLB-load-misses 15.37 +0.3 15.69 perf-stat.overall.cache-miss-rate% 36528779 ± 7% +5.9% 38692724 perf-stat.ps.dTLB-load-misses 121293 +1.8% 123486 proc-vmstat.nr_active_anon 81294 +1.9% 82802 proc-vmstat.nr_anon_pages 31379 -7.4% 29054 ± 3% proc-vmstat.nr_page_table_pages 121293 +1.8% 123486 proc-vmstat.nr_zone_active_anon 1732 ±130% -98.2% 31.00 ±100% proc-vmstat.numa_hint_faults_local 6396 ±104% +179.6% 17884 numa-meminfo.node0.Inactive 6239 ±105% +185.4% 17806 ± 2% numa-meminfo.node0.Inactive(anon) 12314 ± 16% +29.6% 15963 numa-meminfo.node0.Mapped 91839 ± 25% +28.3% 117810 ± 10% numa-meminfo.node1.AnonHugePages 14708 ± 12% -23.6% 11243 ± 2% numa-meminfo.node1.Mapped 62843 ± 57% -60.7% 24674 numa-meminfo.node1.PageTables 8540823 ± 15% -58.5% 3542528 ± 79% sched_debug.cfs_rq:/.MIN_vruntime.max 960.57 ± 5% -17.2% 795.67 ± 3% sched_debug.cfs_rq:/.exec_clock.stddev 4.34 ± 6% +9.3% 4.75 ± 3% sched_debug.cfs_rq:/.load_avg.min 8540823 ± 15% -58.5% 3542528 ± 79% sched_debug.cfs_rq:/.max_vruntime.max 196336 ± 30% +74.9% 343381 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev 170.62 ± 19% +32.7% 226.38 sched_debug.cfs_rq:/.removed.util_avg.max -218268 -209.2% 238323 ± 11% sched_debug.cfs_rq:/.spread0.avg 213438 ± 59% +314.5% 884735 sched_debug.cfs_rq:/.spread0.max -650962 -44.1% -363711 sched_debug.cfs_rq:/.spread0.min 196606 ± 30% +74.6% 343259 ± 7% sched_debug.cfs_rq:/.spread0.stddev 133.29 ± 19% +20.9% 161.21 ± 9% sched_debug.cpu.cpu_load[2].max 67.65 ± 10% +10.0% 74.42 ± 6% sched_debug.cpu.cpu_load[4].max 5021 ± 7% -20.8% 3978 ± 8% sched_debug.cpu.nr_switches.stddev 613.25 ± 39% -35.6% 395.00 interrupts.105:IR-PCI-MSI.1574916-edge.eth1-TxRx-4 538.00 ± 51% +36.2% 733.00 ± 46% interrupts.109:IR-PCI-MSI.1574920-edge.eth1-TxRx-8 394.00 ± 6% +6.1% 418.00 ± 5% interrupts.111:IR-PCI-MSI.1574922-edge.eth1-TxRx-10 345.00 ± 2% +24.1% 428.00 ± 18% interrupts.146:IR-PCI-MSI.1574957-edge.eth1-TxRx-45 631.25 ± 4% +25.1% 790.00 ± 5% interrupts.CPU1.TLB:TLB_shootdowns 394.00 ± 6% +6.1% 418.00 ± 5% interrupts.CPU10.111:IR-PCI-MSI.1574922-edge.eth1-TxRx-10 649.25 ± 10% +12.4% 729.50 ± 4% interrupts.CPU12.TLB:TLB_shootdowns 601.75 ± 4% +32.6% 798.00 ± 7% interrupts.CPU13.TLB:TLB_shootdowns 629.00 ± 9% +24.1% 780.50 interrupts.CPU14.TLB:TLB_shootdowns 601.25 ± 16% +27.7% 767.50 ± 15% interrupts.CPU15.TLB:TLB_shootdowns 635.25 ± 8% +15.7% 735.00 ± 6% interrupts.CPU17.TLB:TLB_shootdowns 604.50 ± 13% +18.3% 715.00 ± 2% interrupts.CPU18.TLB:TLB_shootdowns 635.25 ± 10% +15.7% 735.00 ± 6% interrupts.CPU19.TLB:TLB_shootdowns 607.25 ± 8% +27.7% 775.50 interrupts.CPU2.TLB:TLB_shootdowns 686.50 ± 9% +18.2% 811.50 ± 16% interrupts.CPU21.TLB:TLB_shootdowns 5748 ± 2% -31.4% 3942 ± 32% interrupts.CPU22.NMI:Non-maskable_interrupts 5748 ± 2% -31.4% 3942 ± 32% interrupts.CPU22.PMI:Performance_monitoring_interrupts 651.25 ± 11% +17.2% 763.50 ± 10% interrupts.CPU22.TLB:TLB_shootdowns 5735 ± 3% -31.3% 3941 ± 32% interrupts.CPU23.NMI:Non-maskable_interrupts 5735 ± 3% -31.3% 3941 ± 32% interrupts.CPU23.PMI:Performance_monitoring_interrupts 5679 ± 2% -30.6% 3942 ± 31% interrupts.CPU24.NMI:Non-maskable_interrupts 5679 ± 2% -30.6% 3942 ± 31% interrupts.CPU24.PMI:Performance_monitoring_interrupts 647.50 ± 6% +35.5% 877.50 ± 8% interrupts.CPU24.TLB:TLB_shootdowns 621.00 ± 3% +34.5% 835.50 ± 4% interrupts.CPU26.TLB:TLB_shootdowns 597.75 ± 9% +26.8% 758.00 ± 5% interrupts.CPU27.TLB:TLB_shootdowns 587.25 ± 7% +19.2% 700.00 ± 6% interrupts.CPU37.TLB:TLB_shootdowns 613.25 ± 39% -35.6% 395.00 interrupts.CPU4.105:IR-PCI-MSI.1574916-edge.eth1-TxRx-4 668.25 ± 11% +19.5% 798.50 ± 4% interrupts.CPU40.TLB:TLB_shootdowns 646.25 ± 7% +11.5% 720.50 ± 9% interrupts.CPU43.TLB:TLB_shootdowns 345.00 ± 2% +24.1% 428.00 ± 18% interrupts.CPU45.146:IR-PCI-MSI.1574957-edge.eth1-TxRx-45 587.00 ± 13% +18.2% 694.00 ± 10% interrupts.CPU45.TLB:TLB_shootdowns 616.75 ± 3% +16.9% 721.00 ± 9% interrupts.CPU49.TLB:TLB_shootdowns 680.00 ± 9% +24.9% 849.50 ± 2% interrupts.CPU50.TLB:TLB_shootdowns 661.75 ± 11% +24.1% 821.00 ± 2% interrupts.CPU6.TLB:TLB_shootdowns 606.50 ± 13% +25.2% 759.50 ± 7% interrupts.CPU62.TLB:TLB_shootdowns 611.25 ± 11% +18.2% 722.50 interrupts.CPU63.TLB:TLB_shootdowns 576.50 ± 10% +31.7% 759.50 ± 8% interrupts.CPU64.TLB:TLB_shootdowns 166.25 ±173% +304.5% 672.50 interrupts.CPU65.168:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 5687 ± 4% -30.2% 3969 ± 30% interrupts.CPU66.NMI:Non-maskable_interrupts 5687 ± 4% -30.2% 3969 ± 30% interrupts.CPU66.PMI:Performance_monitoring_interrupts 5683 ± 3% -30.7% 3938 ± 31% interrupts.CPU67.NMI:Non-maskable_interrupts 5683 ± 3% -30.7% 3938 ± 31% interrupts.CPU67.PMI:Performance_monitoring_interrupts 5706 ± 2% -30.7% 3955 ± 32% interrupts.CPU68.NMI:Non-maskable_interrupts 5706 ± 2% -30.7% 3955 ± 32% interrupts.CPU68.PMI:Performance_monitoring_interrupts 610.50 ± 10% +17.3% 716.00 ± 5% interrupts.CPU68.TLB:TLB_shootdowns 5624 ± 2% -29.5% 3967 ± 32% interrupts.CPU70.NMI:Non-maskable_interrupts 5624 ± 2% -29.5% 3967 ± 32% interrupts.CPU70.PMI:Performance_monitoring_interrupts 562.75 ± 15% +14.5% 644.50 ± 11% interrupts.CPU71.TLB:TLB_shootdowns 5672 ± 3% -30.2% 3957 ± 30% interrupts.CPU74.NMI:Non-maskable_interrupts 5672 ± 3% -30.2% 3957 ± 30% interrupts.CPU74.PMI:Performance_monitoring_interrupts 606.00 ± 8% +25.5% 760.50 interrupts.CPU76.TLB:TLB_shootdowns 5739 ± 2% -31.6% 3923 ± 30% interrupts.CPU79.NMI:Non-maskable_interrupts 5739 ± 2% -31.6% 3923 ± 30% interrupts.CPU79.PMI:Performance_monitoring_interrupts 538.00 ± 51% +36.2% 733.00 ± 46% interrupts.CPU8.109:IR-PCI-MSI.1574920-edge.eth1-TxRx-8 655.25 ± 7% +17.1% 767.00 ± 13% interrupts.CPU81.TLB:TLB_shootdowns 575.75 ± 11% +47.8% 851.00 ± 9% interrupts.CPU82.TLB:TLB_shootdowns 684.75 ± 4% +16.8% 800.00 ± 6% interrupts.CPU84.TLB:TLB_shootdowns 30.75 ± 2% -3.6 27.16 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64 29.27 ± 3% -3.6 25.70 ± 3% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 29.82 ± 2% -3.6 26.24 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 29.79 ± 2% -3.6 26.21 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 30.37 ± 2% -3.5 26.83 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 30.37 ± 2% -3.5 26.83 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 30.37 ± 2% -3.5 26.84 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 0.71 ± 3% +0.1 0.84 perf-profile.calltrace.cycles-pp.do_wp_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.63 ± 4% +0.1 0.77 perf-profile.calltrace.cycles-pp.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault.__do_page_fault 0.83 ± 10% +0.2 1.01 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4 0.98 ± 9% +0.2 1.17 ± 2% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.40 ± 57% +0.2 0.60 perf-profile.calltrace.cycles-pp.release_task.wait_consider_task.do_wait.kernel_wait4.__do_sys_wait4 0.42 ± 57% +0.2 0.64 perf-profile.calltrace.cycles-pp.wait_consider_task.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 1.92 ± 5% +0.3 2.17 ± 3% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 1.95 ± 5% +0.3 2.22 ± 3% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 2.12 ± 5% +0.3 2.41 ± 3% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 2.17 ± 5% +0.3 2.46 ± 3% perf-profile.calltrace.cycles-pp.page_fault 2.15 ± 5% +0.3 2.44 ± 3% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 1.81 ± 8% +0.3 2.11 perf-profile.calltrace.cycles-pp.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait 1.99 ± 8% +0.3 2.30 perf-profile.calltrace.cycles-pp.wait 1.78 ± 8% +0.3 2.09 perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.85 ± 8% +0.3 2.16 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.wait 1.84 ± 8% +0.3 2.16 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait 1.80 ± 8% +0.3 2.11 perf-profile.calltrace.cycles-pp.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait 0.27 ±100% +0.3 0.60 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 0.28 ±100% +0.3 0.61 perf-profile.calltrace.cycles-pp.ret_from_fork 0.28 ±100% +0.3 0.61 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 0.29 ±100% +0.3 0.64 perf-profile.calltrace.cycles-pp.__vmalloc_node_range.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.13 ±173% +0.4 0.56 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 0.13 ±173% +0.4 0.56 perf-profile.calltrace.cycles-pp.__get_vm_area_node.__vmalloc_node_range.copy_process._do_fork.do_syscall_64 0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.__vunmap.free_work.process_one_work.worker_thread.kthread 0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.alloc_vmap_area.__get_vm_area_node.__vmalloc_node_range.copy_process._do_fork 0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.free_work.process_one_work.worker_thread.kthread.ret_from_fork 15.78 +0.7 16.50 perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.anon_vma_clone.anon_vma_fork 16.55 +0.7 17.28 perf-profile.calltrace.cycles-pp.down_write.anon_vma_clone.anon_vma_fork.dup_mm.copy_process 16.44 +0.7 17.17 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.anon_vma_clone.anon_vma_fork.dup_mm 20.26 +0.8 21.02 perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_anon_vmas.free_pgtables 20.90 +0.8 21.75 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_anon_vmas.free_pgtables.exit_mmap 21.04 +0.8 21.88 perf-profile.calltrace.cycles-pp.down_write.unlink_anon_vmas.free_pgtables.exit_mmap.mmput 17.93 +0.8 18.77 perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mm.copy_process._do_fork 25.06 ± 2% +1.0 26.04 perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mm.copy_process._do_fork.do_syscall_64 26.77 ± 2% +1.1 27.86 perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe 32.33 +1.3 33.63 perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 32.35 +1.3 33.65 perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 28.27 +1.3 29.60 perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 28.61 +1.4 29.97 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork.fork_test 28.61 +1.4 29.97 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork.fork_test 28.59 +1.4 29.95 perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork.fork_test 29.46 +1.4 30.87 perf-profile.calltrace.cycles-pp.__libc_fork.fork_test 33.46 +1.4 34.91 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 33.46 +1.4 34.91 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.45 +1.5 34.91 perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.45 +1.5 34.91 perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.45 +1.5 34.91 perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 29.95 +1.5 31.41 perf-profile.calltrace.cycles-pp.fork_test 29.64 ± 3% -3.6 26.01 ± 3% perf-profile.children.cycles-pp.intel_idle 30.20 ± 2% -3.6 26.56 ± 3% perf-profile.children.cycles-pp.cpuidle_enter 30.19 ± 2% -3.6 26.56 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 30.75 ± 2% -3.6 27.16 ± 3% perf-profile.children.cycles-pp.do_idle 30.75 ± 2% -3.6 27.16 ± 3% perf-profile.children.cycles-pp.secondary_startup_64 30.75 ± 2% -3.6 27.16 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 30.37 ± 2% -3.5 26.84 ± 3% perf-profile.children.cycles-pp.start_secondary 0.38 ± 6% -0.1 0.33 ± 4% perf-profile.children.cycles-pp.start_kernel 0.06 ± 7% +0.0 0.07 perf-profile.children.cycles-pp.security_vm_enough_memory_mm 0.10 ± 5% +0.0 0.11 perf-profile.children.cycles-pp.___perf_sw_event 0.07 ± 7% +0.0 0.08 perf-profile.children.cycles-pp.put_task_stack 0.09 ± 4% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.rwsem_mark_wake 0.11 ± 7% +0.0 0.13 perf-profile.children.cycles-pp.sched_move_task 0.19 ± 3% +0.0 0.21 perf-profile.children.cycles-pp.dequeue_task_fair 0.18 ± 5% +0.0 0.20 perf-profile.children.cycles-pp.__wake_up_common_lock 0.06 +0.0 0.08 perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.16 ± 2% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.dequeue_entity 0.18 ± 6% +0.0 0.21 perf-profile.children.cycles-pp.unlink_file_vma 0.18 ± 6% +0.0 0.20 perf-profile.children.cycles-pp.__wake_up_common 0.15 ± 3% +0.0 0.18 ± 5% perf-profile.children.cycles-pp.flush_tlb_func_common 0.14 ± 5% +0.0 0.17 ± 3% perf-profile.children.cycles-pp.native_flush_tlb_one_user 0.22 ± 4% +0.0 0.25 ± 4% perf-profile.children.cycles-pp.schedule_idle 0.18 ± 2% +0.0 0.21 ± 4% perf-profile.children.cycles-pp.ptep_clear_flush 0.15 ± 2% +0.0 0.17 ± 2% perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.21 ± 3% +0.0 0.24 ± 4% perf-profile.children.cycles-pp.flush_tlb_mm_range 0.14 ± 8% +0.0 0.17 ± 3% perf-profile.children.cycles-pp.finish_task_switch 0.34 ± 4% +0.0 0.37 ± 2% perf-profile.children.cycles-pp.find_busiest_group 0.53 ± 2% +0.0 0.56 perf-profile.children.cycles-pp.schedule 0.45 ± 3% +0.0 0.48 ± 2% perf-profile.children.cycles-pp.load_balance 0.36 ± 6% +0.0 0.39 ± 2% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.27 ± 2% +0.0 0.30 ± 4% perf-profile.children.cycles-pp.up_write 0.22 ± 9% +0.0 0.26 perf-profile.children.cycles-pp.schedule_tail 0.00 +0.1 0.05 perf-profile.children.cycles-pp.__list_add_valid 0.00 +0.1 0.05 perf-profile.children.cycles-pp.queue_work_on 0.02 ±173% +0.1 0.08 perf-profile.children.cycles-pp.__lru_cache_add 0.38 ± 5% +0.1 0.45 perf-profile.children.cycles-pp.copy_page 0.39 ± 6% +0.1 0.46 perf-profile.children.cycles-pp.copy_user_highpage 0.98 ± 3% +0.1 1.04 perf-profile.children.cycles-pp.__schedule 0.51 ± 9% +0.1 0.60 perf-profile.children.cycles-pp.release_task 0.54 ± 9% +0.1 0.64 perf-profile.children.cycles-pp.wait_consider_task 1.04 ± 4% +0.1 1.14 perf-profile.children.cycles-pp.rwsem_wake 0.37 ± 19% +0.1 0.48 perf-profile.children.cycles-pp.find_vmap_area 0.93 ± 3% +0.1 1.04 perf-profile.children.cycles-pp.wake_up_q 0.41 ± 16% +0.1 0.53 perf-profile.children.cycles-pp.alloc_vmap_area 0.44 ± 15% +0.1 0.56 perf-profile.children.cycles-pp.__get_vm_area_node 0.41 ± 17% +0.1 0.53 perf-profile.children.cycles-pp.__vunmap 0.49 ± 14% +0.1 0.61 perf-profile.children.cycles-pp.kthread 0.48 ± 15% +0.1 0.60 perf-profile.children.cycles-pp.worker_thread 0.43 ± 15% +0.1 0.56 perf-profile.children.cycles-pp.process_one_work 0.41 ± 17% +0.1 0.53 perf-profile.children.cycles-pp.free_work 1.10 ± 5% +0.1 1.23 perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.51 ± 14% +0.1 0.64 perf-profile.children.cycles-pp.__vmalloc_node_range 1.28 ± 4% +0.1 1.42 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 1.08 ± 4% +0.1 1.22 perf-profile.children.cycles-pp.try_to_wake_up 0.93 ± 4% +0.2 1.08 ± 2% perf-profile.children.cycles-pp.wp_page_copy 0.74 ± 12% +0.2 0.91 perf-profile.children.cycles-pp.ret_from_fork 1.05 ± 4% +0.2 1.22 ± 2% perf-profile.children.cycles-pp.do_wp_page 0.98 ± 9% +0.2 1.17 ± 2% perf-profile.children.cycles-pp.queued_read_lock_slowpath 1.00 ± 10% +0.2 1.19 ± 2% perf-profile.children.cycles-pp.queued_write_lock_slowpath 1.08 ± 14% +0.2 1.33 perf-profile.children.cycles-pp._raw_spin_lock 2.01 ± 8% +0.3 2.32 perf-profile.children.cycles-pp.wait 1.81 ± 8% +0.3 2.12 perf-profile.children.cycles-pp.__do_sys_wait4 1.78 ± 8% +0.3 2.09 perf-profile.children.cycles-pp.do_wait 1.80 ± 8% +0.3 2.11 perf-profile.children.cycles-pp.kernel_wait4 3.58 ± 9% +0.7 4.27 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 17.93 +0.9 18.78 perf-profile.children.cycles-pp.anon_vma_clone 25.07 ± 2% +1.0 26.04 perf-profile.children.cycles-pp.anon_vma_fork 26.78 ± 2% +1.1 27.87 perf-profile.children.cycles-pp.dup_mm 32.33 +1.3 33.63 perf-profile.children.cycles-pp.exit_mmap 32.35 +1.3 33.65 perf-profile.children.cycles-pp.mmput 28.27 +1.3 29.61 perf-profile.children.cycles-pp.copy_process 28.59 +1.4 29.95 perf-profile.children.cycles-pp._do_fork 29.50 +1.4 30.92 perf-profile.children.cycles-pp.__libc_fork 33.51 +1.5 34.97 perf-profile.children.cycles-pp.do_exit 33.52 +1.5 34.98 perf-profile.children.cycles-pp.__x64_sys_exit_group 33.52 +1.5 34.98 perf-profile.children.cycles-pp.do_group_exit 30.00 +1.5 31.47 perf-profile.children.cycles-pp.fork_test 64.03 +3.1 67.16 perf-profile.children.cycles-pp.do_syscall_64 64.04 +3.1 67.16 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 29.63 ± 3% -3.6 25.99 ± 3% perf-profile.self.cycles-pp.intel_idle 0.14 ± 5% +0.0 0.17 ± 3% perf-profile.self.cycles-pp.native_flush_tlb_one_user 0.16 ± 11% +0.0 0.19 ± 5% perf-profile.self.cycles-pp.unlock_page 0.27 ± 3% +0.0 0.30 ± 3% perf-profile.self.cycles-pp.up_write 1.12 +0.0 1.17 perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__list_add_valid 0.38 ± 6% +0.1 0.44 perf-profile.self.cycles-pp.copy_page 0.92 ± 6% +0.1 1.04 perf-profile.self.cycles-pp.rwsem_spin_on_owner 3.56 ± 9% +0.7 4.25 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath *************************************************************************************************** lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory ========================================================================================= compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/8000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ex1/disk_src/aim7/0xb000036 commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- %stddev %change %stddev \ | \ 38518 +1.2% 38996 proc-vmstat.nr_slab_reclaimable 233358 ± 29% -83.8% 37823 ± 4% numa-numastat.node0.local_node 233390 ± 29% -70.5% 68854 ± 2% numa-numastat.node0.numa_hit 34.25 ±108% +90501.5% 31031 numa-numastat.node0.other_node 62521 ± 58% +441.7% 338669 numa-numastat.node2.local_node 93578 ± 38% +288.4% 363493 ± 3% numa-numastat.node2.numa_hit 31070 -79.7% 6300 ± 98% numa-numastat.node3.other_node 60884 ± 6% -8.3% 55851 slabinfo.shmem_inode_cache.active_objs 1327 ± 6% -8.4% 1216 slabinfo.shmem_inode_cache.active_slabs 61092 ± 6% -8.4% 55988 slabinfo.shmem_inode_cache.num_objs 1327 ± 6% -8.4% 1216 slabinfo.shmem_inode_cache.num_slabs 771.50 ± 17% -21.4% 606.50 ± 5% slabinfo.skbuff_ext_cache.active_objs 771.50 ± 17% -21.4% 606.50 ± 5% slabinfo.skbuff_ext_cache.num_objs 3.357e+08 -2.1% 3.287e+08 perf-stat.i.cache-references 6168 ± 12% +24.6% 7686 ± 6% perf-stat.i.cycles-between-cache-misses 0.40 +3.1% 0.41 ± 2% perf-stat.i.ipc 1343568 ± 10% +21.3% 1630157 ± 6% perf-stat.i.node-loads 17.33 -4.9% 16.48 perf-stat.overall.MPKI 2.39 -3.4% 2.30 perf-stat.overall.cpi 0.42 +3.6% 0.43 perf-stat.overall.ipc 3.315e+08 -2.1% 3.244e+08 perf-stat.ps.cache-references 7455 +1.4% 7560 perf-stat.ps.minor-faults 1325962 ± 10% +21.2% 1607636 ± 6% perf-stat.ps.node-loads 7454 +1.4% 7559 perf-stat.ps.page-faults 48.50 ± 7% +187.6% 139.50 ± 62% interrupts.145:PCI-MSI.1574927-edge.eth3-TxRx-15 1528 ± 18% +67.9% 2565 ± 35% interrupts.CPU1.RES:Rescheduling_interrupts 1355 ± 8% +20.5% 1632 ± 7% interrupts.CPU11.RES:Rescheduling_interrupts 1058 ± 48% +154.3% 2690 ± 49% interrupts.CPU138.RES:Rescheduling_interrupts 1571 ± 16% +19.7% 1881 ± 11% interrupts.CPU144.RES:Rescheduling_interrupts 1438 ± 17% +18.5% 1704 ± 11% interrupts.CPU145.RES:Rescheduling_interrupts 1440 ± 14% +19.7% 1724 ± 4% interrupts.CPU149.RES:Rescheduling_interrupts 48.50 ± 7% +187.6% 139.50 ± 62% interrupts.CPU15.145:PCI-MSI.1574927-edge.eth3-TxRx-15 1412 ± 14% +11.5% 1574 ± 11% interrupts.CPU155.RES:Rescheduling_interrupts 1407 ± 14% +15.8% 1629 ± 10% interrupts.CPU157.RES:Rescheduling_interrupts 1334 ± 3% +10.2% 1470 interrupts.CPU27.RES:Rescheduling_interrupts 1393 ± 5% +10.5% 1539 ± 7% interrupts.CPU29.RES:Rescheduling_interrupts 1122 ± 22% +354.4% 5099 ± 73% interrupts.CPU34.RES:Rescheduling_interrupts 1132 ± 21% +299.5% 4525 ± 67% interrupts.CPU35.RES:Rescheduling_interrupts 2028 ± 49% -62.6% 758.00 ± 70% interrupts.CPU40.RES:Rescheduling_interrupts 1159 ± 35% +148.2% 2878 ± 48% interrupts.CPU45.RES:Rescheduling_interrupts 1356 ± 16% +19.3% 1618 ± 10% interrupts.CPU71.RES:Rescheduling_interrupts 1.03 ±100% +1.5 2.56 ± 11% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit 0.49 ±173% +2.1 2.56 ± 11% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit 0.98 ±173% +2.7 3.71 ± 22% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 0.98 ±173% +2.7 3.71 ± 22% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.49 ±173% +3.2 3.71 ± 22% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.49 ±173% +3.2 3.71 ± 22% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.00 +3.7 3.71 ± 22% perf-profile.calltrace.cycles-pp.seq_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.98 ±173% +4.2 5.13 ± 11% perf-profile.calltrace.cycles-pp.read 1.03 ±100% +1.5 2.56 ± 11% perf-profile.children.cycles-pp.unmap_vmas 0.49 ±173% +2.1 2.56 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.49 ±173% +2.1 2.56 ± 11% perf-profile.children.cycles-pp.unmap_page_range 0.00 +2.6 2.56 ± 11% perf-profile.children.cycles-pp.mutex_unlock 18.25 ± 10% +3.4 21.66 ± 5% perf-profile.children.cycles-pp.do_group_exit 18.25 ± 10% +3.4 21.66 ± 5% perf-profile.children.cycles-pp.do_exit 0.98 ±173% +4.2 5.13 ± 11% perf-profile.children.cycles-pp.read 0.49 ±173% +2.1 2.56 ± 11% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.00 +2.6 2.56 ± 11% perf-profile.self.cycles-pp.mutex_unlock 3235 ± 3% -59.2% 1318 ± 98% sched_debug.cfs_rq:/.exec_clock.max 10.08 ± 24% +109.8% 21.14 ± 30% sched_debug.cfs_rq:/.load_avg.avg 440.50 ± 11% +75.9% 774.75 ± 32% sched_debug.cfs_rq:/.load_avg.max 51.31 ± 21% +92.3% 98.66 ± 34% sched_debug.cfs_rq:/.load_avg.stddev 416493 ± 3% -69.9% 125500 ± 71% sched_debug.cfs_rq:/.min_vruntime.max 1.36 ±173% +360.0% 6.26 ± 26% sched_debug.cfs_rq:/.removed.load_avg.avg 122.00 ±173% +522.5% 759.50 ± 34% sched_debug.cfs_rq:/.removed.load_avg.max 12.77 ±173% +437.4% 68.61 ± 30% sched_debug.cfs_rq:/.removed.load_avg.stddev 62.75 ±173% +360.3% 288.79 ± 26% sched_debug.cfs_rq:/.removed.runnable_sum.avg 5623 ±173% +522.8% 35019 ± 34% sched_debug.cfs_rq:/.removed.runnable_sum.max 588.63 ±173% +437.5% 3164 ± 30% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 0.55 ±173% +473.9% 3.14 ± 26% sched_debug.cfs_rq:/.removed.util_avg.avg 61.62 ±173% +517.8% 380.75 ± 34% sched_debug.cfs_rq:/.removed.util_avg.max 5.31 ±173% +547.6% 34.40 ± 30% sched_debug.cfs_rq:/.removed.util_avg.stddev 0.79 ± 14% +24.7% 0.99 ± 8% sched_debug.cfs_rq:/.runnable_load_avg.avg 288696 ± 13% -90.7% 26906 ± 96% sched_debug.cfs_rq:/.spread0.max 135.02 ± 3% +32.0% 178.17 ± 9% sched_debug.cfs_rq:/.util_avg.avg 6.14 ± 20% +96.1% 12.04 ± 40% sched_debug.cfs_rq:/.util_est_enqueued.avg 282.25 +40.7% 397.00 ± 24% sched_debug.cfs_rq:/.util_est_enqueued.max 36.69 ± 11% +72.8% 63.38 ± 33% sched_debug.cfs_rq:/.util_est_enqueued.stddev 143.02 ± 6% -65.3% 49.57 ± 83% sched_debug.cpu.clock.stddev 143.02 ± 6% -65.3% 49.57 ± 83% sched_debug.cpu.clock_task.stddev 85.62 ± 18% -35.2% 55.50 ± 4% sched_debug.cpu.cpu_load[4].max 6.55 ± 16% -24.0% 4.98 sched_debug.cpu.cpu_load[4].stddev 0.00 ± 6% -61.0% 0.00 ± 62% sched_debug.cpu.next_balance.stddev 212392 ± 71% -71.1% 61441 ± 5% numa-meminfo.node0.Active 212340 ± 71% -71.1% 61441 ± 5% numa-meminfo.node0.Active(anon) 25308 ± 36% -96.0% 1024 ±100% numa-meminfo.node0.AnonHugePages 203492 ± 74% -69.7% 61607 ± 5% numa-meminfo.node0.AnonPages 96358 ± 8% -66.7% 32077 ± 3% numa-meminfo.node0.KReclaimable 1236693 ± 41% -36.1% 790523 numa-meminfo.node0.MemUsed 96358 ± 8% -66.7% 32077 ± 3% numa-meminfo.node0.SReclaimable 157686 ± 55% -45.7% 85674 numa-meminfo.node0.SUnreclaim 28625 ± 3% -30.4% 19912 numa-meminfo.node0.Shmem 254045 ± 36% -53.6% 117751 numa-meminfo.node0.Slab 1162 ± 27% -76.0% 279.00 ± 81% numa-meminfo.node1.Inactive 1069 ± 25% -82.3% 189.50 ± 72% numa-meminfo.node1.Inactive(anon) 1110 ± 23% -49.7% 558.00 ± 7% numa-meminfo.node1.Shmem 107470 ± 48% +329.4% 461499 ± 6% numa-meminfo.node2.Active 107469 ± 48% +329.3% 461407 ± 6% numa-meminfo.node2.Active(anon) 107671 ± 48% +325.7% 458377 ± 5% numa-meminfo.node2.AnonPages 249548 +8.9% 271751 ± 7% numa-meminfo.node2.FilePages 15548 ± 7% +247.4% 54007 ± 12% numa-meminfo.node2.KReclaimable 5609 ± 2% +2180.6% 127917 numa-meminfo.node2.KernelStack 589997 ± 8% +214.5% 1855314 ± 5% numa-meminfo.node2.MemUsed 2892 ± 57% +9676.0% 282747 ± 8% numa-meminfo.node2.PageTables 15548 ± 7% +247.4% 54007 ± 12% numa-meminfo.node2.SReclaimable 59530 ± 6% +379.2% 285257 numa-meminfo.node2.SUnreclaim 1011 ± 19% +310.3% 4149 ± 53% numa-meminfo.node2.Shmem 75078 ± 5% +351.9% 339265 ± 2% numa-meminfo.node2.Slab 248134 +7.5% 266686 ± 6% numa-meminfo.node2.Unevictable 578.25 ± 71% +132.0% 1341 ± 44% numa-meminfo.node3.Inactive(anon) 23848 ± 39% +116.8% 51695 ± 9% numa-meminfo.node3.KReclaimable 209891 ± 56% -99.0% 2018 ± 35% numa-meminfo.node3.PageTables 23848 ± 39% +116.8% 51695 ± 9% numa-meminfo.node3.SReclaimable 1612 ± 48% +377.7% 7701 ± 26% numa-meminfo.node3.Shmem 52988 ± 71% -70.9% 15424 ± 5% numa-vmstat.node0.nr_active_anon 50771 ± 74% -69.5% 15467 ± 5% numa-vmstat.node0.nr_anon_pages 7156 ± 3% -30.4% 4978 numa-vmstat.node0.nr_shmem 24067 ± 8% -66.7% 8025 ± 3% numa-vmstat.node0.nr_slab_reclaimable 39378 ± 55% -45.5% 21444 numa-vmstat.node0.nr_slab_unreclaimable 52988 ± 71% -70.9% 15424 ± 5% numa-vmstat.node0.nr_zone_active_anon 674305 ± 19% -23.6% 515357 ± 3% numa-vmstat.node0.numa_hit 672585 ± 19% -28.0% 484049 ± 3% numa-vmstat.node0.numa_local 1719 ± 3% +1720.5% 31307 numa-vmstat.node0.numa_other 267.25 ± 25% -82.4% 47.00 ± 72% numa-vmstat.node1.nr_inactive_anon 1267 ± 4% +3.3% 1309 ± 4% numa-vmstat.node1.nr_mapped 277.50 ± 23% -49.9% 139.00 ± 7% numa-vmstat.node1.nr_shmem 267.25 ± 25% -82.4% 47.00 ± 72% numa-vmstat.node1.nr_zone_inactive_anon 26830 ± 48% +330.4% 115485 ± 5% numa-vmstat.node2.nr_active_anon 26878 ± 48% +326.7% 114684 ± 5% numa-vmstat.node2.nr_anon_pages 62390 +8.9% 67941 ± 7% numa-vmstat.node2.nr_file_pages 5612 ± 2% +2183.0% 128119 numa-vmstat.node2.nr_kernel_stack 1238 +6.2% 1315 ± 5% numa-vmstat.node2.nr_mapped 719.50 ± 57% +9745.0% 70835 ± 8% numa-vmstat.node2.nr_page_table_pages 253.25 ± 19% +311.1% 1041 ± 53% numa-vmstat.node2.nr_shmem 3889 ± 7% +247.5% 13517 ± 12% numa-vmstat.node2.nr_slab_reclaimable 14889 ± 6% +379.5% 71397 numa-vmstat.node2.nr_slab_unreclaimable 62037 +7.5% 66670 ± 6% numa-vmstat.node2.nr_unevictable 26830 ± 48% +330.4% 115485 ± 5% numa-vmstat.node2.nr_zone_active_anon 62037 +7.5% 66670 ± 6% numa-vmstat.node2.nr_zone_unevictable 298653 ± 7% +95.0% 582384 numa-vmstat.node2.numa_hit 185598 ± 11% +155.9% 474964 numa-vmstat.node2.numa_local 144.25 ± 71% +132.2% 335.00 ± 45% numa-vmstat.node3.nr_inactive_anon 52258 ± 56% -99.0% 504.50 ± 36% numa-vmstat.node3.nr_page_table_pages 403.50 ± 48% +376.8% 1924 ± 26% numa-vmstat.node3.nr_shmem 5961 ± 39% +116.9% 12927 ± 9% numa-vmstat.node3.nr_slab_reclaimable 144.25 ± 71% +132.2% 335.00 ± 45% numa-vmstat.node3.nr_zone_inactive_anon 112785 -21.2% 88887 ± 6% numa-vmstat.node3.numa_other 32578 ± 10% -13.9% 28035 softirqs.CPU10.TIMER 31748 ± 9% -14.3% 27201 softirqs.CPU100.TIMER 31646 ± 8% -12.3% 27766 softirqs.CPU101.TIMER 31008 ± 7% -12.7% 27078 softirqs.CPU102.TIMER 31788 ± 10% -14.9% 27056 softirqs.CPU103.TIMER 31729 ± 8% -14.7% 27073 softirqs.CPU104.TIMER 31415 ± 7% -13.9% 27037 softirqs.CPU105.TIMER 31606 ± 10% -14.8% 26924 softirqs.CPU106.TIMER 30924 ± 8% -12.7% 26983 softirqs.CPU107.TIMER 31042 ± 7% -13.2% 26952 softirqs.CPU108.TIMER 31113 ± 6% -12.6% 27182 softirqs.CPU109.TIMER 32157 ± 8% -12.8% 28041 softirqs.CPU11.TIMER 31768 ± 7% -15.1% 26969 softirqs.CPU110.TIMER 31708 ± 9% -14.8% 27018 softirqs.CPU111.TIMER 31989 ± 6% -15.3% 27088 softirqs.CPU112.TIMER 32354 ± 7% -16.6% 26968 softirqs.CPU113.TIMER 31293 ± 6% -14.2% 26843 softirqs.CPU114.TIMER 31129 ± 7% -13.7% 26870 softirqs.CPU115.TIMER 31818 ± 5% -14.9% 27074 softirqs.CPU117.TIMER 31706 ± 6% -14.6% 27067 softirqs.CPU118.TIMER 32302 ± 8% -12.7% 28198 ± 2% softirqs.CPU12.TIMER 32852 ± 9% -14.6% 28058 softirqs.CPU13.TIMER 28951 ± 6% -7.4% 26808 softirqs.CPU147.TIMER 29375 ± 7% -7.9% 27053 softirqs.CPU148.TIMER 32964 ± 9% -14.8% 28077 softirqs.CPU15.TIMER 29081 ± 6% -7.2% 26984 softirqs.CPU150.TIMER 29028 ± 6% -7.5% 26853 softirqs.CPU152.TIMER 28918 ± 7% -7.4% 26780 softirqs.CPU158.TIMER 29128 ± 7% -7.9% 26840 softirqs.CPU159.TIMER 32669 ± 9% -13.9% 28130 softirqs.CPU16.TIMER 28807 ± 6% -8.1% 26482 softirqs.CPU162.TIMER 29120 ± 7% -8.8% 26544 softirqs.CPU164.TIMER 33381 ± 9% -16.0% 28032 softirqs.CPU17.TIMER 24424 ± 3% +11.4% 27218 ± 5% softirqs.CPU170.RCU 30548 ± 10% -13.5% 26436 softirqs.CPU171.TIMER 30675 ± 9% -13.7% 26469 softirqs.CPU172.TIMER 30949 ± 10% -13.9% 26650 softirqs.CPU173.TIMER 30559 ± 9% -15.0% 25980 softirqs.CPU174.TIMER 30435 ± 10% -13.8% 26234 softirqs.CPU176.TIMER 30427 ± 10% -14.2% 26104 softirqs.CPU177.TIMER 30221 ± 10% -13.2% 26220 softirqs.CPU178.TIMER 30497 ± 11% -13.4% 26402 softirqs.CPU179.TIMER 32166 ± 8% -13.2% 27924 softirqs.CPU18.TIMER 30188 ± 9% -13.0% 26256 softirqs.CPU180.TIMER 30317 ± 9% -13.6% 26205 softirqs.CPU181.TIMER 30256 ± 9% -13.3% 26227 softirqs.CPU182.TIMER 30373 ± 9% -14.3% 26017 ± 2% softirqs.CPU184.TIMER 30857 ± 8% -14.3% 26434 softirqs.CPU185.TIMER 30779 ± 8% -14.8% 26234 softirqs.CPU186.TIMER 30871 ± 11% -17.4% 25500 ± 3% softirqs.CPU187.TIMER 30105 ± 9% -12.4% 26357 softirqs.CPU188.TIMER 30180 ± 9% -11.8% 26622 softirqs.CPU189.TIMER 32606 ± 8% -11.7% 28778 softirqs.CPU19.TIMER 30348 ± 10% -12.9% 26426 softirqs.CPU190.TIMER 32679 ± 8% -12.8% 28484 ± 2% softirqs.CPU2.TIMER 32633 ± 9% -12.7% 28482 softirqs.CPU20.TIMER 32846 ± 9% -14.5% 28092 softirqs.CPU21.TIMER 32451 ± 7% -13.6% 28038 softirqs.CPU22.TIMER 31888 ± 8% -12.2% 27989 softirqs.CPU23.TIMER 32673 ± 9% -13.3% 28340 softirqs.CPU3.TIMER 30578 ± 7% -10.0% 27535 softirqs.CPU49.TIMER 32571 ± 7% -13.4% 28205 softirqs.CPU5.TIMER 30578 ± 6% -9.5% 27667 softirqs.CPU51.TIMER 30173 ± 7% -8.2% 27697 softirqs.CPU52.TIMER 29841 ± 6% -7.9% 27496 softirqs.CPU53.TIMER 29906 ± 6% -8.0% 27527 softirqs.CPU56.TIMER 29967 ± 5% -7.6% 27701 softirqs.CPU57.TIMER 29938 ± 6% -7.3% 27765 softirqs.CPU58.TIMER 30465 ± 6% -9.6% 27541 softirqs.CPU59.TIMER 32882 ± 6% -14.1% 28238 softirqs.CPU6.TIMER 30412 ± 7% -9.6% 27507 softirqs.CPU60.TIMER 31318 ± 6% -11.7% 27643 softirqs.CPU62.TIMER 31305 ± 7% -11.3% 27776 softirqs.CPU63.TIMER 24337 ± 5% +8.7% 26447 ± 4% softirqs.CPU66.RCU 30129 ± 5% -8.3% 27630 softirqs.CPU66.TIMER 24350 ± 4% +8.2% 26346 ± 5% softirqs.CPU67.RCU 31037 ± 6% -11.7% 27413 softirqs.CPU68.TIMER 30792 ± 7% -9.4% 27893 softirqs.CPU69.TIMER 32250 ± 7% -12.4% 28241 softirqs.CPU7.TIMER 30980 ± 5% -9.7% 27967 softirqs.CPU70.TIMER 30611 ± 5% -10.1% 27526 softirqs.CPU71.TIMER 31699 ± 9% -11.9% 27921 ± 2% softirqs.CPU73.TIMER 31857 ± 10% -13.4% 27585 softirqs.CPU77.TIMER 31624 ± 9% -15.3% 26795 ± 2% softirqs.CPU78.TIMER 31386 ± 9% -12.8% 27368 softirqs.CPU79.TIMER 32576 ± 9% -13.5% 28174 softirqs.CPU8.TIMER 31444 ± 10% -14.0% 27045 softirqs.CPU80.TIMER 31232 ± 10% -12.5% 27325 softirqs.CPU81.TIMER 31572 ± 11% -13.9% 27185 softirqs.CPU82.TIMER 31527 ± 10% -14.1% 27071 softirqs.CPU83.TIMER 31293 ± 9% -13.6% 27025 softirqs.CPU84.TIMER 31232 ± 9% -13.7% 26962 softirqs.CPU85.TIMER 31197 ± 10% -15.1% 26492 ± 2% softirqs.CPU86.TIMER 31215 ± 9% -12.7% 27238 softirqs.CPU87.TIMER 31372 ± 10% -13.0% 27308 softirqs.CPU88.TIMER 31465 ± 9% -14.3% 26968 ± 3% softirqs.CPU89.TIMER 32612 ± 9% -14.3% 27953 softirqs.CPU9.TIMER 31643 ± 8% -16.5% 26422 ± 3% softirqs.CPU90.TIMER 30989 ± 9% -14.4% 26521 ± 2% softirqs.CPU92.TIMER 31242 ± 9% -12.0% 27494 softirqs.CPU93.TIMER 31073 ± 9% -13.4% 26913 softirqs.CPU94.TIMER 25373 -12.6% 22177 ± 3% softirqs.CPU95.RCU 33071 ± 8% -17.2% 27398 softirqs.CPU97.TIMER 26081 ± 6% -9.1% 23705 softirqs.CPU98.RCU 31449 ± 9% -15.7% 26505 softirqs.CPU98.TIMER 31934 ± 8% -14.5% 27300 softirqs.CPU99.TIMER *************************************************************************************************** lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/7000t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/shared_memory/reaim/0x42e commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :2 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 45.00 ± 16% +25.6% 56.50 ± 11% vmstat.procs.r 6.093e+08 ± 19% +41.0% 8.588e+08 ± 32% cpuidle.C1.time 8005497 ± 11% +34.4% 10759454 ± 28% cpuidle.C1.usage 8001963 ± 11% +34.4% 10756594 ± 28% turbostat.C1 4.03 ± 19% +1.6 5.67 ± 32% turbostat.C1% 6.89 ± 29% +48.0% 10.20 ± 12% turbostat.CPU%c6 2.18 ± 26% +47.4% 3.21 ± 23% turbostat.Pkg%pc2 202.00 +9.9% 222.00 proc-vmstat.nr_inactive_file 16125 +2.1% 16469 proc-vmstat.nr_shmem 19096 -2.9% 18541 proc-vmstat.nr_slab_reclaimable 202.00 +9.9% 222.00 proc-vmstat.nr_zone_inactive_file 5269 ± 5% +10.4% 5819 ± 3% proc-vmstat.pgactivate 105576 ± 2% +31.9% 139239 ± 16% softirqs.CPU28.TIMER 6097 ±169% -98.4% 96.50 ± 9% softirqs.CPU29.NET_RX 6060 ±169% +294.5% 23910 softirqs.CPU31.NET_RX 81682 ± 7% +12.4% 91797 softirqs.CPU37.RCU 106031 +9.1% 115712 ± 5% softirqs.CPU4.TIMER 938.75 ± 9% +13.4% 1064 ± 3% slabinfo.dmaengine-unmap-16.active_objs 938.75 ± 9% +13.4% 1064 ± 3% slabinfo.dmaengine-unmap-16.num_objs 558.00 +16.7% 651.00 ± 4% slabinfo.numa_policy.active_objs 558.00 +16.7% 651.00 ± 4% slabinfo.numa_policy.num_objs 153.00 +50.0% 229.50 ± 11% slabinfo.xfrm_dst_cache.active_objs 153.00 +50.0% 229.50 ± 11% slabinfo.xfrm_dst_cache.num_objs 6.594e+09 +1.2% 6.672e+09 perf-stat.i.branch-instructions 15.39 +0.3 15.72 perf-stat.i.cache-miss-rate% 2.28 -3.1% 2.21 perf-stat.i.cpi 9.788e+09 +1.0% 9.889e+09 perf-stat.i.dTLB-loads 21054228 -4.0% 20216327 perf-stat.i.iTLB-load-misses 3.493e+10 +1.2% 3.535e+10 perf-stat.i.instructions 1785 ± 2% +7.5% 1920 perf-stat.i.instructions-per-iTLB-miss 0.45 +2.2% 0.46 perf-stat.i.ipc 14.34 +0.2 14.56 perf-stat.overall.cache-miss-rate% 1661 +5.3% 1749 perf-stat.overall.instructions-per-iTLB-miss 0.46 +1.1% 0.47 perf-stat.overall.ipc 743013 +1.5% 754422 perf-stat.overall.path-length 6.569e+09 +1.2% 6.646e+09 perf-stat.ps.branch-instructions 9.752e+09 +1.0% 9.852e+09 perf-stat.ps.dTLB-loads 3.479e+10 +1.2% 3.522e+10 perf-stat.ps.instructions 1.092e+13 +1.5% 1.109e+13 perf-stat.total.instructions 168.00 ± 2% +8.6% 182.50 ± 5% interrupts.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1 169.25 ± 3% +11.4% 188.50 ± 3% interrupts.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4 12023 ±170% +296.0% 47614 interrupts.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7 1857 ± 57% +157.2% 4777 ± 32% interrupts.CPU10.NMI:Non-maskable_interrupts 1857 ± 57% +157.2% 4777 ± 32% interrupts.CPU10.PMI:Performance_monitoring_interrupts 9964 ± 8% +9.7% 10927 ± 7% interrupts.CPU14.RES:Rescheduling_interrupts 708.75 ±169% +370.2% 3332 ± 96% interrupts.CPU15.NMI:Non-maskable_interrupts 708.75 ±169% +370.2% 3332 ± 96% interrupts.CPU15.PMI:Performance_monitoring_interrupts 9702 ± 4% +9.0% 10579 ± 4% interrupts.CPU16.RES:Rescheduling_interrupts 3535 ± 12% -14.5% 3023 interrupts.CPU18.CAL:Function_call_interrupts 9525 ± 5% +10.5% 10524 interrupts.CPU19.RES:Rescheduling_interrupts 1664 ± 68% +91.4% 3185 interrupts.CPU20.NMI:Non-maskable_interrupts 1664 ± 68% +91.4% 3185 interrupts.CPU20.PMI:Performance_monitoring_interrupts 168.00 ± 2% +8.6% 182.50 ± 5% interrupts.CPU25.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1 1757 ± 58% +172.3% 4785 ± 32% interrupts.CPU27.NMI:Non-maskable_interrupts 1757 ± 58% +172.3% 4785 ± 32% interrupts.CPU27.PMI:Performance_monitoring_interrupts 169.25 ± 3% +11.4% 188.50 ± 3% interrupts.CPU28.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4 189.25 ±126% +1794.1% 3584 ± 82% interrupts.CPU28.NMI:Non-maskable_interrupts 189.25 ±126% +1794.1% 3584 ± 82% interrupts.CPU28.PMI:Performance_monitoring_interrupts 624.75 ±172% +409.3% 3182 ± 97% interrupts.CPU29.NMI:Non-maskable_interrupts 624.75 ±172% +409.3% 3182 ± 97% interrupts.CPU29.PMI:Performance_monitoring_interrupts 12023 ±170% +296.0% 47614 interrupts.CPU31.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7 2004 ± 58% +59.1% 3187 interrupts.CPU36.NMI:Non-maskable_interrupts 2004 ± 58% +59.1% 3187 interrupts.CPU36.PMI:Performance_monitoring_interrupts 1848 ±110% +158.4% 4775 ± 32% interrupts.CPU42.NMI:Non-maskable_interrupts 1848 ±110% +158.4% 4775 ± 32% interrupts.CPU42.PMI:Performance_monitoring_interrupts 9944 ± 7% +16.6% 11595 ± 4% interrupts.CPU42.RES:Rescheduling_interrupts 2432 ± 4% -99.6% 10.00 ± 90% interrupts.CPU44.NMI:Non-maskable_interrupts 2432 ± 4% -99.6% 10.00 ± 90% interrupts.CPU44.PMI:Performance_monitoring_interrupts 623.75 ±173% +663.3% 4761 ± 32% interrupts.CPU6.NMI:Non-maskable_interrupts 623.75 ±173% +663.3% 4761 ± 32% interrupts.CPU6.PMI:Performance_monitoring_interrupts 1193 ±172% +298.6% 4756 ± 32% interrupts.CPU9.NMI:Non-maskable_interrupts 1193 ±172% +298.6% 4756 ± 32% interrupts.CPU9.PMI:Performance_monitoring_interrupts 9355 ± 8% +13.2% 10593 ± 6% interrupts.CPU9.RES:Rescheduling_interrupts 66.50 ± 44% -45.9% 36.00 ± 8% interrupts.CPU9.TLB:TLB_shootdowns 95338 ± 13% +24.4% 118623 ± 10% interrupts.NMI:Non-maskable_interrupts 95338 ± 13% +24.4% 118623 ± 10% interrupts.PMI:Performance_monitoring_interrupts 18276 ± 34% +95.4% 35703 ± 29% sched_debug.cfs_rq:/.load.avg 396888 ± 33% +32.9% 527407 ± 8% sched_debug.cfs_rq:/.load.max 68973 ± 26% +67.5% 115530 ± 15% sched_debug.cfs_rq:/.load.stddev 35.56 ± 10% +27.5% 45.33 ± 4% sched_debug.cfs_rq:/.load_avg.avg 102.31 ± 14% +38.2% 141.35 ± 4% sched_debug.cfs_rq:/.load_avg.stddev 6.07 ± 63% +103.5% 12.35 ± 14% sched_debug.cfs_rq:/.removed.load_avg.avg 26.67 ± 59% +64.5% 43.88 ± 6% sched_debug.cfs_rq:/.removed.load_avg.stddev 280.13 ± 63% +103.3% 569.61 ± 14% sched_debug.cfs_rq:/.removed.runnable_sum.avg 1230 ± 59% +64.4% 2023 ± 6% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 1.61 ± 58% +188.3% 4.64 sched_debug.cfs_rq:/.removed.util_avg.avg 47.17 ± 64% +57.6% 74.33 ± 16% sched_debug.cfs_rq:/.removed.util_avg.max 8.04 ± 60% +110.8% 16.95 ± 8% sched_debug.cfs_rq:/.removed.util_avg.stddev 12.11 ± 33% +76.7% 21.40 ± 13% sched_debug.cfs_rq:/.runnable_load_avg.avg 51.67 ± 26% +68.8% 87.23 sched_debug.cfs_rq:/.runnable_load_avg.stddev 17578 ± 36% +100.8% 35303 ± 29% sched_debug.cfs_rq:/.runnable_weight.avg 380029 ± 35% +37.8% 523711 ± 9% sched_debug.cfs_rq:/.runnable_weight.max 66430 ± 27% +72.8% 114787 ± 15% sched_debug.cfs_rq:/.runnable_weight.stddev 279.47 ± 26% -49.7% 140.62 ± 3% sched_debug.cfs_rq:/.util_avg.avg 6.87 ± 42% -82.1% 1.23 ± 16% sched_debug.cpu.clock.stddev 6.87 ± 42% -82.1% 1.23 ± 16% sched_debug.cpu.clock_task.stddev 12.26 ± 33% +42.1% 17.42 ± 7% sched_debug.cpu.cpu_load[0].avg 51.68 ± 26% +54.0% 79.58 ± 9% sched_debug.cpu.cpu_load[0].stddev 12.43 ± 31% +39.9% 17.39 ± 14% sched_debug.cpu.cpu_load[1].avg 312.42 ± 23% +51.9% 474.58 ± 14% sched_debug.cpu.cpu_load[1].max 50.56 ± 23% +56.7% 79.25 ± 15% sched_debug.cpu.cpu_load[1].stddev 11.92 ± 28% +41.7% 16.90 ± 15% sched_debug.cpu.cpu_load[2].avg 325.07 ± 18% +66.9% 542.67 ± 13% sched_debug.cpu.cpu_load[2].max 49.19 ± 18% +67.7% 82.48 ± 15% sched_debug.cpu.cpu_load[2].stddev 11.52 ± 25% +40.7% 16.21 ± 13% sched_debug.cpu.cpu_load[3].avg 335.89 ± 14% +76.2% 591.83 ± 11% sched_debug.cpu.cpu_load[3].max 49.24 ± 14% +74.6% 85.98 ± 12% sched_debug.cpu.cpu_load[3].stddev 11.46 ± 26% +40.7% 16.12 ± 10% sched_debug.cpu.cpu_load[4].avg 355.26 ± 8% +82.7% 649.08 ± 9% sched_debug.cpu.cpu_load[4].max 51.09 ± 8% +82.0% 92.95 ± 9% sched_debug.cpu.cpu_load[4].stddev 5957 ± 87% -72.0% 1667 sched_debug.cpu.curr->pid.avg 15141 ± 19% -28.5% 10833 sched_debug.cpu.curr->pid.stddev 77073 ± 31% +33.7% 103028 ± 5% sched_debug.cpu.load.stddev 0.18 ± 52% -42.0% 0.10 ± 3% sched_debug.cpu.nr_running.avg 1.30 ± 4% -23.1% 1.00 sched_debug.cpu.nr_running.max 0.35 ± 10% -18.0% 0.28 sched_debug.cpu.nr_running.stddev 8109 ± 11% +49.7% 12141 ± 28% sched_debug.cpu.ttwu_local.max 459.06 ± 14% +101.8% 926.21 ± 44% sched_debug.cpu.ttwu_local.stddev 0.78 ± 3% -0.1 0.64 perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.90 ± 2% -0.1 0.77 perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 0.91 ± 2% -0.1 0.78 perf-profile.calltrace.cycles-pp.page_fault 0.88 ± 3% -0.1 0.75 perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 0.74 ± 4% -0.1 0.61 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.54 ± 5% +0.1 0.60 ± 2% perf-profile.calltrace.cycles-pp.dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop 0.56 ± 6% +0.1 0.64 ± 2% perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64 0.72 ± 6% +0.1 0.80 perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.73 ± 8% +0.1 0.82 ± 7% perf-profile.calltrace.cycles-pp.newseg.ipcget.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.77 ± 7% +0.1 0.86 ± 2% perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.80 ± 7% +0.1 0.90 perf-profile.calltrace.cycles-pp.down_write.shmctl_down.ksys_shmctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.81 ± 7% +0.1 0.91 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.91 ± 5% +0.1 1.02 ± 2% perf-profile.calltrace.cycles-pp.do_smart_wakeup_zero.do_smart_update.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.43 ± 6% +0.1 1.54 perf-profile.calltrace.cycles-pp.shmctl_down.ksys_shmctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.43 ± 6% +0.1 1.56 perf-profile.calltrace.cycles-pp.ksys_shmctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.93 ± 9% +0.1 1.07 ± 6% perf-profile.calltrace.cycles-pp.down_write.do_shmat.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.89 ± 9% +0.1 1.04 ± 5% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.do_shmat.__x64_sys_shmat.do_syscall_64 1.34 ± 7% +0.2 1.50 perf-profile.calltrace.cycles-pp._copy_from_user.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.77 ± 6% +0.2 1.94 perf-profile.calltrace.cycles-pp.do_smart_update.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.40 ± 58% +0.2 0.59 ± 2% perf-profile.calltrace.cycles-pp.__dentry_kill.dentry_kill.dput.__fput.task_work_run 0.41 ± 57% +0.2 0.62 ± 2% perf-profile.calltrace.cycles-pp.perform_atomic_semop.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.00 ± 5% +0.3 2.27 ± 3% perf-profile.calltrace.cycles-pp.do_shmat.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.00 ± 5% +0.3 2.28 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.19 ± 5% +0.4 4.54 perf-profile.calltrace.cycles-pp.ipc_obtain_object_check.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.13 ±173% +0.4 0.52 perf-profile.calltrace.cycles-pp.do_mmap.do_shmat.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.12 ± 12% +0.4 1.50 ± 13% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.shm_close.remove_vma 1.58 ± 9% +0.4 1.98 ± 10% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.shm_close.remove_vma.__do_munmap 1.62 ± 9% +0.4 2.04 ± 10% perf-profile.calltrace.cycles-pp.down_write.shm_close.remove_vma.__do_munmap.ksys_shmdt 1.67 ± 9% +0.4 2.10 ± 9% perf-profile.calltrace.cycles-pp.shm_close.remove_vma.__do_munmap.ksys_shmdt.do_syscall_64 2.16 ± 8% +0.4 2.61 ± 8% perf-profile.calltrace.cycles-pp.remove_vma.__do_munmap.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.69 ± 8% +0.5 3.17 ± 7% perf-profile.calltrace.cycles-pp.__do_munmap.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.73 ± 8% +0.5 3.22 ± 7% perf-profile.calltrace.cycles-pp.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.02 ± 6% +0.5 6.53 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 8.28 ± 32% +3.6 11.89 ± 3% perf-profile.calltrace.cycles-pp.ipcget.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.29 ± 32% +3.6 11.91 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe 49.89 ± 9% +6.4 56.31 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 50.69 ± 9% +6.5 57.20 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 1.49 ± 11% -0.4 1.14 ± 3% perf-profile.children.cycles-pp.apic_timer_interrupt 1.42 ± 11% -0.3 1.08 ± 3% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 1.04 ± 6% -0.3 0.78 perf-profile.children.cycles-pp.page_fault 1.01 ± 5% -0.3 0.75 perf-profile.children.cycles-pp.__do_page_fault 1.02 ± 6% -0.3 0.77 perf-profile.children.cycles-pp.do_page_fault 0.89 ± 5% -0.2 0.65 perf-profile.children.cycles-pp.handle_mm_fault 0.86 ± 5% -0.2 0.61 perf-profile.children.cycles-pp.__handle_mm_fault 0.82 ± 13% -0.2 0.64 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt 0.34 ± 23% -0.2 0.17 perf-profile.children.cycles-pp.menu_select 0.48 ± 10% -0.1 0.39 ± 2% perf-profile.children.cycles-pp.irq_exit 0.38 ± 12% -0.1 0.32 perf-profile.children.cycles-pp.tick_sched_timer 0.11 ± 12% -0.1 0.05 perf-profile.children.cycles-pp.load_balance 0.08 ± 10% -0.1 0.03 ±100% perf-profile.children.cycles-pp.run_rebalance_domains 0.08 ± 11% -0.1 0.03 ±100% perf-profile.children.cycles-pp.update_blocked_averages 0.10 ± 21% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.rebalance_domains 0.10 ± 18% -0.0 0.07 perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.07 ± 13% -0.0 0.05 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.11 ± 3% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.up_write 0.11 ± 8% +0.0 0.12 ± 8% perf-profile.children.cycles-pp.idr_alloc_u32 0.07 ± 10% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.idr_replace 0.11 ± 11% +0.0 0.13 ± 7% perf-profile.children.cycles-pp.idr_alloc_cyclic 0.11 ± 6% +0.0 0.13 perf-profile.children.cycles-pp.shmem_get_unmapped_area 0.10 ± 8% +0.0 0.12 perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.25 ± 4% +0.0 0.27 perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.17 ± 4% +0.0 0.20 ± 2% perf-profile.children.cycles-pp.shm_destroy 0.14 ± 5% +0.0 0.17 ± 5% perf-profile.children.cycles-pp.get_unmapped_area 0.10 ± 15% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.task_tick_fair 0.28 ± 5% +0.0 0.32 perf-profile.children.cycles-pp.arch_stack_walk 0.24 ± 7% +0.0 0.28 perf-profile.children.cycles-pp.newary 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp._cond_resched 1.04 ± 4% +0.1 1.10 ± 2% perf-profile.children.cycles-pp.do_smart_wakeup_zero 0.76 ± 4% +0.1 0.81 perf-profile.children.cycles-pp.__fput 0.81 ± 5% +0.1 0.86 ± 2% perf-profile.children.cycles-pp.task_work_run 0.86 ± 4% +0.1 0.92 perf-profile.children.cycles-pp.exit_to_usermode_loop 0.76 ± 2% +0.1 0.82 ± 7% perf-profile.children.cycles-pp.newseg 0.32 ± 7% +0.1 0.39 perf-profile.children.cycles-pp.ipc_addid 1.47 ± 5% +0.1 1.54 perf-profile.children.cycles-pp._copy_from_user 0.35 ± 5% +0.1 0.43 ± 6% perf-profile.children.cycles-pp.ksys_semget 0.57 +0.1 0.66 ± 2% perf-profile.children.cycles-pp.perform_atomic_semop 0.28 ± 7% +0.1 0.39 ± 19% perf-profile.children.cycles-pp.semctl_down 0.37 ± 5% +0.1 0.48 ± 14% perf-profile.children.cycles-pp.ksys_semctl 2.10 ± 6% +0.2 2.27 ± 3% perf-profile.children.cycles-pp.do_shmat 2.10 ± 5% +0.2 2.28 ± 3% perf-profile.children.cycles-pp.__x64_sys_shmat 1.74 ± 7% +0.4 2.10 ± 9% perf-profile.children.cycles-pp.shm_close 2.25 ± 6% +0.4 2.62 ± 8% perf-profile.children.cycles-pp.remove_vma 0.08 ± 15% -0.1 0.03 ±100% perf-profile.self.cycles-pp.cpuidle_enter_state 0.16 ± 16% -0.1 0.11 perf-profile.self.cycles-pp.menu_select 0.11 ± 4% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.wake_up_q 0.23 ± 3% +0.0 0.25 perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.13 ± 6% +0.0 0.15 ± 3% perf-profile.self.cycles-pp.unwind_next_frame 0.16 ± 4% +0.0 0.18 ± 5% perf-profile.self.cycles-pp.down_write 0.63 ± 2% +0.0 0.66 perf-profile.self.cycles-pp.do_smart_wakeup_zero 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.ktime_get_real_seconds 0.72 ± 4% +0.0 0.77 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.29 ± 6% +0.0 0.34 perf-profile.self.cycles-pp._copy_from_user 0.56 +0.1 0.64 perf-profile.self.cycles-pp.perform_atomic_semop *************************************************************************************************** lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/400t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/disk_src/reaim/0x42e commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:2 dmesg.WARNING:at#for_ip_interrupt_entry/0x 2:4 -50% :2 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :4 25% 1:2 dmesg.WARNING:at_ip__fsnotify_parent/0x *************************************************************************************************** lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory ========================================================================================= compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/directio/1HDD/xfs/x86_64-rhel-7.6/hdd/debian-x86_64-2019-05-14.cgz/lkp-ivb-2ep1/MWUL/fxmark/0x42e commit: 4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation") 00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks") 4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 ---------------- --------------------------- %stddev %change %stddev \ | \ 0.18 ± 17% +30.0% 0.23 fxmark.hdd_xfs_MWUL_12_directio.iowait_util 1145 +0.9% 1156 boot-time.idle 197.75 +2.9% 203.50 vmstat.io.bi 46933823 +15.4% 54152343 ± 11% cpuidle.C1.time 675211 +8.7% 733632 ± 7% cpuidle.C1.usage 441822 ± 74% +107.2% 915380 ± 11% cpuidle.C1E.usage 0.87 ± 6% +14.6% 0.99 ± 5% iostat.cpu.iowait 12.52 +3.9% 13.01 iostat.cpu.system 9725 +1.8% 9900 iostat.sda.wkB/s 250.50 ± 7% +50.7% 377.50 ± 4% slabinfo.numa_policy.active_objs 1586 ± 2% +17.4% 1862 ± 11% slabinfo.proc_inode_cache.active_objs 2208 ± 3% +17.4% 2592 ± 7% slabinfo.proc_inode_cache.num_objs 15301 ± 3% +19.6% 18303 ± 4% softirqs.BLOCK 9332 ± 4% +10.0% 10265 ± 5% softirqs.CPU11.TIMER 9806 ± 5% +76.8% 17339 ± 3% softirqs.CPU12.TIMER 13390 ± 14% +19.5% 15996 softirqs.CPU3.RCU 10236 ± 6% -18.3% 8364 ± 9% softirqs.CPU40.TIMER 5703 +2.3% 5834 proc-vmstat.nr_inactive_file 24468 +0.8% 24675 proc-vmstat.nr_slab_unreclaimable 5703 +2.3% 5834 proc-vmstat.nr_zone_inactive_file 3368 ± 81% +190.8% 9794 ± 8% proc-vmstat.numa_pages_migrated 20369 ± 59% +164.1% 53802 ± 27% proc-vmstat.numa_pte_updates 3368 ± 81% +190.8% 9794 ± 8% proc-vmstat.pgmigrate_success 221862 ± 15% -24.2% 168110 ± 3% perf-stat.i.dTLB-load-misses 3966 +1.6% 4028 perf-stat.i.minor-faults 19254 ± 7% +12.3% 21613 ± 6% perf-stat.i.node-load-misses 2.21 ± 17% +0.5 2.72 perf-stat.i.node-store-miss-rate% 16644 ± 12% +20.2% 20014 ± 6% perf-stat.i.node-store-misses 3966 +1.6% 4028 perf-stat.i.page-faults 219511 ± 15% -24.2% 166480 ± 3% perf-stat.ps.dTLB-load-misses 19050 ± 7% +12.4% 21404 ± 6% perf-stat.ps.node-load-misses 16480 ± 12% +20.3% 19830 ± 6% perf-stat.ps.node-store-misses 32707 ± 12% -49.4% 16539 ± 3% numa-meminfo.node0.Inactive 14732 ± 3% -88.7% 1669 ± 16% numa-meminfo.node0.Inactive(anon) 16642 -10.0% 14973 numa-meminfo.node0.Mapped 14864 ± 3% -88.0% 1778 ± 14% numa-meminfo.node0.Shmem 534994 +10.7% 592351 numa-meminfo.node0.Unevictable 6988 ± 55% +230.7% 23113 numa-meminfo.node1.Inactive 1740 ± 32% +749.7% 14785 ± 2% numa-meminfo.node1.Inactive(anon) 14910 +11.8% 16667 numa-meminfo.node1.Mapped 37981 ± 4% +14.0% 43311 numa-meminfo.node1.SUnreclaim 1788 ± 32% +730.6% 14853 numa-meminfo.node1.Shmem 587633 -9.7% 530463 numa-meminfo.node1.Unevictable 71788 ± 4% +13.3% 81318 interrupts.CAL:Function_call_interrupts 2387 ± 26% +78.0% 4249 ± 18% interrupts.CPU0.CAL:Function_call_interrupts 2160 ± 18% +62.1% 3502 ± 16% interrupts.CPU1.CAL:Function_call_interrupts 2464 ± 15% +67.9% 4137 ± 14% interrupts.CPU2.CAL:Function_call_interrupts 161.25 ± 20% +118.6% 352.50 ± 33% interrupts.CPU21.RES:Rescheduling_interrupts 1705 ± 14% +18.9% 2028 ± 4% interrupts.CPU22.CAL:Function_call_interrupts 2565 ± 22% +43.4% 3679 interrupts.CPU3.CAL:Function_call_interrupts 1151 ± 13% +21.7% 1401 interrupts.CPU3.RES:Rescheduling_interrupts 760.00 ± 14% +143.6% 1851 ± 52% interrupts.CPU4.RES:Rescheduling_interrupts 2400 ± 24% +34.3% 3224 ± 14% interrupts.CPU6.CAL:Function_call_interrupts 2011 ± 3% +32.5% 2665 ± 3% interrupts.CPU7.CAL:Function_call_interrupts 59.75 ±150% +141.0% 144.00 ± 80% interrupts.CPU8.RES:Rescheduling_interrupts 3681 ± 3% -88.7% 416.50 ± 17% numa-vmstat.node0.nr_inactive_anon 3714 ± 3% -88.1% 442.00 ± 15% numa-vmstat.node0.nr_shmem 133747 +10.7% 148087 numa-vmstat.node0.nr_unevictable 3681 ± 3% -88.7% 416.50 ± 17% numa-vmstat.node0.nr_zone_inactive_anon 133747 +10.7% 148087 numa-vmstat.node0.nr_zone_unevictable 566715 ± 5% +11.1% 629665 numa-vmstat.node0.numa_local 434.75 ± 31% +750.1% 3696 ± 2% numa-vmstat.node1.nr_inactive_anon 446.75 ± 32% +731.2% 3713 numa-vmstat.node1.nr_shmem 9486 ± 4% +14.2% 10831 numa-vmstat.node1.nr_slab_unreclaimable 146908 -9.7% 132615 numa-vmstat.node1.nr_unevictable 434.75 ± 31% +750.1% 3696 ± 2% numa-vmstat.node1.nr_zone_inactive_anon 146908 -9.7% 132615 numa-vmstat.node1.nr_zone_unevictable 308139 ± 10% -17.1% 255441 ± 2% numa-vmstat.node1.numa_local 75.28 ± 22% +88.6% 141.96 ± 12% sched_debug.cfs_rq:/.load_avg.avg 210.99 ± 18% +49.2% 314.68 ± 5% sched_debug.cfs_rq:/.load_avg.stddev 4917 ± 6% -16.5% 4106 sched_debug.cfs_rq:/.min_vruntime.stddev 15.85 ±110% +429.8% 83.99 ± 23% sched_debug.cfs_rq:/.removed.load_avg.avg 87.02 ±102% +217.6% 276.43 ± 10% sched_debug.cfs_rq:/.removed.load_avg.stddev 725.97 ±110% +434.6% 3881 ± 23% sched_debug.cfs_rq:/.removed.runnable_sum.avg 3986 ±102% +220.4% 12776 ± 10% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 5.03 ±106% +451.8% 27.76 ± 16% sched_debug.cfs_rq:/.removed.util_avg.avg 167.00 ±100% +164.4% 441.50 ± 17% sched_debug.cfs_rq:/.removed.util_avg.max 27.99 ±100% +237.2% 94.38 sched_debug.cfs_rq:/.removed.util_avg.stddev -3468 -52.9% -1633 sched_debug.cfs_rq:/.spread0.min 4917 ± 6% -16.5% 4104 sched_debug.cfs_rq:/.spread0.stddev 7.26 ± 26% -28.6% 5.18 ± 2% sched_debug.cpu.cpu_load[1].avg 8.21 ± 16% -28.6% 5.86 sched_debug.cpu.cpu_load[2].avg 9.19 ± 6% -20.0% 7.35 sched_debug.cpu.cpu_load[3].avg 10.94 ± 6% -14.5% 9.35 sched_debug.cpu.cpu_load[4].avg 2077 ± 5% +9.0% 2264 sched_debug.cpu.nr_switches.stddev Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Oliver Sang