FYI, we noticed pixz.throughput -9.1% regression due to commit: commit 795ae7a0de6b834a0cc202aa55c190ef81496665 ("mm: scale kswapd watermarks in proportion to memory") https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master in testcase: pixz on test machine: ivb43: 48 threads Ivytown Ivy Bridge-EP with 64G memory with following parameters: cpufreq_governor=performance/nr_threads=100%

3ed3a4f0ddffece9 795ae7a0de6b834a0cc202aa55 ---------------- -------------------------- %stddev %change %stddev \ | \ 78505362 ± 0% -9.1% 71324131 ± 0% pixz.throughput 4530 ± 0% +1.0% 4575 ± 0% pixz.time.percent_of_cpu_this_job_got 14911 ± 0% +2.3% 15251 ± 0% pixz.time.user_time 6586930 ± 0% -7.5% 6093751 ± 1% pixz.time.voluntary_context_switches 49869 ± 1% -9.0% 45401 ± 0% vmstat.system.cs 26406 ± 4% -9.4% 23922 ± 5% numa-meminfo.node0.SReclaimable 4803 ± 85% -87.0% 625.25 ± 16% numa-meminfo.node1.Inactive(anon) 946.75 ± 3% +775.4% 8288 ± 1% proc-vmstat.nr_alloc_batch 2403080 ± 2% -58.4% 999765 ± 0% proc-vmstat.pgalloc_dma32

3ed3a4f0ddffece9 795ae7a0de6b834a0cc202aa55 ---------------- -------------------------- %stddev %change %stddev \ | \ 442409 ± 0% -8.5% 404670 ± 0% will-it-scale.per_process_ops 397397 ± 0% -6.2% 372741 ± 0% will-it-scale.per_thread_ops 0.11 ± 1% -15.1% 0.10 ± 0% will-it-scale.scalability 9933 ± 10% +17.8% 11696 ± 4% will-it-scale.time.involuntary_context_switches 5158470 ± 3% +5.4% 5438873 ± 0% will-it-scale.time.maximum_resident_set_size 10701739 ± 0% -11.6% 9456315 ± 0% will-it-scale.time.minor_page_faults 825.00 ± 0% +7.8% 889.75 ± 0% will-it-scale.time.percent_of_cpu_this_job_got 2484 ± 0% +7.8% 2678 ± 0% will-it-scale.time.system_time 81.98 ± 0% +8.7% 89.08 ± 0% will-it-scale.time.user_time 848972 ± 1% -13.3% 735967 ± 0% will-it-scale.time.voluntary_context_switches 19395253 ± 0% -20.0% 15511908 ± 0% numa-numastat.node0.local_node 19400671 ± 0% -20.0% 15518877 ± 0% numa-numastat.node0.numa_hit

right before the test is running? I wonder if it's mostly filled with cache, and the increase in watermarks causes a higher portion of the anon allocs and frees to spill to the remote node, but never enough to enter the allocator slowpath and waking kswapd to fix it. Another suspect is the fair zone allocator, whose allocation batches increased as well. It shouldn't affect NUMA placement, but I wonder if there is a bug in there that causes false spilling to foreign nodes that was only bounded by the allocation batch of the foreign zone. Mel, does such a symptom sound familiar in any way?

On Thu, Jun 02, 2016 at 12:07:06PM -0400, Johannes Weiner wrote: > Hi, > > On Thu, Jun 02, 2016 at 02:45:07PM +0800, kernel test robot wrote: > > FYI, we noticed pixz.throughput -9.1% regression due to commit: > > > > commit 795ae7a0de6b834a0cc202aa55c190ef81496665 ("mm: scale kswapd watermarks in proportion to memory") > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master > > > > in testcase: pixz > > on test machine: ivb43: 48 threads Ivytown Ivy Bridge-EP with 64G memory with following parameters: cpufreq_governor=performance/nr_threads=100% > > Xiaolong, thanks for the report. > > It looks like the regression stems from a change in NUMA placement: Scratch that, I was misreading the test results. It's just fewer allocations in general that happen during the fixed testing time. I'm stumped by this report. All this patch does other than affect page reclaim (which is not involved here) is increase the size of the round robin batches in the fair zone allocator. That should *reduce* work in the page allocator, if anything. But I also keep failing to reproduce this - having tried it on the third machine now - neither pixz nor will-it-scale/page_fault1 give me that -8-9% regression: 4.5.0-02574-g3ed3a4f: PIXZ-good.log:throughput: 39908733.604941994 PIXZ-good.log:throughput: 37067947.25049969 PIXZ-good.log:throughput: 38604938.39131216 4.5.0-02575-g795ae7a: PIXZ-bad.log:throughput: 39489120.87179377 PIXZ-bad.log:throughput: 39307299.288432725 PIXZ-bad.log:throughput: 38795994.3329781 Is this reliably reproducible with 3ed3a4f vs 795ae7a?

Could I ask you to retry the test with Linus's current head as well as with 795ae7a reverted on top of it? (It's a clean revert.)

Thanks

On Tue, Jun 07, 2016 at 05:56:27PM -0400, Johannes Weiner wrote: >But just to make sure I'm looking at the right code, can you first try >the following patch on top of Linus's current tree and see if that >gets performance back to normal? It's a partial revert of the >watermarks that singles out the fair zone allocator: Seems that this patch doesn't help to gets performance back. I've attached the comparison result among 3ed3a4f, 795ae7ay, v4.7-rc2 and 1fe49ba5 ("mm: revert fairness batching to before the watermarks were") with perf profile information. You can find it via searching 'perf-profile'.

3ed3a4f0ddffece9 795ae7a0de6b834a0cc202aa55 v4.7-rc2 1fe49ba5002a50aefd5b6c4913 ---------------- -------------------------- -------------------------- -------------------------- fail:runs %reproduction fail:runs %reproduction fail:runs %reproduction fail:runs | | | | | | | :4 0% :7 0% :4 50% 2:4 kmsg.DHCP/BOOTP:Reply_not_for_us,op[#]xid[#] :4 50% 2:7 0% :4 0% :4 kmsg.Spurious_LAPIC_timer_interrupt_on_cpu :4 0% :7 14% 1:4 25% 1:4 kmsg.igb#:#:#:exceed_max#second %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 78505362 ± 0% -9.2% 71298182 ± 0% -11.8% 69280014 ± 0% -9.1% 71350485 ± 0% pixz.throughput 5586220 ± 2% -1.6% 5498492 ± 2% +6.5% 5950210 ± 1% +8.4% 6052963 ± 1% pixz.time.involuntary_context_switches 4582198 ± 2% -3.6% 4416275 ± 2% -8.6% 4189304 ± 4% -8.0% 4214839 ± 0% pixz.time.minor_page_faults 4530 ± 0% +1.0% 4575 ± 0% -1.6% 4458 ± 0% -1.3% 4469 ± 0% pixz.time.percent_of_cpu_this_job_got 92.03 ± 0% +5.6% 97.23 ± 11% +31.3% 120.83 ± 1% +30.4% 119.98 ± 0% pixz.time.system_time 14911 ± 0% +2.1% 15218 ± 0% -1.0% 14759 ± 1% -1.0% 14764 ± 0% pixz.time.user_time 6586930 ± 0% -8.4% 6033444 ± 1% -4.4% 6295529 ± 1% -2.6% 6416460 ± 1% pixz.time.voluntary_context_switches 2179703 ± 4% +4.8% 2285049 ± 2% -15.3% 1846752 ± 16% -8.2% 2000913 ± 4% softirqs.RCU 92.03 ± 0% +5.6% 97.23 ± 11% +31.3% 120.83 ± 1% +30.4% 119.98 ± 0% time.system_time 2237 ± 2% -2.9% 2172 ± 7% +16.3% 2601 ± 7% +8.0% 2416 ± 6% uptime.idle 49869 ± 1% -12.6% 43583 ± 8% -18.0% 40917 ± 0% -16.3% 41728 ± 1% vmstat.system.cs 97890 ± 1% -0.0% 97848 ± 3% +7.4% 105143 ± 2% +6.8% 104518 ± 2% vmstat.system.in 105682 ± 1% +0.6% 106297 ± 1% -85.2% 15631 ± 4% -85.1% 15768 ± 1% meminfo.Active(file) 390126 ± 0% -0.2% 389529 ± 0% +23.9% 483296 ± 0% +23.9% 483194 ± 0% meminfo.Inactive 380750 ± 0% -0.2% 380141 ± 0% +24.5% 473891 ± 0% +24.4% 473760 ± 0% meminfo.Inactive(file) 2401 ±107% +76.9% 4247 ± 79% -99.8% 5.75 ± 18% -99.7% 6.75 ± 39% numa-numastat.node0.other_node 2074670 ± 2% -11.3% 1840052 ± 11% -21.1% 1637071 ± 12% -22.5% 1607724 ± 7% numa-numastat.node1.local_node 2081648 ± 2% -11.4% 1844923 ± 11% -21.4% 1637081 ± 12% -22.8% 1607730 ± 7% numa-numastat.node1.numa_hit 6977 ± 36% -30.2% 4871 ± 66% -99.8% 10.50 ± 17% -99.9% 5.50 ± 20% numa-numastat.node1.other_node 13061458 ± 19% -3.3% 12634644 ± 24% +33.5% 17435714 ± 47% +58.3% 20674526 ± 14% cpuidle.C1-IVT.time 193807 ± 15% +26.8% 245657 ± 76% +101.8% 391021 ± 8% +115.5% 417669 ± 20% cpuidle.C1-IVT.usage 8.866e+08 ± 2% -15.6% 7.479e+08 ± 6% +25.0% 1.108e+09 ± 5% +21.0% 1.073e+09 ± 4% cpuidle.C6-IVT.time 93283 ± 0% -13.2% 80988 ± 3% +300.6% 373726 ±121% +20.8% 112719 ± 1% cpuidle.C6-IVT.usage 8559466 ± 20% -39.3% 5195127 ± 40% -98.1% 159481 ±173% -100.0% 97.50 ± 40% cpuidle.POLL.time 771388 ± 9% -53.4% 359081 ± 52% -99.9% 959.00 ±167% -100.0% 40.50 ± 39% cpuidle.POLL.usage 94.35 ± 0% +1.0% 95.28 ± 0% -1.6% 92.81 ± 0% -1.4% 93.00 ± 0% turbostat.%Busy 2824 ± 0% +1.0% 2851 ± 0% -1.6% 2777 ± 0% -1.4% 2784 ± 0% turbostat.Avg_MHz 3.57 ± 3% -20.9% 2.83 ± 6% +18.6% 4.24 ± 6% +9.4% 3.91 ± 4% turbostat.CPU%c1 2.07 ± 3% -8.8% 1.89 ± 10% +42.0% 2.95 ± 13% +48.7% 3.08 ± 4% turbostat.CPU%c6 157.67 ± 0% -0.7% 156.51 ± 0% -1.4% 155.47 ± 0% -1.4% 155.39 ± 0% turbostat.CorWatt 0.17 ± 17% -2.9% 0.17 ± 23% +151.4% 0.44 ± 23% +88.6% 0.33 ± 11% turbostat.Pkg%pc2 192.71 ± 0% -0.8% 191.15 ± 0% -1.4% 190.10 ± 0% -1.3% 190.12 ± 0% turbostat.PkgWatt 22.36 ± 0% -8.4% 20.49 ± 0% -10.3% 20.05 ± 0% -8.1% 20.55 ± 0% turbostat.RAMWatt 53301 ± 2% +0.3% 53439 ± 5% -85.3% 7826 ± 4% -85.2% 7898 ± 1% numa-meminfo.node0.Active(file) 194536 ± 2% +0.8% 196145 ± 2% +24.4% 241970 ± 1% +25.2% 243537 ± 1% numa-meminfo.node0.Inactive 189951 ± 0% -0.1% 189801 ± 1% +24.7% 236921 ± 0% +24.7% 236864 ± 0% numa-meminfo.node0.Inactive(file) 10240 ± 2% -1.0% 10138 ± 3% -16.2% 8580 ± 3% -17.6% 8442 ± 1% numa-meminfo.node0.KernelStack 26406 ± 4% -8.4% 24183 ± 7% -10.2% 23723 ± 2% -4.7% 25152 ± 5% numa-meminfo.node0.SReclaimable 52381 ± 1% +0.9% 52856 ± 3% -85.1% 7804 ± 4% -85.0% 7867 ± 2% numa-meminfo.node1.Active(file) 195602 ± 2% -1.1% 193393 ± 2% +23.4% 241343 ± 1% +22.5% 239683 ± 1% numa-meminfo.node1.Inactive 190797 ± 0% -0.2% 190340 ± 1% +24.2% 236969 ± 0% +24.2% 236897 ± 0% numa-meminfo.node1.Inactive(file) 4188 ± 6% +2.4% 4289 ± 5% +42.2% 5955 ± 4% +45.0% 6073 ± 2% numa-meminfo.node1.KernelStack 22906 ± 4% +10.5% 25314 ± 6% +13.4% 25980 ± 2% +8.5% 24850 ± 5% numa-meminfo.node1.SReclaimable 13324 ± 2% +0.3% 13359 ± 5% -85.3% 1956 ± 4% -85.2% 1974 ± 1% numa-vmstat.node0.nr_active_file 454.25 ± 2% +773.4% 3967 ± 2% +794.3% 4062 ± 3% +194.4% 1337 ± 2% numa-vmstat.node0.nr_alloc_batch 47488 ± 0% -0.1% 47449 ± 1% +24.7% 59229 ± 0% +24.7% 59215 ± 0% numa-vmstat.node0.nr_inactive_file 639.25 ± 2% -1.0% 633.00 ± 3% -16.2% 536.00 ± 3% -17.5% 527.25 ± 1% numa-vmstat.node0.nr_kernel_stack 6600 ± 4% -8.4% 6045 ± 7% -10.2% 5930 ± 2% -4.7% 6287 ± 5% numa-vmstat.node0.nr_slab_reclaimable 69675 ± 3% +3.0% 71759 ± 4% -100.0% 2.50 ± 66% -100.0% 3.75 ± 51% numa-vmstat.node0.numa_other 13094 ± 1% +0.9% 13213 ± 3% -85.1% 1950 ± 4% -85.0% 1966 ± 2% numa-vmstat.node1.nr_active_file 563.00 ± 2% +642.6% 4181 ± 3% +631.5% 4118 ± 4% +162.8% 1479 ± 2% numa-vmstat.node1.nr_alloc_batch 47699 ± 0% -0.2% 47584 ± 1% +24.2% 59241 ± 0% +24.2% 59223 ± 0% numa-vmstat.node1.nr_inactive_file 261.25 ± 6% +2.4% 267.57 ± 5% +42.3% 371.75 ± 4% +45.3% 379.50 ± 2% numa-vmstat.node1.nr_kernel_stack 5726 ± 4% +10.5% 6328 ± 6% +13.4% 6495 ± 2% +8.5% 6212 ± 5% numa-vmstat.node1.nr_slab_reclaimable 1254802 ± 3% -9.6% 1134298 ± 10% -19.6% 1008654 ± 9% -21.0% 990900 ± 5% numa-vmstat.node1.numa_hit 1232554 ± 3% -9.6% 1113884 ± 10% -18.2% 1008648 ± 9% -19.6% 990898 ± 5% numa-vmstat.node1.numa_local 22247 ± 11% -8.2% 20414 ± 16% -100.0% 5.75 ± 18% -100.0% 1.75 ± 24% numa-vmstat.node1.numa_other 26419 ± 1% +0.6% 26573 ± 1% -85.2% 3907 ± 4% -85.1% 3941 ± 1% proc-vmstat.nr_active_file 946.75 ± 3% +764.9% 8188 ± 2% +745.5% 8004 ± 1% +196.1% 2803 ± 1% proc-vmstat.nr_alloc_batch 95188 ± 0% -0.2% 95035 ± 0% +24.5% 118472 ± 0% +24.4% 118440 ± 0% proc-vmstat.nr_inactive_file 3005733 ± 3% -4.4% 2872963 ± 2% -14.6% 2566600 ± 6% -13.9% 2587727 ± 1% proc-vmstat.numa_hint_faults_local 3652636 ± 1% -4.4% 3492233 ± 2% -16.5% 3049926 ± 2% -14.0% 3139498 ± 0% proc-vmstat.numa_hit 3643257 ± 1% -4.4% 3483323 ± 2% -16.3% 3049910 ± 2% -13.8% 3139486 ± 0% proc-vmstat.numa_local 9379 ± 0% -5.0% 8909 ± 12% -99.8% 16.25 ± 7% -99.9% 12.25 ± 27% proc-vmstat.numa_other 4924994 ± 3% +0.9% 4966927 ± 9% +38.2% 6804572 ± 5% +38.7% 6831202 ± 4% proc-vmstat.numa_pages_migrated 8510 ± 0% +1.5% 8638 ± 1% -27.1% 6204 ± 31% -11.2% 7554 ± 1% proc-vmstat.pgactivate 2403080 ± 2% -58.7% 993450 ± 2% -57.0% 1033978 ± 4% -39.3% 1457730 ± 3% proc-vmstat.pgalloc_dma32 15038432 ± 0% +8.1% 16250009 ± 3% +16.9% 17583879 ± 2% +14.9% 17277548 ± 1% proc-vmstat.pgalloc_normal 32128 ± 22% +41.4% 45421 ± 21% +391.6% 157952 ± 9% +333.9% 139392 ± 11% proc-vmstat.pgmigrate_fail 4924994 ± 3% +0.9% 4966927 ± 9% +38.2% 6804572 ± 5% +38.7% 6831202 ± 4% proc-vmstat.pgmigrate_success 25886 ± 2% -1.2% 25585 ± 4% +12.0% 28981 ± 2% +12.5% 29132 ± 2% proc-vmstat.thp_deferred_split_page 632.75 ± 3% -1.6% 622.43 ± 3% -18.1% 518.50 ± 2% -19.4% 510.00 ± 0% slabinfo.RAW.active_objs 632.75 ± 3% -1.6% 622.43 ± 3% -18.1% 518.50 ± 2% -19.4% 510.00 ± 0% slabinfo.RAW.num_objs 1512 ± 1% -0.6% 1502 ± 1% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% slabinfo.UNIX.active_objs 1512 ± 1% -0.6% 1502 ± 1% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% slabinfo.UNIX.num_objs 766.50 ± 10% +7.5% 823.86 ± 10% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% slabinfo.avc_xperms_node.active_objs 766.50 ± 10% +7.5% 823.86 ± 10% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% slabinfo.avc_xperms_node.num_objs 507.00 ± 9% +16.1% 588.57 ± 10% +21.1% 614.00 ± 4% +3.1% 522.75 ± 10% slabinfo.file_lock_cache.active_objs 507.00 ± 9% +16.1% 588.57 ± 10% +21.1% 614.00 ± 4% +3.1% 522.75 ± 10% slabinfo.file_lock_cache.num_objs 13334 ± 4% +6.9% 14255 ± 4% +12.1% 14952 ± 4% +11.8% 14907 ± 11% slabinfo.kmalloc-512.num_objs 357.00 ± 2% +0.7% 359.43 ± 1% +35.2% 482.75 ± 0% +33.9% 478.00 ± 0% slabinfo.kmalloc-8192.num_objs 8080 ± 3% +1.9% 8233 ± 4% +16.8% 9441 ± 5% +17.2% 9470 ± 1% slabinfo.kmalloc-96.active_objs 8125 ± 3% +1.9% 8281 ± 4% +16.8% 9488 ± 5% +17.1% 9511 ± 1% slabinfo.kmalloc-96.num_objs 1112 ± 4% -2.5% 1084 ± 5% +6.6% 1186 ± 11% +11.9% 1244 ± 2% slabinfo.task_group.active_objs 1112 ± 4% -2.5% 1084 ± 5% +6.6% 1186 ± 11% +11.9% 1244 ± 2% slabinfo.task_group.num_objs 18.81 ± 7% +12.4% 21.13 ± 28% +4.5e+06% 837325 ± 3% +4.5e+06% 846688 ± 0% sched_debug.cfs_rq:/.load.avg 90.42 ± 75% +88.7% 170.62 ±137% +1.1e+06% 1028138 ± 0% +1.3e+06% 1157227 ± 19% sched_debug.cfs_rq:/.load.max 10.83 ± 25% +13.6% 12.31 ± 12% +4.6e+06% 500135 ± 62% +5.8e+06% 625582 ± 11% sched_debug.cfs_rq:/.load.min 12.00 ± 81% +96.8% 23.63 ±144% +9.6e+05% 115762 ± 29% +8e+05% 96269 ± 29% sched_debug.cfs_rq:/.load.stddev 26.71 ± 11% -8.9% 24.33 ± 10% +2902.0% 801.76 ± 2% +2935.3% 810.66 ± 0% sched_debug.cfs_rq:/.load_avg.avg 241.42 ± 34% -32.4% 163.29 ± 54% +294.1% 951.38 ± 2% +299.9% 965.33 ± 3% sched_debug.cfs_rq:/.load_avg.max 14.13 ± 5% +4.8% 14.81 ± 4% +3872.3% 561.08 ± 18% +4326.5% 625.25 ± 5% sched_debug.cfs_rq:/.load_avg.min 37.52 ± 37% -35.7% 24.15 ± 48% +103.7% 76.43 ± 19% +49.9% 56.27 ± 7% sched_debug.cfs_rq:/.load_avg.stddev 6864771 ± 0% +1.6% 6971358 ± 0% -97.9% 146805 ± 0% -97.9% 147296 ± 0% sched_debug.cfs_rq:/.min_vruntime.avg 6984488 ± 0% +1.2% 7071775 ± 0% -97.7% 158812 ± 0% -97.7% 160483 ± 1% sched_debug.cfs_rq:/.min_vruntime.max 6522931 ± 1% +1.2% 6598038 ± 1% -97.8% 141019 ± 1% -97.8% 141943 ± 0% sched_debug.cfs_rq:/.min_vruntime.min 80297 ± 7% -5.5% 75882 ± 12% -95.3% 3775 ± 7% -95.4% 3703 ± 9% sched_debug.cfs_rq:/.min_vruntime.stddev 16.76 ± 1% +1.4% 16.98 ± 0% +4570.6% 782.68 ± 2% +4662.4% 798.07 ± 0% sched_debug.cfs_rq:/.runnable_load_avg.avg 28.88 ± 7% +10.6% 31.93 ± 7% +3065.5% 914.04 ± 3% +3138.1% 935.00 ± 1% sched_debug.cfs_rq:/.runnable_load_avg.max 9.54 ± 28% +18.5% 11.31 ± 11% +4105.7% 401.29 ± 62% +5300.0% 515.25 ± 13% sched_debug.cfs_rq:/.runnable_load_avg.min 2.92 ± 11% +2.0% 2.98 ± 10% +3057.4% 92.11 ± 45% +2153.7% 65.75 ± 13% sched_debug.cfs_rq:/.runnable_load_avg.stddev 83900 ± 25% -46.8% 44629 ± 65% -98.9% 894.85 ±155% -99.8% 201.22 ±111% sched_debug.cfs_rq:/.spread0.max -377675 ±-21% +13.7% -429229 ±-20% -95.5% -16912 ± -5% -95.1% -18353 ±-11% sched_debug.cfs_rq:/.spread0.min 80284 ± 7% -5.5% 75895 ± 12% -95.3% 3778 ± 7% -95.4% 3707 ± 9% sched_debug.cfs_rq:/.spread0.stddev 81.92 ± 23% -30.5% 56.96 ± 12% -6.6% 76.55 ± 29% -28.3% 58.74 ± 9% sched_debug.cfs_rq:/.util_avg.stddev 249892 ± 16% -2.1% 244699 ± 33% +94.1% 485129 ± 13% +114.3% 535496 ± 22% sched_debug.cpu.avg_idle.min 149745 ± 9% +13.0% 169186 ± 7% -28.0% 107794 ± 16% -8.4% 137183 ± 70% sched_debug.cpu.avg_idle.stddev 2.94 ± 10% +21.0% 3.56 ± 33% +107.3% 6.10 ± 7% +84.9% 5.44 ± 11% sched_debug.cpu.clock.stddev 2.94 ± 10% +21.0% 3.56 ± 33% +107.3% 6.10 ± 7% +84.9% 5.44 ± 11% sched_debug.cpu.clock_task.stddev 17.64 ± 9% -0.9% 17.48 ± 7% +4333.8% 781.92 ± 2% +4425.8% 798.14 ± 0% sched_debug.cpu.cpu_load[0].avg 69.46 ±103% -20.3% 55.38 ±102% +1216.0% 914.04 ± 3% +1246.3% 935.08 ± 1% sched_debug.cpu.cpu_load[0].max 11.08 ± 24% +25.0% 13.86 ± 12% +3294.7% 376.25 ± 67% +4554.1% 515.83 ± 12% sched_debug.cpu.cpu_load[0].min 8.49 ±115% -28.6% 6.06 ±132% +1028.8% 95.82 ± 43% +674.4% 65.73 ± 13% sched_debug.cpu.cpu_load[0].stddev 17.31 ± 5% -0.1% 17.29 ± 3% +4472.2% 791.32 ± 2% +4547.7% 804.39 ± 0% sched_debug.cpu.cpu_load[1].avg 48.17 ± 72% -8.0% 44.33 ± 60% +1832.6% 930.88 ± 2% +1837.6% 933.29 ± 1% sched_debug.cpu.cpu_load[1].max 12.04 ± 16% +15.5% 13.90 ± 11% +4315.6% 531.71 ± 19% +5030.8% 617.83 ± 4% sched_debug.cpu.cpu_load[1].min 5.37 ± 86% -16.0% 4.51 ± 82% +1297.0% 75.04 ± 37% +890.6% 53.21 ± 14% sched_debug.cpu.cpu_load[1].stddev 17.22 ± 3% -0.2% 17.19 ± 1% +4482.9% 788.99 ± 2% +4559.4% 802.18 ± 0% sched_debug.cpu.cpu_load[2].avg 40.29 ± 36% -4.6% 38.43 ± 32% +2179.5% 918.46 ± 1% +2210.0% 930.75 ± 1% sched_debug.cpu.cpu_load[2].max 12.25 ± 16% +13.1% 13.86 ± 10% +4163.6% 522.29 ± 21% +4879.3% 609.96 ± 5% sched_debug.cpu.cpu_load[2].min 4.29 ± 45% -13.7% 3.70 ± 44% +1627.3% 74.02 ± 36% +1125.0% 52.50 ± 13% sched_debug.cpu.cpu_load[2].stddev 17.16 ± 2% -0.2% 17.13 ± 1% +4483.1% 786.38 ± 2% +4563.1% 800.09 ± 0% sched_debug.cpu.cpu_load[3].avg 36.12 ± 14% -3.7% 34.79 ± 15% +2413.4% 907.96 ± 2% +2461.4% 925.29 ± 1% sched_debug.cpu.cpu_load[3].max 12.38 ± 15% +12.7% 13.95 ± 9% +3985.2% 505.54 ± 25% +4706.1% 594.75 ± 4% sched_debug.cpu.cpu_load[3].min 3.72 ± 21% -14.2% 3.19 ± 22% +1887.6% 73.94 ± 37% +1343.2% 53.69 ± 10% sched_debug.cpu.cpu_load[3].stddev 17.12 ± 1% -0.1% 17.10 ± 1% +4478.8% 783.84 ± 2% +4561.4% 797.98 ± 0% sched_debug.cpu.cpu_load[4].avg 33.42 ± 2% -2.6% 32.55 ± 7% +2573.3% 893.33 ± 3% +2633.2% 913.33 ± 1% sched_debug.cpu.cpu_load[4].max 12.88 ± 12% +10.4% 14.21 ± 7% +3701.3% 489.42 ± 26% +4365.7% 574.96 ± 6% sched_debug.cpu.cpu_load[4].min 3.30 ± 5% -12.2% 2.89 ± 14% +2147.0% 74.10 ± 38% +1571.6% 55.12 ± 9% sched_debug.cpu.cpu_load[4].stddev 1722 ± 32% +14.8% 1977 ± 39% +50.3% 2588 ± 58% +49.2% 2570 ± 17% sched_debug.cpu.curr->pid.min 20.57 ± 7% +5.1% 21.62 ± 27% +4.1e+06% 835527 ± 2% +4.1e+06% 847625 ± 0% sched_debug.cpu.load.avg 169.12 ± 41% +15.5% 195.38 ±117% +6.1e+05% 1027133 ± 0% +7.1e+05% 1200175 ± 14% sched_debug.cpu.load.max 10.88 ± 25% +13.2% 12.31 ± 12% +4.6e+06% 500134 ± 62% +5.8e+06% 625582 ± 11% sched_debug.cpu.load.min 23.05 ± 44% +17.7% 27.14 ±122% +4.9e+05% 113672 ± 31% +4.4e+05% 102223 ± 24% sched_debug.cpu.load.stddev 0.00 ± 2% +2.4% 0.00 ± 1% +31.6% 0.00 ± 14% +20.6% 0.00 ± 17% sched_debug.cpu.next_balance.stddev 1623 ± 9% +4.1% 1689 ± 8% +74.5% 2831 ± 3% +73.9% 2823 ± 7% sched_debug.cpu.nr_load_updates.stddev 159639 ± 1% -11.5% 141259 ± 8% -17.0% 132534 ± 1% -15.7% 134652 ± 1% sched_debug.cpu.nr_switches.avg 11.79 ± 15% +9.0% 12.86 ± 25% +268.6% 43.46 ± 18% +273.9% 44.08 ± 11% sched_debug.cpu.nr_uninterruptible.max -16.00 ±-13% -5.2% -15.17 ±-22% +337.5% -70.00 ±-10% +336.7% -69.88 ±-26% sched_debug.cpu.nr_uninterruptible.min 5.10 ± 9% -1.0% 5.05 ± 8% +414.2% 26.25 ± 16% +399.5% 25.50 ± 10% sched_debug.cpu.nr_uninterruptible.stddev 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.01 ±139% +Inf% 6.56 ± 22% perf-profile.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 ± -1% +Inf% 5.76 ±172% +Inf% 5.79 ±122% +Inf% 16.45 ± 16% perf-profile.cycles-pp.__do_page_fault.do_page_fault.page_fault 0.00 ± -1% +Inf% 1.58 ±162% +Inf% 4.57 ±139% +Inf% 3.30 ± 19% perf-profile.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 ± -1% +Inf% 0.93 ±158% +Inf% 1.05 ±102% +Inf% 3.13 ± 16% perf-profile.cycles-pp.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency 0.00 ± -1% +Inf% 0.40 ±159% +Inf% 0.57 ±104% +Inf% 1.18 ± 6% perf-profile.cycles-pp.__schedule.schedule.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 0.59 ±159% +Inf% 0.83 ±100% +Inf% 2.00 ± 23% perf-profile.cycles-pp.__schedule.schedule.pipe_wait.pipe_write.__vfs_write 0.00 ± -1% +Inf% 6.44 ±159% +Inf% 8.27 ±108% +Inf% 19.49 ± 11% perf-profile.cycles-pp.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 3.43 ±158% +Inf% 4.53 ±100% +Inf% 11.07 ± 16% perf-profile.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 3.18 ±158% +Inf% 3.39 ±102% +Inf% 9.81 ± 22% perf-profile.cycles-pp.__wake_up_common.__wake_up_sync_key.pipe_read.__vfs_read.vfs_read 0.00 ± -1% +Inf% 3.24 ±158% +Inf% 3.44 ±102% +Inf% 10.05 ± 22% perf-profile.cycles-pp.__wake_up_sync_key.pipe_read.__vfs_read.vfs_read.sys_read 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.23 ±141% +Inf% 8.04 ± 21% perf-profile.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function 0.00 ± -1% +Inf% 0.23 ±166% +Inf% 0.34 ±104% +Inf% 0.81 ± 27% perf-profile.cycles-pp.anon_pipe_buf_release.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath 0.02 ± 0% +10478.6% 2.12 ±162% +31737.5% 6.37 ±138% +22075.0% 4.43 ± 18% perf-profile.cycles-pp.apic_timer_interrupt 0.00 ± -1% +Inf% 3.14 ±158% +Inf% 3.38 ±102% +Inf% 9.79 ± 22% perf-profile.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read.__vfs_read 0.00 ± -1% +Inf% 0.24 ±159% +Inf% 0.51 ±100% +Inf% 1.00 ± 24% perf-profile.cycles-pp.bit_cursor.fb_flashcursor.process_one_work.worker_thread.kthread 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 2.84 ±153% +Inf% 0.85 ± 20% perf-profile.cycles-pp.call_console_drivers.constprop.23.console_unlock.vprintk_emit.vprintk_default.printk 27.10 ± 20% -53.0% 12.73 ±105% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.call_cpuidle 0.00 ± -1% +Inf% 4.85 ±181% +Inf% 15.07 ±118% +Inf% 29.07 ± 9% perf-profile.cycles-pp.call_cpuidle.cpu_startup_entry.start_secondary 0.02 ± 19% +5855.6% 1.34 ±175% +5722.2% 1.31 ±107% +14988.9% 3.40 ± 9% perf-profile.cycles-pp.call_function_interrupt 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.84 ±153% +Inf% 0.85 ± 20% perf-profile.cycles-pp.console_unlock.vprintk_emit.vprintk_default.printk.perf_duration_warn 0.00 ± -1% +Inf% 2.23 ±182% +Inf% 2.46 ±131% +Inf% 6.55 ± 31% perf-profile.cycles-pp.copy_page.migrate_misplaced_transhuge_page.do_huge_pmd_numa_page.handle_mm_fault.__do_page_fault 0.00 ± -1% +Inf% 2.42 ±159% +Inf% 3.04 ±100% +Inf% 7.95 ± 16% perf-profile.cycles-pp.copy_page_from_iter.pipe_write.__vfs_write.vfs_write.sys_write 0.00 ± -1% +Inf% 0.35 ±160% +Inf% 0.49 ±104% +Inf% 0.98 ± 19% perf-profile.cycles-pp.copy_page_from_iter_iovec.copy_page_from_iter.pipe_write.__vfs_write.vfs_write 0.00 ± -1% +Inf% 2.25 ±163% +Inf% 3.46 ±114% +Inf% 6.78 ± 17% perf-profile.cycles-pp.copy_page_to_iter.pipe_read.__vfs_read.vfs_read.sys_read 0.00 ± -1% +Inf% 2.04 ±159% +Inf% 2.55 ±101% +Inf% 6.82 ± 16% perf-profile.cycles-pp.copy_user_enhanced_fast_string.copy_page_from_iter.pipe_write.__vfs_write.vfs_write 0.00 ± -1% +Inf% 2.02 ±160% +Inf% 3.12 ±110% +Inf% 6.31 ± 15% perf-profile.cycles-pp.copy_user_enhanced_fast_string.copy_page_to_iter.pipe_read.__vfs_read.vfs_read 28.68 ± 21% -55.2% 12.84 ±105% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.cpu_startup_entry 0.00 ± -1% +Inf% 4.91 ±181% +Inf% 15.14 ±118% +Inf% 29.36 ± 8% perf-profile.cycles-pp.cpu_startup_entry.start_secondary 27.10 ± 20% -53.0% 12.73 ±105% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.cpuidle_enter 0.00 ± -1% +Inf% 4.85 ±181% +Inf% 15.07 ±118% +Inf% 29.07 ± 9% perf-profile.cycles-pp.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary 26.95 ± 20% -53.2% 12.62 ±105% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.cpuidle_enter_state 0.00 ± -1% +Inf% 4.79 ±181% +Inf% 15.03 ±118% +Inf% 28.79 ± 9% perf-profile.cycles-pp.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary 0.00 ± -1% +Inf% 0.34 ±158% +Inf% 0.54 ±103% +Inf% 1.14 ± 28% perf-profile.cycles-pp.deactivate_task.__schedule.schedule.pipe_wait.pipe_write 0.00 ± -1% +Inf% 3.12 ±158% +Inf% 3.37 ±102% +Inf% 9.75 ± 22% perf-profile.cycles-pp.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read 0.00 ± -1% +Inf% 0.29 ±159% +Inf% 0.41 ±100% +Inf% 0.97 ± 30% perf-profile.cycles-pp.dequeue_task_fair.deactivate_task.__schedule.schedule.pipe_wait 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.16 ±173% +Inf% 0.88 ± 39% perf-profile.cycles-pp.do_execveat_common.isra.34.sys_execve.do_syscall_64.return_from_SYSCALL_64.execve 0.00 ± -1% +Inf% 2.92 ±179% +Inf% 3.08 ±131% +Inf% 8.56 ± 30% perf-profile.cycles-pp.do_huge_pmd_numa_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.00 ± -1% +Inf% 5.78 ±172% +Inf% 5.81 ±122% +Inf% 16.48 ± 16% perf-profile.cycles-pp.do_page_fault.page_fault 0.00 ± -1% +Inf% 0.32 ±165% +Inf% 0.17 ±173% +Inf% 0.89 ± 39% perf-profile.cycles-pp.do_syscall_64.return_from_SYSCALL_64.execve 0.00 ± -1% +Inf% 1.78 ±158% +Inf% 0.96 ±141% +Inf% 6.24 ± 22% perf-profile.cycles-pp.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.14 ±140% +Inf% 7.53 ± 22% perf-profile.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.21 ±141% +Inf% 7.86 ± 21% perf-profile.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function 0.02 ± 0% +55407.1% 11.10 ±157% +71087.5% 14.24 ±103% +1.7e+05% 34.11 ± 11% perf-profile.cycles-pp.entry_SYSCALL_64_fastpath 0.03 ± 47% +1120.8% 0.34 ±155% +509.1% 0.17 ±173% +3127.3% 0.89 ± 39% perf-profile.cycles-pp.execve 0.00 ± -1% +Inf% 0.47 ±160% +Inf% 0.64 ±104% +Inf% 1.35 ± 4% perf-profile.cycles-pp.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 0.24 ±159% +Inf% 0.56 ±101% +Inf% 1.20 ± 28% perf-profile.cycles-pp.fb_flashcursor.process_one_work.worker_thread.kthread.ret_from_fork 0.00 ± -1% +Inf% 0.94 ±179% +Inf% 1.05 ±105% +Inf% 2.70 ± 11% perf-profile.cycles-pp.flush_smp_call_function_queue.generic_smp_call_function_single_interrupt.smp_call_function_interrupt.call_function_interrupt 0.00 ± -1% +Inf% 0.39 ±180% +Inf% 0.39 ±107% +Inf% 1.25 ± 13% perf-profile.cycles-pp.flush_tlb_func.flush_smp_call_function_queue.generic_smp_call_function_single_interrupt.smp_call_function_interrupt.call_function_interrupt 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.95 ±115% +Inf% 2.86 ± 21% perf-profile.cycles-pp.flush_tlb_page.ptep_clear_flush.try_to_unmap_one.rmap_walk_anon.rmap_walk 0.00 ± -1% +Inf% 1.12 ±177% +Inf% 1.06 ±104% +Inf% 2.79 ± 11% perf-profile.cycles-pp.generic_smp_call_function_single_interrupt.smp_call_function_interrupt.call_function_interrupt 0.00 ± -1% +Inf% 5.49 ±172% +Inf% 5.56 ±123% +Inf% 15.77 ± 17% perf-profile.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 2.08 ±112% +Inf% 6.05 ± 16% perf-profile.cycles-pp.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0.00 ± -1% +Inf% 1.79 ±163% +Inf% 5.26 ±138% +Inf% 3.69 ± 19% perf-profile.cycles-pp.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 ± -1% +Inf% 0.32 ±159% +Inf% 0.19 ±173% +Inf% 0.91 ± 33% perf-profile.cycles-pp.idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.default_wake_function 24.41 ± 20% -50.5% 12.09 ±104% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.intel_idle 0.00 ± -1% +Inf% 3.67 ±234% +Inf% 4.27 ±165% +Inf% 28.93 ± 9% perf-profile.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry 0.11 ±100% -25.2% 0.08 ±244% +2352.4% 2.57 ±150% +709.5% 0.85 ± 20% perf-profile.cycles-pp.irq_work_interrupt 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.57 ±150% +Inf% 0.85 ± 20% perf-profile.cycles-pp.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.57 ±150% +Inf% 0.85 ± 20% perf-profile.cycles-pp.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt 0.00 ± -1% +Inf% 0.37 ±160% +Inf% 0.37 ±103% +Inf% 1.33 ± 13% perf-profile.cycles-pp.is_module_text_address.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk 0.00 ± -1% +Inf% 0.44 ±160% +Inf% 0.85 ±100% +Inf% 1.86 ± 14% perf-profile.cycles-pp.kthread.ret_from_fork 0.00 ± -1% +Inf% 1.83 ±163% +Inf% 5.35 ±138% +Inf% 3.82 ± 19% perf-profile.cycles-pp.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 ± -1% +Inf% 0.24 ±159% +Inf% 0.47 ±100% +Inf% 0.99 ± 25% perf-profile.cycles-pp.memcpy_erms.mga_imageblit.soft_cursor.bit_cursor.fb_flashcursor 0.00 ± -1% +Inf% 0.24 ±159% +Inf% 0.51 ±100% +Inf% 1.00 ± 24% perf-profile.cycles-pp.mga_imageblit.soft_cursor.bit_cursor.fb_flashcursor.process_one_work 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.51 ±115% +Inf% 4.33 ± 21% perf-profile.cycles-pp.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.00 ± -1% +Inf% 2.79 ±183% +Inf% 3.05 ±132% +Inf% 8.22 ± 30% perf-profile.cycles-pp.migrate_misplaced_transhuge_page.do_huge_pmd_numa_page.handle_mm_fault.__do_page_fault.do_page_fault 0.00 ± -1% +Inf% 0.28 ±244% +Inf% 0.33 ±173% +Inf% 1.07 ± 26% perf-profile.cycles-pp.migrate_page_copy.migrate_misplaced_transhuge_page.do_huge_pmd_numa_page.handle_mm_fault.__do_page_fault 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.23 ±116% +Inf% 3.82 ± 20% perf-profile.cycles-pp.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault.__do_page_fault 0.40 ±162% -77.9% 0.09 ±154% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.mutex_spin_on_owner.isra.4 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.94 ±116% +Inf% 2.82 ± 20% perf-profile.cycles-pp.native_flush_tlb_others.flush_tlb_page.ptep_clear_flush.try_to_unmap_one.rmap_walk_anon 0.01 ±100% +4914.3% 0.50 ±162% +7425.0% 0.75 ±126% +12500.0% 1.26 ± 17% perf-profile.cycles-pp.native_irq_return_iret 0.00 ± -1% +Inf% 0.44 ±188% +Inf% 0.37 ±173% +Inf% 1.20 ± 22% perf-profile.cycles-pp.native_send_call_func_ipi.smp_call_function_many.native_flush_tlb_others.flush_tlb_page.ptep_clear_flush 0.02 ± 19% +25709.5% 5.81 ±171% +25733.3% 5.81 ±122% +73266.7% 16.51 ± 16% perf-profile.cycles-pp.page_fault 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.57 ±150% +Inf% 0.85 ± 20% perf-profile.cycles-pp.perf_duration_warn.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt 0.50 ±104% -6.2% 0.47 ±157% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.pipe_read 0.00 ± -1% +Inf% 6.10 ±159% +Inf% 7.73 ±109% +Inf% 18.29 ± 11% perf-profile.cycles-pp.pipe_read.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 0.80 ±158% +Inf% 1.03 ±100% +Inf% 2.40 ± 24% perf-profile.cycles-pp.pipe_wait.pipe_write.__vfs_write.vfs_write.sys_write 0.00 ± -1% +Inf% 3.41 ±158% +Inf% 2.56 ±147% +Inf% 11.80 ± 18% perf-profile.cycles-pp.pipe_write.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath 2.49 ± 40% -79.0% 0.52 ±151% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.poll_idle 0.00 ± -1% +Inf% 1.69 ±158% +Inf% 0.87 ±139% +Inf% 5.77 ± 22% perf-profile.cycles-pp.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.57 ±150% +Inf% 0.85 ± 20% perf-profile.cycles-pp.printk.perf_duration_warn.irq_work_run_list.irq_work_run.smp_irq_work_interrupt 0.00 ± -1% +Inf% 0.25 ±160% +Inf% 0.60 ±100% +Inf% 1.21 ± 27% perf-profile.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.95 ±115% +Inf% 2.87 ± 21% perf-profile.cycles-pp.ptep_clear_flush.try_to_unmap_one.rmap_walk_anon.rmap_walk.try_to_unmap 0.02 ± 0% +807.1% 0.18 ±209% +1700.0% 0.36 ±102% +4900.0% 1.00 ± 29% perf-profile.cycles-pp.read 1.11 ± 61% -97.6% 0.03 ±216% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.rest_init 0.02 ± 24% +2479.6% 0.45 ±153% +4742.9% 0.85 ±100% +10542.9% 1.86 ± 14% perf-profile.cycles-pp.ret_from_fork 0.00 ± -1% +Inf% 0.32 ±165% +Inf% 0.17 ±173% +Inf% 0.89 ± 39% perf-profile.cycles-pp.return_from_SYSCALL_64.execve 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.98 ±116% +Inf% 2.97 ± 23% perf-profile.cycles-pp.rmap_walk.try_to_unmap.migrate_pages.migrate_misplaced_page.handle_pte_fault 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.97 ±115% +Inf% 2.97 ± 23% perf-profile.cycles-pp.rmap_walk_anon.rmap_walk.try_to_unmap.migrate_pages.migrate_misplaced_page 0.00 ± -1% +Inf% 1.79 ±158% +Inf% 0.97 ±139% +Inf% 6.27 ± 22% perf-profile.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task 0.00 ± -1% +Inf% 0.43 ±159% +Inf% 0.61 ±103% +Inf% 1.27 ± 3% perf-profile.cycles-pp.schedule.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 0.60 ±159% +Inf% 0.88 ±100% +Inf% 2.06 ± 22% perf-profile.cycles-pp.schedule.pipe_wait.pipe_write.__vfs_write.vfs_write 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 2.38 ±136% +Inf% 1.74 ± 20% perf-profile.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.isra.17.tick_sched_timer.__hrtimer_run_queues 0.00 ± -1% +Inf% 0.42 ±159% +Inf% 0.42 ±113% +Inf% 1.28 ± 28% perf-profile.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.default_wake_function.autoremove_wake_function 0.00 ± -1% +Inf% 0.51 ±158% +Inf% 0.53 ±109% +Inf% 1.54 ± 27% perf-profile.cycles-pp.select_task_rq_fair.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 2.69 ±152% +Inf% 0.84 ± 19% perf-profile.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.univ8250_console_write.call_console_drivers.constprop.23 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 2.69 ±152% +Inf% 0.84 ± 19% perf-profile.cycles-pp.serial8250_console_write.univ8250_console_write.call_console_drivers.constprop.23.console_unlock.vprintk_emit 0.00 ± -1% +Inf% 2.04 ±163% +Inf% 6.27 ±138% +Inf% 4.34 ± 18% perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 ± -1% +Inf% 1.28 ±178% +Inf% 1.21 ±105% +Inf% 3.17 ± 8% perf-profile.cycles-pp.smp_call_function_interrupt.call_function_interrupt 0.00 ± -1% +Inf% 0.93 ±177% +Inf% 0.94 ±116% +Inf% 2.80 ± 19% perf-profile.cycles-pp.smp_call_function_many.native_flush_tlb_others.flush_tlb_page.ptep_clear_flush.try_to_unmap_one 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.57 ±150% +Inf% 0.85 ± 20% perf-profile.cycles-pp.smp_irq_work_interrupt.irq_work_interrupt 0.00 ± -1% +Inf% 0.24 ±159% +Inf% 0.51 ±100% +Inf% 1.00 ± 24% perf-profile.cycles-pp.soft_cursor.bit_cursor.fb_flashcursor.process_one_work.worker_thread 1.11 ± 61% -97.6% 0.03 ±216% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.start_kernel 0.00 ± -1% +Inf% 0.32 ±165% +Inf% 0.17 ±173% +Inf% 0.89 ± 39% perf-profile.cycles-pp.sys_execve.do_syscall_64.return_from_SYSCALL_64.execve 0.00 ± -1% +Inf% 6.82 ±159% +Inf% 8.52 ±108% +Inf% 20.43 ± 11% perf-profile.cycles-pp.sys_read.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 3.45 ±158% +Inf% 4.57 ±100% +Inf% 11.18 ± 16% perf-profile.cycles-pp.sys_write.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 0.49 ±159% +Inf% 0.64 ±103% +Inf% 1.44 ± 5% perf-profile.cycles-pp.syscall_return_slowpath.entry_SYSCALL_64_fastpath 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.70 ±141% +Inf% 1.16 ± 25% perf-profile.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.isra.17.tick_sched_timer 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 3.39 ±134% +Inf% 2.78 ± 22% perf-profile.cycles-pp.tick_sched_handle.isra.17.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.local_apic_timer_interrupt 0.00 ± -1% +Inf% 1.35 ±162% +Inf% 3.59 ±135% +Inf% 2.89 ± 22% perf-profile.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.98 ±116% +Inf% 3.00 ± 23% perf-profile.cycles-pp.try_to_unmap.migrate_pages.migrate_misplaced_page.handle_pte_fault.handle_mm_fault 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 0.95 ±115% +Inf% 2.94 ± 23% perf-profile.cycles-pp.try_to_unmap_one.rmap_walk_anon.rmap_walk.try_to_unmap.migrate_pages 0.00 ± -1% +Inf% 3.11 ±158% +Inf% 1.66 ±143% +Inf% 10.32 ± 21% perf-profile.cycles-pp.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 1.33 ±143% +Inf% 8.29 ± 21% perf-profile.cycles-pp.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 2.69 ±152% +Inf% 0.84 ± 19% perf-profile.cycles-pp.uart_console_write.serial8250_console_write.univ8250_console_write.call_console_drivers.constprop.23.console_unlock 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 2.69 ±152% +Inf% 0.84 ± 19% perf-profile.cycles-pp.univ8250_console_write.call_console_drivers.constprop.23.console_unlock.vprintk_emit.vprintk_default 0.00 ± -1% +NaN% 0.00 ± -1% +Inf% 3.30 ±133% +Inf% 2.70 ± 22% perf-profile.cycles-pp.update_process_times.tick_sched_handle.isra.17.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 ± -1% +Inf% 6.74 ±159% +Inf% 8.48 ±108% +Inf% 20.19 ± 11% perf-profile.cycles-pp.vfs_read.sys_read.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 3.45 ±158% +Inf% 4.55 ±100% +Inf% 11.15 ± 16% perf-profile.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.84 ±153% +Inf% 0.85 ± 20% perf-profile.cycles-pp.vprintk_default.printk.perf_duration_warn.irq_work_run_list.irq_work_run 0.00 ± -1% +Inf% 0.08 ±244% +Inf% 2.84 ±153% +Inf% 0.85 ± 20% perf-profile.cycles-pp.vprintk_emit.vprintk_default.printk.perf_duration_warn.irq_work_run_list 0.00 ± -1% +Inf% 0.07 ±244% +Inf% 2.59 ±152% +Inf% 0.81 ± 19% perf-profile.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.univ8250_console_write 0.00 ± -1% +Inf% 0.25 ±160% +Inf% 0.61 ±100% +Inf% 1.23 ± 26% perf-profile.cycles-pp.worker_thread.kthread.ret_from_fork 1.11 ± 61% -85.6% 0.16 ±199% -87.8% 0.14 ±173% -85.8% 0.16 ±173% perf-profile.cycles-pp.x86_64_start_kernel 1.11 ± 61% -97.6% 0.03 ±216% -100.0% 0.00 ± -1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.x86_64_start_reservations