May 2020 - LKP - Ml01.01.Org

[mm/lru] 94a3fe59f5: reaim.jobs_per_min -37.0% regression

by kernel test robot

Greeting, FYI, we noticed a -37.0% regression of reaim.jobs_per_min due to commit: commit: 94a3fe59f55ec752729e959b37547f2e38171ee2 ("mm/lru: replace pgdat lru_lock with lruvec lock") https://github.com/alexshi/linux.git lrulock in testcase: reaim on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory with following parameters: runtime: 300s nr_task: 1000t test: mem_rtns_1 cpufreq_governor: performance ucode: 0x42e test-description: REAIM is an updated and improved version of AIM 7 benchmark. test-url: https://sourceforge.net/projects/re-aim-7/ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/1000t/debian-x86_64-20191114.cgz/300s/lkp-ivb-2ep1/mem_rtns_1/reaim/0x42e commit: b3f1a029a5 ("mm/thp: take PageLRU before get lru_lock in THP") 94a3fe59f5 ("mm/lru: replace pgdat lru_lock with lruvec lock") b3f1a029a53275c5 94a3fe59f55ec752729e959b375 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at_ip___perf_sw_event/0x 5:4 -63% 2:4 perf-profile.calltrace.cycles-pp.error_entry 5:4 -66% 2:4 perf-profile.children.cycles-pp.error_entry 4:4 -54% 2:4 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 1205 +95.0% 2350 reaim.child_systime 649.37 ± 2% -8.2% 595.96 ± 3% reaim.child_utime 154712 -37.0% 97459 reaim.jobs_per_min 154.71 -37.0% 97.46 reaim.jobs_per_min_child 69.09 ± 2% +10.9% 76.60 reaim.jti 156710 -37.2% 98409 reaim.max_jobs_per_min 38.78 +58.7% 61.57 reaim.parent_time 30.30 ± 5% -24.6% 22.83 ± 5% reaim.std_dev_percent 9.13 ± 4% +30.0% 11.86 ± 5% reaim.std_dev_time 327.12 -2.7% 318.36 reaim.time.elapsed_time 327.12 -2.7% 318.36 reaim.time.elapsed_time.max 3407153 +5.8% 3604244 reaim.time.involuntary_context_switches 1.26e+09 -37.5% 7.879e+08 reaim.time.minor_page_faults 4536 +2.0% 4627 reaim.time.percent_of_cpu_this_job_got 9644 +21.9% 11753 reaim.time.system_time 5195 ± 2% -42.6% 2979 ± 3% reaim.time.user_time 31698 -37.1% 19952 reaim.time.voluntary_context_switches 800000 -37.5% 500000 reaim.workload 5.51 -2.0 3.50 ± 4% mpstat.cpu.all.idle% 61.36 +15.5 76.90 mpstat.cpu.all.sys% 33.12 ± 3% -13.5 19.59 ± 3% mpstat.cpu.all.usr% 60.50 +25.2% 75.75 vmstat.cpu.sy 32.50 ± 2% -40.8% 19.25 ± 4% vmstat.cpu.us 11628 +7.0% 12441 vmstat.system.cs 7.134e+08 ± 12% -39.3% 4.328e+08 ± 9% cpuidle.C6.time 1090874 ± 24% -45.5% 594048 ± 3% cpuidle.C6.usage 1982 ± 12% -16.3% 1660 ± 15% cpuidle.POLL.time 1257 ± 14% -23.0% 968.75 ± 11% cpuidle.POLL.usage 8436 ± 47% -60.8% 3308 ± 64% numa-meminfo.node0.Inactive 9026 ± 49% -65.8% 3084 ± 43% numa-meminfo.node0.Shmem 11237 ± 36% +45.4% 16339 ± 13% numa-meminfo.node1.Inactive 29038 ± 12% +28.0% 37160 ± 4% numa-meminfo.node1.Shmem 6.35e+08 -37.3% 3.983e+08 numa-numastat.node0.local_node 6.35e+08 -37.3% 3.983e+08 numa-numastat.node0.numa_hit 6.35e+08 -38.2% 3.925e+08 numa-numastat.node1.local_node 6.35e+08 -38.2% 3.925e+08 numa-numastat.node1.numa_hit 2255 ± 49% -65.8% 771.50 ± 43% numa-vmstat.node0.nr_shmem 3.167e+08 -36.8% 2e+08 numa-vmstat.node0.numa_hit 3.166e+08 -36.9% 1.999e+08 numa-vmstat.node0.numa_local 7277 ± 11% +28.2% 9326 ± 4% numa-vmstat.node1.nr_shmem 3.171e+08 -37.8% 1.973e+08 numa-vmstat.node1.numa_hit 3.17e+08 -37.8% 1.973e+08 numa-vmstat.node1.numa_local 113934 +5.6% 120326 proc-vmstat.nr_active_anon 108475 +5.4% 114311 proc-vmstat.nr_anon_pages 23259 +4.4% 24280 proc-vmstat.nr_kernel_stack 9161 +3.8% 9513 ± 2% proc-vmstat.nr_page_table_pages 9518 ± 2% +6.2% 10107 proc-vmstat.nr_shmem 113934 +5.6% 120326 proc-vmstat.nr_zone_active_anon 1.27e+09 -37.7% 7.908e+08 proc-vmstat.numa_hit 1.27e+09 -37.7% 7.908e+08 proc-vmstat.numa_local 8536 ± 4% +7.9% 9207 ± 2% proc-vmstat.pgactivate 1.27e+09 -37.7% 7.91e+08 proc-vmstat.pgalloc_normal 1.261e+09 -37.5% 7.887e+08 proc-vmstat.pgfault 1.27e+09 -37.7% 7.909e+08 proc-vmstat.pgfree 201.75 ± 4% -15.4% 170.75 ± 4% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1 172.25 ± 3% +27.3% 219.25 ± 5% interrupts.40:PCI-MSI.2621446-edge.eth0-TxRx-5 4125 +46.1% 6026 ± 31% interrupts.CPU1.NMI:Non-maskable_interrupts 4125 +46.1% 6026 ± 31% interrupts.CPU1.PMI:Performance_monitoring_interrupts 5101 ± 33% +56.0% 7956 ± 5% interrupts.CPU17.NMI:Non-maskable_interrupts 5101 ± 33% +56.0% 7956 ± 5% interrupts.CPU17.PMI:Performance_monitoring_interrupts 201.75 ± 4% -15.4% 170.75 ± 4% interrupts.CPU25.36:PCI-MSI.2621442-edge.eth0-TxRx-1 172.25 ± 3% +27.3% 219.25 ± 5% interrupts.CPU29.40:PCI-MSI.2621446-edge.eth0-TxRx-5 5151 ± 34% +40.2% 7221 ± 24% interrupts.CPU35.NMI:Non-maskable_interrupts 5151 ± 34% +40.2% 7221 ± 24% interrupts.CPU35.PMI:Performance_monitoring_interrupts 8042 ± 3% -33.7% 5330 ± 24% interrupts.CPU41.NMI:Non-maskable_interrupts 8042 ± 3% -33.7% 5330 ± 24% interrupts.CPU41.PMI:Performance_monitoring_interrupts 6.75 ± 38% +788.9% 60.00 ± 97% interrupts.CPU47.TLB:TLB_shootdowns 2041 ± 36% -54.4% 930.50 ± 36% interrupts.CPU5.RES:Rescheduling_interrupts 5121 ± 33% +38.2% 7080 ± 24% interrupts.CPU7.NMI:Non-maskable_interrupts 5121 ± 33% +38.2% 7080 ± 24% interrupts.CPU7.PMI:Performance_monitoring_interrupts 6201 ± 2% -19.7% 4977 ± 2% slabinfo.Acpi-State.active_objs 6201 ± 2% -19.7% 4977 ± 2% slabinfo.Acpi-State.num_objs 453.25 ± 8% -21.6% 355.50 ± 9% slabinfo.buffer_head.active_objs 453.25 ± 8% -21.6% 355.50 ± 9% slabinfo.buffer_head.num_objs 6376 ± 3% -13.7% 5502 ± 4% slabinfo.cred_jar.active_objs 6383 ± 3% -13.6% 5513 ± 4% slabinfo.cred_jar.num_objs 6716 -21.3% 5284 slabinfo.files_cache.active_objs 6719 -21.4% 5284 slabinfo.files_cache.num_objs 719.00 ± 7% -16.2% 602.50 ± 12% slabinfo.khugepaged_mm_slot.active_objs 719.00 ± 7% -16.2% 602.50 ± 12% slabinfo.khugepaged_mm_slot.num_objs 6879 ± 4% -11.2% 6111 ± 4% slabinfo.pid.active_objs 6884 ± 4% -11.2% 6111 ± 4% slabinfo.pid.num_objs 6480 ± 3% -14.4% 5544 ± 4% slabinfo.task_delay_info.active_objs 6486 ± 4% -14.4% 5553 ± 4% slabinfo.task_delay_info.num_objs 14736 -20.8% 11675 slabinfo.vmap_area.active_objs 14751 -20.8% 11678 slabinfo.vmap_area.num_objs 20408 ± 7% +10.6% 22579 ± 7% sched_debug.cfs_rq:/.load.avg 6716 ± 18% +57.6% 10583 ± 5% sched_debug.cfs_rq:/.load.min 6.29 ± 17% +91.4% 12.04 ± 4% sched_debug.cfs_rq:/.load_avg.min 58.90 ± 9% -25.3% 44.00 ± 16% sched_debug.cfs_rq:/.load_avg.stddev 14228052 ± 8% -24.8% 10697869 ± 4% sched_debug.cfs_rq:/.min_vruntime.avg 23097210 ± 13% -34.5% 15120258 ± 6% sched_debug.cfs_rq:/.min_vruntime.max 5257692 ± 5% +11.3% 5849957 sched_debug.cfs_rq:/.min_vruntime.min 6940140 ± 19% -52.5% 3295666 ± 14% sched_debug.cfs_rq:/.min_vruntime.stddev 146.00 ± 21% -36.6% 92.50 ± 8% sched_debug.cfs_rq:/.nr_spread_over.max 78.71 ± 11% -60.2% 31.29 ± 61% sched_debug.cfs_rq:/.removed.util_avg.max 14.71 ± 32% -60.9% 5.75 ± 67% sched_debug.cfs_rq:/.removed.util_avg.stddev 21948 ± 2% -23.4% 16803 ± 4% sched_debug.cfs_rq:/.runnable_avg.max 4977 ± 23% +47.9% 7363 ± 4% sched_debug.cfs_rq:/.runnable_avg.min 5958 ± 12% -61.7% 2283 ± 3% sched_debug.cfs_rq:/.runnable_avg.stddev 9908803 ± 28% -43.2% 5623802 ± 39% sched_debug.cfs_rq:/.spread0.max 6940230 ± 19% -52.5% 3295710 ± 14% sched_debug.cfs_rq:/.spread0.stddev 681342 ± 5% +14.4% 779335 sched_debug.cpu.avg_idle.min 79838 ± 8% -24.6% 60164 ± 8% sched_debug.cpu.avg_idle.stddev 8049 -26.7% 5900 sched_debug.cpu.curr->pid.avg 9116 -18.9% 7392 sched_debug.cpu.curr->pid.max 20.88 ± 2% -24.2% 15.83 ± 2% sched_debug.cpu.nr_running.max 4.83 ± 24% +48.3% 7.17 ± 4% sched_debug.cpu.nr_running.min 5.73 ± 12% -62.5% 2.15 ± 3% sched_debug.cpu.nr_running.stddev 277.77 ± 2% -28.2% 199.56 ± 3% sched_debug.cpu.sched_goidle.avg 37.71 ± 7% -44.9% 20.79 ± 13% sched_debug.cpu.sched_goidle.min 453.12 ± 23% -22.1% 352.78 ± 8% sched_debug.cpu.sched_goidle.stddev 605.38 ± 9% +32.2% 800.29 ± 13% sched_debug.cpu.ttwu_count.min 4783 ± 6% +7.2% 5126 ± 7% sched_debug.cpu.ttwu_local.max 446.88 ± 12% +25.1% 559.04 ± 13% sched_debug.cpu.ttwu_local.min 48667 -13.0% 42357 ± 3% softirqs.CPU0.RCU 48109 ± 3% -12.2% 42263 softirqs.CPU1.RCU 45884 ± 2% -8.5% 41973 ± 2% softirqs.CPU10.RCU 45812 ± 3% -11.0% 40759 ± 2% softirqs.CPU12.RCU 46392 ± 5% -10.3% 41605 ± 6% softirqs.CPU13.RCU 142526 ± 11% -15.6% 120363 softirqs.CPU13.TIMER 46414 ± 2% -14.0% 39897 ± 5% softirqs.CPU14.RCU 46927 ± 2% -12.9% 40857 ± 3% softirqs.CPU15.RCU 133925 -10.0% 120494 softirqs.CPU15.TIMER 53821 ± 3% -18.8% 43687 ± 3% softirqs.CPU16.RCU 134184 ± 2% -10.3% 120380 softirqs.CPU16.TIMER 53916 ± 3% -17.6% 44423 ± 2% softirqs.CPU17.RCU 134441 ± 2% -10.3% 120628 softirqs.CPU17.TIMER 52967 ± 2% -17.9% 43476 ± 3% softirqs.CPU18.RCU 132945 -10.1% 119531 softirqs.CPU18.TIMER 51522 ± 2% -16.5% 43046 ± 2% softirqs.CPU19.RCU 131626 ± 2% -9.5% 119115 softirqs.CPU19.TIMER 49163 ± 2% -13.2% 42695 ± 2% softirqs.CPU2.RCU 128909 ± 2% -7.9% 118668 softirqs.CPU2.TIMER 48837 -14.0% 42010 ± 2% softirqs.CPU20.RCU 132063 ± 2% -9.4% 119584 softirqs.CPU20.TIMER 48447 ± 3% -13.6% 41873 softirqs.CPU21.RCU 137596 ± 6% -12.5% 120449 softirqs.CPU21.TIMER 46040 ± 2% -12.3% 40369 ± 3% softirqs.CPU22.RCU 45230 ± 3% -9.6% 40896 ± 2% softirqs.CPU23.RCU 131061 ± 2% -8.6% 119738 softirqs.CPU23.TIMER 46841 -9.8% 42242 ± 2% softirqs.CPU24.RCU 128110 ± 2% -7.8% 118082 softirqs.CPU24.TIMER 47572 ± 4% -11.6% 42066 ± 2% softirqs.CPU25.RCU 129325 ± 3% -8.6% 118182 softirqs.CPU25.TIMER 47809 ± 2% -11.0% 42550 ± 2% softirqs.CPU26.RCU 128440 ± 2% -8.3% 117735 softirqs.CPU26.TIMER 47788 -10.9% 42588 softirqs.CPU27.RCU 51548 ± 3% -15.0% 43816 softirqs.CPU28.RCU 51330 ± 4% -15.9% 43177 ± 3% softirqs.CPU29.RCU 49637 ± 2% -13.8% 42767 ± 2% softirqs.CPU3.RCU 128539 ± 2% -7.7% 118608 softirqs.CPU3.TIMER 50774 ± 4% -14.3% 43508 ± 2% softirqs.CPU30.RCU 49221 ± 4% -12.9% 42848 softirqs.CPU31.RCU 45018 -9.3% 40844 softirqs.CPU32.RCU 51392 -18.8% 41713 ± 3% softirqs.CPU36.RCU 51223 ± 3% -16.3% 42849 softirqs.CPU37.RCU 135708 ± 5% -12.2% 119086 softirqs.CPU37.TIMER 53334 ± 6% -15.0% 45337 ± 4% softirqs.CPU38.RCU 52300 ± 2% -14.8% 44567 ± 3% softirqs.CPU39.RCU 133077 ± 2% -9.1% 120957 softirqs.CPU39.TIMER 52743 ± 4% -16.3% 44170 ± 2% softirqs.CPU4.RCU 54340 ± 4% -19.9% 43539 ± 3% softirqs.CPU40.RCU 134788 -10.1% 121238 softirqs.CPU40.TIMER 53250 -18.9% 43204 ± 2% softirqs.CPU41.RCU 133791 ± 2% -10.3% 119973 softirqs.CPU41.TIMER 53406 ± 2% -18.6% 43465 ± 3% softirqs.CPU42.RCU 132429 ± 2% -9.8% 119501 softirqs.CPU42.TIMER 52077 -17.0% 43203 ± 2% softirqs.CPU43.RCU 130943 ± 2% -9.1% 119012 softirqs.CPU43.TIMER 50560 -15.9% 42500 softirqs.CPU44.RCU 132414 ± 2% -10.2% 118853 softirqs.CPU44.TIMER 49609 -14.9% 42238 ± 2% softirqs.CPU45.RCU 143782 ± 12% -17.2% 119074 softirqs.CPU45.TIMER 47657 -12.7% 41599 ± 2% softirqs.CPU46.RCU 131243 ± 2% -7.6% 121305 ± 3% softirqs.CPU46.TIMER 46877 ± 2% -11.6% 41455 ± 3% softirqs.CPU47.RCU 52986 ± 4% -19.9% 42432 ± 8% softirqs.CPU5.RCU 52465 ± 3% -16.5% 43793 ± 2% softirqs.CPU6.RCU 130012 -8.5% 118905 softirqs.CPU6.TIMER 50693 ± 4% -15.4% 42868 softirqs.CPU7.RCU 48012 ± 4% -12.1% 42208 ± 2% softirqs.CPU8.RCU 47855 ± 3% -11.4% 42419 ± 2% softirqs.CPU9.RCU 130592 -8.9% 118936 softirqs.CPU9.TIMER 2361466 -13.8% 2034755 ± 2% softirqs.RCU 250859 ± 5% -13.7% 216455 softirqs.SCHED 40.61 -40.6 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region 39.78 -39.8 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu.tlb_finish_mmu 73.27 -17.2 56.06 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 72.91 -17.0 55.88 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 69.73 -15.6 54.17 perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 67.44 -14.5 52.91 perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 59.81 -11.0 48.83 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 47.89 -10.7 37.18 perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 47.67 -10.6 37.07 perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk 44.85 -9.2 35.62 perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap 5.44 ± 2% -5.4 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.pagevec_lru_move_fn.lru_add_drain_cpu.lru_add_drain.unmap_region 5.39 ± 2% -5.4 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.pagevec_lru_move_fn.lru_add_drain_cpu.lru_add_drain 4.93 -2.4 2.52 perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_anonymous_page.handle_mm_fault.do_page_fault.page_fault 4.79 -2.3 2.51 perf-profile.calltrace.cycles-pp.__split_vma.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.53 -2.2 2.29 perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.handle_mm_fault.do_page_fault 3.96 ± 2% -2.0 2.00 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.handle_mm_fault 3.50 ± 2% -1.7 1.80 perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode 3.94 ± 2% -1.5 2.43 ± 2% perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 2.85 ± 2% -1.4 1.46 perf-profile.calltrace.cycles-pp.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page 2.70 ± 2% -1.3 1.38 perf-profile.calltrace.cycles-pp.clear_page_erms.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma 3.40 ± 2% -1.3 2.12 ± 2% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__x64_sys_brk 2.36 ± 2% -1.1 1.21 ± 2% perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 2.09 -1.0 1.05 perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap 2.13 ± 2% -1.0 1.11 ± 3% perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.__do_munmap.__x64_sys_brk.do_syscall_64 1.69 ± 2% -0.8 0.86 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 1.36 ± 2% -0.7 0.66 perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region 1.42 -0.7 0.77 perf-profile.calltrace.cycles-pp.anon_vma_clone.__split_vma.__do_munmap.__x64_sys_brk.do_syscall_64 1.36 ± 2% -0.6 0.71 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 1.27 ± 2% -0.6 0.66 ± 4% perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 1.14 ± 2% -0.6 0.57 perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu 1.13 ± 2% -0.6 0.58 ± 5% perf-profile.calltrace.cycles-pp.unlink_anon_vmas.free_pgtables.unmap_region.__do_munmap.__x64_sys_brk 1.07 -0.5 0.57 ± 3% perf-profile.calltrace.cycles-pp.mem_cgroup_uncharge_list.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region 0.62 ± 3% -0.1 0.56 perf-profile.calltrace.cycles-pp.page_remove_rmap.unmap_page_range.unmap_vmas.unmap_region.__do_munmap 6.09 +2.2 8.25 ± 3% perf-profile.calltrace.cycles-pp.lru_add_drain.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 6.05 +2.2 8.22 ± 3% perf-profile.calltrace.cycles-pp.lru_add_drain_cpu.lru_add_drain.unmap_region.__do_munmap.__x64_sys_brk 5.87 +2.2 8.12 ± 3% perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.lru_add_drain_cpu.lru_add_drain.unmap_region.__do_munmap 0.00 +7.5 7.51 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.pagevec_lru_move_fn.lru_add_drain_cpu 0.00 +7.6 7.61 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.pagevec_lru_move_fn.lru_add_drain_cpu.lru_add_drain 0.00 +7.7 7.70 ± 3% perf-profile.calltrace.cycles-pp.lock_page_lruvec_irqsave.pagevec_lru_move_fn.lru_add_drain_cpu.lru_add_drain.unmap_region 16.60 +22.0 38.64 perf-profile.calltrace.cycles-pp.page_fault 15.91 +22.4 38.32 perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 14.24 +23.2 37.42 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault 12.81 +23.8 36.62 perf-profile.calltrace.cycles-pp.do_anonymous_page.handle_mm_fault.do_page_fault.page_fault 5.52 ± 2% +27.1 32.65 perf-profile.calltrace.cycles-pp.__lru_cache_add.do_anonymous_page.handle_mm_fault.do_page_fault.page_fault 5.38 ± 2% +27.2 32.58 perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.do_anonymous_page.handle_mm_fault.do_page_fault 0.00 +30.2 30.22 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.pagevec_lru_move_fn.__lru_cache_add 0.00 +30.7 30.69 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.pagevec_lru_move_fn.__lru_cache_add.do_anonymous_page 0.00 +31.1 31.05 perf-profile.calltrace.cycles-pp.lock_page_lruvec_irqsave.pagevec_lru_move_fn.__lru_cache_add.do_anonymous_page.handle_mm_fault 0.00 +32.4 32.42 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.release_pages.tlb_flush_mmu 0.00 +32.8 32.83 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.lock_page_lruvec_irqsave.release_pages.tlb_flush_mmu.tlb_finish_mmu 0.00 +33.1 33.12 perf-profile.calltrace.cycles-pp.lock_page_lruvec_irqsave.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region 73.34 -17.2 56.14 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 73.00 -17.0 55.95 perf-profile.children.cycles-pp.do_syscall_64 69.75 -15.6 54.18 perf-profile.children.cycles-pp.__x64_sys_brk 67.49 -14.5 52.95 perf-profile.children.cycles-pp.__do_munmap 59.84 -11.0 48.86 perf-profile.children.cycles-pp.unmap_region 47.91 -10.7 37.19 perf-profile.children.cycles-pp.tlb_finish_mmu 47.71 -10.6 37.09 perf-profile.children.cycles-pp.tlb_flush_mmu 45.12 -9.2 35.96 perf-profile.children.cycles-pp.release_pages 4.96 -2.4 2.54 perf-profile.children.cycles-pp.alloc_pages_vma 4.81 -2.3 2.53 perf-profile.children.cycles-pp.__split_vma 4.63 ± 3% -2.3 2.35 ± 3% perf-profile.children.cycles-pp.__alloc_pages_nodemask 4.05 ± 4% -2.0 2.05 ± 2% perf-profile.children.cycles-pp.get_page_from_freelist 3.50 ± 2% -1.7 1.80 perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 3.97 ± 2% -1.5 2.44 ± 2% perf-profile.children.cycles-pp.unmap_vmas 2.90 ± 4% -1.4 1.48 ± 2% perf-profile.children.cycles-pp.prep_new_page 2.76 ± 4% -1.4 1.41 ± 2% perf-profile.children.cycles-pp.clear_page_erms 3.47 ± 2% -1.3 2.15 ± 2% perf-profile.children.cycles-pp.unmap_page_range 2.37 ± 2% -1.2 1.22 ± 2% perf-profile.children.cycles-pp.prepare_exit_to_usermode 2.37 ± 2% -1.1 1.24 ± 2% perf-profile.children.cycles-pp.__vma_adjust 2.12 -1.1 1.06 perf-profile.children.cycles-pp.flush_tlb_mm_range 1.86 ± 2% -0.9 0.95 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 1.37 ± 2% -0.7 0.67 perf-profile.children.cycles-pp.flush_tlb_func_common 1.44 -0.7 0.78 perf-profile.children.cycles-pp.anon_vma_clone 1.36 ± 2% -0.6 0.72 perf-profile.children.cycles-pp.entry_SYSCALL_64 1.33 ± 2% -0.6 0.70 ± 2% perf-profile.children.cycles-pp.native_irq_return_iret 1.29 ± 2% -0.6 0.67 ± 4% perf-profile.children.cycles-pp.free_pgtables 1.16 ± 2% -0.6 0.58 perf-profile.children.cycles-pp.native_flush_tlb_one_user 1.16 ± 2% -0.6 0.59 ± 4% perf-profile.children.cycles-pp.unlink_anon_vmas 1.09 -0.5 0.58 ± 3% perf-profile.children.cycles-pp.mem_cgroup_uncharge_list 1.00 ± 3% -0.5 0.50 ± 3% perf-profile.children.cycles-pp.vm_area_dup 1.07 ± 7% -0.5 0.61 ± 6% perf-profile.children.cycles-pp.find_vma 0.92 ± 4% -0.5 0.46 ± 2% perf-profile.children.cycles-pp.kmem_cache_alloc 0.94 ± 2% -0.4 0.49 ± 2% perf-profile.children.cycles-pp.__perf_sw_event 0.95 -0.4 0.51 ± 3% perf-profile.children.cycles-pp.___might_sleep 0.94 -0.4 0.51 ± 2% perf-profile.children.cycles-pp.down_write 0.88 ± 2% -0.4 0.45 ± 5% perf-profile.children.cycles-pp.free_unref_page_list 0.77 ± 6% -0.4 0.39 perf-profile.children.cycles-pp.mem_cgroup_charge 0.78 ± 6% -0.4 0.42 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid 0.73 ± 2% -0.3 0.38 ± 2% perf-profile.children.cycles-pp.___perf_sw_event 0.84 ± 5% -0.3 0.49 perf-profile.children.cycles-pp.do_brk_flags 0.71 ± 2% -0.3 0.38 ± 3% perf-profile.children.cycles-pp.remove_vma 0.70 -0.3 0.39 ± 4% perf-profile.children.cycles-pp._cond_resched 0.60 ± 6% -0.3 0.30 ± 6% perf-profile.children.cycles-pp.kmem_cache_free 0.59 ± 2% -0.3 0.30 ± 4% perf-profile.children.cycles-pp.__might_sleep 0.58 ± 13% -0.3 0.29 ± 5% perf-profile.children.cycles-pp._raw_spin_lock 0.89 ± 2% -0.2 0.64 perf-profile.children.cycles-pp.__mod_lruvec_state 0.94 ± 8% -0.2 0.71 ± 5% perf-profile.children.cycles-pp.apic_timer_interrupt 0.51 ± 2% -0.2 0.28 ± 5% perf-profile.children.cycles-pp.down_write_killable 0.88 ± 9% -0.2 0.65 ± 5% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.46 -0.2 0.24 ± 4% perf-profile.children.cycles-pp.uncharge_page 0.46 ± 2% -0.2 0.25 ± 4% perf-profile.children.cycles-pp.cpumask_any_but 0.40 ± 3% -0.2 0.21 ± 2% perf-profile.children.cycles-pp.up_write 0.73 ± 9% -0.2 0.54 ± 6% perf-profile.children.cycles-pp.hrtimer_interrupt 0.36 ± 4% -0.2 0.18 ± 4% perf-profile.children.cycles-pp.uncharge_batch 0.34 ± 2% -0.2 0.16 ± 6% perf-profile.children.cycles-pp.free_pages_and_swap_cache 0.42 ± 11% -0.2 0.25 ± 14% perf-profile.children.cycles-pp.vmacache_find 0.33 ± 2% -0.2 0.17 ± 4% perf-profile.children.cycles-pp.rcu_all_qs 0.34 ± 8% -0.2 0.17 ± 7% perf-profile.children.cycles-pp.free_unref_page_prepare 0.31 -0.2 0.15 ± 3% perf-profile.children.cycles-pp.__mod_node_page_state 0.33 ± 2% -0.2 0.17 ± 4% perf-profile.children.cycles-pp.__vma_link_rb 0.57 ± 10% -0.1 0.42 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.33 ± 2% -0.1 0.18 ± 2% perf-profile.children.cycles-pp.anon_vma_interval_tree_remove 0.27 ± 3% -0.1 0.14 ± 6% perf-profile.children.cycles-pp.sync_regs 0.28 ± 3% -0.1 0.15 ± 4% perf-profile.children.cycles-pp.anon_vma_interval_tree_insert 0.27 ± 3% -0.1 0.14 ± 3% perf-profile.children.cycles-pp.__vma_rb_erase 0.26 ± 14% -0.1 0.14 ± 11% perf-profile.children.cycles-pp.__count_memcg_events 0.24 ± 7% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.tlb_gather_mmu 0.23 ± 4% -0.1 0.11 ± 7% perf-profile.children.cycles-pp.cgroup_throttle_swaprate 0.24 ± 2% -0.1 0.12 ± 3% perf-profile.children.cycles-pp.up_read 0.29 ± 6% -0.1 0.18 ± 2% perf-profile.children.cycles-pp.perf_event_mmap 0.27 ± 8% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.vma_merge 0.42 ± 14% -0.1 0.31 ± 9% perf-profile.children.cycles-pp.tick_sched_timer 0.22 ± 4% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.free_unref_page_commit 0.18 ± 4% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.__rb_insert_augmented 0.23 ± 3% -0.1 0.13 ± 3% perf-profile.children.cycles-pp.anon_vma_chain_link 0.21 ± 2% -0.1 0.11 ± 7% perf-profile.children.cycles-pp._find_next_bit 0.19 ± 8% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.36 ± 3% -0.1 0.27 perf-profile.children.cycles-pp.__list_add_valid 0.17 ± 32% -0.1 0.07 ± 11% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm 0.20 ± 5% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.__mod_zone_page_state 0.15 ± 2% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics 0.11 ± 10% -0.1 0.03 ±100% perf-profile.children.cycles-pp.userfaultfd_unmap_complete 0.18 ± 7% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.sync_mm_rss 0.17 ± 5% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.downgrade_write 0.15 ± 5% -0.1 0.08 ± 6% perf-profile.children.cycles-pp.userfaultfd_unmap_prep 0.15 ± 4% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.__mod_memcg_state 0.13 ± 6% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.memcg_check_events 0.25 ± 12% -0.1 0.18 ± 10% perf-profile.children.cycles-pp.scheduler_tick 0.63 ± 3% -0.1 0.57 ± 2% perf-profile.children.cycles-pp.page_remove_rmap 0.18 ± 4% -0.1 0.12 ± 8% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.14 ± 3% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.perf_iterate_sb 0.12 ± 14% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.try_charge 0.12 ± 11% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.lru_cache_add_active_or_unevictable 0.11 ± 4% -0.1 0.05 ± 9% perf-profile.children.cycles-pp.down_read_trylock 0.11 ± 4% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.page_mapping 0.12 ± 6% -0.1 0.07 perf-profile.children.cycles-pp.get_task_policy 0.12 ± 3% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.unmap_single_vma 0.12 ± 12% -0.0 0.07 ± 19% perf-profile.children.cycles-pp.vma_adjust_trans_huge 0.11 ± 6% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.get_unmapped_area 0.11 ± 4% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.09 ± 8% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.pmd_devmap_trans_unstable 0.06 -0.0 0.03 ±100% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.09 ± 4% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.vmacache_update 0.09 ± 13% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.__softirqentry_text_start 0.10 ± 15% -0.0 0.08 ± 11% perf-profile.children.cycles-pp.irq_exit 0.06 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.clockevents_program_event 1.23 +0.0 1.27 perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.52 ± 7% +0.1 0.61 ± 6% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.27 ± 24% +0.6 0.83 ± 3% perf-profile.children.cycles-pp.mem_cgroup_page_lruvec 6.23 +2.1 8.33 ± 3% perf-profile.children.cycles-pp.lru_add_drain 6.14 +2.1 8.29 ± 3% perf-profile.children.cycles-pp.lru_add_drain_cpu 49.23 +21.0 70.22 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 50.16 +21.0 71.20 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 16.64 +22.0 38.66 perf-profile.children.cycles-pp.page_fault 15.95 +22.4 38.35 perf-profile.children.cycles-pp.do_page_fault 14.31 +23.1 37.45 perf-profile.children.cycles-pp.handle_mm_fault 12.85 +23.8 36.64 perf-profile.children.cycles-pp.do_anonymous_page 5.53 ± 2% +27.1 32.67 perf-profile.children.cycles-pp.__lru_cache_add 11.27 +29.5 40.74 perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.00 +71.9 71.92 perf-profile.children.cycles-pp.lock_page_lruvec_irqsave 2.99 -1.4 1.58 perf-profile.self.cycles-pp.do_syscall_64 2.73 ± 4% -1.3 1.40 ± 2% perf-profile.self.cycles-pp.clear_page_erms 2.24 ± 2% -1.1 1.11 perf-profile.self.cycles-pp.prepare_exit_to_usermode 1.85 ± 2% -0.9 0.95 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 1.86 -0.8 1.08 ± 2% perf-profile.self.cycles-pp.unmap_page_range 1.36 ± 2% -0.6 0.72 perf-profile.self.cycles-pp.entry_SYSCALL_64 1.33 ± 2% -0.6 0.70 ± 2% perf-profile.self.cycles-pp.native_irq_return_iret 1.15 ± 2% -0.6 0.58 perf-profile.self.cycles-pp.native_flush_tlb_one_user 1.13 -0.6 0.58 perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode 1.11 -0.5 0.60 ± 3% perf-profile.self.cycles-pp.handle_mm_fault 0.91 ± 2% -0.4 0.47 ± 2% perf-profile.self.cycles-pp.__vma_adjust 0.92 -0.4 0.50 ± 3% perf-profile.self.cycles-pp.___might_sleep 0.76 ± 7% -0.4 0.41 ± 4% perf-profile.self.cycles-pp.__list_del_entry_valid 0.67 ± 5% -0.3 0.35 perf-profile.self.cycles-pp.get_page_from_freelist 0.68 ± 3% -0.3 0.38 ± 3% perf-profile.self.cycles-pp.__do_munmap 0.61 ± 3% -0.3 0.31 ± 3% perf-profile.self.cycles-pp.__x64_sys_brk 0.59 ± 5% -0.3 0.29 ± 6% perf-profile.self.cycles-pp.kmem_cache_free 0.61 ± 3% -0.3 0.32 ± 2% perf-profile.self.cycles-pp.___perf_sw_event 0.62 ± 5% -0.3 0.34 ± 2% perf-profile.self.cycles-pp.kmem_cache_alloc 0.57 ± 13% -0.3 0.29 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.53 ± 2% -0.3 0.27 ± 3% perf-profile.self.cycles-pp.__might_sleep 0.56 ± 4% -0.3 0.31 ± 4% perf-profile.self.cycles-pp.find_vma 0.55 ± 2% -0.2 0.31 ± 3% perf-profile.self.cycles-pp.do_page_fault 0.46 -0.2 0.24 ± 4% perf-profile.self.cycles-pp.uncharge_page 0.38 ± 9% -0.2 0.20 ± 8% perf-profile.self.cycles-pp.__alloc_pages_nodemask 0.40 ± 2% -0.2 0.21 ± 6% perf-profile.self.cycles-pp.down_write 0.37 ± 3% -0.2 0.20 ± 2% perf-profile.self.cycles-pp.up_write 0.41 ± 12% -0.2 0.23 ± 9% perf-profile.self.cycles-pp.vmacache_find 0.35 ± 2% -0.2 0.18 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.32 ± 6% -0.2 0.15 ± 2% perf-profile.self.cycles-pp.unlink_anon_vmas 0.35 ± 8% -0.2 0.19 ± 2% perf-profile.self.cycles-pp.anon_vma_clone 0.33 ± 8% -0.2 0.17 ± 8% perf-profile.self.cycles-pp.free_unref_page_prepare 0.33 ± 3% -0.2 0.17 perf-profile.self.cycles-pp.vm_area_dup 0.32 ± 2% -0.2 0.17 ± 2% perf-profile.self.cycles-pp.__vma_link_rb 0.30 -0.2 0.14 ± 5% perf-profile.self.cycles-pp.__mod_node_page_state 0.33 ± 3% -0.1 0.18 ± 2% perf-profile.self.cycles-pp.do_anonymous_page 0.20 ± 4% -0.1 0.06 perf-profile.self.cycles-pp.page_fault 0.31 ± 3% -0.1 0.17 ± 4% perf-profile.self.cycles-pp.tlb_flush_mmu 0.29 ± 2% -0.1 0.15 ± 8% perf-profile.self.cycles-pp._cond_resched 0.27 ± 6% -0.1 0.14 ± 3% perf-profile.self.cycles-pp.flush_tlb_mm_range 0.29 ± 2% -0.1 0.15 ± 3% perf-profile.self.cycles-pp.anon_vma_interval_tree_remove 0.27 ± 6% -0.1 0.14 ± 7% perf-profile.self.cycles-pp.free_unref_page_list 0.25 ± 3% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.sync_regs 0.22 ± 3% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.flush_tlb_func_common 0.17 ± 7% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.pagevec_lru_move_fn 0.27 ± 3% -0.1 0.14 ± 3% perf-profile.self.cycles-pp.anon_vma_interval_tree_insert 0.27 ± 4% -0.1 0.14 perf-profile.self.cycles-pp.__vma_rb_erase 0.29 ± 3% -0.1 0.16 perf-profile.self.cycles-pp.mem_cgroup_uncharge_list 0.26 ± 4% -0.1 0.14 ± 3% perf-profile.self.cycles-pp.rcu_all_qs 0.23 ± 6% -0.1 0.11 ± 9% perf-profile.self.cycles-pp.tlb_gather_mmu 0.25 ± 16% -0.1 0.13 ± 9% perf-profile.self.cycles-pp.__count_memcg_events 0.21 ± 2% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.free_pages_and_swap_cache 0.27 ± 2% -0.1 0.15 ± 7% perf-profile.self.cycles-pp.down_write_killable 0.23 ± 3% -0.1 0.12 ± 7% perf-profile.self.cycles-pp.up_read 0.84 ± 4% -0.1 0.73 perf-profile.self.cycles-pp.release_pages 0.26 ± 4% -0.1 0.15 ± 5% perf-profile.self.cycles-pp.lru_add_drain_cpu 0.22 ± 5% -0.1 0.11 ± 3% perf-profile.self.cycles-pp.uncharge_batch 0.24 -0.1 0.13 ± 6% perf-profile.self.cycles-pp.cpumask_any_but 0.26 ± 5% -0.1 0.15 ± 2% perf-profile.self.cycles-pp.mem_cgroup_charge 0.23 ± 2% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.__perf_sw_event 0.17 ± 32% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm 0.18 ± 4% -0.1 0.08 ± 10% perf-profile.self.cycles-pp.__mod_zone_page_state 0.17 ± 6% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.__rb_insert_augmented 0.20 ± 4% -0.1 0.11 ± 7% perf-profile.self.cycles-pp._find_next_bit 0.18 ± 8% -0.1 0.09 ± 7% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.20 ± 5% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.unmap_vmas 0.18 ± 4% -0.1 0.10 ± 9% perf-profile.self.cycles-pp.cgroup_throttle_swaprate 0.19 ± 4% -0.1 0.10 perf-profile.self.cycles-pp.__split_vma 0.34 ± 2% -0.1 0.26 perf-profile.self.cycles-pp.__list_add_valid 0.18 ± 2% -0.1 0.10 ± 7% perf-profile.self.cycles-pp.tlb_finish_mmu 0.19 ± 6% -0.1 0.11 ± 6% perf-profile.self.cycles-pp.unmap_region 0.16 ± 9% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.alloc_pages_vma 0.10 ± 10% -0.1 0.03 ±100% perf-profile.self.cycles-pp.userfaultfd_unmap_complete 0.14 ± 3% -0.1 0.07 perf-profile.self.cycles-pp.userfaultfd_unmap_prep 0.17 ± 5% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.sync_mm_rss 0.14 ± 6% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.__lru_cache_add 0.14 ± 8% -0.1 0.07 perf-profile.self.cycles-pp.prep_new_page 0.14 ± 6% -0.1 0.07 ± 11% perf-profile.self.cycles-pp.free_pgd_range 0.15 ± 5% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.downgrade_write 0.13 ± 6% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.__mod_memcg_state 0.11 ± 7% -0.1 0.05 ± 8% perf-profile.self.cycles-pp.memcg_check_events 0.18 ± 2% -0.1 0.12 ± 7% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.08 ± 8% -0.1 0.03 ±100% perf-profile.self.cycles-pp.vmacache_update 0.13 ± 7% -0.1 0.08 ± 6% perf-profile.self.cycles-pp.free_unref_page_commit 0.12 ± 16% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.try_charge 0.11 ± 11% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.lru_cache_add_active_or_unevictable 0.72 ± 2% -0.1 0.67 ± 2% perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.09 ± 9% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.remove_vma 0.11 ± 7% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.get_task_policy 0.10 ± 4% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.down_read_trylock 0.10 ± 8% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.page_mapping 0.12 ± 3% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.unmap_single_vma 0.11 ± 13% -0.0 0.07 ± 13% perf-profile.self.cycles-pp.vma_adjust_trans_huge 0.08 ± 8% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.anon_vma_chain_link 0.10 ± 8% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.perf_iterate_sb 0.11 ± 4% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.10 ± 7% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.free_pgtables 0.09 ± 7% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.pmd_devmap_trans_unstable 0.08 ± 10% -0.0 0.05 perf-profile.self.cycles-pp.perf_event_mmap 0.35 ± 2% +0.0 0.39 ± 2% perf-profile.self.cycles-pp.page_remove_rmap 0.94 +0.1 1.00 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.22 ± 11% +0.1 0.29 ± 6% perf-profile.self.cycles-pp.page_add_new_anon_rmap 0.00 +0.1 0.13 ± 5% perf-profile.self.cycles-pp.lock_page_lruvec_irqsave 0.26 ± 25% +0.6 0.82 ± 3% perf-profile.self.cycles-pp.mem_cgroup_page_lruvec 49.23 +21.0 70.21 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath reaim.time.minor_page_faults 1.3e+09 +----------------------------------------------------------------+ 1.25e+09 |.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.| | | 1.2e+09 |-+ | 1.15e+09 |-+ | 1.1e+09 |-O O O O O O O O O O O O O O O O O O O O | 1.05e+09 |-+ | | | 1e+09 |-+ | 9.5e+08 |-+ | 9e+08 |-+ | 8.5e+08 |-+ | | | 8e+08 |-+ O O O O | 7.5e+08 +----------------------------------------------------------------+ reaim.time.voluntary_context_switches 32000 +-------------------------------------------------------------------+ | + + | 30000 |-+ | | | 28000 |-O O O O O O O O O O O O O O O O O O O O | | | 26000 |-+ | | | 24000 |-+ | | | 22000 |-+ | | | 20000 |-+ O O O O | | | 18000 +-------------------------------------------------------------------+ reaim.parent_time 65 +----------------------------------------------------------------------+ | | 60 |-+ O O O O | | | | | 55 |-+ | | | 50 |-+ | | | 45 |-+ | | | | O O O O O O O O O O O O O O O O O O O O | 40 |.+.+..+.+.+.+..+.+.+.+.+..+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+..+.+.| | | 35 +----------------------------------------------------------------------+ reaim.jobs_per_min 160000 +------------------------------------------------------------------+ |.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.| 150000 |-+ | | O O O O O O O O O O O O O O O O O O O O | 140000 |-+ | | | 130000 |-+ | | | 120000 |-+ | | | 110000 |-+ | | | 100000 |-+ O O O | | O | 90000 +------------------------------------------------------------------+ reaim.jobs_per_min_child 160 +---------------------------------------------------------------------+ |.+.+..+.+.+.+.+..+.+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+.+..+.+.| 150 |-+ | | O O O O O O O O O O O O O O O O O O O O | 140 |-+ | | | 130 |-+ | | | 120 |-+ | | | 110 |-+ | | | 100 |-+ O O O | | O | 90 +---------------------------------------------------------------------+ reaim.max_jobs_per_min 160000 +------------------------------------------------------------------+ |.+.+.+.+.+. + +.+.+.+.+..+.+ +.+.+ +.+. +.+.+.+.| 150000 |-+ O O O O O | | O O O O O O O O O O O O O O O | 140000 |-+ | | | 130000 |-+ | | | 120000 |-+ | | | 110000 |-+ | | | 100000 |-+ O O O O | | | 90000 +------------------------------------------------------------------+ reaim.workload 800000 +------------------------------------------------------------------+ | | 750000 |-+ | | | | | 700000 |-O O O O O O O O O O O O O O O O O O O O | | | 650000 |-+ | | | 600000 |-+ | | | | | 550000 |-+ | | | 500000 +------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

8 months, 2 weeks

1
0
0 / 0

[f2fs] da55a3ad36: filebench.sum_bytes_mb/s -44.4% regression

by kernel test robot

Greeting, FYI, we noticed a -44.4% regression of filebench.sum_bytes_mb/s due to commit: commit: da55a3ad3669f17a05e00ec819901293c9cc7bf3 ("[PATCH V4] f2fs: Avoid double lock for cp_rwsem during checkpoint") url: https://github.com/0day-ci/linux/commits/Sayali-Lokhande/f2fs-Avoid-doubl... base: https://git.kernel.org/cgit/linux/kernel/git/jaegeuk/f2fs.git dev-test in testcase: filebench on test machine: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory with following parameters: disk: 1HDD fs: f2fs test: createfiles.f cpufreq_governor: performance ucode: 0x27 If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/1HDD/f2fs/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/lkp-hsw-d01/createfiles.f/filebench/0x27 commit: dcd94277cf ("f2fs: add compressed/gc data read IO stat") da55a3ad36 ("f2fs: Avoid double lock for cp_rwsem during checkpoint") dcd94277cf7b4778 da55a3ad3669f17a05e00ec8199 ---------------- --------------------------- %stddev %change %stddev \ | \ 390.50 -44.4% 216.93 ± 28% filebench.sum_bytes_mb/s 74982 -44.5% 41652 ± 28% filebench.sum_operations/s 0.19 ± 3% +55.6% 0.29 ± 29% filebench.sum_time_ms/op 24992 -44.4% 13884 ± 28% filebench.sum_writes/s 11.21 ± 2% +24.5% 13.96 ± 3% filebench.time.elapsed_time 11.21 ± 2% +24.5% 13.96 ± 3% filebench.time.elapsed_time.max 2599052 +8.1% 2808701 filebench.time.file_system_outputs 34.50 -16.9% 28.67 ± 3% filebench.time.percent_of_cpu_this_job_got 19623 ± 3% +14.0% 22364 filebench.time.voluntary_context_switches 12.67 +2.8% 13.02 boot-time.dhcp 8.25 +3.3 11.54 mpstat.cpu.all.iowait% 4.25 ± 4% -0.5 3.73 ± 4% mpstat.cpu.all.sys% 3.84 ± 2% -0.6 3.25 ± 3% mpstat.cpu.all.usr% 10853 ± 3% +21.6% 13198 ± 6% softirqs.CPU0.TIMER 9855 +21.3% 11958 ± 8% softirqs.CPU4.TIMER 9749 ± 3% +36.2% 13275 ± 21% softirqs.CPU6.TIMER 30472 +11.1% 33865 softirqs.SCHED 957129 ± 18% -33.2% 639243 ± 28% cpuidle.C1.time 39341259 ± 22% -66.3% 13274567 ± 93% cpuidle.C6.time 45789 ± 18% -60.4% 18150 ± 87% cpuidle.C6.usage 32265 ± 19% -40.2% 19301 ± 12% cpuidle.POLL.time 8071 ± 12% -15.2% 6845 ± 11% cpuidle.POLL.usage 84.00 -2.4% 82.00 vmstat.cpu.id 7.00 +42.9% 10.00 vmstat.cpu.wa 14528 -15.9% 12222 ± 2% vmstat.io.bi 20968 +73.5% 36378 ± 6% vmstat.io.bo 1387456 +17.2% 1626013 vmstat.memory.cache 4842 +15.5% 5594 ± 2% interrupts.19:IR-IO-APIC.19-fasteoi.ata_piix 166.00 ± 74% +183.1% 470.00 ± 42% interrupts.28:IR-PCI-MSI.409600-edge.eth0 27387 ± 5% +178.5% 76272 ± 17% interrupts.CPU2.LOC:Local_timer_interrupts 30881 ± 20% +63.4% 50460 ± 25% interrupts.CPU3.LOC:Local_timer_interrupts 166.00 ± 74% +183.1% 470.00 ± 42% interrupts.CPU4.28:IR-PCI-MSI.409600-edge.eth0 75.50 ± 17% +32.0% 99.67 ± 3% interrupts.CPU4.CAL:Function_call_interrupts 175.50 -19.5% 141.33 ± 17% interrupts.CPU4.RES:Rescheduling_interrupts 116.00 ± 17% +53.7% 178.33 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts 82.00 ± 6% +33.7% 109.67 ± 4% interrupts.CPU7.RES:Rescheduling_interrupts 373139 ± 2% +20.3% 448922 ± 4% interrupts.LOC:Local_timer_interrupts 84.66 -2.7% 82.38 iostat.cpu.idle 7.19 +43.5% 10.33 ± 3% iostat.cpu.iowait 4.33 -8.0% 3.98 ± 5% iostat.cpu.system 3.81 -13.2% 3.31 ± 3% iostat.cpu.user 1.25 +14.0% 1.42 ± 3% iostat.sda.avgqu-sz 17.43 +1308.1% 245.35 ±131% iostat.sda.await.max 313.22 -14.1% 269.02 ± 4% iostat.sda.r/s 14565 -15.9% 12254 ± 2% iostat.sda.rkB/s 3322 -16.0% 2789 ± 2% iostat.sda.rrqm/s 3.09 ± 3% +467.6% 17.52 ±109% iostat.sda.svctm 8.71 +2685.7% 242.77 ±133% iostat.sda.svctm.max 69.97 ± 2% +7.7% 75.33 ± 3% iostat.sda.util 35.44 ± 2% +94.3% 68.86 ± 7% iostat.sda.w/s 17.43 +1308.1% 245.35 ±131% iostat.sda.w_await.max 22983 +66.0% 38143 ± 6% iostat.sda.wkB/s 319440 +25.3% 400307 ± 2% meminfo.Active 280118 ± 2% +12.6% 315548 meminfo.Active(anon) 39321 ± 4% +115.6% 84758 ± 6% meminfo.Active(file) 66008 +64.7% 108726 ± 3% meminfo.AnonHugePages 279917 ± 2% +12.7% 315339 meminfo.AnonPages 1339189 +16.9% 1565959 meminfo.Cached 161579 ± 2% +98.1% 320046 ± 8% meminfo.Dirty 201499 ± 3% +90.1% 383044 ± 6% meminfo.Inactive 155133 ± 4% +115.6% 334509 ± 6% meminfo.Inactive(file) 63886 +36.5% 87191 ± 2% meminfo.KReclaimable 1819283 +15.7% 2104828 meminfo.Memused 63886 +36.5% 87191 ± 2% meminfo.SReclaimable 98391 +23.8% 121806 ± 2% meminfo.Slab 502.00 ± 13% +84.8% 927.67 ± 10% meminfo.Writeback 219588 -16.8% 182704 ± 3% meminfo.max_used_kB 70801 +11.4% 78906 proc-vmstat.nr_active_anon 9773 ± 3% +116.4% 21146 ± 7% proc-vmstat.nr_active_file 70754 +11.4% 78854 proc-vmstat.nr_anon_pages 324216 ± 3% +14.1% 369827 ± 4% proc-vmstat.nr_dirtied 40081 +99.1% 79787 ± 9% proc-vmstat.nr_dirty 334411 +17.0% 391272 proc-vmstat.nr_file_pages 1558377 -4.5% 1487609 proc-vmstat.nr_free_pages 11461 +5.7% 12118 ± 2% proc-vmstat.nr_inactive_anon 38579 ± 3% +116.3% 83463 ± 7% proc-vmstat.nr_inactive_file 3993 +1.6% 4055 proc-vmstat.nr_kernel_stack 14932 +4.4% 15587 ± 2% proc-vmstat.nr_mapped 1043 +4.0% 1085 proc-vmstat.nr_page_table_pages 11558 +5.8% 12228 ± 2% proc-vmstat.nr_shmem 15979 +36.3% 21783 ± 3% proc-vmstat.nr_slab_reclaimable 134.50 ± 5% +78.7% 240.33 ± 3% proc-vmstat.nr_writeback 71015 ± 2% +106.0% 146295 ± 3% proc-vmstat.nr_written 70801 +11.5% 78912 proc-vmstat.nr_zone_active_anon 9771 ± 3% +116.4% 21145 ± 7% proc-vmstat.nr_zone_active_file 11460 +5.7% 12117 ± 2% proc-vmstat.nr_zone_inactive_anon 38572 ± 3% +116.4% 83466 ± 7% proc-vmstat.nr_zone_inactive_file 40223 +99.0% 80023 ± 9% proc-vmstat.nr_zone_write_pending 285084 ± 2% +105.4% 585492 ± 3% proc-vmstat.pgpgout 6835 ± 2% -11.2% 6072 ± 9% slabinfo.anon_vma_chain.num_objs 51857 +21.8% 63172 ± 2% slabinfo.dentry.active_objs 2509 +20.9% 3035 ± 2% slabinfo.dentry.active_slabs 52713 +20.9% 63747 ± 2% slabinfo.dentry.num_objs 2509 +20.9% 3035 ± 2% slabinfo.dentry.num_slabs 1385 ± 9% +123.0% 3090 ± 12% slabinfo.f2fs_extent_node.active_objs 1562 ± 6% +102.7% 3167 ± 12% slabinfo.f2fs_extent_node.num_objs 11274 ± 8% +91.6% 21599 ± 6% slabinfo.f2fs_extent_tree.active_objs 224.00 ± 8% +90.2% 426.00 ± 6% slabinfo.f2fs_extent_tree.active_slabs 11471 ± 8% +89.6% 21747 ± 6% slabinfo.f2fs_extent_tree.num_objs 224.00 ± 8% +90.2% 426.00 ± 6% slabinfo.f2fs_extent_tree.num_slabs 10359 ± 3% +110.1% 21760 ± 6% slabinfo.f2fs_inode_cache.active_objs 372.50 ± 3% +109.0% 778.67 ± 6% slabinfo.f2fs_inode_cache.active_slabs 10437 ± 3% +109.0% 21817 ± 6% slabinfo.f2fs_inode_cache.num_objs 372.50 ± 3% +109.0% 778.67 ± 6% slabinfo.f2fs_inode_cache.num_slabs 43654 ± 4% +40.1% 61180 slabinfo.f2fs_nat_entry.active_objs 341.50 ± 4% +40.5% 479.67 slabinfo.f2fs_nat_entry.active_slabs 43751 ± 4% +40.5% 61453 slabinfo.f2fs_nat_entry.num_objs 341.50 ± 4% +40.5% 479.67 slabinfo.f2fs_nat_entry.num_slabs 7680 ± 6% -8.9% 6997 ± 6% slabinfo.kmalloc-8.active_objs 7680 ± 6% -8.9% 6997 ± 6% slabinfo.kmalloc-8.num_objs 22811 +54.6% 35261 ± 3% slabinfo.radix_tree_node.active_objs 855.00 +49.5% 1278 ± 3% slabinfo.radix_tree_node.active_slabs 23949 +49.5% 35802 ± 3% slabinfo.radix_tree_node.num_objs 855.00 +49.5% 1278 ± 3% slabinfo.radix_tree_node.num_slabs 2.53 ±100% +137.9% 6.03 ± 11% sched_debug.cfs_rq:/.exec_clock.avg 131763 ± 2% -30.7% 91313 ± 11% sched_debug.cfs_rq:/.load.avg 43314 ± 20% -100.0% 0.00 sched_debug.cfs_rq:/.load.min 58283 ± 18% +45.6% 84867 ± 7% sched_debug.cfs_rq:/.load.stddev 533.31 -20.8% 422.50 ± 7% sched_debug.cfs_rq:/.load_avg.avg 6147 -20.9% 4861 ± 11% sched_debug.cfs_rq:/.min_vruntime.min 1.00 -29.2% 0.71 ± 8% sched_debug.cfs_rq:/.nr_running.avg 1.00 -100.0% 0.00 sched_debug.cfs_rq:/.nr_running.min 128.00 -100.0% 0.00 sched_debug.cfs_rq:/.removed.load_avg.avg 1024 -100.0% 0.00 sched_debug.cfs_rq:/.removed.load_avg.max 338.66 -100.0% 0.00 sched_debug.cfs_rq:/.removed.load_avg.stddev 128.00 -100.0% 0.00 sched_debug.cfs_rq:/.removed.runnable_avg.avg 1024 -100.0% 0.00 sched_debug.cfs_rq:/.removed.runnable_avg.max 338.66 -100.0% 0.00 sched_debug.cfs_rq:/.removed.runnable_avg.stddev 20.81 ± 92% -100.0% 0.00 sched_debug.cfs_rq:/.removed.util_avg.avg 166.50 ± 92% -100.0% 0.00 sched_debug.cfs_rq:/.removed.util_avg.max 55.06 ± 92% -100.0% 0.00 sched_debug.cfs_rq:/.removed.util_avg.stddev 113.25 ± 35% -61.3% 43.88 ± 70% sched_debug.cfs_rq:/.util_est_enqueued.avg 470.50 ± 8% -42.3% 271.67 ± 68% sched_debug.cfs_rq:/.util_est_enqueued.max 1.50 ± 33% -100.0% 0.00 sched_debug.cfs_rq:/.util_est_enqueued.min 170.87 ± 16% -46.2% 91.94 ± 68% sched_debug.cfs_rq:/.util_est_enqueued.stddev 821833 ± 21% -18.7% 668072 ± 11% sched_debug.cpu.avg_idle.max 271414 ± 14% -12.8% 236616 ± 10% sched_debug.cpu.avg_idle.stddev 0.96 ± 42% -62.0% 0.37 ± 55% sched_debug.cpu.clock.stddev 0.96 ± 42% -62.0% 0.37 ± 55% sched_debug.cpu.clock_task.stddev 1106 ± 13% -22.8% 853.96 ± 7% sched_debug.cpu.curr->pid.avg 240.88 ± 56% +127.4% 547.88 ± 5% sched_debug.cpu.curr->pid.stddev 0.00 ± 2% -28.8% 0.00 ± 15% sched_debug.cpu.next_balance.stddev 1.56 ± 28% -54.7% 0.71 ± 8% sched_debug.cpu.nr_running.avg 3.00 ± 33% -66.7% 1.00 sched_debug.cpu.nr_running.max 3552 ± 13% +41.3% 5019 ± 18% sched_debug.cpu.nr_switches.min filebench.sum_operations_s 80000 +-------------------------------------------------------------------+ |+.++++.+++.+++ +++ ++.++ ++ +++ ++ +.++ ++++.+++.+ + +| 70000 |-+ : : : : : :: : : :: : : : : : :| 60000 |-+ : : : : : :: : : :: : : : : : :| | : : : : : :: : : :: : : : : : :| 50000 |O+ OO OO O :O: OO: O: :O: : : : : : : : : : : : | | : : : : : : : : : : : : : : : : : | 40000 |-+ : : : : : : : : : : : : : : : : : | | :: : : :: : : :: : : :: : :: | 30000 |-+ :: : : :: : : :: : : :: : :: | 20000 |-+ :: : : O :: O: : :: : : :: : :: | | O : :: : : : : : : : : : | 10000 |-+ : :: : : : : : : : : : | | : :: : : : : : : : : : | 0 +-------------------------------------------------------------------+ filebench.sum_writes_s 25000 +-------------------------------------------------------------------+ | : : : : : :: : : :: : : : : : :| | : : : : : :: : : :: : : : : : :| 20000 |-+ : : : : : :: : : :: : : : : : :| | : : : : : : : : : : : : : : : : : | |O OO OO O :O: OO: O: :O: : : : : : : : : : : : | 15000 |-+ : : : : : : : : : : : : : : : : : | | :: : : :: : : : : : : :: :::: | 10000 |-+ :: : : :: : : :: : : :: : :: | | :: : : O :: O: : :: : : :: : :: | | :: : : :: : : :: : : :: : :: | 5000 |-+ O : :: : : : : : : : : : | | : :: : : : : : : : : : | | : :: : : : : : : : : : | 0 +-------------------------------------------------------------------+ filebench.sum_bytes_mb_s 400 +---------------------------------------------------------------------+ | : : : : : : : : : : : : : : : : :| 350 |-+ : : : : : : : : : : : : : : : : :| 300 |-+ : : : : : : : : : : : : : : : : :| | : : : : : : : : : : : : : : : : : | 250 |O+ OO O OO :O:O : O:O :O: : : : : : : : : : : : | | : : : : : : : : : : : : : : : : : | 200 |-+ :: : : :: : : :: : : :: : ::: | | : : : : : : : : : : :: :: | 150 |-+ : : : O : O: : : : : : :: :: | 100 |-+ : : : : : : : : : : :: :: | | O : :: : : : : : : : : : | 50 |-+ : :: : : : : : : : : : | | : :: : : : : : : : : : | 0 +---------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

8 months, 2 weeks

1
0
0 / 0

[eventfd] a4ef93e263: will-it-scale.per_process_ops -2.2% regression

by kernel test robot

Greeting, FYI, we noticed a -2.2% regression of will-it-scale.per_process_ops due to commit: commit: a4ef93e263d720a988a208c56aad22300b944ee6 ("[PATCH v4] eventfd: convert to f_op->read_iter()") url: https://github.com/0day-ci/linux/commits/Jens-Axboe/eventfd-convert-to-f_... in testcase: will-it-scale on test machine: 104 threads Skylake with 192G memory with following parameters: nr_task: 100% mode: process test: eventfd1 cpufreq_governor: performance ucode: 0x2000065 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-skl-fpga01/eventfd1/will-it-scale/0x2000065 commit: 052c467cb5 (" block-5.7-2020-05-01") a4ef93e263 ("eventfd: convert to f_op->read_iter()") 052c467cb58748e3 a4ef93e263d720a988a208c56aa ---------------- --------------------------- %stddev %change %stddev \ | \ 586005 -2.2% 573349 will-it-scale.per_process_ops 60944537 -2.2% 59628423 will-it-scale.workload 1611 -100.0% 0.00 meminfo.Mlocked 36.10 ± 2% +2.9% 37.15 boot-time.boot 3321 +2.5% 3403 boot-time.idle 49320 ±107% -80.6% 9578 ± 2% cpuidle.POLL.time 3633 ± 3% -23.7% 2773 ± 2% cpuidle.POLL.usage 11182 ±141% +139.9% 26832 ± 25% numa-numastat.node0.other_node 341388 ± 7% +6.7% 364292 ± 3% numa-numastat.node1.local_node 52.67 +6.3% 56.00 vmstat.cpu.sy 45.00 -6.7% 42.00 vmstat.cpu.us 1431 ± 2% +5.8% 1514 ± 2% vmstat.system.cs 15631 ± 61% -86.3% 2134 ± 25% numa-meminfo.node0.Inactive 17411 ± 53% -83.2% 2933 ± 26% numa-meminfo.node0.Shmem 12939 ± 75% +105.5% 26586 numa-meminfo.node1.Inactive 12675 ± 77% +109.0% 26486 ± 2% numa-meminfo.node1.Inactive(anon) 39471 ± 16% +12.9% 44547 ± 7% numa-meminfo.node1.KReclaimable 6841 ± 10% +8.1% 7396 ± 6% numa-meminfo.node1.PageTables 39471 ± 16% +12.9% 44547 ± 7% numa-meminfo.node1.SReclaimable 61920 ± 17% +30.8% 80987 numa-meminfo.node1.Shmem 121373 +9.3% 132631 ± 4% numa-meminfo.node1.Slab 211.67 ± 11% -100.0% 0.00 numa-vmstat.node0.nr_mlock 4352 ± 53% -83.2% 733.00 ± 26% numa-vmstat.node0.nr_shmem 155270 ± 10% +10.2% 171086 ± 3% numa-vmstat.node0.numa_other 3168 ± 76% +110.5% 6668 ± 2% numa-vmstat.node1.nr_inactive_anon 190.33 ± 13% -100.0% 0.00 numa-vmstat.node1.nr_mlock 1710 ± 10% +8.0% 1848 ± 6% numa-vmstat.node1.nr_page_table_pages 15498 ± 17% +30.9% 20281 numa-vmstat.node1.nr_shmem 9868 ± 16% +12.9% 11137 ± 7% numa-vmstat.node1.nr_slab_reclaimable 3168 ± 76% +110.5% 6668 ± 2% numa-vmstat.node1.nr_zone_inactive_anon 4141 ± 7% +11.5% 4616 ± 3% slabinfo.eventpoll_pwq.active_objs 4141 ± 7% +11.5% 4616 ± 3% slabinfo.eventpoll_pwq.num_objs 970.67 ± 9% +25.3% 1216 slabinfo.kmalloc-rcl-128.active_objs 970.67 ± 9% +25.3% 1216 slabinfo.kmalloc-rcl-128.num_objs 4484 ± 3% +17.9% 5287 ± 10% slabinfo.kmalloc-rcl-64.active_objs 4484 ± 3% +17.9% 5287 ± 10% slabinfo.kmalloc-rcl-64.num_objs 1119 ± 3% +15.1% 1288 ± 9% slabinfo.skbuff_ext_cache.active_objs 1119 ± 3% +15.1% 1288 ± 9% slabinfo.skbuff_ext_cache.num_objs 2484 +7.4% 2668 ± 3% slabinfo.trace_event_file.active_objs 2484 +7.4% 2668 ± 3% slabinfo.trace_event_file.num_objs 88548 +1.7% 90036 proc-vmstat.nr_active_anon 7035 +0.8% 7092 proc-vmstat.nr_inactive_anon 8842 +1.7% 8995 proc-vmstat.nr_mapped 403.00 -100.0% 0.00 proc-vmstat.nr_mlock 19852 ± 2% +5.8% 21001 proc-vmstat.nr_shmem 19893 +1.7% 20225 proc-vmstat.nr_slab_reclaimable 88548 +1.7% 90036 proc-vmstat.nr_zone_active_anon 7035 +0.8% 7092 proc-vmstat.nr_zone_inactive_anon 731288 +1.3% 740501 proc-vmstat.numa_hit 697623 +1.3% 706849 proc-vmstat.numa_local 14860 ± 92% +58.6% 23562 ± 51% proc-vmstat.numa_pte_updates 18740 ± 3% +9.9% 20592 proc-vmstat.pgactivate 800446 +1.3% 811159 proc-vmstat.pgalloc_normal 9767 ± 12% +10.2% 10765 ± 15% sched_debug.cfs_rq:/.load.avg 210597 +11.3% 234483 sched_debug.cfs_rq:/.min_vruntime.stddev 1.93 ± 19% +27.7% 2.46 ± 3% sched_debug.cfs_rq:/.nr_spread_over.avg 59.22 ± 12% +30.6% 77.33 ± 16% sched_debug.cfs_rq:/.nr_spread_over.max 8.10 ± 17% +29.6% 10.49 ± 8% sched_debug.cfs_rq:/.nr_spread_over.stddev 210512 +11.3% 234373 sched_debug.cfs_rq:/.spread0.stddev 62.54 ± 8% +24.8% 78.03 ± 15% sched_debug.cfs_rq:/.util_avg.stddev 78.05 ± 5% +22.8% 95.81 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev 19567 ± 14% +49.4% 29227 sched_debug.cpu.nr_switches.max 3236 ± 4% +14.1% 3692 sched_debug.cpu.nr_switches.stddev 783.06 ± 2% +11.7% 874.67 ± 7% sched_debug.cpu.sched_count.min 293.67 ± 7% +20.6% 354.17 ± 9% sched_debug.cpu.ttwu_count.min 618.29 ± 2% +10.6% 683.94 ± 3% sched_debug.cpu.ttwu_local.avg 280.56 ± 5% +18.2% 331.58 ± 8% sched_debug.cpu.ttwu_local.min 884.00 ± 57% +223.9% 2863 ± 66% interrupts.38:PCI-MSI.67633153-edge.eth0-TxRx-0 381.33 ± 44% +1845.1% 7417 ± 42% interrupts.39:PCI-MSI.67633154-edge.eth0-TxRx-1 2568 ± 95% -89.2% 277.50 ± 4% interrupts.40:PCI-MSI.67633155-edge.eth0-TxRx-2 225.33 ± 20% +328.3% 965.00 ± 32% interrupts.41:PCI-MSI.67633156-edge.eth0-TxRx-3 1789 ± 21% +30.2% 2328 interrupts.CPU0.RES:Rescheduling_interrupts 327.00 ± 6% +130.6% 754.00 ± 57% interrupts.CPU100.RES:Rescheduling_interrupts 315.67 ± 3% +53.3% 484.00 ± 27% interrupts.CPU101.RES:Rescheduling_interrupts 1094 ± 14% +126.0% 2473 ± 13% interrupts.CPU2.RES:Rescheduling_interrupts 6226 ± 26% +26.4% 7868 interrupts.CPU26.NMI:Non-maskable_interrupts 6226 ± 26% +26.4% 7868 interrupts.CPU26.PMI:Performance_monitoring_interrupts 1447 ± 39% +28.5% 1859 ± 26% interrupts.CPU26.RES:Rescheduling_interrupts 1550 ± 17% +23.9% 1920 ± 18% interrupts.CPU27.RES:Rescheduling_interrupts 4915 ± 28% +20.1% 5902 ± 33% interrupts.CPU29.NMI:Non-maskable_interrupts 4915 ± 28% +20.1% 5902 ± 33% interrupts.CPU29.PMI:Performance_monitoring_interrupts 884.00 ± 57% +223.9% 2863 ± 66% interrupts.CPU30.38:PCI-MSI.67633153-edge.eth0-TxRx-0 381.33 ± 44% +1845.1% 7417 ± 42% interrupts.CPU31.39:PCI-MSI.67633154-edge.eth0-TxRx-1 2568 ± 95% -89.2% 277.50 ± 4% interrupts.CPU32.40:PCI-MSI.67633155-edge.eth0-TxRx-2 225.33 ± 20% +328.3% 965.00 ± 32% interrupts.CPU33.41:PCI-MSI.67633156-edge.eth0-TxRx-3 6228 ± 26% +26.3% 7868 interrupts.CPU39.NMI:Non-maskable_interrupts 6228 ± 26% +26.3% 7868 interrupts.CPU39.PMI:Performance_monitoring_interrupts 6228 ± 26% +26.4% 7869 interrupts.CPU40.NMI:Non-maskable_interrupts 6228 ± 26% +26.4% 7869 interrupts.CPU40.PMI:Performance_monitoring_interrupts 6228 ± 26% +26.3% 7867 interrupts.CPU45.NMI:Non-maskable_interrupts 6228 ± 26% +26.3% 7867 interrupts.CPU45.PMI:Performance_monitoring_interrupts 6227 ± 26% +26.3% 7867 interrupts.CPU47.NMI:Non-maskable_interrupts 6227 ± 26% +26.3% 7867 interrupts.CPU47.PMI:Performance_monitoring_interrupts 6227 ± 26% +26.3% 7867 interrupts.CPU48.NMI:Non-maskable_interrupts 6227 ± 26% +26.3% 7867 interrupts.CPU48.PMI:Performance_monitoring_interrupts 6227 ± 26% +26.4% 7868 interrupts.CPU50.NMI:Non-maskable_interrupts 6227 ± 26% +26.4% 7868 interrupts.CPU50.PMI:Performance_monitoring_interrupts 6226 ± 26% +26.3% 7865 interrupts.CPU51.NMI:Non-maskable_interrupts 6226 ± 26% +26.3% 7865 interrupts.CPU51.PMI:Performance_monitoring_interrupts 1951 ± 6% -13.4% 1689 ± 3% interrupts.CPU52.RES:Rescheduling_interrupts 467.00 ± 10% +360.0% 2148 ± 18% interrupts.CPU57.RES:Rescheduling_interrupts 357.67 ± 5% +95.9% 700.50 ± 17% interrupts.CPU58.RES:Rescheduling_interrupts 335.67 ± 5% +14.1% 383.00 ± 12% interrupts.CPU60.RES:Rescheduling_interrupts 466.67 ± 44% +150.1% 1167 ± 16% interrupts.CPU62.RES:Rescheduling_interrupts 6210 ± 26% +26.5% 7857 interrupts.CPU69.NMI:Non-maskable_interrupts 6210 ± 26% +26.5% 7857 interrupts.CPU69.PMI:Performance_monitoring_interrupts 799.33 ± 54% -59.7% 322.00 interrupts.CPU7.RES:Rescheduling_interrupts 329.67 ± 5% +167.5% 882.00 ± 60% interrupts.CPU8.RES:Rescheduling_interrupts 6161 ± 35% +51.3% 9321 interrupts.CPU80.RES:Rescheduling_interrupts 4911 ± 22% -61.2% 1907 ± 55% interrupts.CPU81.RES:Rescheduling_interrupts 868.00 ± 53% +80.0% 1562 ± 34% interrupts.CPU82.RES:Rescheduling_interrupts 499.33 ± 13% -21.6% 391.50 ± 2% interrupts.CPU83.RES:Rescheduling_interrupts 328.33 ± 4% +148.1% 814.50 ± 60% interrupts.CPU9.RES:Rescheduling_interrupts 355.67 ± 13% +10.5% 393.00 ± 12% interrupts.CPU93.RES:Rescheduling_interrupts 730440 ± 3% +8.0% 788906 interrupts.NMI:Non-maskable_interrupts 730440 ± 3% +8.0% 788906 interrupts.PMI:Performance_monitoring_interrupts 24238 ± 7% +14.6% 27769 ± 10% softirqs.CPU0.RCU 106850 ± 2% +25.2% 133780 ± 20% softirqs.CPU1.TIMER 27066 ± 11% +18.5% 32064 ± 15% softirqs.CPU10.RCU 25790 ± 10% +20.2% 30999 ± 13% softirqs.CPU100.RCU 25624 ± 9% +21.7% 31186 ± 14% softirqs.CPU101.RCU 25892 ± 12% +19.9% 31034 ± 14% softirqs.CPU102.RCU 26207 ± 10% +18.1% 30961 ± 14% softirqs.CPU103.RCU 26890 ± 10% +18.1% 31769 ± 15% softirqs.CPU11.RCU 26960 ± 10% +18.7% 31993 ± 14% softirqs.CPU12.RCU 27170 ± 11% +19.5% 32473 ± 15% softirqs.CPU13.RCU 27089 ± 10% +20.0% 32496 ± 14% softirqs.CPU14.RCU 27534 ± 16% +31.3% 36166 ± 13% softirqs.CPU15.RCU 29615 ± 10% +23.8% 36651 ± 13% softirqs.CPU16.RCU 29695 ± 10% +24.7% 37023 ± 14% softirqs.CPU17.RCU 29884 ± 10% +23.9% 37016 ± 14% softirqs.CPU18.RCU 29957 ± 10% +22.7% 36761 ± 14% softirqs.CPU19.RCU 27903 ± 11% +17.1% 32662 ± 15% softirqs.CPU2.RCU 29475 ± 10% +22.8% 36187 ± 13% softirqs.CPU20.RCU 29223 ± 10% +24.7% 36438 ± 14% softirqs.CPU21.RCU 29403 ± 10% +24.7% 36680 ± 14% softirqs.CPU22.RCU 29223 ± 10% +24.5% 36377 ± 14% softirqs.CPU23.RCU 29439 ± 11% +23.8% 36447 ± 14% softirqs.CPU24.RCU 29644 ± 11% +26.4% 37466 ± 15% softirqs.CPU25.RCU 27156 ± 13% +19.6% 32470 ± 14% softirqs.CPU26.RCU 27077 ± 12% +15.5% 31267 ± 13% softirqs.CPU27.RCU 27006 ± 12% +17.6% 31759 ± 14% softirqs.CPU29.RCU 27206 ± 10% +19.1% 32405 ± 15% softirqs.CPU3.RCU 26797 ± 9% +26.4% 33871 ± 12% softirqs.CPU30.RCU 27088 ± 9% +24.2% 33643 ± 13% softirqs.CPU31.RCU 27162 ± 9% +24.1% 33721 ± 14% softirqs.CPU32.RCU 25150 ± 14% +31.9% 33177 ± 13% softirqs.CPU33.RCU 26739 ± 8% +23.9% 33128 ± 13% softirqs.CPU34.RCU 26626 ± 8% +26.5% 33670 ± 10% softirqs.CPU35.RCU 27097 ± 7% +22.5% 33195 ± 13% softirqs.CPU36.RCU 27004 ± 9% +23.5% 33344 ± 14% softirqs.CPU37.RCU 25138 +31.9% 33163 ± 14% softirqs.CPU38.RCU 26487 ± 8% +25.4% 33217 ± 14% softirqs.CPU39.RCU 27006 ± 11% +19.7% 32321 ± 16% softirqs.CPU4.RCU 26603 ± 8% +24.6% 33136 ± 12% softirqs.CPU40.RCU 26385 ± 9% +24.7% 32911 ± 12% softirqs.CPU41.RCU 27351 ± 11% +23.2% 33684 ± 14% softirqs.CPU42.RCU 27019 ± 9% +22.0% 32960 ± 13% softirqs.CPU43.RCU 26723 ± 10% +25.9% 33654 ± 12% softirqs.CPU44.RCU 29057 ± 11% +16.6% 33876 ± 12% softirqs.CPU45.RCU 29413 ± 11% +15.8% 34050 ± 13% softirqs.CPU46.RCU 29157 ± 11% +15.7% 33739 ± 12% softirqs.CPU47.RCU 29199 ± 12% +16.1% 33895 ± 12% softirqs.CPU48.RCU 28857 ± 10% +16.4% 33578 ± 13% softirqs.CPU49.RCU 27418 ± 10% +17.4% 32199 ± 16% softirqs.CPU5.RCU 28971 ± 11% +16.9% 33869 ± 12% softirqs.CPU50.RCU 28954 ± 13% +17.1% 33912 ± 12% softirqs.CPU51.RCU 30469 ± 8% +22.4% 37280 ± 9% softirqs.CPU52.RCU 105919 ± 2% +27.1% 134605 ± 18% softirqs.CPU52.TIMER 31184 ± 9% +23.6% 38533 ± 13% softirqs.CPU54.RCU 30675 ± 8% +24.9% 38314 ± 13% softirqs.CPU55.RCU 30626 ± 7% +23.2% 37723 ± 13% softirqs.CPU56.RCU 31019 ± 8% +20.4% 37347 ± 12% softirqs.CPU57.RCU 30597 ± 9% +22.8% 37588 ± 13% softirqs.CPU58.RCU 30243 ± 8% +24.8% 37751 ± 13% softirqs.CPU59.RCU 26980 ± 10% +19.0% 32102 ± 16% softirqs.CPU6.RCU 26808 ± 10% +22.9% 32935 ± 14% softirqs.CPU60.RCU 27006 ± 10% +23.9% 33453 ± 16% softirqs.CPU61.RCU 27337 ± 11% +21.7% 33259 ± 14% softirqs.CPU62.RCU 27593 ± 11% +21.1% 33417 ± 15% softirqs.CPU63.RCU 27796 ± 10% +21.0% 33641 ± 14% softirqs.CPU64.RCU 27087 ± 11% +22.1% 33072 ± 13% softirqs.CPU65.RCU 27109 ± 10% +21.5% 32925 ± 13% softirqs.CPU66.RCU 25097 ± 21% +31.9% 33094 ± 13% softirqs.CPU67.RCU 27755 ± 10% +19.6% 33194 ± 13% softirqs.CPU68.RCU 27476 ± 10% +22.2% 33569 ± 14% softirqs.CPU69.RCU 27402 ± 11% +19.1% 32628 ± 16% softirqs.CPU7.RCU 27829 ± 10% +20.8% 33604 ± 14% softirqs.CPU70.RCU 28217 ± 11% +19.3% 33669 ± 13% softirqs.CPU71.RCU 27146 ± 10% +21.5% 32977 ± 13% softirqs.CPU72.RCU 27327 ± 9% +19.6% 32674 ± 14% softirqs.CPU73.RCU 27173 ± 11% +22.7% 33345 ± 13% softirqs.CPU74.RCU 26143 ± 9% +28.8% 33669 ± 13% softirqs.CPU75.RCU 25936 ± 8% +28.8% 33398 ± 14% softirqs.CPU76.RCU 25725 ± 7% +30.7% 33626 ± 13% softirqs.CPU77.RCU 30496 ± 10% +21.6% 37072 ± 12% softirqs.CPU78.RCU 30408 ± 10% +20.5% 36633 ± 13% softirqs.CPU79.RCU 27088 ± 10% +19.4% 32345 ± 15% softirqs.CPU8.RCU 29710 ± 10% +38.3% 41102 ± 11% softirqs.CPU80.RCU 29133 ± 10% +21.6% 35420 ± 12% softirqs.CPU81.RCU 28787 ± 11% +21.3% 34923 ± 12% softirqs.CPU83.RCU 28669 ± 10% +21.8% 34907 ± 12% softirqs.CPU84.RCU 28342 ± 12% +23.3% 34938 ± 12% softirqs.CPU85.RCU 28874 ± 11% +20.9% 34922 ± 14% softirqs.CPU86.RCU 28600 ± 10% +21.2% 34651 ± 13% softirqs.CPU87.RCU 28668 ± 11% +20.7% 34590 ± 13% softirqs.CPU88.RCU 28553 ± 11% +20.9% 34534 ± 14% softirqs.CPU89.RCU 27215 ± 10% +19.7% 32580 ± 15% softirqs.CPU9.RCU 23437 +31.1% 30737 ± 13% softirqs.CPU90.RCU 25261 ± 9% +22.8% 31030 ± 13% softirqs.CPU91.RCU 25566 ± 10% +22.8% 31395 ± 13% softirqs.CPU92.RCU 25677 ± 10% +21.0% 31073 ± 13% softirqs.CPU93.RCU 25746 ± 11% +22.1% 31435 ± 14% softirqs.CPU94.RCU 25492 ± 9% +20.9% 30816 ± 13% softirqs.CPU95.RCU 25219 ± 9% +22.8% 30958 ± 13% softirqs.CPU96.RCU 25537 ± 10% +22.3% 31221 ± 13% softirqs.CPU97.RCU 25615 ± 10% +22.2% 31306 ± 13% softirqs.CPU98.RCU 25599 ± 9% +21.8% 31169 ± 13% softirqs.CPU99.RCU 2884148 ± 10% +21.7% 3508969 ± 14% softirqs.RCU 6.21 -6.2 0.00 perf-profile.calltrace.cycles-pp.eventfd_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 50.14 -3.3 46.80 perf-profile.calltrace.cycles-pp.write 12.15 -1.7 10.41 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 13.59 -1.7 11.90 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 26.88 -1.5 25.42 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 9.36 -1.4 7.92 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read 7.73 -1.4 6.31 perf-profile.calltrace.cycles-pp.eventfd_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 27.40 -1.4 25.99 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 4.15 -1.3 2.85 perf-profile.calltrace.cycles-pp._copy_from_user.eventfd_write.vfs_write.ksys_write.do_syscall_64 2.20 -1.0 1.22 perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.11 -0.6 11.48 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write 1.22 -0.5 0.72 ± 2% perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled._copy_from_user.eventfd_write.vfs_write.ksys_write 1.65 -0.5 1.19 perf-profile.calltrace.cycles-pp.__might_fault._copy_from_user.eventfd_write.vfs_write.ksys_write 8.29 -0.4 7.89 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write 2.99 -0.3 2.73 perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.76 -0.2 0.55 perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault._copy_from_user.eventfd_write.vfs_write 2.31 -0.2 2.12 perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.34 -0.1 1.23 ± 2% perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64 1.28 -0.1 1.21 perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.63 -0.0 0.58 perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 -0.0 0.70 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.eventfd_write.vfs_write.ksys_write.do_syscall_64 0.73 ± 3% +0.2 0.91 perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.66 +0.2 0.84 perf-profile.calltrace.cycles-pp.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 0.81 ± 2% +0.2 1.02 perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 13.25 +0.4 13.70 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.00 +0.6 0.57 perf-profile.calltrace.cycles-pp.__might_sleep.__might_fault._copy_to_iter.eventfd_read.new_sync_read 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault._copy_to_iter.eventfd_read.new_sync_read 0.00 +0.7 0.73 perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyout._copy_to_iter.eventfd_read.new_sync_read 0.00 +0.8 0.84 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.eventfd_read.new_sync_read.vfs_read.ksys_read 14.68 +1.2 15.90 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.63 +1.4 2.08 perf-profile.calltrace.cycles-pp.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.00 +1.5 1.48 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.eventfd_read.new_sync_read.vfs_read 0.00 +1.8 1.77 perf-profile.calltrace.cycles-pp.__might_fault._copy_to_iter.eventfd_read.new_sync_read.vfs_read 11.98 +1.8 13.82 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read 27.66 +2.7 30.34 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 28.20 +3.1 31.31 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 51.23 +3.3 54.56 perf-profile.calltrace.cycles-pp.read 0.00 +4.5 4.50 perf-profile.calltrace.cycles-pp._copy_to_iter.eventfd_read.new_sync_read.vfs_read.ksys_read 0.00 +7.1 7.06 perf-profile.calltrace.cycles-pp.eventfd_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 0.00 +8.2 8.20 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 50.38 -3.3 47.03 perf-profile.children.cycles-pp.write 19.51 -1.9 17.61 perf-profile.children.cycles-pp.syscall_return_via_sysret 12.23 -1.7 10.48 perf-profile.children.cycles-pp.vfs_write 13.63 -1.7 11.94 perf-profile.children.cycles-pp.ksys_write 7.79 -1.4 6.37 perf-profile.children.cycles-pp.eventfd_write 4.24 -1.3 2.93 perf-profile.children.cycles-pp._copy_from_user 2.89 -1.0 1.86 perf-profile.children.cycles-pp.fsnotify 5.41 -0.5 4.94 perf-profile.children.cycles-pp.security_file_permission 1.91 -0.3 1.60 perf-profile.children.cycles-pp._raw_spin_lock_irq 0.34 ± 2% -0.2 0.15 ± 3% perf-profile.children.cycles-pp.__vfs_read 2.62 -0.2 2.45 perf-profile.children.cycles-pp.common_file_perm 0.15 ± 5% -0.1 0.07 ± 7% perf-profile.children.cycles-pp.read@plt 0.50 -0.1 0.43 perf-profile.children.cycles-pp.fpregs_assert_state_consistent 1.24 -0.1 1.18 perf-profile.children.cycles-pp.fsnotify_parent 1.08 -0.1 1.02 perf-profile.children.cycles-pp.apparmor_file_permission 0.16 ± 5% -0.1 0.11 perf-profile.children.cycles-pp.__vfs_write 0.45 -0.0 0.41 perf-profile.children.cycles-pp.aa_file_perm 0.45 -0.0 0.43 perf-profile.children.cycles-pp.hrtimer_interrupt 0.06 -0.0 0.05 perf-profile.children.cycles-pp.write@plt 1.45 ± 2% +0.2 1.61 perf-profile.children.cycles-pp.__fget_light 1.51 +0.2 1.68 perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.00 +0.2 0.17 ± 2% perf-profile.children.cycles-pp.iov_iter_init 1.74 ± 2% +0.2 1.92 perf-profile.children.cycles-pp.__fdget_pos 0.74 +0.2 0.93 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.66 +0.2 0.85 perf-profile.children.cycles-pp.__x64_sys_write 0.77 ± 2% +0.2 0.97 perf-profile.children.cycles-pp.__might_sleep 0.99 +0.3 1.28 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 13.32 +0.5 13.77 perf-profile.children.cycles-pp.vfs_read 2.59 +0.5 3.07 perf-profile.children.cycles-pp.__might_fault 6.33 +0.8 7.16 perf-profile.children.cycles-pp.eventfd_read 14.70 +1.2 15.93 perf-profile.children.cycles-pp.ksys_read 22.24 +1.2 23.48 perf-profile.children.cycles-pp.entry_SYSCALL_64 0.64 ± 2% +1.5 2.12 perf-profile.children.cycles-pp.__x64_sys_read 0.00 +1.5 1.51 perf-profile.children.cycles-pp.copyout 54.64 +1.5 56.16 perf-profile.children.cycles-pp.do_syscall_64 55.67 +1.7 57.39 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 51.46 +3.3 54.77 perf-profile.children.cycles-pp.read 0.00 +4.6 4.60 perf-profile.children.cycles-pp._copy_to_iter 0.00 +8.2 8.24 perf-profile.children.cycles-pp.new_sync_read 19.48 -1.9 17.59 perf-profile.self.cycles-pp.syscall_return_via_sysret 2.98 -1.3 1.64 perf-profile.self.cycles-pp.eventfd_read 2.82 -1.0 1.80 perf-profile.self.cycles-pp.fsnotify 2.17 ± 2% -0.9 1.31 perf-profile.self.cycles-pp.write 1.84 -0.3 1.54 perf-profile.self.cycles-pp._raw_spin_lock_irq 0.31 ± 4% -0.2 0.14 ± 3% perf-profile.self.cycles-pp.__vfs_read 0.61 ± 2% -0.2 0.46 perf-profile.self.cycles-pp._copy_from_user 2.12 -0.1 1.98 perf-profile.self.cycles-pp.common_file_perm 1.49 -0.1 1.35 perf-profile.self.cycles-pp.read 1.18 -0.1 1.07 perf-profile.self.cycles-pp.security_file_permission 0.84 ± 2% -0.1 0.73 ± 2% perf-profile.self.cycles-pp.vfs_read 0.15 ± 5% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.read@plt 0.48 -0.1 0.41 perf-profile.self.cycles-pp.fpregs_assert_state_consistent 2.71 -0.1 2.65 perf-profile.self.cycles-pp.eventfd_write 0.85 -0.1 0.79 perf-profile.self.cycles-pp.vfs_write 1.18 -0.1 1.13 perf-profile.self.cycles-pp.fsnotify_parent 0.95 -0.0 0.90 perf-profile.self.cycles-pp.apparmor_file_permission 0.12 ± 3% -0.0 0.08 perf-profile.self.cycles-pp.__vfs_write 0.45 -0.0 0.41 perf-profile.self.cycles-pp.aa_file_perm 0.06 -0.0 0.05 perf-profile.self.cycles-pp.write@plt 0.34 +0.0 0.37 perf-profile.self.cycles-pp.__fdget_pos 0.56 +0.1 0.61 perf-profile.self.cycles-pp.ksys_write 0.00 +0.2 0.16 perf-profile.self.cycles-pp.iov_iter_init 1.36 ± 2% +0.2 1.52 perf-profile.self.cycles-pp.__fget_light 0.72 +0.2 0.89 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.58 +0.2 0.77 perf-profile.self.cycles-pp.__x64_sys_write 1.02 +0.2 1.21 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.70 ± 2% +0.2 0.91 perf-profile.self.cycles-pp.__might_sleep 0.64 +0.3 0.90 perf-profile.self.cycles-pp.__might_fault 0.00 +0.3 0.29 perf-profile.self.cycles-pp.copyout 1.19 +0.3 1.53 perf-profile.self.cycles-pp.copy_user_generic_unrolled 24.37 +0.4 24.79 perf-profile.self.cycles-pp.do_syscall_64 0.53 +0.5 1.04 perf-profile.self.cycles-pp.ksys_read 0.00 +0.9 0.92 perf-profile.self.cycles-pp.new_sync_read 20.35 +1.3 21.62 perf-profile.self.cycles-pp.entry_SYSCALL_64 0.00 +1.3 1.32 perf-profile.self.cycles-pp._copy_to_iter 0.56 ± 2% +1.4 1.98 perf-profile.self.cycles-pp.__x64_sys_read will-it-scale.per_process_ops 600000 +------------------------------------------------------------------+ | O O O O O O O : O : : O O : : : | 500000 |-+ : : : : : : | | : : : : : : | | : : : : : : : :| 400000 |-+ : : : : : : : :| | : : : : : : : :| 300000 |-+ : : : : : : : :| | : : : : : : : :| 200000 |-+ : : : : : : : :| | : : : : : : : :| | : : : : | 100000 |-+ : : : : | | : : : : | 0 +------------------------------------------------------------------+ will-it-scale.workload 7e+07 +-------------------------------------------------------------------+ | | 6e+07 |.+.+.+.+.+.+.+.+.+.+.+.+ O + +.+.+.+.+.+.+ +.+.+.+.+.+.+.+.+ | | : : : : : : | 5e+07 |-+ : : : : : : | | : :: : : : : | 4e+07 |-+ : : : : : : : :| | : : : : : : : :| 3e+07 |-+ : : : : : : : :| | : : : : : : : :| 2e+07 |-+ : : : : : : : :| | :: :: :: ::| 1e+07 |-+ : : : : | | : : : : | 0 +-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

8 months, 2 weeks

1
0
0 / 0

b3df6626cb ("module,jump_label: Hook to UNFORMED notifier"): WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:887 jump_label_test

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git modules/WIP commit b3df6626cb74b9b3f7693decf4d13d3c756939df Author: Peter Zijlstra <peterz(a)infradead.org> AuthorDate: Mon Apr 20 15:38:39 2020 +0200 Commit: Peter Zijlstra <peterz(a)infradead.org> CommitDate: Thu Apr 30 20:15:02 2020 +0200 module,jump_label: Hook to UNFORMED notifier Signed-off-by: Peter Zijlstra (Intel) <peterz(a)infradead.org> 45016484c3 module: Add MODULE_STATE_UNFORMED callback b3df6626cb module,jump_label: Hook to UNFORMED notifier 65d09e3e7c module,static_call: Hook to UNFORMED notifier +-------------------------------------------------+------------+------------+------------+ | | 45016484c3 | b3df6626cb | 65d09e3e7c | +-------------------------------------------------+------------+------------+------------+ | boot_successes | 46 | 0 | 0 | | boot_failures | 4 | 30 | 30 | | BUG:kernel_hang_in_boot_stage | 2 | 0 | 1 | | BUG:kernel_hang_in_test_stage | 1 | | | | BUG:kernel_timeout_in_boot_stage | 1 | 0 | 1 | | kernel_BUG_at_arch/x86/kernel/jump_label.c | 0 | 18 | 18 | | PANIC:early_exception | 0 | 18 | 18 | | EIP:__jump_label_set_jump_code | 0 | 18 | 18 | | WARNING:at_kernel/jump_label.c:#jump_label_test | 0 | 12 | 10 | | EIP:jump_label_test | 0 | 12 | 10 | | kernel_BUG_at_mm/vmalloc.c | 0 | 1 | | | invalid_opcode:#[##] | 0 | 1 | | | EIP:alloc_vmap_area | 0 | 1 | | | Kernel_panic-not_syncing:Fatal_exception | 0 | 1 | | +-------------------------------------------------+------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 5.158028] Speculative Store Bypass: Vulnerable [ 5.158418] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 5.159929] random: get_random_u32 called from copy_process+0x212/0x11fb with crng_init=0 [ 5.160962] Performance Events: unsupported p6 CPU model 60 no PMU driver, software events only. [ 5.161847] ------------[ cut here ]------------ [ 5.162239] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:887 jump_label_test+0x6e/0xaf [ 5.163063] Modules linked in: [ 5.163375] CPU: 0 PID: 1 Comm: swapper Not tainted 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.163998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.164677] EIP: jump_label_test+0x6e/0xaf [ 5.165010] Code: e8 fe b5 75 fe b8 4c 2d d7 82 e8 8f b5 75 fe a1 e0 5d 7d 82 85 c0 74 02 0f 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 <0f> 0b e9 02 00 00 00 eb 02 0f 0b e9 02 00 00 00 eb 02 0f 0b 3e 8d [ 5.167510] EAX: 00000001 EBX: 00000002 ECX: 00000000 EDX: 00000000 [ 5.168418] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.169272] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.170243] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.171202] Call Trace: [ 5.171618] ? jump_label_init_module+0xf/0xf [ 5.172226] do_one_initcall+0x70/0x159 [ 5.172833] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.173544] ? proc_register+0x9f/0xde [ 5.174065] ? rest_init+0xba/0xba [ 5.174615] ? proc_create_seq_private+0x39/0x3b [ 5.175247] ? init_mm_internals+0x81/0x86 [ 5.175834] kernel_init_freeable+0x55/0x1a0 [ 5.176449] ? rest_init+0xba/0xba [ 5.176921] kernel_init+0x8/0xd0 [ 5.177425] ret_from_fork+0x1c/0x28 [ 5.177911] random: get_random_bytes called from init_oops_id+0x23/0x3b with crng_init=0 [ 5.177917] ---[ end trace 396ae3e6b10ea155 ]--- [ 5.179990] ------------[ cut here ]------------ [ 5.179990] ------------[ cut here ]------------ [ 5.180802] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:888 jump_label_test+0x77/0xaf [ 5.182432] Modules linked in: [ 5.182896] CPU: 0 PID: 1 Comm: swapper Tainted: G W 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.184237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.185477] EIP: jump_label_test+0x77/0xaf [ 5.186037] Code: 82 e8 8f b5 75 fe a1 e0 5d 7d 82 85 c0 74 02 0f 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 0f 0b e9 02 00 00 00 eb 02 <0f> 0b e9 02 00 00 00 eb 02 0f 0b 3e 8d 74 26 00 0f 0b b8 e0 5d 7d [ 5.188663] EAX: 00000001 EBX: 00000002 ECX: 00000000 EDX: 00000000 [ 5.189558] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.190443] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.191415] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.192241] Call Trace: [ 5.192689] ? jump_label_init_module+0xf/0xf [ 5.193330] do_one_initcall+0x70/0x159 [ 5.193876] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.194581] ? proc_register+0x9f/0xde [ 5.195104] ? rest_init+0xba/0xba [ 5.195607] ? proc_create_seq_private+0x39/0x3b [ 5.196233] ? init_mm_internals+0x81/0x86 [ 5.196826] kernel_init_freeable+0x55/0x1a0 [ 5.197443] ? rest_init+0xba/0xba [ 5.197906] kernel_init+0x8/0xd0 [ 5.198400] ret_from_fork+0x1c/0x28 [ 5.198905] ---[ end trace 396ae3e6b10ea156 ]--- [ 5.199634] ------------[ cut here ]------------ [ 5.199634] ------------[ cut here ]------------ [ 5.200264] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:889 jump_label_test+0x80/0xaf [ 5.201626] Modules linked in: [ 5.202052] CPU: 0 PID: 1 Comm: swapper Tainted: G W 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.203368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.204520] EIP: jump_label_test+0x80/0xaf [ 5.205074] Code: 7d 82 85 c0 74 02 0f 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 0f 0b e9 02 00 00 00 eb 02 0f 0b e9 02 00 00 00 eb 02 <0f> 0b 3e 8d 74 26 00 0f 0b b8 e0 5d 7d 82 e8 4f b5 75 fe b8 4c 2d [ 5.210145] EAX: 00000001 EBX: 00000002 ECX: 00000000 EDX: 00000000 [ 5.211048] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.211926] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.212855] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.213715] Call Trace: [ 5.214069] ? jump_label_init_module+0xf/0xf [ 5.214702] do_one_initcall+0x70/0x159 [ 5.215223] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.215867] ? proc_register+0x9f/0xde [ 5.216442] ? rest_init+0xba/0xba [ 5.216899] ? proc_create_seq_private+0x39/0x3b [ 5.217541] ? init_mm_internals+0x81/0x86 [ 5.218089] kernel_init_freeable+0x55/0x1a0 [ 5.218743] ? rest_init+0xba/0xba [ 5.219200] kernel_init+0x8/0xd0 [ 5.219712] ret_from_fork+0x1c/0x28 [ 5.220192] ---[ end trace 396ae3e6b10ea157 ]--- [ 5.220927] ------------[ cut here ]------------ [ 5.220927] ------------[ cut here ]------------ [ 5.221573] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:890 jump_label_test+0x87/0xaf [ 5.222920] Modules linked in: [ 5.223362] CPU: 0 PID: 1 Comm: swapper Tainted: G W 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.224618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.225771] EIP: jump_label_test+0x87/0xaf [ 5.226338] Code: 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 0f 0b e9 02 00 00 00 eb 02 0f 0b e9 02 00 00 00 eb 02 0f 0b 3e 8d 74 26 00 <0f> 0b b8 e0 5d 7d 82 e8 4f b5 75 fe b8 4c 2d d7 82 e8 aa b5 75 fe [ 5.228849] EAX: 00000001 EBX: 00000002 ECX: 00000000 EDX: 00000000 [ 5.229737] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.230639] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.231561] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.232407] Call Trace: [ 5.232752] ? jump_label_init_module+0xf/0xf [ 5.233354] do_one_initcall+0x70/0x159 [ 5.233878] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.234516] ? proc_register+0x9f/0xde [ 5.235038] ? rest_init+0xba/0xba [ 5.235513] ? proc_create_seq_private+0x39/0x3b [ 5.236128] ? init_mm_internals+0x81/0x86 [ 5.236693] kernel_init_freeable+0x55/0x1a0 [ 5.237303] ? rest_init+0xba/0xba [ 5.237763] kernel_init+0x8/0xd0 [ 5.238209] ret_from_fork+0x1c/0x28 [ 5.238755] ---[ end trace 396ae3e6b10ea158 ]--- [ 5.239473] ------------[ cut here ]------------ [ 5.239473] ------------[ cut here ]------------ [ 5.240102] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:887 jump_label_test+0x6e/0xaf [ 5.241433] Modules linked in: [ 5.241856] CPU: 0 PID: 1 Comm: swapper Tainted: G W 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.243086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.244229] EIP: jump_label_test+0x6e/0xaf [ 5.244832] Code: e8 fe b5 75 fe b8 4c 2d d7 82 e8 8f b5 75 fe a1 e0 5d 7d 82 85 c0 74 02 0f 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 <0f> 0b e9 02 00 00 00 eb 02 0f 0b e9 02 00 00 00 eb 02 0f 0b 3e 8d [ 5.247443] EAX: 00000001 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 5.248304] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.249144] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.250094] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.250970] Call Trace: [ 5.251337] ? jump_label_init_module+0xf/0xf [ 5.251926] do_one_initcall+0x70/0x159 [ 5.252467] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.253087] ? proc_register+0x9f/0xde [ 5.253614] ? rest_init+0xba/0xba [ 5.254079] ? proc_create_seq_private+0x39/0x3b [ 5.254745] ? init_mm_internals+0x81/0x86 [ 5.255329] kernel_init_freeable+0x55/0x1a0 [ 5.255909] ? rest_init+0xba/0xba [ 5.256421] kernel_init+0x8/0xd0 [ 5.256875] ret_from_fork+0x1c/0x28 [ 5.257413] ---[ end trace 396ae3e6b10ea159 ]--- [ 5.258058] ------------[ cut here ]------------ [ 5.258058] ------------[ cut here ]------------ [ 5.258788] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:888 jump_label_test+0x77/0xaf [ 5.260063] Modules linked in: [ 5.260511] CPU: 0 PID: 1 Comm: swapper Tainted: G W 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.261687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.262815] EIP: jump_label_test+0x77/0xaf [ 5.263381] Code: 82 e8 8f b5 75 fe a1 e0 5d 7d 82 85 c0 74 02 0f 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 0f 0b e9 02 00 00 00 eb 02 <0f> 0b e9 02 00 00 00 eb 02 0f 0b 3e 8d 74 26 00 0f 0b b8 e0 5d 7d [ 5.279139] EAX: 00000001 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 5.279957] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.280761] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.281609] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.282408] Call Trace: [ 5.282751] ? jump_label_init_module+0xf/0xf [ 5.283349] do_one_initcall+0x70/0x159 [ 5.283814] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.284408] ? proc_register+0x9f/0xde [ 5.284868] ? rest_init+0xba/0xba [ 5.285364] ? proc_create_seq_private+0x39/0x3b [ 5.285987] ? init_mm_internals+0x81/0x86 [ 5.286617] kernel_init_freeable+0x55/0x1a0 [ 5.287256] ? rest_init+0xba/0xba [ 5.287785] kernel_init+0x8/0xd0 [ 5.288268] ret_from_fork+0x1c/0x28 [ 5.288815] ---[ end trace 396ae3e6b10ea15a ]--- [ 5.289570] ------------[ cut here ]------------ [ 5.289570] ------------[ cut here ]------------ [ 5.290205] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:889 jump_label_test+0x80/0xaf [ 5.291599] Modules linked in: [ 5.292022] CPU: 0 PID: 1 Comm: swapper Tainted: G W 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.293253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.294395] EIP: jump_label_test+0x80/0xaf [ 5.294953] Code: 7d 82 85 c0 74 02 0f 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 0f 0b e9 02 00 00 00 eb 02 0f 0b e9 02 00 00 00 eb 02 <0f> 0b 3e 8d 74 26 00 0f 0b b8 e0 5d 7d 82 e8 4f b5 75 fe b8 4c 2d [ 5.297487] EAX: 00000001 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 5.298333] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.299155] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.300053] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.300955] Call Trace: [ 5.301369] ? jump_label_init_module+0xf/0xf [ 5.301942] do_one_initcall+0x70/0x159 [ 5.302507] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.303143] ? proc_register+0x9f/0xde [ 5.303704] ? rest_init+0xba/0xba [ 5.304181] ? proc_create_seq_private+0x39/0x3b [ 5.304856] ? init_mm_internals+0x81/0x86 [ 5.305462] kernel_init_freeable+0x55/0x1a0 [ 5.306048] ? rest_init+0xba/0xba [ 5.306578] kernel_init+0x8/0xd0 [ 5.307021] ret_from_fork+0x1c/0x28 [ 5.307530] ---[ end trace 396ae3e6b10ea15b ]--- [ 5.308207] ------------[ cut here ]------------ [ 5.308207] ------------[ cut here ]------------ [ 5.308878] WARNING: CPU: 0 PID: 1 at kernel/jump_label.c:890 jump_label_test+0x87/0xaf [ 5.310213] Modules linked in: [ 5.310686] CPU: 0 PID: 1 Comm: swapper Tainted: G W 5.7.0-rc3-00080-gb3df6626cb74b #1 [ 5.311888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.313010] EIP: jump_label_test+0x87/0xaf [ 5.313605] Code: 0b a1 4c 2d d7 82 85 c0 75 02 0f 0b 3e 8d 74 26 00 0f 0b e9 02 00 00 00 eb 02 0f 0b e9 02 00 00 00 eb 02 0f 0b 3e 8d 74 26 00 <0f> 0b b8 e0 5d 7d 82 e8 4f b5 75 fe b8 4c 2d d7 82 e8 aa b5 75 fe [ 5.316097] EAX: 00000001 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 5.316903] ESI: 00000000 EDI: ffffffff EBP: eea2df2c ESP: eea2df28 [ 5.317706] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 5.318662] CR0: 80050033 CR2: ffffffff CR3: 02a4a000 CR4: 001406b0 [ 5.319543] Call Trace: [ 5.319897] ? jump_label_init_module+0xf/0xf [ 5.320530] do_one_initcall+0x70/0x159 [ 5.321066] ? _raw_write_lock_irqsave+0x21/0x5f [ 5.321735] ? proc_register+0x9f/0xde [ 5.322255] ? rest_init+0xba/0xba [ 5.322771] ? proc_create_seq_private+0x39/0x3b [ 5.323418] ? init_mm_internals+0x81/0x86 [ 5.323990] kernel_init_freeable+0x55/0x1a0 [ 5.324609] ? rest_init+0xba/0xba [ 5.325073] kernel_init+0x8/0xd0 [ 5.325561] ret_from_fork+0x1c/0x28 [ 5.326050] ---[ end trace 396ae3e6b10ea15c ]--- [ 5.329024] NMI watchdog: Perf NMI watchdog permanently disabled # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start ec0b68313a2832827cb5b9f75f1934273d3d59df e4a08b64261ab411b15580c369a3b8fbed28bbc1 -- git bisect good 12d79f830dfe5fd94b001fcd112a8a0d11c9a189 # 16:33 G 10 0 0 2 Merge 'linux-review/dillon-minfei-gmail-com/add-drm-panel-driver-for-stm32f429-dicovery-board-the-change-details/20200430-182711' into devel-catchup-202005010322 git bisect bad e6dedfc09dd96c3a78864983120ecc08c1301558 # 16:33 B 0 8 27 3 Merge 'jirislaby/devel' into devel-catchup-202005010322 git bisect bad 627ef6592f91a72dafb60914d52731154f376f1c # 16:33 B 0 15 38 7 Merge 'linux-review/Sayali-Lokhande/f2fs-Avoid-double-lock-for-cp_rwsem-during-checkpoint/20200501-024326' into devel-catchup-202005010322 git bisect good b29241196f75972bf872fa27a3b7f7a4468e0a8b # 16:33 G 16 0 0 0 Merge 'linux-review/Alexey-Budankov/perf-make-Perf-tool-aware-of-SELinux-access-control/20200430-170113' into devel-catchup-202005010322 git bisect bad 4a9bc2f7aa6c040facf9b5f9fe1dcb75f9374027 # 16:33 B 0 13 35 6 Merge 'peterz-queue/master' into devel-catchup-202005010322 git bisect good f204f56a167186bcd08704ef61dec10f2052246f # 16:33 G 14 0 0 2 Merge 'peterz-queue/sched/rbtree' into devel-catchup-202005010322 git bisect good 122ae21dbf1753a37634f65278c6aa26832d7d87 # 16:53 G 11 0 1 2 Merge branch 'modules/core' git bisect good aef192e32985594c0a6fd55a5fa567233fb05e06 # 19:06 G 11 0 0 0 sched: Remove sched_set_*() return value git bisect bad ed632ce9e1b239cfbf4da24d8a40b997ab6ffdbb # 20:20 B 0 1 19 2 ftrace: Merge ftrace_module_{init,enable}() git bisect good bbc7333976ebd77ca29f69c15ffc588a34df0253 # 21:55 G 11 0 1 1 tracepoint: Optimize using static_call() git bisect good e227b884ad2ffc03713e864f7c4508381a5234cd # 23:22 G 11 0 1 1 static_call: Allow early init git bisect good fc2542d1c7e40ddf019fe2f270f51190b326fda0 # 00:48 G 11 0 0 1 objtool: Clean up elf_write() condition git bisect bad b3df6626cb74b9b3f7693decf4d13d3c756939df # 00:55 B 0 1 35 18 module,jump_label: Hook to UNFORMED notifier git bisect good 45016484c3d05c58e1bbceaa4619b4fddbb4b1fb # 01:43 G 11 0 0 1 module: Add MODULE_STATE_UNFORMED callback # first bad commit: [b3df6626cb74b9b3f7693decf4d13d3c756939df] module,jump_label: Hook to UNFORMED notifier git bisect good 45016484c3d05c58e1bbceaa4619b4fddbb4b1fb # 02:24 G 33 0 3 4 module: Add MODULE_STATE_UNFORMED callback # extra tests with debug options git bisect bad b3df6626cb74b9b3f7693decf4d13d3c756939df # 02:38 B 0 1 17 0 module,jump_label: Hook to UNFORMED notifier # extra tests on head commit of peterz-queue/modules/WIP git bisect bad 65d09e3e7c7d4248e4695f27ae0c12fe63e1c45e # 02:55 B 0 1 35 18 module,static_call: Hook to UNFORMED notifier # bad: [65d09e3e7c7d4248e4695f27ae0c12fe63e1c45e] module,static_call: Hook to UNFORMED notifier # extra tests on revert first bad commit git bisect good 7f78694123d47caa9e3640116dbe00a3e45be4df # 04:08 G 11 0 2 4 Revert "module,jump_label: Hook to UNFORMED notifier" # good: [7f78694123d47caa9e3640116dbe00a3e45be4df] Revert "module,jump_label: Hook to UNFORMED notifier" --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/lkp@lists.01.org

8 months, 2 weeks

1
0
0 / 0

[blktrace] d106a3fdba: BUG:kernel_NULL_pointer_dereference,address

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-7): commit: d106a3fdbac9d93aec07a533059439de56c0216b ("[PATCH v3 4/6] blktrace: fix debugfs use after free") url: https://github.com/0day-ci/linux/commits/Luis-Chamberlain/block-fix-blktr... base: https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git for-next in testcase: blktests with following parameters: disk: 1SSD test: block-group1 on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 24.328966] BUG: kernel NULL pointer dereference, address: 00000000000000a0 [ 24.330808] #PF: supervisor write access in kernel mode [ 24.332468] #PF: error_code(0x0002) - not-present page [ 24.334138] PGD 8000000117ff4067 P4D 8000000117ff4067 PUD 1bc789067 PMD 0 [ 24.336046] Oops: 0002 [#1] SMP PTI [ 24.341721] CPU: 0 PID: 1019 Comm: check Not tainted 5.7.0-rc2-00034-gd106a3fdbac9d #1 [ 24.343858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 24.346026] RIP: 0010:down_write+0x28/0x50 [ 24.347665] Code: 5d ff 66 66 66 66 90 53 31 d2 be fa 05 00 00 48 89 fb 48 c7 c7 b0 fe 73 ad e8 04 41 60 ff e8 9f d6 ff ff 31 c0 ba 01 00 00 00 <f0> 48 0f b1 13 75 0f 65 48 8b 04 25 c0 8b 01 00 48 89 43 08 5b c3 [ 24.352502] RSP: 0000:ffffa03100877d58 EFLAGS: 00010246 [ 24.354357] RAX: 0000000000000000 RBX: 00000000000000a0 RCX: ffffff8100000000 [ 24.356428] RDX: 0000000000000001 RSI: 00000000000005fa RDI: ffffffffad73feb0 [ 24.358407] RBP: ffffa03100877db8 R08: 0000000000000064 R09: 0000000000000001 [ 24.360462] R10: ffffa03100877e18 R11: 0000000000000335 R12: 0000000000000000 [ 24.364395] R13: ffff8be7bc5480a8 R14: 0000000000000001 R15: ffff8be7cec98e60 [ 24.366404] FS: 00007f23b0625740(0000) GS:ffff8be83fc00000(0000) knlGS:0000000000000000 [ 24.368469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 24.370360] CR2: 00000000000000a0 CR3: 00000001bff04000 CR4: 00000000000406f0 [ 24.372338] Call Trace: [ 24.377936] simple_recursive_removal+0x4f/0x2d0 [ 24.379620] ? debugfs_remove+0x60/0x60 [ 24.381070] debugfs_remove+0x40/0x60 [ 24.382528] blk_mq_debugfs_unregister_hctx+0x15/0x30 [ 24.384095] blk_mq_exit_queue+0xd0/0x110 [ 24.385561] blk_cleanup_queue+0x94/0xe0 [ 24.386958] __scsi_remove_device+0x66/0x150 [ 24.388384] scsi_remove_device+0x21/0x30 [ 24.389788] sdev_store_delete+0x4f/0x90 [ 24.391127] kernfs_fop_write+0x124/0x1b0 [ 24.392485] vfs_write+0xbe/0x1d0 [ 24.393732] ksys_write+0xa1/0xe0 [ 24.395028] do_syscall_64+0x5b/0x1f0 [ 24.396341] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 24.418166] RIP: 0033:0x7f23b0716643 [ 24.419487] Code: 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 48 89 54 24 18 [ 24.423748] RSP: 002b:00007ffd535d5cc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 24.425787] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f23b0716643 [ 24.427710] RDX: 0000000000000002 RSI: 0000000001aa6a08 RDI: 0000000000000001 [ 24.429674] RBP: 0000000001aa6a08 R08: 000000000000000a R09: 0000000000000001 [ 24.431601] R10: 0000000001b7d5e8 R11: 0000000000000246 R12: 0000000000000002 [ 24.433531] R13: 00007f23b07e56a0 R14: 0000000000000002 R15: 00007f23b07e58a0 [ 24.435466] Modules linked in: scsi_debug loop dm_multipath dm_mod intel_rapl_msr intel_rapl_common crct10dif_pclmul bochs_drm crc32_pclmul drm_vram_helper crc32c_intel drm_ttm_helper ghash_clmulni_intel ttm sd_mod t10_pi sg ppdev drm_kms_helper snd_pcm ata_generic pata_acpi syscopyarea sysfillrect snd_timer sysimgblt fb_sys_fops snd aesni_intel crypto_simd drm cryptd glue_helper ata_piix soundcore libata pcspkr joydev serio_raw parport_pc virtio_scsi i2c_piix4 parport floppy ip_tables [ 24.445381] CR2: 00000000000000a0 [ 24.447202] ---[ end trace 83e08183a0d38f71 ]--- To reproduce: # build kernel cd linux cp config-5.7.0-rc2-00034-gd106a3fdbac9d .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

8 months, 2 weeks

1
0
0 / 0

[x86/head/64] 73ee00b89d: System_halted

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-7): commit: 73ee00b89dea2b766d7bcae234f8017586197beb ("x86/head/64: Load IDT earlier") https://git.kernel.org/cgit/linux/kernel/git/joro/linux.git sev-es-client-v5.7-rc3 in testcase: boot on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> pages->pgt_buf_offset: 0x0000000000000000 pages->pgt_buf_size: 0x0000000000000000 Error: kernel_ident_mapping_init() failed -- System haltedBUG: kernel hang in boot stage Linux version 5.7.0-rc3-00036-g73ee00b89dea2 #1 Command line: ip=::::vm-snb-69::dhcp root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-69/boot-1-debian-i386-20191205.cgz-73ee00b89dea2b766d7bcae234f8017586197beb-20200507-3770-13h0iuq-2.yaml ARCH=x86_64 kconfig=x86_64-randconfig-b003-20200506 branch=linux-devel/devel-hourly-2020050700 commit=73ee00b89dea2b766d7bcae234f8017586197beb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-b003-20200506/gcc-7/73ee00b89dea2b766d7bcae234f8017586197beb/vmlinuz-5.7.0-rc3-00036-g73ee00b89dea2 max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-i386-20191205.cgz/x86_64-randconfig-b003-20200506/gcc-7/73ee00b89dea2b766d7bcae234f8017586197beb/3 LKP_SERVER=inn selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw rcuperf.shutdown=0 watchdog_thresh=60 Elapsed time: 420 To reproduce: # build kernel cd linux cp config-5.7.0-rc3-00036-g73ee00b89dea2 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

8 months, 3 weeks

1
0
0 / 0

[mm] 9266baf8ee: vm-scalability.median -17.0% regression

by kernel test robot

Greeting, FYI, we noticed a -17.0% regression of vm-scalability.median due to commit: commit: 9266baf8eea588a78e5a0aa4ccd6cae36ff7349d ("mm: Optimized hugepage zeroing & copying from user") url: https://github.com/0day-ci/linux/commits/UPDATE-20200501-184514/Prathu-Ba... in testcase: vm-scalability on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory with following parameters: runtime: 300s size: 8T test: anon-w-seq-hugetlb cpufreq_governor: performance ucode: 0xca test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us. test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/ In addition to that, the commit also has significant impact on the following tests: +------------------+----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -4.9% regression | | test machine | 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=16 | | | test=page_fault1 | | | ucode=0x21 | +------------------+----------------------------------------------------------------------+ | testcase: change | vm-scalability: vm-scalability.median 8.4% improvement | | test machine | 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | runtime=300s | | | size=8T | | | test=anon-w-seq-mt | | | ucode=0x21 | +------------------+----------------------------------------------------------------------+ | testcase: change | vm-scalability: vm-scalability.median -2.6% regression | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | runtime=300s | | | size=8T | | | test=anon-cow-seq-hugetlb | | | ucode=0x500002c | +------------------+----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -3.5% regression | | test machine | 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=100% | | | test=page_fault1 | | | ucode=0x21 | +------------------+----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -5.3% regression | | test machine | 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | test=page_fault1 | | | ucode=0x21 | +------------------+----------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-cfl-e1/anon-w-seq-hugetlb/vm-scalability/0xca commit: 8632e9b564 (" hyperv-fixes for 5.7-rc1") 9266baf8ee ("mm: Optimized hugepage zeroing & copying from user") 8632e9b5645bbc23 9266baf8eea588a78e5a0aa4ccd ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 0.00 +35.0% 0.00 ± 41% vm-scalability.free_time 885731 -17.0% 735379 vm-scalability.median 14052443 -16.6% 11724633 vm-scalability.throughput 130581 ± 2% -9.0% 118824 ± 3% vm-scalability.time.involuntary_context_switches 2059540 -15.5% 1739286 vm-scalability.time.minor_page_faults 1466 +1.4% 1487 vm-scalability.time.percent_of_cpu_this_job_got 1430 +17.8% 1685 vm-scalability.time.system_time 2993 -6.5% 2797 vm-scalability.time.user_time 16555 -12.1% 14554 vm-scalability.time.voluntary_context_switches 3.937e+09 -15.6% 3.323e+09 vm-scalability.workload 322.62 -0.9% 319.64 boot-time.idle 5.46 ± 2% -1.0 4.50 ± 2% mpstat.cpu.all.idle% 31.64 +5.1 36.71 mpstat.cpu.all.sys% 62.00 -6.5% 58.00 vmstat.cpu.us 2220 ± 2% -10.8% 1980 vmstat.system.cs 1140 ± 5% -12.6% 997.00 ± 7% slabinfo.kmalloc-rcl-96.active_objs 1140 ± 5% -12.6% 997.00 ± 7% slabinfo.kmalloc-rcl-96.num_objs 6311 -9.9% 5685 slabinfo.vmap_area.active_objs 6311 -9.9% 5685 slabinfo.vmap_area.num_objs 132283 ± 9% -12.8% 115384 softirqs.CPU0.TIMER 137945 ± 7% -12.9% 120092 softirqs.CPU4.TIMER 26018 ± 4% -11.9% 22933 ± 9% softirqs.CPU5.RCU 134170 ± 8% -11.2% 119154 softirqs.CPU8.TIMER 354273 ± 7% -14.5% 303080 ± 7% cpuidle.C1.time 1473619 ± 37% -29.0% 1045628 ± 2% cpuidle.C1E.time 4610710 ± 51% -48.9% 2357842 ± 23% cpuidle.C3.time 23383 ± 41% -39.8% 14070 ± 11% cpuidle.C3.usage 2.605e+08 -16.8% 2.166e+08 cpuidle.C6.time 552177 -16.6% 460777 cpuidle.C6.usage 16488 ± 9% -30.4% 11472 ± 14% cpuidle.POLL.usage 8308 ± 5% -11.4% 7364 ± 6% interrupts.CAL:Function_call_interrupts 479.00 ± 2% -12.6% 418.50 ± 3% interrupts.CPU0.CAL:Function_call_interrupts 514.50 ± 4% -13.0% 447.50 ± 9% interrupts.CPU1.CAL:Function_call_interrupts 529.50 ± 4% -15.3% 448.75 ± 6% interrupts.CPU10.CAL:Function_call_interrupts 528.50 ± 4% -13.0% 459.75 ± 6% interrupts.CPU14.CAL:Function_call_interrupts 526.25 ± 6% -15.7% 443.50 ± 9% interrupts.CPU2.CAL:Function_call_interrupts 523.75 ± 3% -9.6% 473.25 ± 6% interrupts.CPU5.CAL:Function_call_interrupts 518.75 ± 6% -10.9% 462.25 ± 6% interrupts.CPU6.CAL:Function_call_interrupts 3608 ± 30% +101.4% 7268 ± 20% interrupts.CPU6.RES:Rescheduling_interrupts 81528 ± 4% -12.2% 71606 ± 5% interrupts.RES:Rescheduling_interrupts 373.33 ± 28% +47.7% 551.38 ± 5% sched_debug.cfs_rq:/.load_avg.max 90.00 ± 34% +50.7% 135.66 sched_debug.cfs_rq:/.load_avg.stddev 13.32 ± 13% +23.0% 16.39 ± 6% sched_debug.cfs_rq:/.nr_spread_over.avg 4816 ± 17% -31.8% 3286 ± 22% sched_debug.cpu.curr->pid.min 39104 ± 3% -10.7% 34937 sched_debug.cpu.nr_switches.max 7502 ± 7% -12.5% 6566 ± 3% sched_debug.cpu.nr_switches.stddev 34374 ± 5% -14.3% 29462 ± 5% sched_debug.cpu.sched_count.max 6717 ± 12% -16.4% 5617 ± 4% sched_debug.cpu.sched_count.stddev 2224 ± 5% -13.4% 1925 ± 4% sched_debug.cpu.sched_goidle.avg 1468 ± 5% -19.5% 1181 ± 3% sched_debug.cpu.sched_goidle.min 17663 ± 7% -14.0% 15196 ± 4% sched_debug.cpu.ttwu_count.max 3887 ± 16% -16.2% 3258 ± 4% sched_debug.cpu.ttwu_count.stddev 11677 ± 6% -18.3% 9544 ± 9% sched_debug.cpu.ttwu_local.max 2596 ± 12% -17.1% 2151 ± 6% sched_debug.cpu.ttwu_local.stddev 1709412 -15.6% 1442880 proc-vmstat.htlb_buddy_alloc_success 167.50 -11.0% 149.00 proc-vmstat.nr_dirtied 387471 +2.2% 395950 proc-vmstat.nr_dirty_background_threshold 775891 +2.2% 792870 proc-vmstat.nr_dirty_threshold 3943955 +2.2% 4030727 proc-vmstat.nr_free_pages 8037 -2.4% 7847 proc-vmstat.nr_inactive_anon 7418 -1.2% 7328 proc-vmstat.nr_mapped 11491 -1.8% 11288 proc-vmstat.nr_shmem 155.50 -11.3% 138.00 proc-vmstat.nr_written 8037 -2.4% 7847 proc-vmstat.nr_zone_inactive_anon 2351084 -13.1% 2042972 proc-vmstat.numa_hit 2351084 -13.1% 2042972 proc-vmstat.numa_local 8.759e+08 -15.6% 7.394e+08 proc-vmstat.pgalloc_normal 2451008 -13.1% 2129664 proc-vmstat.pgfault 8.749e+08 -15.7% 7.373e+08 proc-vmstat.pgfree 25.20 ± 2% -25.2 0.00 perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.hugetlb_no_page.hugetlb_fault.handle_mm_fault 24.59 ± 2% -24.6 0.00 perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.hugetlb_no_page.hugetlb_fault 26.17 -4.7 21.52 ± 2% perf-profile.calltrace.cycles-pp.do_rw_once 3.15 ± 5% -0.5 2.61 ± 5% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_fresh_huge_page.alloc_surplus_huge_page.hugetlb_acct_memory.hugetlb_reserve_pages 3.15 ± 5% -0.5 2.60 ± 5% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_fresh_huge_page.alloc_surplus_huge_page.hugetlb_acct_memory 3.17 ± 5% -0.5 2.63 ± 5% perf-profile.calltrace.cycles-pp.alloc_fresh_huge_page.alloc_surplus_huge_page.hugetlb_acct_memory.hugetlb_reserve_pages.hugetlbfs_file_mmap 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mmap64 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.calltrace.cycles-pp.mmap64 3.18 ± 5% -0.5 2.64 ± 5% perf-profile.calltrace.cycles-pp.alloc_surplus_huge_page.hugetlb_acct_memory.hugetlb_reserve_pages.hugetlbfs_file_mmap.mmap_region 3.20 ± 4% -0.5 2.66 ± 5% perf-profile.calltrace.cycles-pp.hugetlb_acct_memory.hugetlb_reserve_pages.hugetlbfs_file_mmap.mmap_region.do_mmap 3.20 ± 4% -0.5 2.66 ± 5% perf-profile.calltrace.cycles-pp.hugetlbfs_file_mmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 3.20 ± 4% -0.5 2.66 ± 5% perf-profile.calltrace.cycles-pp.hugetlb_reserve_pages.hugetlbfs_file_mmap.mmap_region.do_mmap.vm_mmap_pgoff 2.79 ± 5% -0.5 2.33 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.alloc_fresh_huge_page.alloc_surplus_huge_page 2.78 ± 5% -0.5 2.33 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.alloc_fresh_huge_page 1.51 ± 8% -0.3 1.18 ± 3% perf-profile.calltrace.cycles-pp.ret_from_fork 1.51 ± 8% -0.3 1.18 ± 3% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 1.49 ± 8% -0.3 1.17 ± 3% perf-profile.calltrace.cycles-pp.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread.ret_from_fork 1.50 ± 8% -0.3 1.18 ± 3% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 1.50 ± 8% -0.3 1.18 ± 3% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 1.45 ± 8% -0.3 1.14 ± 3% perf-profile.calltrace.cycles-pp.memcpy_erms.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread 1.05 ± 5% -0.3 0.79 ± 5% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write._fini 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.write._fini 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.devkmsg_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp._fini 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write.ksys_write 0.89 ± 6% -0.2 0.66 ± 5% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write 0.83 ± 7% -0.2 0.60 ± 5% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit 1.42 ± 3% -0.2 1.24 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.42 ± 3% -0.2 1.24 ± 6% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.42 ± 3% -0.2 1.24 ± 6% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.41 ± 2% -0.2 1.23 ± 6% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 1.41 ± 2% -0.2 1.23 ± 6% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 1.39 ± 2% -0.2 1.21 ± 6% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit 1.39 ± 2% -0.2 1.21 ± 6% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit 1.39 ± 2% -0.2 1.21 ± 6% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput 1.38 ± 2% -0.2 1.20 ± 6% perf-profile.calltrace.cycles-pp.__free_huge_page.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap 1.44 ± 2% -0.2 1.27 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 1.44 ± 2% -0.2 1.27 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.05 ± 3% -0.2 0.88 ± 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__free_huge_page.release_pages.tlb_flush_mmu 1.05 ± 3% -0.2 0.88 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.__free_huge_page.release_pages.tlb_flush_mmu.tlb_finish_mmu 0.78 ± 7% -0.1 0.66 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64 0.74 ± 8% -0.1 0.62 ± 6% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 0.75 ± 8% -0.1 0.63 ± 6% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 0.75 ± 8% -0.1 0.63 ± 6% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.75 ± 8% -0.1 0.63 ± 6% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 0.74 ± 8% -0.1 0.62 ± 6% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +0.9 0.95 perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.hugetlb_fault.handle_mm_fault.do_page_fault 0.00 +1.0 0.97 perf-profile.calltrace.cycles-pp.__mutex_lock.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault 27.03 ± 2% +5.1 32.11 ± 4% perf-profile.calltrace.cycles-pp.hugetlb_no_page.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault 26.48 ± 2% +5.3 31.80 ± 4% perf-profile.calltrace.cycles-pp.clear_huge_page.hugetlb_no_page.hugetlb_fault.handle_mm_fault.do_page_fault 25.54 +5.8 31.33 ± 2% perf-profile.calltrace.cycles-pp.page_fault.do_access 25.48 +5.8 31.28 ± 2% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.do_access 25.41 +5.8 31.22 ± 2% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault.do_access 25.38 +5.8 31.19 ± 2% perf-profile.calltrace.cycles-pp.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault.do_access 89.02 +9.2 98.24 ± 2% perf-profile.calltrace.cycles-pp.do_access 25.46 ± 2% -25.5 0.00 perf-profile.children.cycles-pp.clear_subpage 24.72 ± 2% -24.7 0.00 perf-profile.children.cycles-pp.clear_page_erms 5.90 ± 2% -1.0 4.92 ± 2% perf-profile.children.cycles-pp.do_syscall_64 5.90 ± 2% -1.0 4.93 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 0.73 ± 3% -0.7 0.06 ± 7% perf-profile.children.cycles-pp.___might_sleep 3.91 ± 3% -0.6 3.29 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.children.cycles-pp.mmap64 3.17 ± 5% -0.5 2.64 ± 5% perf-profile.children.cycles-pp.alloc_fresh_huge_page 3.16 ± 5% -0.5 2.62 ± 5% perf-profile.children.cycles-pp.get_page_from_freelist 3.21 ± 4% -0.5 2.68 ± 5% perf-profile.children.cycles-pp.vm_mmap_pgoff 3.18 ± 5% -0.5 2.65 ± 5% perf-profile.children.cycles-pp.alloc_surplus_huge_page 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.children.cycles-pp.hugetlbfs_file_mmap 3.20 ± 4% -0.5 2.67 ± 5% perf-profile.children.cycles-pp.hugetlb_reserve_pages 3.20 ± 4% -0.5 2.68 ± 5% perf-profile.children.cycles-pp.ksys_mmap_pgoff 3.20 ± 4% -0.5 2.68 ± 5% perf-profile.children.cycles-pp.do_mmap 3.20 ± 4% -0.5 2.68 ± 5% perf-profile.children.cycles-pp.mmap_region 3.16 ± 5% -0.5 2.63 ± 5% perf-profile.children.cycles-pp.__alloc_pages_nodemask 3.20 ± 4% -0.5 2.68 ± 5% perf-profile.children.cycles-pp.hugetlb_acct_memory 2.84 ± 4% -0.5 2.35 ± 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 1.51 ± 8% -0.3 1.18 ± 3% perf-profile.children.cycles-pp.kthread 1.49 ± 8% -0.3 1.17 ± 3% perf-profile.children.cycles-pp.memcpy_erms 1.49 ± 8% -0.3 1.17 ± 3% perf-profile.children.cycles-pp.drm_fb_helper_dirty_work 1.51 ± 8% -0.3 1.19 ± 3% perf-profile.children.cycles-pp.ret_from_fork 1.50 ± 8% -0.3 1.18 ± 3% perf-profile.children.cycles-pp.process_one_work 1.50 ± 8% -0.3 1.18 ± 3% perf-profile.children.cycles-pp.worker_thread 1.18 ± 4% -0.3 0.89 ± 2% perf-profile.children.cycles-pp.console_unlock 1.01 ± 5% -0.3 0.75 perf-profile.children.cycles-pp.serial8250_console_write 1.19 ± 4% -0.3 0.92 ± 3% perf-profile.children.cycles-pp.vprintk_emit 1.08 ± 4% -0.3 0.83 ± 6% perf-profile.children.cycles-pp.write 1.09 ± 5% -0.3 0.84 ± 7% perf-profile.children.cycles-pp.new_sync_write 1.10 ± 5% -0.3 0.85 ± 7% perf-profile.children.cycles-pp.ksys_write 1.10 ± 5% -0.3 0.85 ± 7% perf-profile.children.cycles-pp.vfs_write 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.children.cycles-pp.devkmsg_write 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.children.cycles-pp._fini 1.06 ± 5% -0.3 0.80 ± 6% perf-profile.children.cycles-pp.devkmsg_emit 0.94 ± 6% -0.2 0.69 perf-profile.children.cycles-pp.uart_console_write 0.81 ± 5% -0.2 0.60 ± 2% perf-profile.children.cycles-pp.wait_for_xmitr 0.79 ± 5% -0.2 0.58 ± 2% perf-profile.children.cycles-pp.io_serial_in 0.75 ± 5% -0.2 0.55 perf-profile.children.cycles-pp.serial8250_console_putchar 1.43 ± 2% -0.2 1.25 ± 6% perf-profile.children.cycles-pp.__x64_sys_exit_group 1.43 ± 2% -0.2 1.25 ± 6% perf-profile.children.cycles-pp.do_group_exit 1.43 ± 2% -0.2 1.25 ± 6% perf-profile.children.cycles-pp.do_exit 1.39 ± 2% -0.2 1.21 ± 6% perf-profile.children.cycles-pp.tlb_finish_mmu 1.39 ± 2% -0.2 1.21 ± 6% perf-profile.children.cycles-pp.tlb_flush_mmu 1.42 ± 2% -0.2 1.25 ± 6% perf-profile.children.cycles-pp.mmput 1.42 ± 2% -0.2 1.25 ± 6% perf-profile.children.cycles-pp.exit_mmap 1.39 ± 2% -0.2 1.21 ± 6% perf-profile.children.cycles-pp.release_pages 1.38 ± 2% -0.2 1.20 ± 6% perf-profile.children.cycles-pp.__free_huge_page 1.19 ± 3% -0.2 1.02 ± 7% perf-profile.children.cycles-pp._raw_spin_lock 0.78 ± 7% -0.1 0.65 ± 4% perf-profile.children.cycles-pp.cpuidle_enter 0.78 ± 7% -0.1 0.65 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state 0.78 ± 7% -0.1 0.66 ± 4% perf-profile.children.cycles-pp.secondary_startup_64 0.78 ± 7% -0.1 0.66 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 0.78 ± 7% -0.1 0.66 ± 4% perf-profile.children.cycles-pp.do_idle 0.75 ± 8% -0.1 0.63 ± 6% perf-profile.children.cycles-pp.start_secondary 0.74 ± 7% -0.1 0.63 ± 4% perf-profile.children.cycles-pp.intel_idle 0.20 ± 8% -0.1 0.15 ± 7% perf-profile.children.cycles-pp.io_serial_out 0.48 ± 5% -0.1 0.43 ± 5% perf-profile.children.cycles-pp.tick_sched_handle 0.46 ± 6% -0.0 0.42 ± 6% perf-profile.children.cycles-pp.update_process_times 0.15 ± 7% -0.0 0.12 ± 5% perf-profile.children.cycles-pp.alloc_huge_page 0.08 ± 5% -0.0 0.07 perf-profile.children.cycles-pp.update_curr 0.43 ± 4% +0.6 1.01 ± 8% perf-profile.children.cycles-pp.mutex_spin_on_owner 0.44 ± 5% +0.6 1.04 ± 7% perf-profile.children.cycles-pp.__mutex_lock 27.03 ± 2% +5.1 32.17 ± 4% perf-profile.children.cycles-pp.hugetlb_no_page 26.78 ± 2% +5.2 31.95 ± 4% perf-profile.children.cycles-pp.clear_huge_page 27.75 ± 2% +5.7 33.48 ± 4% perf-profile.children.cycles-pp.page_fault 27.68 ± 2% +5.7 33.42 ± 4% perf-profile.children.cycles-pp.do_page_fault 27.61 ± 2% +5.7 33.35 ± 4% perf-profile.children.cycles-pp.handle_mm_fault 27.54 ± 2% +5.7 33.28 ± 4% perf-profile.children.cycles-pp.hugetlb_fault 64.78 +5.9 70.72 ± 2% perf-profile.children.cycles-pp.do_access 24.19 ± 2% -24.2 0.00 perf-profile.self.cycles-pp.clear_page_erms 13.76 -3.1 10.71 ± 2% perf-profile.self.cycles-pp.do_access 3.91 ± 3% -0.6 3.28 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.46 ± 4% -0.4 0.04 ± 58% perf-profile.self.cycles-pp.___might_sleep 1.47 ± 8% -0.3 1.16 ± 3% perf-profile.self.cycles-pp.memcpy_erms 0.78 ± 4% -0.2 0.57 ± 2% perf-profile.self.cycles-pp.io_serial_in 0.74 ± 7% -0.1 0.63 ± 4% perf-profile.self.cycles-pp.intel_idle 0.30 ± 6% -0.1 0.24 ± 3% perf-profile.self.cycles-pp.get_page_from_freelist 0.20 ± 8% -0.1 0.15 ± 7% perf-profile.self.cycles-pp.io_serial_out 0.43 ± 5% +0.6 1.00 ± 8% perf-profile.self.cycles-pp.mutex_spin_on_owner 0.32 ± 2% +31.2 31.57 ± 4% perf-profile.self.cycles-pp.clear_huge_page vm-scalability.time.user_time 3050 +--------------------------------------------------------------------+ | | 3000 |-+ .+.. .+.. .+.. .+. .+.. .+..+.. .+.. .+..| |..+.+..+..+ +..+ +..+ +. +..+ +..+ + +..+ | | | 2950 |-+ | | | 2900 |-+ | | | 2850 |-+ | | | | O O O O | 2800 |-+O O O O O O O O O O O O O O O O O O O O | | | 2750 +--------------------------------------------------------------------+ vm-scalability.time.system_time 1750 +--------------------------------------------------------------------+ | | 1700 |-+O O O O O O O O O O O O O O O | | O O O O O O O O O | 1650 |-+ | | | 1600 |-+ | | | 1550 |-+ | | | 1500 |-+ | | | 1450 |..+.+..+..+. .+..+. .+.+..+.+..+..+. .+. .+..| | +. +..+..+.+..+. +..+. +..+..+ | 1400 +--------------------------------------------------------------------+ vm-scalability.time.percent_of_cpu_this_job_got 1490 +--------------------------------------------------------------------+ | O O O O O O O | | O O O O O O O O O O O O O | 1485 |-+ O O | | | | | 1480 |-+ | | | 1475 |-+ | | | | | 1470 |-+ .+ +. .+.. .+. + | |.. .+..+. + .+ .+.. .. +..+ +..+. .+. +.. : + | | + +..+. + .+ + +. +.. : +| 1465 +--------------------------------------------------------------------+ vm-scalability.time.minor_page_faults 2.1e+06 +----------------------------------------------------------------+ |..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+.. .+..+.| 2.05e+06 |-+ + | | | 2e+06 |-+ | 1.95e+06 |-+ | | | 1.9e+06 |-+ | | | 1.85e+06 |-+ | 1.8e+06 |-+ | | | 1.75e+06 |-+ O | | O O O O O O O O O O O O O O O O O O O O O O O | 1.7e+06 +----------------------------------------------------------------+ vm-scalability.time.voluntary_context_switches 17000 +-------------------------------------------------------------------+ | +.. .+..+.. .+. .+..+.+..+.+..+..+. .+.. | 16500 |..+.+..+. .. + +.+. +. +. +.+..+.+..| | + | | | 16000 |-+ | | | 15500 |-+ | | | 15000 |-+ O | | O O O | | O O O O O O O O | 14500 |-+O O O O O O O O O O O O | | | 14000 +-------------------------------------------------------------------+ vm-scalability.throughput 1.45e+07 +----------------------------------------------------------------+ | | 1.4e+07 |..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.+..+.| | | | | 1.35e+07 |-+ | | | 1.3e+07 |-+ | | | 1.25e+07 |-+ | | | | | 1.2e+07 |-+ | | O O O O O O O O O O O O O O O O O O O O O O O O | 1.15e+07 +----------------------------------------------------------------+ vm-scalability.median 900000 +------------------------------------------------------------------+ |..+.+..+.+..+.+..+..+.+..+.+..+..+.+..+.+..+.+..+..+.+..+.+..+.+..| 880000 |-+ | 860000 |-+ | | | 840000 |-+ | 820000 |-+ | | | 800000 |-+ | 780000 |-+ | | | 760000 |-+ | 740000 |-+ | | O O O O O O O O O O O O O O O O O O O O O O O O | 720000 +------------------------------------------------------------------+ vm-scalability.workload 4e+09 +-----------------------------------------------------------------+ |..+.+..+.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+.+..+.+..+.+..+.+..| 3.9e+09 |-+ | | | 3.8e+09 |-+ | | | 3.7e+09 |-+ | | | 3.6e+09 |-+ | | | 3.5e+09 |-+ | | | 3.4e+09 |-+ | | O | 3.3e+09 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-ivb-d02: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/16/debian-x86_64-20191114.cgz/lkp-ivb-d02/page_fault1/will-it-scale/0x21 commit: 8632e9b564 (" hyperv-fixes for 5.7-rc1") 9266baf8ee ("mm: Optimized hugepage zeroing & copying from user") 8632e9b5645bbc23 9266baf8eea588a78e5a0aa4ccd ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 0% 1:4 perf-profile.children.cycles-pp.error_entry %stddev %change %stddev \ | \ 238214 -4.9% 226603 will-it-scale.per_process_ops 3811432 -4.9% 3625669 will-it-scale.workload 25486 ± 8% +16.8% 29758 ± 7% softirqs.CPU3.RCU 20.96 ± 2% +5.5% 22.11 ± 2% boot-time.boot 55.42 ± 2% +6.0% 58.77 ± 2% boot-time.idle 4186 ± 16% +29.2% 5409 ± 12% interrupts.CPU2.RES:Rescheduling_interrupts 5630 ± 7% -14.9% 4788 ± 12% interrupts.CPU3.RES:Rescheduling_interrupts 98.92 ± 25% -27.7% 71.56 ± 18% sched_debug.cfs_rq:/.load_avg.stddev 515190 ± 15% +21.8% 627617 ± 8% sched_debug.cpu.avg_idle.avg 11670 ± 6% +17.0% 13659 sched_debug.cpu.ttwu_count.min 9696 ± 7% +13.9% 11046 ± 4% sched_debug.cpu.ttwu_local.min 313013 +1.0% 316254 proc-vmstat.nr_active_anon 309576 +1.1% 313054 proc-vmstat.nr_anon_pages 313008 +1.0% 316246 proc-vmstat.nr_zone_active_anon 22747924 -4.8% 21653355 proc-vmstat.numa_hit 22747924 -4.8% 21653355 proc-vmstat.numa_local 1.15e+09 -4.9% 1.093e+09 proc-vmstat.pgalloc_normal 20511827 -4.9% 19515779 proc-vmstat.pgfault 1.15e+09 -5.0% 1.093e+09 proc-vmstat.pgfree 2206073 -4.9% 2097199 proc-vmstat.thp_fault_alloc 82.50 -82.5 0.00 perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 80.56 -80.6 0.00 perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault 3.16 -0.3 2.87 ± 3% perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault 3.10 -0.3 2.81 ± 3% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault 3.12 -0.3 2.84 ± 3% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 93.63 +0.3 93.98 perf-profile.calltrace.cycles-pp.page_fault 93.45 +0.4 93.81 perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 92.88 +0.4 93.31 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault 92.65 +0.5 93.12 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 85.28 +0.6 85.83 perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault 89.69 +0.7 90.36 perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 82.68 -82.7 0.00 perf-profile.children.cycles-pp.clear_subpage 82.08 -80.9 1.19 perf-profile.children.cycles-pp.clear_page_erms 0.81 -0.6 0.20 ± 4% perf-profile.children.cycles-pp.rcu_all_qs 5.07 -0.4 4.64 perf-profile.children.cycles-pp.get_page_from_freelist 5.20 -0.4 4.79 perf-profile.children.cycles-pp.__alloc_pages_nodemask 4.92 -0.4 4.54 ± 2% perf-profile.children.cycles-pp.alloc_pages_vma 1.08 -0.1 0.96 ± 4% perf-profile.children.cycles-pp._cond_resched 0.40 ± 11% -0.1 0.31 ± 4% perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay 1.90 -0.1 1.81 perf-profile.children.cycles-pp.prep_new_page 0.47 ± 4% -0.1 0.41 ± 6% perf-profile.children.cycles-pp.unmap_vmas 0.46 ± 4% -0.1 0.41 ± 6% perf-profile.children.cycles-pp.unmap_page_range 0.38 ± 8% -0.0 0.33 ± 6% perf-profile.children.cycles-pp.pte_alloc_one 0.10 ± 7% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate 0.16 ± 5% -0.0 0.13 ± 19% perf-profile.children.cycles-pp._raw_spin_lock 0.13 ± 6% -0.0 0.10 ± 11% perf-profile.children.cycles-pp.zap_huge_pmd 93.72 +0.3 94.05 perf-profile.children.cycles-pp.page_fault 93.53 +0.3 93.88 perf-profile.children.cycles-pp.do_page_fault 92.94 +0.4 93.36 perf-profile.children.cycles-pp.handle_mm_fault 92.72 +0.5 93.18 perf-profile.children.cycles-pp.__handle_mm_fault 89.69 +0.7 90.36 perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page 85.41 +1.1 86.54 perf-profile.children.cycles-pp.clear_huge_page 81.23 -80.1 1.17 ± 2% perf-profile.self.cycles-pp.clear_page_erms 0.60 ± 3% -0.4 0.15 ± 2% perf-profile.self.cycles-pp.rcu_all_qs 1.56 ± 5% -0.4 1.12 ± 4% perf-profile.self.cycles-pp.___might_sleep 2.44 -0.3 2.18 ± 4% perf-profile.self.cycles-pp.get_page_from_freelist 2.07 ± 4% -0.1 1.96 perf-profile.self.cycles-pp.__free_pages_ok 0.08 ± 6% -0.0 0.05 perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate 0.31 ± 8% +0.1 0.46 ± 2% perf-profile.self.cycles-pp._cond_resched 0.21 ± 8% +83.9 84.10 perf-profile.self.cycles-pp.clear_huge_page *************************************************************************************************** lkp-ivb-d02: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-ivb-d02/anon-w-seq-mt/vm-scalability/0x21 commit: 8632e9b564 (" hyperv-fixes for 5.7-rc1") 9266baf8ee ("mm: Optimized hugepage zeroing & copying from user") 8632e9b5645bbc23 9266baf8eea588a78e5a0aa4ccd ---------------- --------------------------- %stddev %change %stddev \ | \ 0.00 +1.8% 0.00 vm-scalability.free_time 1148096 +8.4% 1244718 vm-scalability.median 4588586 +7.8% 4948645 vm-scalability.throughput 1475456 +7.5% 1586296 vm-scalability.time.minor_page_faults 270.13 -15.6% 227.88 ± 2% vm-scalability.time.system_time 868.34 +4.7% 908.82 vm-scalability.time.user_time 1.326e+09 +7.8% 1.43e+09 vm-scalability.workload 0.01 ± 37% +0.0 0.02 ± 76% mpstat.cpu.all.soft% 23.34 -3.5 19.88 ± 2% mpstat.cpu.all.sys% 73.00 +5.1% 76.75 vmstat.cpu.us 6969 +5.2% 7329 ± 2% vmstat.system.cs 627528 ± 2% -16.5% 524174 ± 4% sched_debug.cpu.avg_idle.avg -12.75 +25.5% -16.00 sched_debug.cpu.nr_uninterruptible.min 10.19 ± 6% +24.2% 12.65 ± 18% sched_debug.cpu.nr_uninterruptible.stddev 19223068 ± 15% -21.8% 15037169 ± 10% cpuidle.C6.time 33333 ± 3% -16.3% 27901 ± 6% cpuidle.C6.usage 8465 ± 39% -58.8% 3487 ± 15% cpuidle.POLL.time 2649 ± 23% -54.6% 1203 ± 21% cpuidle.POLL.usage 26729 ± 7% +78.6% 47740 ± 39% softirqs.CPU0.RCU 4574 ± 15% +79.7% 8219 ± 55% softirqs.CPU1.NET_RX 28386 ± 10% +63.3% 46347 ± 40% softirqs.CPU1.RCU 27498 ± 6% +75.2% 48178 ± 40% softirqs.CPU2.RCU 27869 ± 5% +73.8% 48433 ± 39% softirqs.CPU3.RCU 4577 ± 15% +79.6% 8222 ± 55% softirqs.NET_RX 110484 ± 4% +72.6% 190699 ± 39% softirqs.RCU 3455928 ± 2% -36.0% 2212394 meminfo.Active 3452870 ± 2% -36.0% 2209318 meminfo.Active(anon) 3281881 ± 2% -37.5% 2049676 meminfo.AnonHugePages 3425083 ± 2% -35.9% 2195653 meminfo.AnonPages 52607 ± 16% +165.2% 139515 ± 2% meminfo.CmaFree 3259754 ± 2% +38.2% 4504331 meminfo.MemAvailable 3368788 ± 2% +36.9% 4613377 meminfo.MemFree 4662470 ± 2% -26.7% 3417882 meminfo.Memused 10268 -24.1% 7794 meminfo.PageTables 8448 ± 16% +82.5% 15416 ± 55% interrupts.27:PCI-MSI.1572864-edge.eth0 4211 -13.1% 3661 interrupts.CAL:Function_call_interrupts 1058 -14.5% 904.50 ± 2% interrupts.CPU0.CAL:Function_call_interrupts 8448 ± 16% +82.5% 15416 ± 55% interrupts.CPU1.27:PCI-MSI.1572864-edge.eth0 1042 ± 4% -14.6% 889.75 ± 3% interrupts.CPU1.CAL:Function_call_interrupts 6167 +52.0% 9373 ± 29% interrupts.CPU1.NMI:Non-maskable_interrupts 6167 +52.0% 9373 ± 29% interrupts.CPU1.PMI:Performance_monitoring_interrupts 1076 -10.1% 968.25 ± 2% interrupts.CPU2.CAL:Function_call_interrupts 1034 ± 2% -13.1% 898.75 ± 4% interrupts.CPU3.CAL:Function_call_interrupts 6172 +53.3% 9464 ± 32% interrupts.CPU3.NMI:Non-maskable_interrupts 6172 +53.3% 9464 ± 32% interrupts.CPU3.PMI:Performance_monitoring_interrupts 825701 ± 3% -33.6% 548074 proc-vmstat.nr_active_anon 810670 ± 3% -32.9% 543797 proc-vmstat.nr_anon_pages 1516 ± 3% -34.5% 992.25 proc-vmstat.nr_anon_transparent_hugepages 85789 ± 3% +30.6% 112073 proc-vmstat.nr_dirty_background_threshold 171789 ± 3% +30.6% 224421 proc-vmstat.nr_dirty_threshold 15623 ± 19% +127.4% 35525 proc-vmstat.nr_free_cma 879784 ± 3% +31.6% 1157608 proc-vmstat.nr_free_pages 3558 -0.7% 3535 proc-vmstat.nr_kernel_stack 2481 ± 2% -21.7% 1944 proc-vmstat.nr_page_table_pages 825696 ± 3% -33.6% 548054 proc-vmstat.nr_zone_active_anon 2326268 +6.5% 2476331 proc-vmstat.numa_hit 2326268 +6.5% 2476331 proc-vmstat.numa_local 72116983 +5.5% 76117229 ± 3% proc-vmstat.pgalloc_dma32 2.237e+08 +8.6% 2.429e+08 proc-vmstat.pgalloc_normal 1859214 +5.9% 1969537 proc-vmstat.pgfault 2.948e+08 +8.1% 3.186e+08 proc-vmstat.pgfree 574297 +7.8% 619284 proc-vmstat.thp_fault_alloc 21.56 ± 7% -21.6 0.00 perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 20.80 ± 7% -20.8 0.00 perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault 23.47 ± 7% -5.2 18.26 ± 2% perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 22.35 ± 7% -5.2 17.18 ± 2% perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault 22.65 ± 3% -4.5 18.18 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault.do_access 22.74 ± 3% -4.5 18.27 perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.do_access 22.61 ± 3% -4.5 18.14 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault.do_access 22.77 ± 3% -4.5 18.31 perf-profile.calltrace.cycles-pp.page_fault.do_access 0.73 ± 8% -0.1 0.63 ± 5% perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault 0.72 ± 8% -0.1 0.62 ± 5% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 0.70 ± 9% -0.1 0.61 ± 5% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault 62.06 ± 2% +2.1 64.16 perf-profile.calltrace.cycles-pp.do_rw_once 21.78 ± 7% -21.8 0.00 perf-profile.children.cycles-pp.clear_subpage 21.35 ± 7% -21.3 0.06 ± 16% perf-profile.children.cycles-pp.clear_page_erms 23.62 ± 7% -5.2 18.45 ± 2% perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page 23.93 ± 6% -5.1 18.82 ± 2% perf-profile.children.cycles-pp.do_page_fault 23.82 ± 6% -5.1 18.73 ± 2% perf-profile.children.cycles-pp.handle_mm_fault 23.98 ± 6% -5.1 18.89 ± 2% perf-profile.children.cycles-pp.page_fault 23.77 ± 7% -5.1 18.69 ± 2% perf-profile.children.cycles-pp.__handle_mm_fault 22.54 ± 7% -5.0 17.52 perf-profile.children.cycles-pp.clear_huge_page 48.63 -1.0 47.68 perf-profile.children.cycles-pp.do_access 1.40 ± 8% -0.4 1.03 ± 19% perf-profile.children.cycles-pp.apic_timer_interrupt 1.30 ± 7% -0.4 0.95 ± 21% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 1.09 ± 9% -0.3 0.75 ± 22% perf-profile.children.cycles-pp.hrtimer_interrupt 0.91 ± 11% -0.3 0.64 ± 22% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.67 ± 9% -0.2 0.46 ± 20% perf-profile.children.cycles-pp.tick_sched_timer 0.58 ± 10% -0.2 0.39 ± 18% perf-profile.children.cycles-pp.tick_sched_handle 0.56 ± 10% -0.2 0.37 ± 20% perf-profile.children.cycles-pp.update_process_times 0.21 ± 6% -0.1 0.07 ± 17% perf-profile.children.cycles-pp.rcu_all_qs 0.34 ± 6% -0.1 0.23 ± 12% perf-profile.children.cycles-pp.scheduler_tick 0.46 ± 13% -0.1 0.35 ± 8% perf-profile.children.cycles-pp.___might_sleep 0.80 ± 7% -0.1 0.71 ± 6% perf-profile.children.cycles-pp.alloc_pages_vma 0.25 ± 8% -0.1 0.17 ± 14% perf-profile.children.cycles-pp.task_tick_fair 0.28 ± 9% -0.1 0.21 ± 5% perf-profile.children.cycles-pp._cond_resched 0.15 ± 11% -0.0 0.11 ± 22% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.08 ± 8% +0.0 0.12 ± 10% perf-profile.children.cycles-pp.__do_execve_file 0.08 ± 8% +0.0 0.12 ± 12% perf-profile.children.cycles-pp.__x64_sys_execve 0.08 ± 10% +0.0 0.12 ± 8% perf-profile.children.cycles-pp.execve 0.03 ±102% +0.0 0.08 ± 19% perf-profile.children.cycles-pp.search_binary_handler 0.01 ±173% +0.1 0.06 ± 6% perf-profile.children.cycles-pp.drm_fb_helper_dirty 0.11 ± 33% +0.1 0.18 ± 16% perf-profile.children.cycles-pp.handle_pte_fault 1.27 ± 7% +0.2 1.47 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 1.27 ± 7% +0.2 1.47 ± 4% perf-profile.children.cycles-pp.do_syscall_64 63.21 +3.4 66.66 perf-profile.children.cycles-pp.do_rw_once 20.97 ± 6% -20.9 0.06 ± 16% perf-profile.self.cycles-pp.clear_page_erms 0.44 ± 16% -0.2 0.28 ± 11% perf-profile.self.cycles-pp.___might_sleep 0.16 ± 13% -0.1 0.05 ± 58% perf-profile.self.cycles-pp.rcu_all_qs 0.08 ± 23% -0.1 0.03 ±100% perf-profile.self.cycles-pp.task_tick_fair 0.06 ± 14% +0.0 0.08 ± 10% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 17.35 ± 5% +1.8 19.11 perf-profile.self.cycles-pp.do_access 55.41 +3.1 58.55 perf-profile.self.cycles-pp.do_rw_once 0.09 ± 27% +16.8 16.94 ± 2% perf-profile.self.cycles-pp.clear_huge_page *************************************************************************************************** lkp-csl-2sp6: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-csl-2sp6/anon-cow-seq-hugetlb/vm-scalability/0x500002c commit: 8632e9b564 (" hyperv-fixes for 5.7-rc1") 9266baf8ee ("mm: Optimized hugepage zeroing & copying from user") 8632e9b5645bbc23 9266baf8eea588a78e5a0aa4ccd ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 75% 3:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :4 75% 3:4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 361704 -2.6% 352245 vm-scalability.median 36376079 -4.6% 34694751 vm-scalability.throughput 4265227 -5.4% 4036714 vm-scalability.time.minor_page_faults 13472 -2.7% 13115 vm-scalability.time.system_time 8845 +2.1% 9035 vm-scalability.time.user_time 354016 -5.3% 335133 vm-scalability.time.voluntary_context_switches 9.105e+09 -5.4% 8.616e+09 vm-scalability.workload 4848 -2.0% 4751 vmstat.system.cs 2477460 ± 3% -8.2% 2274167 ± 4% numa-numastat.node0.local_node 2508460 ± 3% -8.3% 2301032 ± 4% numa-numastat.node0.numa_hit 1386 ± 4% -7.1% 1288 ± 5% slabinfo.Acpi-Parse.active_objs 1386 ± 4% -7.1% 1288 ± 5% slabinfo.Acpi-Parse.num_objs 3371 ± 11% +26.7% 4271 ± 10% slabinfo.eventpoll_pwq.active_objs 3371 ± 11% +26.7% 4271 ± 10% slabinfo.eventpoll_pwq.num_objs 812.33 ± 9% -15.8% 684.33 ± 4% slabinfo.kmem_cache_node.active_objs 852.33 ± 9% -14.9% 725.00 ± 4% slabinfo.kmem_cache_node.num_objs 3993284 -5.4% 3778602 proc-vmstat.htlb_buddy_alloc_success 179.00 -3.5% 172.67 proc-vmstat.nr_dirtied 45924 -1.1% 45436 proc-vmstat.nr_shmem 166.00 -3.6% 160.00 proc-vmstat.nr_written 5075790 -4.8% 4832494 proc-vmstat.numa_hit 5044600 -4.8% 4801279 proc-vmstat.numa_local 2.046e+09 -5.4% 1.936e+09 proc-vmstat.pgalloc_normal 5095148 -4.6% 4860017 proc-vmstat.pgfault 2.045e+09 -5.3% 1.936e+09 proc-vmstat.pgfree 1299 ± 71% +185.8% 3715 ± 35% sched_debug.cfs_rq:/.load.min 0.12 ± 71% +181.8% 0.34 ± 35% sched_debug.cfs_rq:/.nr_running.min 0.29 ± 5% -33.3% 0.19 ± 11% sched_debug.cfs_rq:/.nr_running.stddev 1164 ± 6% +20.0% 1397 ± 4% sched_debug.cfs_rq:/.util_avg.max 244.31 ± 4% -12.5% 213.67 ± 2% sched_debug.cfs_rq:/.util_avg.stddev 0.12 ± 71% +181.8% 0.34 ± 35% sched_debug.cfs_rq:/.util_est_enqueued.min 8.71 ± 35% +62.1% 14.13 ± 20% sched_debug.cpu.clock.stddev 8.71 ± 35% +62.1% 14.13 ± 20% sched_debug.cpu.clock_task.stddev 2263 ± 13% -33.6% 1503 ± 12% sched_debug.cpu.curr->pid.stddev 1.18 +30.2% 1.53 ± 9% sched_debug.cpu.nr_running.max 0.12 ± 71% +236.4% 0.41 ± 16% sched_debug.cpu.nr_running.min 0.30 ± 2% -24.7% 0.23 ± 12% sched_debug.cpu.nr_running.stddev 0.06 ± 18% -75.4% 0.02 ± 40% sched_debug.cpu.nr_uninterruptible.avg -50.38 -19.7% -40.47 sched_debug.cpu.nr_uninterruptible.min 9247 ± 13% +20.9% 11175 ± 7% sched_debug.cpu.ttwu_count.max 13116 ± 9% +12.9% 14811 ± 6% softirqs.CPU34.SCHED 13634 ± 9% -14.8% 11609 ± 10% softirqs.CPU4.SCHED 123480 ± 6% -10.8% 110147 softirqs.CPU4.TIMER 123091 ± 6% -10.4% 110308 softirqs.CPU52.TIMER 12976 ± 13% -13.9% 11174 ± 14% softirqs.CPU53.SCHED 12600 ± 10% -12.2% 11058 ± 15% softirqs.CPU56.SCHED 13147 ± 13% -14.2% 11275 ± 13% softirqs.CPU57.SCHED 13284 ± 14% -18.0% 10898 ± 15% softirqs.CPU58.SCHED 121769 ± 6% -10.3% 109258 softirqs.CPU58.TIMER 13140 ± 12% -19.6% 10561 ± 15% softirqs.CPU60.SCHED 13017 ± 13% -13.4% 11274 ± 12% softirqs.CPU64.SCHED 12951 ± 12% -17.2% 10724 ± 14% softirqs.CPU68.SCHED 13327 ± 14% -15.8% 11216 ± 15% softirqs.CPU69.SCHED 12891 ± 14% +14.7% 14789 ± 7% softirqs.CPU84.SCHED 13383 ± 9% -10.6% 11968 ± 9% softirqs.CPU9.SCHED 39.20 -39.2 0.00 perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault 38.70 -38.7 0.00 perf-profile.calltrace.cycles-pp.copy_page.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault 83.46 -17.1 66.37 ± 9% perf-profile.calltrace.cycles-pp.do_access 52.26 -12.6 39.62 ± 9% perf-profile.calltrace.cycles-pp.page_fault.do_access 52.22 -12.6 39.59 ± 9% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.do_access 52.16 -12.6 39.54 ± 9% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault.do_access 52.13 -12.6 39.52 ± 9% perf-profile.calltrace.cycles-pp.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault.do_access 14.42 ± 2% -3.6 10.85 ± 7% perf-profile.calltrace.cycles-pp.do_rw_once 0.69 ± 7% -0.3 0.37 ± 70% perf-profile.calltrace.cycles-pp.alloc_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.copy_page.copy_user_huge_page.hugetlb_cow.hugetlb_fault 39.37 +6.0 45.35 perf-profile.calltrace.cycles-pp.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault 40.32 +6.1 46.44 perf-profile.calltrace.cycles-pp.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault 6.40 ± 36% +18.0 24.41 ± 23% perf-profile.calltrace.cycles-pp.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault 6.41 ± 36% +18.0 24.44 ± 23% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault 6.42 ± 36% +18.0 24.47 ± 23% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 6.42 ± 36% +18.1 24.48 ± 23% perf-profile.calltrace.cycles-pp.page_fault 0.00 +44.6 44.64 perf-profile.calltrace.cycles-pp.copy_page.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault 39.21 -39.2 0.00 perf-profile.children.cycles-pp.copy_subpage 72.47 -16.0 56.47 ± 9% perf-profile.children.cycles-pp.do_access 25.92 ± 2% -4.8 21.11 ± 8% perf-profile.children.cycles-pp.do_rw_once 0.11 +0.0 0.13 ± 6% perf-profile.children.cycles-pp.native_irq_return_iret 0.07 ± 11% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.native_write_msr 0.11 ± 4% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.update_and_free_page 0.28 ± 5% +0.0 0.32 perf-profile.children.cycles-pp.process_one_work 0.27 ± 4% +0.0 0.31 perf-profile.children.cycles-pp.drm_fb_helper_dirty_work 0.37 ± 3% +0.0 0.41 ± 6% perf-profile.children.cycles-pp.task_tick_fair 0.29 ± 4% +0.0 0.33 perf-profile.children.cycles-pp.ret_from_fork 0.29 ± 4% +0.0 0.33 perf-profile.children.cycles-pp.kthread 0.28 ± 5% +0.0 0.32 perf-profile.children.cycles-pp.worker_thread 0.27 ± 4% +0.0 0.32 perf-profile.children.cycles-pp.memcpy_erms 0.09 ± 24% +0.0 0.13 ± 14% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.02 ±141% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.00 +0.1 0.05 perf-profile.children.cycles-pp.run_local_timers 0.02 ±141% +0.1 0.07 ± 23% perf-profile.children.cycles-pp.menu_select 0.58 ± 2% +0.1 0.66 ± 2% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.56 ± 2% +0.1 0.64 perf-profile.children.cycles-pp.get_page_from_freelist 0.62 ± 3% +0.1 0.72 perf-profile.children.cycles-pp.alloc_fresh_huge_page 0.05 ± 74% +0.1 0.14 ± 32% perf-profile.children.cycles-pp.start_kernel 1.02 ± 2% +0.1 1.13 ± 6% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.71 +0.1 0.82 ± 3% perf-profile.children.cycles-pp.alloc_surplus_huge_page 0.84 +0.1 0.96 ± 3% perf-profile.children.cycles-pp.alloc_huge_page 1.26 ± 2% +0.1 1.40 ± 2% perf-profile.children.cycles-pp.hrtimer_interrupt 1.47 ± 2% +0.1 1.62 perf-profile.children.cycles-pp.smp_apic_timer_interrupt 1.91 +0.2 2.14 perf-profile.children.cycles-pp.apic_timer_interrupt 58.58 ± 3% +5.4 63.99 ± 4% perf-profile.children.cycles-pp.handle_mm_fault 58.60 ± 3% +5.4 64.02 ± 4% perf-profile.children.cycles-pp.hugetlb_fault 58.70 ± 3% +5.4 64.12 ± 4% perf-profile.children.cycles-pp.page_fault 58.65 ± 3% +5.4 64.08 ± 4% perf-profile.children.cycles-pp.do_page_fault 39.38 +6.0 45.36 perf-profile.children.cycles-pp.copy_user_huge_page 39.05 +6.0 45.07 perf-profile.children.cycles-pp.copy_page 40.33 +6.1 46.44 perf-profile.children.cycles-pp.hugetlb_cow 25.22 ± 2% -4.6 20.61 ± 8% perf-profile.self.cycles-pp.do_rw_once 8.36 -2.1 6.29 ± 8% perf-profile.self.cycles-pp.do_access 0.11 +0.0 0.13 ± 6% perf-profile.self.cycles-pp.native_irq_return_iret 0.07 ± 11% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.native_write_msr 0.11 ± 4% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.update_and_free_page 0.27 ± 5% +0.0 0.31 perf-profile.self.cycles-pp.memcpy_erms 0.00 +0.1 0.05 perf-profile.self.cycles-pp.run_local_timers 0.33 +0.1 0.40 perf-profile.self.cycles-pp.get_page_from_freelist 0.00 +0.1 0.08 ± 10% perf-profile.self.cycles-pp.clear_huge_page 0.00 +0.1 0.12 perf-profile.self.cycles-pp.copy_user_huge_page 38.43 +5.9 44.38 perf-profile.self.cycles-pp.copy_page 291.00 ± 9% -11.9% 256.33 ± 3% interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 297.00 ±116% -99.0% 3.00 ±141% interrupts.46:PCI-MSI.31981579-edge.i40e-eth0-TxRx-10 296.67 ±116% -99.1% 2.67 ±141% interrupts.CPU10.46:PCI-MSI.31981579-edge.i40e-eth0-TxRx-10 501.00 ± 3% +22.0% 611.00 ± 8% interrupts.CPU12.RES:Rescheduling_interrupts 6041 ± 7% -37.7% 3766 ± 20% interrupts.CPU2.NMI:Non-maskable_interrupts 6041 ± 7% -37.7% 3766 ± 20% interrupts.CPU2.PMI:Performance_monitoring_interrupts 5468 ± 8% +10.8% 6057 ± 10% interrupts.CPU21.NMI:Non-maskable_interrupts 5468 ± 8% +10.8% 6057 ± 10% interrupts.CPU21.PMI:Performance_monitoring_interrupts 1547 ± 63% +186.7% 4435 ± 17% interrupts.CPU24.TLB:TLB_shootdowns 585.67 -23.1% 450.33 ± 20% interrupts.CPU25.CAL:Function_call_interrupts 6241 ± 4% -23.6% 4767 ± 26% interrupts.CPU25.NMI:Non-maskable_interrupts 6241 ± 4% -23.6% 4767 ± 26% interrupts.CPU25.PMI:Performance_monitoring_interrupts 1089 ± 68% -56.9% 469.67 ± 2% interrupts.CPU26.RES:Rescheduling_interrupts 6424 -11.8% 5667 ± 4% interrupts.CPU28.NMI:Non-maskable_interrupts 6424 -11.8% 5667 ± 4% interrupts.CPU28.PMI:Performance_monitoring_interrupts 6076 ± 8% -22.9% 4684 ± 20% interrupts.CPU30.NMI:Non-maskable_interrupts 6076 ± 8% -22.9% 4684 ± 20% interrupts.CPU30.PMI:Performance_monitoring_interrupts 895.33 ± 73% +105.6% 1840 ± 3% interrupts.CPU30.TLB:TLB_shootdowns 6224 ± 9% -23.3% 4777 ± 26% interrupts.CPU31.NMI:Non-maskable_interrupts 6224 ± 9% -23.3% 4777 ± 26% interrupts.CPU31.PMI:Performance_monitoring_interrupts 529.00 ± 6% -27.2% 385.00 ± 26% interrupts.CPU38.RES:Rescheduling_interrupts 970.00 ± 69% +103.5% 1974 ± 9% interrupts.CPU38.TLB:TLB_shootdowns 4676 ± 26% +33.3% 6233 ± 10% interrupts.CPU4.NMI:Non-maskable_interrupts 4676 ± 26% +33.3% 6233 ± 10% interrupts.CPU4.PMI:Performance_monitoring_interrupts 518.67 ± 22% -25.4% 387.00 ± 25% interrupts.CPU40.RES:Rescheduling_interrupts 450.67 ± 20% -23.0% 347.00 ± 21% interrupts.CPU41.RES:Rescheduling_interrupts 6379 ± 7% -8.6% 5832 ± 7% interrupts.CPU42.NMI:Non-maskable_interrupts 6379 ± 7% -8.6% 5832 ± 7% interrupts.CPU42.PMI:Performance_monitoring_interrupts 509.67 ± 13% -34.7% 332.67 ± 22% interrupts.CPU42.RES:Rescheduling_interrupts 527.33 ± 9% -44.6% 292.00 ± 19% interrupts.CPU43.RES:Rescheduling_interrupts 545.33 ± 27% -41.1% 321.33 ± 10% interrupts.CPU45.RES:Rescheduling_interrupts 507.67 ± 25% -34.6% 332.00 ± 22% interrupts.CPU46.RES:Rescheduling_interrupts 435.00 ± 22% -21.5% 341.67 ± 6% interrupts.CPU48.RES:Rescheduling_interrupts 1142 ± 65% -72.2% 317.33 ± 26% interrupts.CPU56.RES:Rescheduling_interrupts 602.00 ± 64% -56.0% 265.00 ± 5% interrupts.CPU60.RES:Rescheduling_interrupts 282.00 ± 6% -16.2% 236.33 ± 2% interrupts.CPU61.RES:Rescheduling_interrupts 64.33 ± 67% +293.8% 253.33 ± 43% interrupts.CPU61.TLB:TLB_shootdowns 771.00 ± 62% -66.1% 261.33 ± 19% interrupts.CPU66.RES:Rescheduling_interrupts 81.00 ± 83% +135.0% 190.33 ± 27% interrupts.CPU66.TLB:TLB_shootdowns 4613 ± 21% +35.3% 6242 ± 10% interrupts.CPU68.NMI:Non-maskable_interrupts 4613 ± 21% +35.3% 6242 ± 10% interrupts.CPU68.PMI:Performance_monitoring_interrupts 4684 ± 35% +35.8% 6359 ± 8% interrupts.CPU69.NMI:Non-maskable_interrupts 4684 ± 35% +35.8% 6359 ± 8% interrupts.CPU69.PMI:Performance_monitoring_interrupts 291.00 ± 9% -11.9% 256.33 ± 3% interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 102.00 ± 71% +105.2% 209.33 ± 26% interrupts.CPU70.TLB:TLB_shootdowns 245.00 ± 61% +156.5% 628.33 ± 5% interrupts.CPU72.TLB:TLB_shootdowns 140.67 ± 53% +183.9% 399.33 ± 47% interrupts.CPU74.TLB:TLB_shootdowns 6394 ± 2% -8.2% 5870 ± 6% interrupts.CPU76.NMI:Non-maskable_interrupts 6394 ± 2% -8.2% 5870 ± 6% interrupts.CPU76.PMI:Performance_monitoring_interrupts 173.33 ± 54% +251.9% 610.00 ± 32% interrupts.CPU76.TLB:TLB_shootdowns 74.00 ± 36% +544.6% 477.00 ± 14% interrupts.CPU77.TLB:TLB_shootdowns 317.00 ± 24% -26.4% 233.33 ± 20% interrupts.CPU79.RES:Rescheduling_interrupts 114.67 ±103% +126.7% 260.00 ± 23% interrupts.CPU79.TLB:TLB_shootdowns 152.00 ± 85% +110.3% 319.67 ± 11% interrupts.CPU82.TLB:TLB_shootdowns 284.67 -31.6% 194.67 ± 32% interrupts.CPU83.RES:Rescheduling_interrupts 267.67 ± 8% -19.7% 215.00 ± 17% interrupts.CPU84.RES:Rescheduling_interrupts 99.67 ±115% +134.1% 233.33 ± 40% interrupts.CPU84.TLB:TLB_shootdowns 76.00 ± 87% +200.4% 228.33 ± 32% interrupts.CPU86.TLB:TLB_shootdowns 6324 ± 6% -21.9% 4936 ± 29% interrupts.CPU87.NMI:Non-maskable_interrupts 6324 ± 6% -21.9% 4936 ± 29% interrupts.CPU87.PMI:Performance_monitoring_interrupts 38.67 ±112% +550.0% 251.33 ± 58% interrupts.CPU87.TLB:TLB_shootdowns 720.33 ± 66% -69.8% 217.67 ± 19% interrupts.CPU89.RES:Rescheduling_interrupts 309.67 ± 14% -35.5% 199.67 ± 19% interrupts.CPU90.RES:Rescheduling_interrupts 31.67 ± 70% +533.7% 200.67 ± 15% interrupts.CPU92.TLB:TLB_shootdowns 6189 ± 9% -24.8% 4656 ± 29% interrupts.CPU94.NMI:Non-maskable_interrupts 6189 ± 9% -24.8% 4656 ± 29% interrupts.CPU94.PMI:Performance_monitoring_interrupts 73.00 ± 65% +444.3% 397.33 ± 24% interrupts.CPU94.TLB:TLB_shootdowns 6313 ± 5% -9.7% 5703 ± 4% interrupts.CPU95.NMI:Non-maskable_interrupts 6313 ± 5% -9.7% 5703 ± 4% interrupts.CPU95.PMI:Performance_monitoring_interrupts 49044 ± 10% -15.0% 41667 ± 12% interrupts.RES:Rescheduling_interrupts *************************************************************************************************** lkp-ivb-d01: 8 threads Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz with 16G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-ivb-d01/page_fault1/will-it-scale/0x21 commit: 8632e9b564 (" hyperv-fixes for 5.7-rc1") 9266baf8ee ("mm: Optimized hugepage zeroing & copying from user") 8632e9b5645bbc23 9266baf8eea588a78e5a0aa4ccd ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.RIP:cpuidle_enter_state :4 25% 1:4 dmesg.RIP:simple_getattr :4 25% 1:4 kmsg.b135d8f>]usb_hcd_irq 1:4 -25% :4 kmsg.c48>]usb_hcd_irq 1:4 -25% :4 kmsg.ca#>]usb_hcd_irq 1:4 -25% :4 kmsg.cb32>]usb_hcd_irq :4 25% 1:4 kmsg.d428ae1>]usb_hcd_irq 1:4 -25% :4 kmsg.ddef>]usb_hcd_irq %stddev %change %stddev \ | \ 534714 -3.5% 515893 will-it-scale.per_thread_ops 2517464 -3.5% 2428934 will-it-scale.time.minor_page_faults 518727 -4.8% 493620 will-it-scale.time.voluntary_context_switches 4277722 -3.5% 4127153 will-it-scale.workload 9481 -1.3% 9358 vmstat.system.cs 2764362 +196.4% 8193435 ± 28% cpuidle.C1.time 63195 ± 3% +126.3% 143040 ± 41% cpuidle.C1.usage 18199 ± 7% +12.8% 20532 ± 9% softirqs.CPU3.RCU 17571 ± 10% +21.0% 21255 ± 7% softirqs.CPU5.RCU 248287 -6.0% 233281 interrupts.CAL:Function_call_interrupts 2273 ± 17% -18.5% 1852 interrupts.CPU1.RES:Rescheduling_interrupts 1965 ± 7% -7.7% 1815 ± 3% interrupts.CPU2.RES:Rescheduling_interrupts 4.43 ± 7% -19.6% 3.56 ± 15% sched_debug.cfs_rq:/.nr_spread_over.avg 32.42 ± 12% -41.5% 18.96 ± 32% sched_debug.cfs_rq:/.nr_spread_over.max 10.63 ± 12% -39.0% 6.48 ± 25% sched_debug.cfs_rq:/.nr_spread_over.stddev 5397479 -3.2% 5222071 proc-vmstat.numa_hit 5397479 -3.2% 5222071 proc-vmstat.numa_local 1.289e+09 -3.5% 1.243e+09 proc-vmstat.pgalloc_normal 2902685 -2.9% 2817148 proc-vmstat.pgfault 1.289e+09 -3.5% 1.243e+09 proc-vmstat.pgfree 2511246 -3.5% 2422977 proc-vmstat.thp_fault_alloc 88.56 -88.6 0.00 perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 86.61 -86.6 0.00 perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault 90.95 -0.4 90.57 perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault 2.48 ± 3% -0.2 2.28 ± 2% perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.37 ± 3% -0.2 2.17 ± 2% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap 2.52 ± 3% -0.2 2.31 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.munmap 2.52 ± 3% -0.2 2.31 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 2.52 ± 3% -0.2 2.31 ± 2% perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 2.52 ± 3% -0.2 2.31 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 2.46 ± 3% -0.2 2.25 ± 2% perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 2.52 ± 3% -0.2 2.32 ± 2% perf-profile.calltrace.cycles-pp.munmap 2.42 ± 3% -0.2 2.22 ± 2% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 2.42 ± 3% -0.2 2.22 ± 2% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap 2.25 ± 3% -0.2 2.04 ± 3% perf-profile.calltrace.cycles-pp.__free_pages_ok.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region 3.34 -0.1 3.22 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault 3.36 -0.1 3.25 perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault 3.35 -0.1 3.24 perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 0.61 ± 3% +0.2 0.78 ± 2% perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 95.16 +0.3 95.45 perf-profile.calltrace.cycles-pp.page_fault 94.98 +0.3 95.28 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault 95.12 +0.3 95.42 perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 94.95 +0.3 95.26 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 94.89 +0.3 95.22 perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 88.69 -88.7 0.00 perf-profile.children.cycles-pp.clear_subpage 87.05 -86.9 0.12 ± 9% perf-profile.children.cycles-pp.clear_page_erms 0.88 ± 3% -0.8 0.10 ± 15% perf-profile.children.cycles-pp.rcu_all_qs 2.85 ± 3% -0.2 2.63 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 2.85 ± 3% -0.2 2.63 ± 3% perf-profile.children.cycles-pp.do_syscall_64 2.49 ± 3% -0.2 2.28 ± 2% perf-profile.children.cycles-pp.__do_munmap 2.43 ± 3% -0.2 2.22 ± 2% perf-profile.children.cycles-pp.tlb_finish_mmu 2.52 ± 3% -0.2 2.31 ± 2% perf-profile.children.cycles-pp.__x64_sys_munmap 2.46 ± 3% -0.2 2.26 ± 2% perf-profile.children.cycles-pp.unmap_region 2.43 ± 3% -0.2 2.22 ± 2% perf-profile.children.cycles-pp.tlb_flush_mmu 2.38 ± 3% -0.2 2.18 ± 2% perf-profile.children.cycles-pp.release_pages 2.52 ± 3% -0.2 2.32 ± 2% perf-profile.children.cycles-pp.munmap 2.52 ± 3% -0.2 2.31 ± 2% perf-profile.children.cycles-pp.__vm_munmap 2.25 ± 3% -0.2 2.05 ± 3% perf-profile.children.cycles-pp.__free_pages_ok 3.58 -0.1 3.43 perf-profile.children.cycles-pp.__alloc_pages_nodemask 3.55 -0.1 3.41 perf-profile.children.cycles-pp.get_page_from_freelist 0.95 ± 4% -0.1 0.84 ± 2% perf-profile.children.cycles-pp._cond_resched 3.37 -0.1 3.27 perf-profile.children.cycles-pp.alloc_pages_vma 0.68 -0.1 0.59 ± 2% perf-profile.children.cycles-pp.apic_timer_interrupt 0.33 ± 3% -0.1 0.27 ± 8% perf-profile.children.cycles-pp.hrtimer_interrupt 0.44 ± 2% -0.1 0.37 ± 3% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.26 ± 3% -0.1 0.20 ± 9% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.19 ± 3% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.tick_sched_timer 0.16 ± 5% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.tick_sched_handle 0.22 ± 5% -0.0 0.19 ± 6% perf-profile.children.cycles-pp.pte_alloc_one 0.15 ± 5% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.update_process_times 95.14 +0.3 95.44 perf-profile.children.cycles-pp.do_page_fault 95.01 +0.3 95.31 perf-profile.children.cycles-pp.handle_mm_fault 95.19 +0.3 95.49 perf-profile.children.cycles-pp.page_fault 94.98 +0.3 95.29 perf-profile.children.cycles-pp.__handle_mm_fault 94.89 +0.3 95.22 perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page 91.02 +0.5 91.52 perf-profile.children.cycles-pp.clear_huge_page 86.50 -86.4 0.12 ± 9% perf-profile.self.cycles-pp.clear_page_erms 0.57 ± 4% -0.5 0.08 ± 13% perf-profile.self.cycles-pp.rcu_all_qs 1.26 ± 4% -0.5 0.78 ± 3% perf-profile.self.cycles-pp.___might_sleep 2.16 ± 3% -0.2 1.96 ± 3% perf-profile.self.cycles-pp.__free_pages_ok 2.09 -0.1 2.00 perf-profile.self.cycles-pp.get_page_from_freelist 0.36 ± 4% +0.1 0.50 ± 4% perf-profile.self.cycles-pp._cond_resched 0.11 ± 11% +89.6 89.74 perf-profile.self.cycles-pp.clear_huge_page *************************************************************************************************** lkp-ivb-d02: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/lkp-ivb-d02/page_fault1/will-it-scale/0x21 commit: 8632e9b564 (" hyperv-fixes for 5.7-rc1") 9266baf8ee ("mm: Optimized hugepage zeroing & copying from user") 8632e9b5645bbc23 9266baf8eea588a78e5a0aa4ccd ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:4 2% 0:4 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 2061889 -5.3% 1951865 will-it-scale.per_process_ops 2171644 -5.7% 2048044 will-it-scale.per_thread_ops 0.23 +2.1% 0.23 will-it-scale.scalability 1232399 -5.2% 1168571 will-it-scale.time.minor_page_faults 16.41 -6.3% 15.38 will-it-scale.time.user_time 57433 -6.5% 53713 will-it-scale.time.voluntary_context_switches 32371162 -5.1% 30719792 will-it-scale.workload 1138201 ± 16% -10.8% 1015835 ± 16% cpuidle.C1E.usage 532.00 ± 11% -22.6% 412.00 ± 20% slabinfo.skbuff_head_cache.num_objs 24570 +2.8% 25263 interrupts.CAL:Function_call_interrupts 1823 ± 20% +26.7% 2309 ± 11% interrupts.CPU0.RES:Rescheduling_interrupts 28931 ± 11% -15.5% 24434 ± 14% softirqs.CPU1.SCHED 14949 ± 11% +17.2% 17527 ± 6% softirqs.CPU2.RCU 14575349 -4.8% 13875471 proc-vmstat.numa_hit 14575349 -4.8% 13875471 proc-vmstat.numa_local 1.228e+09 -5.0% 1.167e+09 proc-vmstat.pgalloc_normal 12186704 -4.7% 11609076 proc-vmstat.pgfault 1.228e+09 -5.0% 1.167e+09 proc-vmstat.pgfree 2374618 -5.0% 2256894 proc-vmstat.thp_fault_alloc 0.00 +2e+11% 2041 ± 82% sched_debug.cfs_rq:/.MIN_vruntime.avg 0.00 +7.5e+11% 7472 ± 94% sched_debug.cfs_rq:/.MIN_vruntime.max 266202 ± 2% +16.4% 309784 ± 5% sched_debug.cfs_rq:/.load.avg 0.00 +2e+11% 2041 ± 82% sched_debug.cfs_rq:/.max_vruntime.avg 0.00 +7.5e+11% 7472 ± 94% sched_debug.cfs_rq:/.max_vruntime.max 155407 ± 7% -19.4% 125275 ± 15% sched_debug.cfs_rq:/.min_vruntime.min 58058 ± 16% +28.5% 74586 ± 10% sched_debug.cfs_rq:/.min_vruntime.stddev 4.17 ± 36% +46.0% 6.08 ± 17% sched_debug.cfs_rq:/.nr_spread_over.max 22843 ±100% +333.2% 98964 ± 51% sched_debug.cfs_rq:/.spread0.max 58060 ± 16% +28.5% 74587 ± 10% sched_debug.cfs_rq:/.spread0.stddev 58.60 ± 8% -58.6 0.00 perf-profile.calltrace.cycles-pp.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 57.50 ± 8% -57.5 0.00 perf-profile.calltrace.cycles-pp.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault 0.67 ± 4% -0.1 0.59 ± 6% perf-profile.calltrace.cycles-pp.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.handle_pte_fault 0.65 ± 3% -0.1 0.57 ± 7% perf-profile.calltrace.cycles-pp.clear_page_erms.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma 58.68 ± 8% -58.7 0.00 perf-profile.children.cycles-pp.clear_subpage 58.41 ± 8% -57.7 0.69 ± 7% perf-profile.children.cycles-pp.clear_page_erms 0.08 ± 8% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.free_pages_and_swap_cache 0.10 ± 11% +0.0 0.13 ± 3% perf-profile.children.cycles-pp._raw_spin_lock 0.00 +0.1 0.09 ± 29% perf-profile.children.cycles-pp.forkshell 0.23 ± 19% +0.3 0.50 ± 5% perf-profile.children.cycles-pp.rcu_all_qs 57.92 ± 8% -57.2 0.69 ± 6% perf-profile.self.cycles-pp.clear_page_erms 0.08 ± 5% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.free_pages_and_swap_cache 0.18 ± 13% +0.1 0.26 ± 7% perf-profile.self.cycles-pp.rcu_all_qs 0.72 ± 13% +60.5 61.22 ± 6% perf-profile.self.cycles-pp.clear_huge_page Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

8 months, 3 weeks

1
0
0 / 0

a530431919: will-it-scale.per_process_ops 2097.3% improvement

by kernel test robot

Greeting, FYI, we noticed a 2097.3% improvement of will-it-scale.per_process_ops due to commit: commit: a5304319194e6f6fdae62fca227dd9c588dc155f ("new overcommit patch") git://bee.sh.intel.com/git/feng/linux.git master in testcase: will-it-scale on test machine: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory with following parameters: nr_task: 100% mode: process test: mmap2 cpufreq_governor: performance ucode: 0x16 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale In addition to that, the commit also has significant impact on the following tests: Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-hsw-4ex1/mmap2/will-it-scale/0x16 commit: 591e8cfa8d ("percpu counter init debug") a530431919 ("new overcommit patch") 591e8cfa8d9bfbd7 a5304319194e6f6fdae62fca227 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 100% 4:5 dmesg.WARNING:at_ip_perf_event_mmap_output/0x :4 125% 5:5 kmsg.mm_compute_batch()]:vm_committed_as_batch= :4 125% 5:5 kmsg.mmap_init()]:totalram_pages :4 25% 1:5 kmsg.overcommit_policy_handler()]:enter :4 25% 1:5 kmsg.overcommit_policy_handler()]:ret=#,sysctl_overcommit_memory= %stddev %change %stddev \ | \ 3914 +2097.3% 86000 will-it-scale.per_process_ops 563689 +2097.0% 12384186 will-it-scale.workload 366.00 ± 11% +353.9% 1661 ±138% cpuidle.POLL.usage 14648666 -69.8% 4427855 ± 18% meminfo.Committed_AS 58393 ± 2% -13.3% 50624 ± 3% meminfo.Shmem 230557 ± 4% -16.0% 193663 ± 6% numa-numastat.node0.local_node 234056 -10.5% 209536 ± 7% numa-numastat.node0.numa_hit 0.00 ± 27% -0.0 0.00 ± 95% mpstat.cpu.all.soft% 98.61 -18.7 79.93 mpstat.cpu.all.sys% 0.88 +18.6 19.50 mpstat.cpu.all.usr% 97.00 -18.6% 79.00 vmstat.cpu.sy 1.00 +1800.0% 19.00 vmstat.cpu.us 1736 +17.4% 2038 vmstat.system.cs 93861 -2.7% 91354 proc-vmstat.nr_active_anon 84501 -0.7% 83922 proc-vmstat.nr_anon_pages 14581 ± 2% -13.1% 12668 ± 3% proc-vmstat.nr_shmem 93861 -2.7% 91354 proc-vmstat.nr_zone_active_anon 14148 ± 4% -17.4% 11686 ± 6% proc-vmstat.pgactivate 192.98 ± 4% +14.7% 221.40 ± 5% sched_debug.cfs_rq:/.exec_clock.stddev 860540 -13.4% 744887 ± 3% sched_debug.cpu.avg_idle.avg 160835 ± 11% +43.0% 230044 ± 30% sched_debug.cpu.avg_idle.stddev 6893 ± 3% -14.3% 5905 ± 5% sched_debug.cpu.curr->pid.max 3903 ± 6% -9.4% 3538 ± 3% sched_debug.cpu.nr_switches.stddev 27022 ± 28% -26.8% 19770 ± 10% sched_debug.cpu.sched_count.max 625.54 -12.8% 545.47 ± 3% sched_debug.cpu.sched_count.min 809.80 +19.9% 970.79 sched_debug.cpu.ttwu_count.avg 12351 ± 28% -54.5% 5616 ± 9% sched_debug.cpu.ttwu_local.max 1426 ± 14% -42.2% 824.77 ± 9% sched_debug.cpu.ttwu_local.stddev 2859 ± 62% -81.9% 517.80 ± 26% numa-vmstat.node0.nr_shmem 8638 ± 12% -36.3% 5503 ± 28% numa-vmstat.node0.nr_slab_reclaimable 17011 ± 9% -19.3% 13730 ± 8% numa-vmstat.node0.nr_slab_unreclaimable 525558 ± 10% -16.3% 439768 ± 10% numa-vmstat.node0.numa_hit 468320 ± 14% -24.8% 352170 ± 13% numa-vmstat.node0.numa_local 57237 ± 43% +53.0% 87597 ± 10% numa-vmstat.node0.numa_other 390.75 ± 44% +469.8% 2226 ± 50% numa-vmstat.node1.nr_inactive_anon 537.00 ± 21% +351.2% 2423 ± 45% numa-vmstat.node1.nr_shmem 3913 ± 17% +48.6% 5816 ± 17% numa-vmstat.node1.nr_slab_reclaimable 10522 ± 8% +21.9% 12823 ± 7% numa-vmstat.node1.nr_slab_unreclaimable 390.75 ± 44% +469.8% 2226 ± 50% numa-vmstat.node1.nr_zone_inactive_anon 1180 ± 5% +11.4% 1314 ± 8% numa-vmstat.node2.nr_page_table_pages 34556 ± 12% -36.3% 22017 ± 28% numa-meminfo.node0.KReclaimable 34556 ± 12% -36.3% 22017 ± 28% numa-meminfo.node0.SReclaimable 68046 ± 9% -19.3% 54925 ± 8% numa-meminfo.node0.SUnreclaim 11438 ± 62% -81.9% 2072 ± 26% numa-meminfo.node0.Shmem 102603 ± 9% -25.0% 76943 ± 14% numa-meminfo.node0.Slab 1653 ± 40% +447.3% 9049 ± 49% numa-meminfo.node1.Inactive 1564 ± 44% +469.3% 8907 ± 50% numa-meminfo.node1.Inactive(anon) 15652 ± 17% +48.6% 23267 ± 17% numa-meminfo.node1.KReclaimable 15652 ± 17% +48.6% 23267 ± 17% numa-meminfo.node1.SReclaimable 42090 ± 8% +21.9% 51296 ± 7% numa-meminfo.node1.SUnreclaim 2149 ± 21% +350.9% 9692 ± 45% numa-meminfo.node1.Shmem 57744 ± 10% +29.1% 74563 ± 9% numa-meminfo.node1.Slab 4721 ± 5% +11.4% 5258 ± 8% numa-meminfo.node2.PageTables 24916677 ± 58% -82.6% 4341483 ± 71% syscalls.sys_mmap.max 310375 -97.9% 6381 syscalls.sys_mmap.med 6654 ± 2% -36.5% 4225 syscalls.sys_mmap.min 22376668 ± 14% +9e+06 31377874 syscalls.sys_mmap.noise.100% 6.697e+08 ± 5% -6.1e+08 63646115 syscalls.sys_mmap.noise.2% 28502502 ± 11% +1.1e+07 39699989 syscalls.sys_mmap.noise.25% 5.019e+08 ± 8% -4.4e+08 62729306 syscalls.sys_mmap.noise.5% 25892627 ± 11% +6.4e+06 32291830 syscalls.sys_mmap.noise.50% 24968119 ± 12% +6.6e+06 31591907 syscalls.sys_mmap.noise.75% 315822 -95.8% 13123 syscalls.sys_munmap.med 29554 -71.9% 8313 syscalls.sys_munmap.min 973315 ± 58% +5.9e+07 59986702 ± 3% syscalls.sys_munmap.noise.100% 6.307e+08 ± 5% -5.2e+08 1.126e+08 ± 12% syscalls.sys_munmap.noise.2% 1383937 ± 48% +6e+07 61689601 ± 4% syscalls.sys_munmap.noise.25% 4.596e+08 ± 8% -3.5e+08 1.075e+08 ± 16% syscalls.sys_munmap.noise.5% 1133158 ± 55% +5.9e+07 60240681 ± 3% syscalls.sys_munmap.noise.50% 1082700 ± 56% +5.9e+07 60107504 ± 3% syscalls.sys_munmap.noise.75% 2.494e+09 -100.0% 178358 ± 13% syscalls.sys_open.max 12050 +39.4% 16799 ± 2% syscalls.sys_open.med 7854 +40.2% 11015 ± 8% syscalls.sys_open.min 1.193e+12 ± 9% -1.2e+12 5.255e+08 ± 20% syscalls.sys_open.noise.100% 1.193e+12 ± 9% -1.2e+12 7.052e+08 ± 15% syscalls.sys_open.noise.2% 1.193e+12 ± 9% -1.2e+12 6.652e+08 ± 16% syscalls.sys_open.noise.25% 1.193e+12 ± 9% -1.2e+12 7.031e+08 ± 15% syscalls.sys_open.noise.5% 1.193e+12 ± 9% -1.2e+12 6.046e+08 ± 18% syscalls.sys_open.noise.50% 1.193e+12 ± 9% -1.2e+12 5.414e+08 ± 20% syscalls.sys_open.noise.75% 2.77e+09 -99.1% 24131810 ± 81% syscalls.sys_read.max 6.986e+11 ± 11% -6.7e+11 2.758e+10 ±132% syscalls.sys_read.noise.100% 6.986e+11 ± 11% -6.7e+11 2.761e+10 ±131% syscalls.sys_read.noise.2% 6.986e+11 ± 11% -6.7e+11 2.76e+10 ±131% syscalls.sys_read.noise.25% 6.986e+11 ± 11% -6.7e+11 2.761e+10 ±131% syscalls.sys_read.noise.5% 6.986e+11 ± 11% -6.7e+11 2.759e+10 ±131% syscalls.sys_read.noise.50% 6.986e+11 ± 11% -6.7e+11 2.758e+10 ±131% syscalls.sys_read.noise.75% 483.75 ±105% -66.3% 163.00 ± 12% interrupts.58:PCI-MSI.1572877-edge.eth0-TxRx-13 3327 -32.7% 2240 ± 33% interrupts.CPU0.NMI:Non-maskable_interrupts 3327 -32.7% 2240 ± 33% interrupts.CPU0.PMI:Performance_monitoring_interrupts 3245 ± 3% -29.8% 2279 ± 35% interrupts.CPU101.NMI:Non-maskable_interrupts 3245 ± 3% -29.8% 2279 ± 35% interrupts.CPU101.PMI:Performance_monitoring_interrupts 2857 ± 24% -20.1% 2282 ± 35% interrupts.CPU102.NMI:Non-maskable_interrupts 2857 ± 24% -20.1% 2282 ± 35% interrupts.CPU102.PMI:Performance_monitoring_interrupts 2916 ± 24% -21.5% 2288 ± 35% interrupts.CPU103.NMI:Non-maskable_interrupts 2916 ± 24% -21.5% 2288 ± 35% interrupts.CPU103.PMI:Performance_monitoring_interrupts 2906 ± 24% -32.7% 1957 ± 34% interrupts.CPU104.NMI:Non-maskable_interrupts 2906 ± 24% -32.7% 1957 ± 34% interrupts.CPU104.PMI:Performance_monitoring_interrupts 14.75 ± 60% +720.3% 121.00 ± 78% interrupts.CPU104.RES:Rescheduling_interrupts 2808 ± 24% -18.7% 2284 ± 35% interrupts.CPU105.NMI:Non-maskable_interrupts 2808 ± 24% -18.7% 2284 ± 35% interrupts.CPU105.PMI:Performance_monitoring_interrupts 2909 ± 24% -21.9% 2273 ± 34% interrupts.CPU107.NMI:Non-maskable_interrupts 2909 ± 24% -21.9% 2273 ± 34% interrupts.CPU107.PMI:Performance_monitoring_interrupts 2468 ± 26% +156.7% 6336 ± 34% interrupts.CPU108.RES:Rescheduling_interrupts 3327 -41.1% 1958 ± 33% interrupts.CPU109.NMI:Non-maskable_interrupts 3327 -41.1% 1958 ± 33% interrupts.CPU109.PMI:Performance_monitoring_interrupts 2314 ± 30% -29.8% 1625 interrupts.CPU110.NMI:Non-maskable_interrupts 2314 ± 30% -29.8% 1625 interrupts.CPU110.PMI:Performance_monitoring_interrupts 2506 ± 31% -35.2% 1624 interrupts.CPU111.NMI:Non-maskable_interrupts 2506 ± 31% -35.2% 1624 interrupts.CPU111.PMI:Performance_monitoring_interrupts 2499 ± 33% -21.7% 1958 ± 33% interrupts.CPU112.NMI:Non-maskable_interrupts 2499 ± 33% -21.7% 1958 ± 33% interrupts.CPU112.PMI:Performance_monitoring_interrupts 2913 ± 24% -22.0% 2272 ± 34% interrupts.CPU113.NMI:Non-maskable_interrupts 2913 ± 24% -22.0% 2272 ± 34% interrupts.CPU113.PMI:Performance_monitoring_interrupts 2515 ± 32% -22.5% 1950 ± 34% interrupts.CPU114.NMI:Non-maskable_interrupts 2515 ± 32% -22.5% 1950 ± 34% interrupts.CPU114.PMI:Performance_monitoring_interrupts 2500 ± 33% -22.1% 1948 ± 33% interrupts.CPU117.NMI:Non-maskable_interrupts 2500 ± 33% -22.1% 1948 ± 33% interrupts.CPU117.PMI:Performance_monitoring_interrupts 3329 -31.2% 2290 ± 35% interrupts.CPU119.NMI:Non-maskable_interrupts 3329 -31.2% 2290 ± 35% interrupts.CPU119.PMI:Performance_monitoring_interrupts 16.50 ± 78% +2280.6% 392.80 ±134% interrupts.CPU119.RES:Rescheduling_interrupts 2882 ± 24% -32.2% 1955 ± 33% interrupts.CPU120.NMI:Non-maskable_interrupts 2882 ± 24% -32.2% 1955 ± 33% interrupts.CPU120.PMI:Performance_monitoring_interrupts 2886 ± 24% -43.8% 1622 interrupts.CPU122.NMI:Non-maskable_interrupts 2886 ± 24% -43.8% 1622 interrupts.CPU122.PMI:Performance_monitoring_interrupts 21.75 ± 81% +168.5% 58.40 ± 57% interrupts.CPU122.RES:Rescheduling_interrupts 2500 ± 33% -34.6% 1636 interrupts.CPU123.NMI:Non-maskable_interrupts 2500 ± 33% -34.6% 1636 interrupts.CPU123.PMI:Performance_monitoring_interrupts 19.50 ±111% +223.1% 63.00 ± 76% interrupts.CPU123.RES:Rescheduling_interrupts 2913 ± 24% -32.9% 1956 ± 33% interrupts.CPU125.NMI:Non-maskable_interrupts 2913 ± 24% -32.9% 1956 ± 33% interrupts.CPU125.PMI:Performance_monitoring_interrupts 2914 ± 24% -43.8% 1636 interrupts.CPU126.NMI:Non-maskable_interrupts 2914 ± 24% -43.8% 1636 interrupts.CPU126.PMI:Performance_monitoring_interrupts 1962 ± 17% +307.7% 8003 ± 29% interrupts.CPU126.RES:Rescheduling_interrupts 2500 ± 33% -21.4% 1964 ± 33% interrupts.CPU127.NMI:Non-maskable_interrupts 2500 ± 33% -21.4% 1964 ± 33% interrupts.CPU127.PMI:Performance_monitoring_interrupts 1207 ± 11% +171.6% 3279 ± 16% interrupts.CPU127.RES:Rescheduling_interrupts 2893 ± 24% -32.1% 1963 ± 33% interrupts.CPU128.NMI:Non-maskable_interrupts 2893 ± 24% -32.1% 1963 ± 33% interrupts.CPU128.PMI:Performance_monitoring_interrupts 2915 ± 24% -43.8% 1638 interrupts.CPU129.NMI:Non-maskable_interrupts 2915 ± 24% -43.8% 1638 interrupts.CPU129.PMI:Performance_monitoring_interrupts 483.75 ±105% -66.3% 163.00 ± 12% interrupts.CPU13.58:PCI-MSI.1572877-edge.eth0-TxRx-13 2858 ± 24% -43.1% 1627 interrupts.CPU130.NMI:Non-maskable_interrupts 2858 ± 24% -43.1% 1627 interrupts.CPU130.PMI:Performance_monitoring_interrupts 2915 ± 24% -44.0% 1633 interrupts.CPU131.NMI:Non-maskable_interrupts 2915 ± 24% -44.0% 1633 interrupts.CPU131.PMI:Performance_monitoring_interrupts 2481 ± 32% -34.2% 1633 interrupts.CPU132.NMI:Non-maskable_interrupts 2481 ± 32% -34.2% 1633 interrupts.CPU132.PMI:Performance_monitoring_interrupts 2918 ± 24% -33.0% 1956 ± 33% interrupts.CPU133.NMI:Non-maskable_interrupts 2918 ± 24% -33.0% 1956 ± 33% interrupts.CPU133.PMI:Performance_monitoring_interrupts 2501 ± 33% -21.5% 1963 ± 33% interrupts.CPU134.NMI:Non-maskable_interrupts 2501 ± 33% -21.5% 1963 ± 33% interrupts.CPU134.PMI:Performance_monitoring_interrupts 2495 ± 33% -21.4% 1960 ± 33% interrupts.CPU135.NMI:Non-maskable_interrupts 2495 ± 33% -21.4% 1960 ± 33% interrupts.CPU135.PMI:Performance_monitoring_interrupts 2501 ± 33% -21.7% 1958 ± 33% interrupts.CPU136.NMI:Non-maskable_interrupts 2501 ± 33% -21.7% 1958 ± 33% interrupts.CPU136.PMI:Performance_monitoring_interrupts 2916 ± 24% -44.0% 1631 interrupts.CPU137.NMI:Non-maskable_interrupts 2916 ± 24% -44.0% 1631 interrupts.CPU137.PMI:Performance_monitoring_interrupts 2493 ± 32% -34.4% 1636 interrupts.CPU138.NMI:Non-maskable_interrupts 2493 ± 32% -34.4% 1636 interrupts.CPU138.PMI:Performance_monitoring_interrupts 2913 ± 24% -43.7% 1639 interrupts.CPU139.NMI:Non-maskable_interrupts 2913 ± 24% -43.7% 1639 interrupts.CPU139.PMI:Performance_monitoring_interrupts 2493 ± 33% -34.4% 1635 interrupts.CPU140.NMI:Non-maskable_interrupts 2493 ± 33% -34.4% 1635 interrupts.CPU140.PMI:Performance_monitoring_interrupts 3293 ± 2% -30.4% 2292 ± 35% interrupts.CPU141.NMI:Non-maskable_interrupts 3293 ± 2% -30.4% 2292 ± 35% interrupts.CPU141.PMI:Performance_monitoring_interrupts 2913 ± 24% -43.9% 1635 interrupts.CPU142.NMI:Non-maskable_interrupts 2913 ± 24% -43.9% 1635 interrupts.CPU142.PMI:Performance_monitoring_interrupts 2887 ± 24% -43.6% 1629 interrupts.CPU143.NMI:Non-maskable_interrupts 2887 ± 24% -43.6% 1629 interrupts.CPU143.PMI:Performance_monitoring_interrupts 3300 -24.0% 2508 ± 28% interrupts.CPU15.NMI:Non-maskable_interrupts 3300 -24.0% 2508 ± 28% interrupts.CPU15.PMI:Performance_monitoring_interrupts 2497 ± 32% -8.2% 2294 ± 34% interrupts.CPU2.NMI:Non-maskable_interrupts 2497 ± 32% -8.2% 2294 ± 34% interrupts.CPU2.PMI:Performance_monitoring_interrupts 318.25 ± 13% +148.1% 789.60 ± 83% interrupts.CPU22.RES:Rescheduling_interrupts 2475 ± 32% -16.0% 2079 ± 26% interrupts.CPU24.NMI:Non-maskable_interrupts 2475 ± 32% -16.0% 2079 ± 26% interrupts.CPU24.PMI:Performance_monitoring_interrupts 283.50 +18.2% 335.20 ± 8% interrupts.CPU25.RES:Rescheduling_interrupts 288.50 ± 3% +26.7% 365.60 ± 22% interrupts.CPU30.RES:Rescheduling_interrupts 3297 -23.1% 2533 ± 28% interrupts.CPU31.NMI:Non-maskable_interrupts 3297 -23.1% 2533 ± 28% interrupts.CPU31.PMI:Performance_monitoring_interrupts 288.75 ± 3% +20.5% 347.80 ± 8% interrupts.CPU32.RES:Rescheduling_interrupts 3293 -24.6% 2483 ± 29% interrupts.CPU36.NMI:Non-maskable_interrupts 3293 -24.6% 2483 ± 29% interrupts.CPU36.PMI:Performance_monitoring_interrupts 873.00 ± 20% +110.4% 1837 ± 72% interrupts.CPU37.RES:Rescheduling_interrupts 289.00 ± 3% +14.5% 330.80 ± 8% interrupts.CPU43.RES:Rescheduling_interrupts 281.00 +15.4% 324.40 ± 8% interrupts.CPU45.RES:Rescheduling_interrupts 280.00 +49.9% 419.80 ± 43% interrupts.CPU47.RES:Rescheduling_interrupts 2905 ± 24% -21.3% 2285 ± 34% interrupts.CPU5.NMI:Non-maskable_interrupts 2905 ± 24% -21.3% 2285 ± 34% interrupts.CPU5.PMI:Performance_monitoring_interrupts 2015 ± 29% +51.0% 3044 ± 11% interrupts.CPU50.NMI:Non-maskable_interrupts 2015 ± 29% +51.0% 3044 ± 11% interrupts.CPU50.PMI:Performance_monitoring_interrupts 299.00 ± 11% +30.2% 389.20 ± 22% interrupts.CPU50.RES:Rescheduling_interrupts 273.25 +33.6% 365.00 ± 17% interrupts.CPU51.RES:Rescheduling_interrupts 276.75 ± 2% +25.1% 346.20 ± 9% interrupts.CPU52.RES:Rescheduling_interrupts 273.25 +14.6% 313.20 ± 8% interrupts.CPU53.RES:Rescheduling_interrupts 940.00 ± 15% +102.9% 1907 ± 45% interrupts.CPU54.RES:Rescheduling_interrupts 940.25 ± 44% +100.2% 1882 ± 33% interrupts.CPU55.RES:Rescheduling_interrupts 284.50 ± 18% +76.8% 503.00 ± 17% interrupts.CPU57.RES:Rescheduling_interrupts 2486 ± 32% -21.3% 1957 ± 32% interrupts.CPU6.NMI:Non-maskable_interrupts 2486 ± 32% -21.3% 1957 ± 32% interrupts.CPU6.PMI:Performance_monitoring_interrupts 332.00 ± 14% +77.2% 588.40 ± 36% interrupts.CPU6.RES:Rescheduling_interrupts 74.50 ± 17% +358.3% 341.40 ± 12% interrupts.CPU60.RES:Rescheduling_interrupts 41.00 ± 8% +728.8% 339.80 ± 14% interrupts.CPU61.RES:Rescheduling_interrupts 37.25 ± 62% +762.3% 321.20 ± 11% interrupts.CPU62.RES:Rescheduling_interrupts 19.50 ± 38% +1468.2% 305.80 ± 13% interrupts.CPU63.RES:Rescheduling_interrupts 25.00 ± 44% +1076.0% 294.00 ± 4% interrupts.CPU64.RES:Rescheduling_interrupts 12.25 ± 65% +2355.5% 300.80 ± 10% interrupts.CPU65.RES:Rescheduling_interrupts 34.00 ±100% +733.5% 283.40 ± 4% interrupts.CPU66.RES:Rescheduling_interrupts 13.25 ± 33% +2730.2% 375.00 ± 35% interrupts.CPU67.RES:Rescheduling_interrupts 16.50 ± 58% +1713.3% 299.20 ± 5% interrupts.CPU68.RES:Rescheduling_interrupts 8.25 ± 5% +3323.0% 282.40 ± 7% interrupts.CPU69.RES:Rescheduling_interrupts 2899 ± 24% -35.5% 1869 ± 24% interrupts.CPU7.NMI:Non-maskable_interrupts 2899 ± 24% -35.5% 1869 ± 24% interrupts.CPU7.PMI:Performance_monitoring_interrupts 60.00 ±130% +450.3% 330.20 ± 10% interrupts.CPU70.RES:Rescheduling_interrupts 11.00 ± 28% +3301.8% 374.20 ± 42% interrupts.CPU71.RES:Rescheduling_interrupts 1412 ± 15% +90.0% 2684 ± 36% interrupts.CPU73.RES:Rescheduling_interrupts 3334 -31.7% 2277 ± 35% interrupts.CPU80.NMI:Non-maskable_interrupts 3334 -31.7% 2277 ± 35% interrupts.CPU80.PMI:Performance_monitoring_interrupts 24.00 ± 75% +420.8% 125.00 ± 75% interrupts.CPU80.RES:Rescheduling_interrupts 15.75 ± 58% +326.7% 67.20 ± 51% interrupts.CPU87.RES:Rescheduling_interrupts 8.00 ± 38% +1342.5% 115.40 ±109% interrupts.CPU88.RES:Rescheduling_interrupts 2914 ± 24% -21.3% 2293 ± 34% interrupts.CPU90.NMI:Non-maskable_interrupts 2914 ± 24% -21.3% 2293 ± 34% interrupts.CPU90.PMI:Performance_monitoring_interrupts 2164 ± 14% +173.3% 5915 ± 15% interrupts.CPU90.RES:Rescheduling_interrupts 1632 ± 16% +93.0% 3150 ± 51% interrupts.CPU91.RES:Rescheduling_interrupts 2946 ± 19% -22.4% 2285 ± 35% interrupts.CPU97.NMI:Non-maskable_interrupts 2946 ± 19% -22.4% 2285 ± 35% interrupts.CPU97.PMI:Performance_monitoring_interrupts 373529 ± 4% -7.8% 344395 ± 6% interrupts.NMI:Non-maskable_interrupts 373529 ± 4% -7.8% 344395 ± 6% interrupts.PMI:Performance_monitoring_interrupts 51237 +88.0% 96315 ± 2% interrupts.RES:Rescheduling_interrupts 5287 ± 15% +282.2% 20206 ± 6% softirqs.CPU0.RCU 138547 -22.8% 107022 ± 10% softirqs.CPU0.TIMER 6630 ± 27% +253.6% 23446 ± 4% softirqs.CPU1.RCU 140113 ± 3% -26.7% 102747 ± 9% softirqs.CPU1.TIMER 4969 ± 6% +359.8% 22851 ± 5% softirqs.CPU10.RCU 138404 -28.2% 99420 ± 7% softirqs.CPU10.TIMER 4325 ± 3% +447.2% 23667 ± 3% softirqs.CPU100.RCU 136595 -29.8% 95856 ± 4% softirqs.CPU100.TIMER 4138 ± 4% +479.7% 23990 ± 6% softirqs.CPU101.RCU 136139 -30.1% 95124 ± 6% softirqs.CPU101.TIMER 4271 +478.6% 24718 ± 3% softirqs.CPU102.RCU 136243 -27.7% 98498 ± 8% softirqs.CPU102.TIMER 4849 ± 15% +387.9% 23658 ± 8% softirqs.CPU103.RCU 139376 ± 3% -32.1% 94681 ± 6% softirqs.CPU103.TIMER 4297 ± 2% +450.9% 23676 ± 5% softirqs.CPU104.RCU 136431 -29.6% 96072 ± 6% softirqs.CPU104.TIMER 4269 ± 2% +452.4% 23583 ± 6% softirqs.CPU105.RCU 136558 -31.0% 94184 ± 4% softirqs.CPU105.TIMER 4440 ± 7% +437.8% 23880 ± 5% softirqs.CPU106.RCU 142857 ± 8% -33.7% 94741 ± 5% softirqs.CPU106.TIMER 4463 ± 4% +429.6% 23639 ± 10% softirqs.CPU107.RCU 136709 -29.2% 96832 ± 6% softirqs.CPU107.TIMER 6331 ± 3% +233.7% 21125 ± 13% softirqs.CPU108.RCU 135662 -27.9% 97825 ± 5% softirqs.CPU108.TIMER 4925 ± 4% +318.2% 20598 ± 9% softirqs.CPU109.RCU 135795 -29.0% 96392 ± 5% softirqs.CPU109.TIMER 4848 +398.6% 24170 ± 4% softirqs.CPU11.RCU 138262 -27.6% 100033 ± 10% softirqs.CPU11.TIMER 4333 ± 3% +347.5% 19395 ± 11% softirqs.CPU110.RCU 135677 -30.2% 94669 ± 4% softirqs.CPU110.TIMER 4188 +360.3% 19279 ± 11% softirqs.CPU111.RCU 135991 -29.1% 96443 ± 5% softirqs.CPU111.TIMER 4137 +447.4% 22649 ± 2% softirqs.CPU112.RCU 135979 -31.8% 92804 ± 3% softirqs.CPU112.TIMER 4793 ± 20% +419.1% 24878 ± 3% softirqs.CPU113.RCU 138987 ± 3% -31.2% 95615 ± 3% softirqs.CPU113.TIMER 4160 +490.2% 24557 ± 3% softirqs.CPU114.RCU 136169 -28.9% 96809 ± 5% softirqs.CPU114.TIMER 4268 ± 2% +459.9% 23900 ± 3% softirqs.CPU115.RCU 136571 -30.6% 94823 ± 3% softirqs.CPU115.TIMER 4143 +508.2% 25200 ± 4% softirqs.CPU116.RCU 136010 -27.2% 98984 ± 6% softirqs.CPU116.TIMER 4079 +483.2% 23788 ± 3% softirqs.CPU117.RCU 135911 -30.4% 94604 ± 4% softirqs.CPU117.TIMER 4160 +463.9% 23459 ± 8% softirqs.CPU118.RCU 135993 -30.3% 94833 ± 4% softirqs.CPU118.TIMER 4141 +474.9% 23806 ± 8% softirqs.CPU119.RCU 136062 -30.0% 95278 ± 5% softirqs.CPU119.TIMER 4913 +390.4% 24095 ± 4% softirqs.CPU12.RCU 137946 -28.5% 98689 ± 6% softirqs.CPU12.TIMER 4172 +471.4% 23840 ± 7% softirqs.CPU120.RCU 136107 -30.1% 95107 ± 4% softirqs.CPU120.TIMER 4114 +455.3% 22846 ± 6% softirqs.CPU121.RCU 135816 -32.0% 92302 ± 2% softirqs.CPU121.TIMER 4268 ± 5% +432.9% 22746 ± 6% softirqs.CPU122.RCU 136460 -31.3% 93800 ± 4% softirqs.CPU122.TIMER 4078 +467.5% 23148 ± 8% softirqs.CPU123.RCU 135872 -30.6% 94243 ± 4% softirqs.CPU123.TIMER 4118 +452.5% 22753 ± 7% softirqs.CPU124.RCU 135803 -31.0% 93667 ± 3% softirqs.CPU124.TIMER 4185 ± 2% +447.4% 22911 ± 9% softirqs.CPU125.RCU 136541 -31.2% 93986 ± 3% softirqs.CPU125.TIMER 6053 ± 5% +190.0% 17553 ± 7% softirqs.CPU126.RCU 135833 -28.9% 96597 ± 4% softirqs.CPU126.TIMER 5025 ± 5% +239.7% 17069 ± 3% softirqs.CPU127.RCU 136541 -26.6% 100156 ± 7% softirqs.CPU127.TIMER 4214 ± 2% +420.6% 21941 ± 6% softirqs.CPU128.RCU 136031 -27.8% 98197 ± 6% softirqs.CPU128.TIMER 4195 ± 5% +414.5% 21585 ± 3% softirqs.CPU129.RCU 136489 -29.8% 95855 ± 4% softirqs.CPU129.TIMER 4960 ± 5% +377.9% 23706 ± 9% softirqs.CPU13.RCU 140323 -30.8% 97063 ± 4% softirqs.CPU13.TIMER 4431 ± 9% +393.7% 21875 ± 3% softirqs.CPU130.RCU 137060 -28.4% 98071 ± 3% softirqs.CPU130.TIMER 4198 ± 7% +452.1% 23180 ± 5% softirqs.CPU131.RCU 136486 -25.6% 101562 ± 8% softirqs.CPU131.TIMER 4046 ± 2% +462.4% 22757 ± 8% softirqs.CPU132.RCU 135873 -27.9% 98007 ± 5% softirqs.CPU132.TIMER 4057 ± 4% +419.1% 21059 ± 11% softirqs.CPU133.RCU 136031 -29.6% 95739 ± 3% softirqs.CPU133.TIMER 3910 +452.9% 21620 ± 5% softirqs.CPU134.RCU 135597 -28.9% 96351 ± 5% softirqs.CPU134.TIMER 4055 ± 2% +448.8% 22254 ± 5% softirqs.CPU135.RCU 136076 -28.8% 96819 ± 5% softirqs.CPU135.TIMER 3966 +451.1% 21856 ± 7% softirqs.CPU136.RCU 135683 -29.1% 96182 ± 4% softirqs.CPU136.TIMER 3992 +431.1% 21203 ± 9% softirqs.CPU137.RCU 136026 -30.5% 94595 ± 3% softirqs.CPU137.TIMER 4080 ± 4% +430.0% 21624 ± 4% softirqs.CPU138.RCU 136484 -29.2% 96670 ± 6% softirqs.CPU138.TIMER 4494 ± 18% +389.9% 22017 ± 5% softirqs.CPU139.RCU 138770 ± 3% -29.4% 97991 ± 6% softirqs.CPU139.TIMER 4653 +361.8% 21490 ± 18% softirqs.CPU14.RCU 3993 ± 2% +451.3% 22013 softirqs.CPU140.RCU 136073 -28.2% 97707 ± 6% softirqs.CPU140.TIMER 4026 +444.0% 21904 ± 4% softirqs.CPU141.RCU 136385 -29.9% 95587 ± 4% softirqs.CPU141.TIMER 3985 ± 5% +448.3% 21851 ± 2% softirqs.CPU142.RCU 136087 -27.7% 98450 ± 7% softirqs.CPU142.TIMER 4627 ± 14% +372.7% 21870 ± 5% softirqs.CPU143.RCU 136035 -28.9% 96772 ± 6% softirqs.CPU143.TIMER 4799 ± 2% +344.2% 21320 ± 17% softirqs.CPU15.RCU 138235 -29.3% 97740 ± 5% softirqs.CPU15.TIMER 4562 +311.8% 18789 ± 6% softirqs.CPU16.RCU 137587 -30.3% 95932 ± 4% softirqs.CPU16.TIMER 4525 +323.3% 19156 ± 7% softirqs.CPU17.RCU 137792 -28.7% 98217 ± 6% softirqs.CPU17.TIMER 5479 ± 15% +335.0% 23834 softirqs.CPU18.RCU 136367 -29.0% 96823 ± 6% softirqs.CPU18.TIMER 4958 ± 2% +391.1% 24350 ± 11% softirqs.CPU19.RCU 137655 -27.7% 99527 ± 5% softirqs.CPU19.TIMER 5274 ± 16% +297.5% 20963 ± 5% softirqs.CPU2.RCU 137577 -28.7% 98107 ± 5% softirqs.CPU2.TIMER 4725 +400.3% 23643 ± 6% softirqs.CPU20.RCU 137392 -30.2% 95849 ± 5% softirqs.CPU20.TIMER 4617 +436.6% 24776 ± 7% softirqs.CPU21.RCU 136300 -29.8% 95672 ± 5% softirqs.CPU21.TIMER 4764 +451.7% 26284 ± 8% softirqs.CPU22.RCU 137033 -27.6% 99238 ± 8% softirqs.CPU22.TIMER 6308 ± 43% +293.4% 24816 ± 12% softirqs.CPU23.RCU 140392 ± 3% -30.3% 97872 softirqs.CPU23.TIMER 4927 ± 11% +400.6% 24665 ± 6% softirqs.CPU24.RCU 139074 -30.0% 97390 ± 5% softirqs.CPU24.TIMER 7114 ± 37% +249.5% 24865 ± 3% softirqs.CPU25.RCU 141916 ± 3% -31.8% 96789 ± 4% softirqs.CPU25.TIMER 4626 ± 2% +409.3% 23559 ± 15% softirqs.CPU26.RCU 137006 -29.4% 96673 ± 4% softirqs.CPU26.TIMER 5054 ± 11% +396.7% 25104 ± 5% softirqs.CPU27.RCU 138652 -30.9% 95812 ± 4% softirqs.CPU27.TIMER 4680 +439.9% 25268 ± 4% softirqs.CPU28.RCU 137452 -29.7% 96599 ± 5% softirqs.CPU28.TIMER 5082 ± 13% +403.3% 25581 ± 7% softirqs.CPU29.RCU 138018 -30.3% 96185 ± 6% softirqs.CPU29.TIMER 4815 ± 2% +345.2% 21435 ± 2% softirqs.CPU3.RCU 137173 -27.6% 99381 ± 7% softirqs.CPU3.TIMER 5319 ± 20% +409.5% 27101 ± 2% softirqs.CPU30.RCU 137782 -27.8% 99418 ± 8% softirqs.CPU30.TIMER 4774 ± 8% +426.7% 25147 ± 9% softirqs.CPU31.RCU 143863 ± 8% -33.7% 95442 ± 5% softirqs.CPU31.TIMER 4888 ± 16% +305.4% 19819 ± 2% softirqs.CPU32.RCU 137941 -29.6% 97088 ± 6% softirqs.CPU32.TIMER 4475 +343.8% 19863 ± 5% softirqs.CPU33.RCU 137302 -31.1% 94646 ± 4% softirqs.CPU33.TIMER 4426 +357.4% 20247 ± 5% softirqs.CPU34.RCU 137165 -30.1% 95855 ± 4% softirqs.CPU34.TIMER 4446 +356.0% 20272 ± 8% softirqs.CPU35.RCU 137232 -28.6% 97944 ± 6% softirqs.CPU35.TIMER 4997 ± 4% +358.4% 22905 ± 13% softirqs.CPU36.RCU 4922 ± 2% +384.7% 23857 ± 7% softirqs.CPU37.RCU 136734 -29.0% 97055 ± 5% softirqs.CPU37.TIMER 4717 ± 2% +387.4% 22992 ± 8% softirqs.CPU38.RCU 137181 -29.7% 96415 ± 3% softirqs.CPU38.TIMER 4647 ± 2% +408.6% 23637 ± 9% softirqs.CPU39.RCU 136982 -29.2% 97049 ± 5% softirqs.CPU39.TIMER 4939 ± 4% +329.0% 21192 ± 4% softirqs.CPU4.RCU 138222 -28.8% 98401 ± 3% softirqs.CPU4.TIMER 4950 ± 12% +340.0% 21777 ± 4% softirqs.CPU40.RCU 138031 -32.1% 93722 ± 2% softirqs.CPU40.TIMER 4771 ± 4% +404.9% 24091 ± 3% softirqs.CPU41.RCU 144227 ± 7% -33.1% 96559 ± 3% softirqs.CPU41.TIMER 4563 +412.7% 23395 ± 4% softirqs.CPU42.RCU 137339 -29.6% 96632 ± 4% softirqs.CPU42.TIMER 4646 +398.8% 23175 ± 4% softirqs.CPU43.RCU 136891 -29.9% 95967 ± 3% softirqs.CPU43.TIMER 4806 ± 10% +402.0% 24125 ± 6% softirqs.CPU44.RCU 136734 -27.7% 98846 ± 5% softirqs.CPU44.TIMER 4525 ± 2% +406.0% 22897 ± 5% softirqs.CPU45.RCU 136959 -30.3% 95475 ± 4% softirqs.CPU45.TIMER 4498 +395.9% 22309 ± 8% softirqs.CPU46.RCU 136904 -31.2% 94168 ± 3% softirqs.CPU46.TIMER 4503 ± 2% +432.8% 23993 ± 9% softirqs.CPU47.RCU 136710 -29.9% 95900 ± 4% softirqs.CPU47.TIMER 4425 +384.9% 21458 ± 7% softirqs.CPU48.RCU 136875 -30.1% 95640 ± 5% softirqs.CPU48.TIMER 4606 ± 10% +359.6% 21167 ± 4% softirqs.CPU49.RCU 136784 -32.0% 93017 ± 2% softirqs.CPU49.TIMER 5049 ± 5% +349.6% 22701 ± 4% softirqs.CPU5.RCU 139913 -27.9% 100937 ± 6% softirqs.CPU5.TIMER 4518 ± 3% +357.4% 20668 ± 5% softirqs.CPU50.RCU 137194 -31.1% 94542 ± 4% softirqs.CPU50.TIMER 4482 ± 3% +375.1% 21298 ± 8% softirqs.CPU51.RCU 136835 -30.5% 95111 ± 4% softirqs.CPU51.TIMER 4601 ± 5% +356.2% 20991 ± 6% softirqs.CPU52.RCU 137235 -31.4% 94077 ± 3% softirqs.CPU52.TIMER 4388 ± 2% +390.0% 21503 ± 9% softirqs.CPU53.RCU 137098 -30.8% 94877 ± 3% softirqs.CPU53.TIMER 5139 ± 2% +329.3% 22063 ± 6% softirqs.CPU54.RCU 142302 ± 7% -32.3% 96357 ± 4% softirqs.CPU54.TIMER 4635 ± 2% +391.1% 22766 ± 2% softirqs.CPU55.RCU 137116 -26.3% 101030 ± 8% softirqs.CPU55.TIMER 4545 ± 6% +376.5% 21657 ± 3% softirqs.CPU56.RCU 136972 -28.0% 98650 ± 6% softirqs.CPU56.TIMER 4474 ± 2% +379.5% 21454 ± 4% softirqs.CPU57.RCU 136672 -29.4% 96557 ± 4% softirqs.CPU57.TIMER 4462 ± 2% +375.3% 21209 ± 2% softirqs.CPU58.RCU 137068 -28.8% 97548 ± 5% softirqs.CPU58.TIMER 4428 ± 2% +409.2% 22548 ± 4% softirqs.CPU59.RCU 136853 -25.0% 102707 ± 9% softirqs.CPU59.TIMER 5073 ± 10% +338.2% 22233 ± 3% softirqs.CPU6.RCU 138835 -27.5% 100624 ± 8% softirqs.CPU6.TIMER 4459 +404.0% 22476 ± 6% softirqs.CPU60.RCU 137198 -28.3% 98362 ± 5% softirqs.CPU60.TIMER 4515 +365.8% 21031 ± 9% softirqs.CPU61.RCU 137075 -29.4% 96807 ± 3% softirqs.CPU61.TIMER 4531 ± 2% +378.4% 21678 ± 5% softirqs.CPU62.RCU 136885 -29.0% 97237 ± 5% softirqs.CPU62.TIMER 4594 ± 8% +373.9% 21769 ± 5% softirqs.CPU63.RCU 136859 -28.7% 97641 ± 5% softirqs.CPU63.TIMER 4364 +367.8% 20419 ± 7% softirqs.CPU64.RCU 136627 -28.8% 97324 ± 4% softirqs.CPU64.TIMER 4366 ± 2% +369.3% 20494 ± 9% softirqs.CPU65.RCU 136854 -30.4% 95291 ± 3% softirqs.CPU65.TIMER 4442 ± 2% +369.6% 20861 ± 3% softirqs.CPU66.RCU 137199 -28.5% 98129 ± 5% softirqs.CPU66.TIMER 4533 ± 4% +362.7% 20977 ± 4% softirqs.CPU67.RCU 137119 -27.9% 98833 ± 6% softirqs.CPU67.TIMER 4403 +372.8% 20822 ± 2% softirqs.CPU68.RCU 136962 -28.0% 98567 ± 6% softirqs.CPU68.TIMER 4392 +364.7% 20411 ± 5% softirqs.CPU69.RCU 137157 -29.9% 96152 ± 4% softirqs.CPU69.TIMER 4859 ± 5% +318.0% 20313 ± 9% softirqs.CPU7.RCU 4559 ± 7% +358.6% 20911 softirqs.CPU70.RCU 137080 -27.8% 99010 ± 7% softirqs.CPU70.TIMER 4447 ± 2% +358.8% 20404 ± 5% softirqs.CPU71.RCU 137002 -28.0% 98583 ± 6% softirqs.CPU71.TIMER 6710 +271.2% 24909 ± 3% softirqs.CPU72.RCU 136653 -24.3% 103408 ± 9% softirqs.CPU72.TIMER 5874 ± 10% +303.0% 23672 ± 4% softirqs.CPU73.RCU 136843 -26.2% 101045 ± 9% softirqs.CPU73.TIMER 4692 +363.9% 21769 ± 4% softirqs.CPU74.RCU 136743 -30.3% 95297 ± 4% softirqs.CPU74.TIMER 4462 ± 2% +405.5% 22554 ± 3% softirqs.CPU75.RCU 136540 -28.1% 98159 ± 7% softirqs.CPU75.TIMER 4412 +389.9% 21616 ± 4% softirqs.CPU76.RCU 136935 -30.3% 95476 ± 4% softirqs.CPU76.TIMER 4365 +429.0% 23090 ± 4% softirqs.CPU77.RCU 137111 -26.9% 100166 ± 6% softirqs.CPU77.TIMER 4707 ± 10% +382.8% 22726 ± 3% softirqs.CPU78.RCU 137006 -27.5% 99374 ± 9% softirqs.CPU78.TIMER 4473 ± 5% +375.1% 21254 ± 7% softirqs.CPU79.RCU 136840 -30.3% 95422 ± 4% softirqs.CPU79.TIMER 4716 +373.1% 22310 softirqs.CPU8.RCU 137943 -26.6% 101184 ± 8% softirqs.CPU8.TIMER 4321 ± 2% +468.7% 24578 ± 4% softirqs.CPU80.RCU 136728 -26.8% 100149 ± 8% softirqs.CPU80.TIMER 4444 ± 6% +419.8% 23101 ± 5% softirqs.CPU81.RCU 136703 -29.2% 96768 ± 7% softirqs.CPU81.TIMER 4267 ± 2% +480.9% 24791 ± 6% softirqs.CPU82.RCU 136766 -28.2% 98171 ± 5% softirqs.CPU82.TIMER 4516 ± 3% +444.7% 24603 ± 5% softirqs.CPU83.RCU 136651 -28.4% 97833 ± 7% softirqs.CPU83.TIMER 4448 ± 3% +452.6% 24583 ± 5% softirqs.CPU84.RCU 137162 -29.2% 97148 ± 6% softirqs.CPU84.TIMER 4520 ± 3% +467.5% 25652 ± 9% softirqs.CPU85.RCU 136831 -26.0% 101197 ± 4% softirqs.CPU85.TIMER 4412 +419.2% 22907 ± 14% softirqs.CPU86.RCU 136808 -29.2% 96877 ± 5% softirqs.CPU86.TIMER 4798 ± 12% +385.1% 23279 ± 15% softirqs.CPU87.RCU 4701 ± 12% +412.6% 24100 ± 5% softirqs.CPU88.RCU 137117 -30.7% 95063 ± 4% softirqs.CPU88.TIMER 4699 ± 6% +419.0% 24392 ± 6% softirqs.CPU89.RCU 138163 -29.5% 97423 ± 5% softirqs.CPU89.TIMER 4977 ± 6% +331.8% 21495 ± 5% softirqs.CPU9.RCU 138622 -29.6% 97572 ± 7% softirqs.CPU9.TIMER 6462 ± 2% +257.4% 23100 ± 4% softirqs.CPU90.RCU 135954 -28.8% 96797 ± 6% softirqs.CPU90.TIMER 5143 +343.5% 22811 ± 13% softirqs.CPU91.RCU 135915 -28.0% 97879 ± 8% softirqs.CPU91.TIMER 4440 +366.6% 20722 ± 6% softirqs.CPU92.RCU 136329 -30.2% 95115 ± 5% softirqs.CPU92.TIMER 4363 ± 3% +384.5% 21139 ± 11% softirqs.CPU93.RCU 136605 -31.1% 94134 ± 3% softirqs.CPU93.TIMER 4294 ± 3% +375.8% 20433 ± 4% softirqs.CPU94.RCU 135773 -30.1% 94847 ± 4% softirqs.CPU94.TIMER 4251 +397.6% 21157 ± 6% softirqs.CPU95.RCU 135656 -30.2% 94714 ± 3% softirqs.CPU95.TIMER 4160 ± 4% +473.6% 23860 ± 7% softirqs.CPU96.RCU 136103 -29.9% 95443 ± 4% softirqs.CPU96.TIMER 4730 ± 17% +404.1% 23847 ± 5% softirqs.CPU97.RCU 138826 ± 3% -31.0% 95808 ± 4% softirqs.CPU97.TIMER 4331 ± 6% +425.9% 22779 ± 13% softirqs.CPU98.RCU 4182 +464.9% 23626 ± 4% softirqs.CPU99.RCU 136510 -30.6% 94756 ± 5% softirqs.CPU99.TIMER 667629 +386.1% 3245313 ± 2% softirqs.RCU 19758579 -29.0% 14035086 ± 4% softirqs.TIMER 47.78 -47.8 0.00 perf-profile.calltrace.cycles-pp.percpu_counter_add_batch.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 47.66 -47.7 0.00 perf-profile.calltrace.cycles-pp.__vm_enough_memory.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 47.66 -47.7 0.00 perf-profile.calltrace.cycles-pp.percpu_counter_add_batch.__vm_enough_memory.mmap_region.do_mmap.vm_mmap_pgoff 47.17 -47.2 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.percpu_counter_add_batch.__do_munmap.__vm_munmap.__x64_sys_munmap 47.09 -47.1 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.percpu_counter_add_batch.__do_munmap.__vm_munmap 47.05 -47.0 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.percpu_counter_add_batch.__vm_enough_memory.mmap_region.do_mmap 46.97 -47.0 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.percpu_counter_add_batch.__vm_enough_memory.mmap_region 48.48 -31.4 17.11 perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 48.73 -26.8 21.97 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 48.87 -23.8 25.09 perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 48.94 -22.7 26.25 perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 49.35 -13.9 35.50 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 49.37 -13.6 35.78 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mmap64 49.43 -9.2 40.23 perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe 49.61 -8.0 41.63 perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 49.63 -7.6 42.03 perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 49.62 -7.2 42.40 perf-profile.calltrace.cycles-pp.mmap64 0.00 +0.5 0.53 ± 2% perf-profile.calltrace.cycles-pp.common_file_perm.security_mmap_file.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 0.00 +0.5 0.54 ± 2% perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.58 ± 2% perf-profile.calltrace.cycles-pp.down_write.vma_link.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +0.6 0.59 perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.unmap_region.__do_munmap 0.00 +0.6 0.63 perf-profile.calltrace.cycles-pp.security_vm_enough_memory_mm.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.00 +0.6 0.63 ± 2% perf-profile.calltrace.cycles-pp.strlen.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.vmacache_find.find_vma.__do_munmap.__vm_munmap.__x64_sys_munmap 0.00 +0.7 0.65 ± 3% perf-profile.calltrace.cycles-pp.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.7 0.67 perf-profile.calltrace.cycles-pp.kfree.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +0.7 0.72 perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.shmem_mmap.mmap_region.do_mmap 0.00 +0.8 0.80 perf-profile.calltrace.cycles-pp.kmem_cache_alloc.vm_area_alloc.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +0.8 0.81 ± 3% perf-profile.calltrace.cycles-pp.ftrace_syscall_exit.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 0.00 +0.8 0.83 perf-profile.calltrace.cycles-pp.__vma_rb_erase.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 0.00 +0.8 0.85 perf-profile.calltrace.cycles-pp.ftrace_syscall_enter.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 0.00 +0.9 0.86 perf-profile.calltrace.cycles-pp.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap 0.00 +0.9 0.92 ± 2% perf-profile.calltrace.cycles-pp.ftrace_syscall_exit.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 0.00 +0.9 0.93 perf-profile.calltrace.cycles-pp.touch_atime.shmem_mmap.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +0.9 0.94 ± 3% perf-profile.calltrace.cycles-pp.prepend_name.prepend_path.d_path.perf_event_mmap.mmap_region 0.00 +1.0 0.95 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_trace.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +1.0 1.02 perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.__do_munmap.__vm_munmap 50.03 +1.1 51.10 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 0.00 +1.1 1.09 ± 3% perf-profile.calltrace.cycles-pp.ftrace_syscall_enter.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 0.00 +1.1 1.13 perf-profile.calltrace.cycles-pp.perf_event_mmap_output.perf_iterate_sb.perf_event_mmap.mmap_region.do_mmap 0.00 +1.2 1.18 perf-profile.calltrace.cycles-pp.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 0.00 +1.2 1.18 perf-profile.calltrace.cycles-pp.__vma_link_rb.vma_link.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +1.3 1.26 perf-profile.calltrace.cycles-pp.shmem_mmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.00 +1.3 1.28 ± 2% perf-profile.calltrace.cycles-pp.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 0.00 +1.3 1.33 perf-profile.calltrace.cycles-pp.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64 50.05 +1.3 51.39 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.munmap 0.00 +1.3 1.35 perf-profile.calltrace.cycles-pp.remove_vma.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 0.00 +1.4 1.38 perf-profile.calltrace.cycles-pp.vm_area_alloc.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.00 +1.6 1.55 perf-profile.calltrace.cycles-pp.security_mmap_file.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.6 1.55 perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 0.00 +1.6 1.63 ± 2% perf-profile.calltrace.cycles-pp.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap 0.00 +1.6 1.63 perf-profile.calltrace.cycles-pp.find_vma.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 0.00 +1.7 1.70 perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff 0.00 +1.9 1.92 perf-profile.calltrace.cycles-pp.prepend_path.d_path.perf_event_mmap.mmap_region.do_mmap 0.00 +2.3 2.31 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.munmap 0.00 +2.4 2.37 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.mmap64 0.00 +2.4 2.41 perf-profile.calltrace.cycles-pp.d_path.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +2.6 2.55 perf-profile.calltrace.cycles-pp.perf_iterate_sb.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff 0.00 +2.7 2.69 perf-profile.calltrace.cycles-pp.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.00 +2.7 2.71 perf-profile.calltrace.cycles-pp.rcu_all_qs._cond_resched.unmap_page_range.unmap_vmas.unmap_region 0.00 +2.8 2.77 perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.00 +3.0 3.02 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.munmap 0.00 +3.1 3.12 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.mmap64 0.00 +3.6 3.59 perf-profile.calltrace.cycles-pp.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 0.00 +5.1 5.10 perf-profile.calltrace.cycles-pp._cond_resched.unmap_page_range.unmap_vmas.unmap_region.__do_munmap 0.00 +5.6 5.62 ± 2% perf-profile.calltrace.cycles-pp.free_p4d_range.free_pgd_range.unmap_region.__do_munmap.__vm_munmap 0.00 +6.0 6.04 ± 2% perf-profile.calltrace.cycles-pp.free_pgd_range.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 50.28 +7.1 57.40 perf-profile.calltrace.cycles-pp.munmap 0.00 +8.5 8.47 perf-profile.calltrace.cycles-pp.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.00 +10.1 10.11 perf-profile.calltrace.cycles-pp.___might_sleep.unmap_page_range.unmap_vmas.unmap_region.__do_munmap 0.97 +23.2 24.20 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap 1.07 +24.4 25.45 perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 1.38 +33.0 34.38 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 95.45 -95.1 0.37 ± 2% perf-profile.children.cycles-pp.percpu_counter_add_batch 94.23 -94.2 0.00 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 94.05 -94.1 0.00 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 47.66 -47.4 0.31 perf-profile.children.cycles-pp.__vm_enough_memory 48.52 -31.3 17.24 perf-profile.children.cycles-pp.mmap_region 48.73 -26.7 22.05 perf-profile.children.cycles-pp.do_mmap 48.88 -23.7 25.16 perf-profile.children.cycles-pp.vm_mmap_pgoff 48.94 -22.6 26.30 perf-profile.children.cycles-pp.ksys_mmap_pgoff 99.44 -12.7 86.75 perf-profile.children.cycles-pp.do_syscall_64 99.47 -12.2 87.28 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 49.43 -9.1 40.34 perf-profile.children.cycles-pp.__do_munmap 49.62 -7.9 41.69 perf-profile.children.cycles-pp.__vm_munmap 49.63 -7.6 42.07 perf-profile.children.cycles-pp.__x64_sys_munmap 49.63 -6.9 42.74 perf-profile.children.cycles-pp.mmap64 0.05 ± 8% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.irq_exit 0.00 +0.1 0.05 perf-profile.children.cycles-pp.uprobe_munmap 0.00 +0.1 0.05 perf-profile.children.cycles-pp.__remove_shared_vm_struct 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.vm_area_free 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.munmap@plt 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.__softirqentry_text_start 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.mmap@plt 0.00 +0.1 0.06 ± 10% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.00 +0.1 0.06 perf-profile.children.cycles-pp.__vma_link_list 0.14 ± 3% +0.1 0.20 ± 3% perf-profile.children.cycles-pp.up_read 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.rb_next 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.is_file_shm_hugepages 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.fput 0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64 0.00 +0.1 0.08 ± 6% perf-profile.children.cycles-pp.ima_file_mmap 0.00 +0.1 0.08 perf-profile.children.cycles-pp.get_mmap_base 0.13 ± 5% +0.1 0.21 ± 7% perf-profile.children.cycles-pp.task_tick_fair 0.00 +0.1 0.08 ± 7% perf-profile.children.cycles-pp.fget 0.19 ± 3% +0.1 0.27 ± 9% perf-profile.children.cycles-pp.scheduler_tick 0.00 +0.1 0.10 ± 5% perf-profile.children.cycles-pp.common_mmap 0.00 +0.1 0.10 ± 3% perf-profile.children.cycles-pp.get_align_mask 0.00 +0.1 0.11 ± 3% perf-profile.children.cycles-pp.aa_file_perm 0.00 +0.1 0.12 ± 8% perf-profile.children.cycles-pp.vma_wants_writenotify 0.00 +0.1 0.12 ± 3% perf-profile.children.cycles-pp.should_failslab 0.00 +0.1 0.12 ± 6% perf-profile.children.cycles-pp.__x86_indirect_thunk_r12 0.00 +0.1 0.12 ± 3% perf-profile.children.cycles-pp.vm_pgprot_modify 0.00 +0.1 0.14 ± 5% perf-profile.children.cycles-pp.uprobe_mmap 0.00 +0.1 0.14 ± 5% perf-profile.children.cycles-pp.tlb_flush_mmu 0.08 +0.1 0.22 ± 14% perf-profile.children.cycles-pp.clockevents_program_event 0.00 +0.1 0.15 ± 5% perf-profile.children.cycles-pp.vmacache_update 0.00 +0.2 0.15 ± 7% perf-profile.children.cycles-pp.may_expand_vm 0.25 ± 3% +0.2 0.40 ± 6% perf-profile.children.cycles-pp.update_process_times 0.00 +0.2 0.16 ± 2% perf-profile.children.cycles-pp.cap_capable 0.00 +0.2 0.16 ± 4% perf-profile.children.cycles-pp.unlink_anon_vmas 0.00 +0.2 0.16 ± 2% perf-profile.children.cycles-pp.timestamp_truncate 0.25 ± 2% +0.2 0.42 ± 6% perf-profile.children.cycles-pp.tick_sched_handle 0.09 ± 9% +0.2 0.25 ± 3% perf-profile.children.cycles-pp.vma_merge 0.00 +0.2 0.17 ± 7% perf-profile.children.cycles-pp.__x86_indirect_thunk_r15 0.01 ±173% +0.2 0.19 ± 16% perf-profile.children.cycles-pp.ktime_get 0.00 +0.2 0.18 ± 3% perf-profile.children.cycles-pp.__vma_link_file 0.00 +0.2 0.18 ± 4% perf-profile.children.cycles-pp.vma_interval_tree_insert 0.28 +0.2 0.46 ± 5% perf-profile.children.cycles-pp.tick_sched_timer 0.39 ± 2% +0.2 0.60 ± 5% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.00 +0.2 0.22 ± 3% perf-profile.children.cycles-pp.__rb_insert_augmented 0.00 +0.2 0.22 ± 4% perf-profile.children.cycles-pp.cap_mmap_file 0.00 +0.2 0.25 perf-profile.children.cycles-pp.userfaultfd_unmap_prep 0.00 +0.2 0.25 ± 4% perf-profile.children.cycles-pp.unmap_single_vma 0.00 +0.3 0.26 ± 4% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.00 +0.3 0.27 ± 3% perf-profile.children.cycles-pp.vma_interval_tree_remove 0.00 +0.3 0.28 ± 2% perf-profile.children.cycles-pp.downgrade_write 0.00 +0.3 0.28 ± 3% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.00 +0.3 0.29 ± 4% perf-profile.children.cycles-pp.cap_mmap_addr 0.00 +0.3 0.30 ± 4% perf-profile.children.cycles-pp.apparmor_mmap_file 0.00 +0.3 0.30 ± 4% perf-profile.children.cycles-pp.tlb_gather_mmu 0.00 +0.3 0.31 ± 3% perf-profile.children.cycles-pp.__x64_sys_mmap 0.00 +0.3 0.33 ± 6% perf-profile.children.cycles-pp.path_noexec 0.00 +0.3 0.34 ± 2% perf-profile.children.cycles-pp.lru_add_drain 0.80 ± 2% +0.3 1.14 ± 4% perf-profile.children.cycles-pp.apic_timer_interrupt 0.00 +0.4 0.36 ± 2% perf-profile.children.cycles-pp.vma_set_page_prot 0.56 ± 2% +0.4 0.92 ± 6% perf-profile.children.cycles-pp.hrtimer_interrupt 0.63 ± 2% +0.4 1.02 ± 5% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.00 +0.4 0.39 ± 2% perf-profile.children.cycles-pp.userfaultfd_unmap_complete 0.00 +0.4 0.42 ± 4% perf-profile.children.cycles-pp.kmem_cache_free 0.00 +0.4 0.44 ± 3% perf-profile.children.cycles-pp.current_time 0.00 +0.4 0.44 ± 5% perf-profile.children.cycles-pp.security_mmap_addr 0.00 +0.5 0.46 ± 2% perf-profile.children.cycles-pp.tlb_finish_mmu 0.00 +0.5 0.49 ± 2% perf-profile.children.cycles-pp.cap_vm_enough_memory 0.00 +0.5 0.50 ± 5% perf-profile.children.cycles-pp.__fget_files 0.00 +0.5 0.50 ± 3% perf-profile.children.cycles-pp.fput_many 0.00 +0.5 0.54 ± 2% perf-profile.children.cycles-pp.common_file_perm 0.01 ±173% +0.6 0.61 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.00 +0.6 0.60 ± 3% perf-profile.children.cycles-pp.ring_buffer_lock_reserve 0.00 +0.6 0.64 perf-profile.children.cycles-pp.security_vm_enough_memory_mm 0.00 +0.6 0.64 perf-profile.children.cycles-pp.strlen 0.00 +0.7 0.66 perf-profile.children.cycles-pp.vmacache_find 0.00 +0.7 0.67 perf-profile.children.cycles-pp.kfree 0.00 +0.7 0.73 perf-profile.children.cycles-pp.atime_needs_update 0.06 ± 7% +0.8 0.86 perf-profile.children.cycles-pp.kmem_cache_alloc 0.01 ±173% +0.8 0.83 perf-profile.children.cycles-pp.__vma_rb_erase 0.00 +0.8 0.83 ± 2% perf-profile.children.cycles-pp.up_write 0.01 ±173% +0.9 0.87 perf-profile.children.cycles-pp.vm_unmapped_area 0.00 +0.9 0.89 ± 4% perf-profile.children.cycles-pp.trace_buffer_lock_reserve 0.01 ±173% +0.9 0.94 perf-profile.children.cycles-pp.touch_atime 0.03 ±100% +1.0 0.98 perf-profile.children.cycles-pp.__might_sleep 0.05 ± 9% +1.0 1.02 perf-profile.children.cycles-pp.kmem_cache_alloc_trace 0.00 +1.0 0.99 ± 3% perf-profile.children.cycles-pp.prepend_name 0.04 ± 57% +1.0 1.04 perf-profile.children.cycles-pp.unlink_file_vma 0.05 +1.1 1.19 perf-profile.children.cycles-pp.__vma_link_rb 0.05 +1.2 1.22 perf-profile.children.cycles-pp.down_write 0.00 +1.2 1.18 ± 2% perf-profile.children.cycles-pp.perf_event_mmap_output 0.05 +1.2 1.27 ± 2% perf-profile.children.cycles-pp.down_write_killable 0.07 ± 7% +1.2 1.28 perf-profile.children.cycles-pp.shmem_mmap 0.11 ± 4% +1.3 1.38 perf-profile.children.cycles-pp.vm_area_alloc 0.06 ± 7% +1.3 1.39 perf-profile.children.cycles-pp.remove_vma 0.08 ± 5% +1.5 1.56 perf-profile.children.cycles-pp.security_mmap_file 0.07 ± 6% +1.5 1.59 perf-profile.children.cycles-pp.free_pgtables 0.07 ± 5% +1.6 1.70 perf-profile.children.cycles-pp.find_vma 0.09 ± 4% +1.7 1.75 perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.08 ± 5% +1.7 1.80 ± 2% perf-profile.children.cycles-pp.ftrace_syscall_exit 0.08 ± 6% +1.9 1.94 perf-profile.children.cycles-pp.prepend_path 0.07 ± 5% +1.9 1.98 perf-profile.children.cycles-pp.ftrace_syscall_enter 0.11 ± 4% +2.3 2.42 perf-profile.children.cycles-pp.d_path 0.11 ± 7% +2.4 2.48 perf-profile.children.cycles-pp.syscall_slow_exit_work 0.08 ± 5% +2.5 2.58 perf-profile.children.cycles-pp.perf_iterate_sb 0.13 ± 3% +2.6 2.69 perf-profile.children.cycles-pp.shmem_get_unmapped_area 0.12 ± 3% +2.8 2.87 perf-profile.children.cycles-pp.vma_link 0.12 ± 3% +2.9 2.99 perf-profile.children.cycles-pp.syscall_trace_enter 0.10 ± 4% +3.0 3.12 perf-profile.children.cycles-pp.rcu_all_qs 0.17 ± 4% +3.4 3.61 perf-profile.children.cycles-pp.get_unmapped_area 0.22 +5.1 5.28 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.17 ± 5% +5.5 5.64 ± 2% perf-profile.children.cycles-pp.free_p4d_range 0.18 ± 3% +5.9 6.06 ± 2% perf-profile.children.cycles-pp.free_pgd_range 0.22 ± 3% +5.9 6.14 perf-profile.children.cycles-pp._cond_resched 0.19 ± 3% +6.0 6.16 perf-profile.children.cycles-pp.entry_SYSCALL_64 50.30 +7.5 57.81 perf-profile.children.cycles-pp.munmap 0.35 ± 3% +8.2 8.56 perf-profile.children.cycles-pp.perf_event_mmap 0.49 ± 3% +10.4 10.86 perf-profile.children.cycles-pp.___might_sleep 1.04 +23.6 24.67 perf-profile.children.cycles-pp.unmap_page_range 1.07 +24.4 25.48 perf-profile.children.cycles-pp.unmap_vmas 1.38 +33.1 34.47 perf-profile.children.cycles-pp.unmap_region 94.05 -94.1 0.00 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__vma_link_list 0.14 ± 3% +0.1 0.19 ± 2% perf-profile.self.cycles-pp.up_read 0.00 +0.1 0.05 ± 9% perf-profile.self.cycles-pp.mmap@plt 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.lru_add_drain 0.00 +0.1 0.06 perf-profile.self.cycles-pp.should_failslab 0.00 +0.1 0.07 ± 7% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64 0.00 +0.1 0.07 ± 11% perf-profile.self.cycles-pp.fget 0.00 +0.1 0.07 ± 12% perf-profile.self.cycles-pp.ima_file_mmap 0.00 +0.1 0.07 perf-profile.self.cycles-pp.get_mmap_base 0.00 +0.1 0.08 ± 5% perf-profile.self.cycles-pp.get_align_mask 0.00 +0.1 0.09 ± 4% perf-profile.self.cycles-pp.common_mmap 0.00 +0.1 0.09 ± 8% perf-profile.self.cycles-pp.__x86_indirect_thunk_r12 0.00 +0.1 0.10 ± 5% perf-profile.self.cycles-pp.tlb_flush_mmu 0.00 +0.1 0.10 ± 11% perf-profile.self.cycles-pp.vma_wants_writenotify 0.00 +0.1 0.11 ± 5% perf-profile.self.cycles-pp.aa_file_perm 0.00 +0.1 0.11 ± 3% perf-profile.self.cycles-pp.vm_pgprot_modify 0.00 +0.1 0.12 ± 6% perf-profile.self.cycles-pp.unlink_file_vma 0.00 +0.1 0.12 ± 8% perf-profile.self.cycles-pp.uprobe_mmap 0.00 +0.1 0.13 ± 3% perf-profile.self.cycles-pp.security_vm_enough_memory_mm 0.00 +0.1 0.13 ± 4% perf-profile.self.cycles-pp.vma_set_page_prot 0.00 +0.1 0.13 ± 3% perf-profile.self.cycles-pp.vmacache_update 0.00 +0.1 0.14 ± 7% perf-profile.self.cycles-pp.may_expand_vm 0.00 +0.1 0.14 ± 8% perf-profile.self.cycles-pp.security_mmap_addr 0.00 +0.1 0.14 ± 5% perf-profile.self.cycles-pp.unlink_anon_vmas 0.00 +0.1 0.15 ± 5% perf-profile.self.cycles-pp.__vma_link_file 0.00 +0.1 0.15 ± 5% perf-profile.self.cycles-pp.timestamp_truncate 0.00 +0.1 0.15 ± 2% perf-profile.self.cycles-pp.cap_capable 0.09 ± 9% +0.2 0.24 ± 4% perf-profile.self.cycles-pp.vma_merge 0.00 +0.2 0.16 ± 16% perf-profile.self.cycles-pp.ktime_get 0.00 +0.2 0.16 ± 3% perf-profile.self.cycles-pp.free_pgtables 0.00 +0.2 0.16 ± 6% perf-profile.self.cycles-pp.__x86_indirect_thunk_r15 0.00 +0.2 0.16 ± 2% perf-profile.self.cycles-pp.__vm_enough_memory 0.00 +0.2 0.17 ± 5% perf-profile.self.cycles-pp.vma_interval_tree_insert 0.00 +0.2 0.19 ± 5% perf-profile.self.cycles-pp.touch_atime 0.00 +0.2 0.19 ± 5% perf-profile.self.cycles-pp.cap_mmap_file 0.00 +0.2 0.21 ± 4% perf-profile.self.cycles-pp.__rb_insert_augmented 0.00 +0.2 0.21 ± 5% perf-profile.self.cycles-pp.unmap_single_vma 0.00 +0.2 0.21 ± 4% perf-profile.self.cycles-pp.vma_interval_tree_remove 0.00 +0.2 0.22 ± 4% perf-profile.self.cycles-pp.remove_vma 0.00 +0.2 0.22 ± 3% perf-profile.self.cycles-pp.current_time 0.00 +0.2 0.24 ± 2% perf-profile.self.cycles-pp.userfaultfd_unmap_prep 0.00 +0.2 0.25 ± 4% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.00 +0.3 0.26 ± 5% perf-profile.self.cycles-pp.cap_mmap_addr 0.00 +0.3 0.26 ± 2% perf-profile.self.cycles-pp.__x64_sys_mmap 0.00 +0.3 0.26 ± 3% perf-profile.self.cycles-pp.apparmor_mmap_file 0.00 +0.3 0.26 ± 3% perf-profile.self.cycles-pp.downgrade_write 0.00 +0.3 0.26 ± 3% perf-profile.self.cycles-pp.lru_add_drain_cpu 0.00 +0.3 0.27 ± 2% perf-profile.self.cycles-pp.unmap_vmas 0.00 +0.3 0.29 perf-profile.self.cycles-pp.vma_link 0.00 +0.3 0.29 ± 3% perf-profile.self.cycles-pp.tlb_gather_mmu 0.00 +0.3 0.29 ± 3% perf-profile.self.cycles-pp.atime_needs_update 0.00 +0.3 0.30 ± 7% perf-profile.self.cycles-pp.path_noexec 0.00 +0.3 0.30 ± 3% perf-profile.self.cycles-pp.cap_vm_enough_memory 0.00 +0.3 0.30 ± 2% perf-profile.self.cycles-pp.__x64_sys_munmap 0.00 +0.3 0.31 perf-profile.self.cycles-pp.tlb_finish_mmu 0.00 +0.3 0.32 perf-profile.self.cycles-pp.unmap_region 0.00 +0.3 0.33 ± 3% perf-profile.self.cycles-pp.ksys_mmap_pgoff 0.00 +0.3 0.33 ± 4% perf-profile.self.cycles-pp.trace_buffer_lock_reserve 0.00 +0.3 0.34 perf-profile.self.cycles-pp.vm_mmap_pgoff 0.00 +0.3 0.34 ± 4% perf-profile.self.cycles-pp.__vm_munmap 0.00 +0.3 0.35 ± 2% perf-profile.self.cycles-pp.shmem_mmap 0.00 +0.4 0.36 ± 3% perf-profile.self.cycles-pp.userfaultfd_unmap_complete 0.00 +0.4 0.38 ± 3% perf-profile.self.cycles-pp.kmem_cache_free 0.00 +0.4 0.41 perf-profile.self.cycles-pp.security_mmap_file 0.00 +0.4 0.42 ± 3% perf-profile.self.cycles-pp.free_pgd_range 0.00 +0.4 0.42 ± 2% perf-profile.self.cycles-pp.common_file_perm 0.00 +0.5 0.46 ± 3% perf-profile.self.cycles-pp.get_unmapped_area 0.00 +0.5 0.47 ± 2% perf-profile.self.cycles-pp.kmem_cache_alloc 0.00 +0.5 0.48 ± 3% perf-profile.self.cycles-pp.fput_many 0.00 +0.5 0.48 perf-profile.self.cycles-pp.d_path 0.00 +0.5 0.49 ± 3% perf-profile.self.cycles-pp.vm_area_alloc 0.00 +0.5 0.49 ± 3% perf-profile.self.cycles-pp.munmap 0.00 +0.5 0.49 ± 4% perf-profile.self.cycles-pp.__fget_files 0.00 +0.5 0.50 ± 2% perf-profile.self.cycles-pp.down_write_killable 0.00 +0.5 0.51 perf-profile.self.cycles-pp.down_write 0.00 +0.5 0.53 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.00 +0.5 0.55 ± 4% perf-profile.self.cycles-pp.ring_buffer_lock_reserve 0.00 +0.6 0.56 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.59 perf-profile.self.cycles-pp.kmem_cache_alloc_trace 0.00 +0.6 0.62 perf-profile.self.cycles-pp.vmacache_find 0.00 +0.6 0.63 perf-profile.self.cycles-pp.mmap64 0.00 +0.6 0.63 ± 2% perf-profile.self.cycles-pp.kfree 0.00 +0.6 0.63 ± 2% perf-profile.self.cycles-pp.strlen 0.00 +0.7 0.69 ± 8% perf-profile.self.cycles-pp.syscall_slow_exit_work 0.00 +0.7 0.70 perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown 0.00 +0.8 0.77 perf-profile.self.cycles-pp.up_write 0.01 ±173% +0.8 0.82 perf-profile.self.cycles-pp.__vma_rb_erase 0.01 ±173% +0.8 0.85 perf-profile.self.cycles-pp.vm_unmapped_area 0.01 ±173% +0.9 0.89 perf-profile.self.cycles-pp.find_vma 0.00 +0.9 0.88 perf-profile.self.cycles-pp.__might_sleep 0.01 ±173% +0.9 0.90 perf-profile.self.cycles-pp.shmem_get_unmapped_area 0.00 +0.9 0.90 ± 2% perf-profile.self.cycles-pp.do_mmap 0.00 +0.9 0.94 perf-profile.self.cycles-pp.prepend_path 0.00 +1.0 0.97 ± 3% perf-profile.self.cycles-pp.prepend_name 0.01 ±173% +1.0 1.00 perf-profile.self.cycles-pp.syscall_trace_enter 0.00 +1.1 1.12 ± 2% perf-profile.self.cycles-pp.perf_event_mmap_output 0.05 +1.1 1.17 perf-profile.self.cycles-pp.__vma_link_rb 0.05 +1.2 1.25 perf-profile.self.cycles-pp.perf_event_mmap 0.06 ± 6% +1.2 1.26 ± 3% perf-profile.self.cycles-pp.ftrace_syscall_exit 0.07 ± 11% +1.2 1.29 perf-profile.self.cycles-pp.__do_munmap 0.11 ± 8% +1.2 1.34 perf-profile.self.cycles-pp.mmap_region 0.06 ± 9% +1.3 1.37 ± 2% perf-profile.self.cycles-pp.perf_iterate_sb 0.06 ± 7% +1.4 1.50 perf-profile.self.cycles-pp.ftrace_syscall_enter 0.08 ± 6% +2.3 2.35 perf-profile.self.cycles-pp.rcu_all_qs 0.11 ± 7% +2.9 2.99 perf-profile.self.cycles-pp._cond_resched 0.22 +5.0 5.27 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.17 ± 5% +5.3 5.50 perf-profile.self.cycles-pp.entry_SYSCALL_64 0.16 ± 6% +5.4 5.54 ± 2% perf-profile.self.cycles-pp.free_p4d_range 0.47 ± 3% +9.6 10.07 perf-profile.self.cycles-pp.___might_sleep 0.46 +10.5 10.94 perf-profile.self.cycles-pp.unmap_page_range 0.53 +11.7 12.21 perf-profile.self.cycles-pp.do_syscall_64 will-it-scale.per_process_ops 90000 +-------------------------------------------------------------------+ | O O O O | 80000 |-+ | 70000 |-+ | | | 60000 |-+ | 50000 |-+ | | | 40000 |-+ | 30000 |-+ | | | 20000 |-+ | 10000 |-+ | |.....+....+.....+.....+....+.....+.....+....+.....+.....+....+.....| 0 +-------------------------------------------------------------------+ will-it-scale.workload 1.4e+07 +-----------------------------------------------------------------+ | | 1.2e+07 |-+ O O O O | | | 1e+07 |-+ | | | 8e+06 |-+ | | | 6e+06 |-+ | | | 4e+06 |-+ | | | 2e+06 |-+ | |.....+....+.....+....+.....+....+.....+....+.....+....+.....+....| 0 +-----------------------------------------------------------------+ syscalls.sys_mmap.min 7000 +--------------------------------------------------------------------+ |.....+.. . ...+.....+.. +..... .+....+.....| 6500 |-+ +.. +..... ... | | +. | | | 6000 |-+ | | | 5500 |-+ | | | 5000 |-+ | | | | | 4500 |-+ | | O O O O | 4000 +--------------------------------------------------------------------+ syscalls.sys_mmap.med 350000 +------------------------------------------------------------------+ | | 300000 |.....+....+.....+....+..... ..+.....+....+.....+....+.....| | +.....+.. | 250000 |-+ | | | 200000 |-+ | | | 150000 |-+ | | | 100000 |-+ | | | 50000 |-+ | | | 0 +------------------------------------------------------------------+ syscalls.sys_mmap.noise.5_ 6e+08 +-------------------------------------------------------------------+ | +.... | 5e+08 |-+ +.....+.....+....+.....+.....+....+.... .+....+.....| | . ... | | +. | 4e+08 |-+ | | | 3e+08 |-+ | | | 2e+08 |-+ | | | | | 1e+08 |-+ | | O O O O | 0 +-------------------------------------------------------------------+ syscalls.sys_mmap.noise.2_ 8e+08 +-------------------------------------------------------------------+ |.....+... | 7e+08 |-+ . ...+.....+....+.....+.....+....+..... ...+....+.....| 6e+08 |-+ +.. +.. | | | 5e+08 |-+ | | | 4e+08 |-+ | | | 3e+08 |-+ | 2e+08 |-+ | | | 1e+08 |-+ | | O O O O | 0 +-------------------------------------------------------------------+ syscalls.sys_munmap.min 35000 +-------------------------------------------------------------------+ | | 30000 |..... ..+..... ...+.... ..+.....+..... | | +.. +.. +.....+.....+.. +....+.....| | | 25000 |-+ | | | 20000 |-+ | | | 15000 |-+ | | | | | 10000 |-+ O O O O | | | 5000 +-------------------------------------------------------------------+ syscalls.sys_munmap.med 350000 +------------------------------------------------------------------+ |.....+....+.....+....+.... .+.....+....+.....+....+.....| 300000 |-+ . .. | | +.....+. | 250000 |-+ | | | 200000 |-+ | | | 150000 |-+ | | | 100000 |-+ | | | 50000 |-+ | | O O O O | 0 +------------------------------------------------------------------+ syscalls.sys_munmap.noise.100_ 7e+07 +-------------------------------------------------------------------+ | | 6e+07 |-+ O O O O | | | 5e+07 |-+ | | | 4e+07 |-+ | | | 3e+07 |-+ | | | 2e+07 |-+ | | | 1e+07 |-+ | | ...+.... | 0 +-------------------------------------------------------------------+ syscalls.sys_munmap.noise.75_ 7e+07 +-------------------------------------------------------------------+ | | 6e+07 |-+ O O O O | | | 5e+07 |-+ | | | 4e+07 |-+ | | | 3e+07 |-+ | | | 2e+07 |-+ | | | 1e+07 |-+ | | ...+.... | 0 +-------------------------------------------------------------------+ syscalls.sys_munmap.noise.50_ 7e+07 +-------------------------------------------------------------------+ | | 6e+07 |-+ O O O O | | | 5e+07 |-+ | | | 4e+07 |-+ | | | 3e+07 |-+ | | | 2e+07 |-+ | | | 1e+07 |-+ | | ...+.... | 0 +-------------------------------------------------------------------+ syscalls.sys_munmap.noise.25_ 7e+07 +-------------------------------------------------------------------+ | O O O | 6e+07 |-+ O | | | 5e+07 |-+ | | | 4e+07 |-+ | | | 3e+07 |-+ | | | 2e+07 |-+ | | | 1e+07 |-+ | |..... ...+....+.....+..... | 0 +-------------------------------------------------------------------+ syscalls.sys_munmap.noise.5_ 6e+08 +-----------------------------------------------------------------+ 5.5e+08 |..... | | +.... | 5e+08 |-+ +.....+....+.....+....+.....+....+.... .+.....+....| 4.5e+08 |-+ . .. | 4e+08 |-+ +. | 3.5e+08 |-+ | | | 3e+08 |-+ | 2.5e+08 |-+ | 2e+08 |-+ | 1.5e+08 |-+ | | O O O | 1e+08 |-+ O | 5e+07 +-----------------------------------------------------------------+ syscalls.sys_munmap.noise.2_ 8e+08 +-------------------------------------------------------------------+ | | 7e+08 |.....+.... | 6e+08 |-+ +.....+.....+....+.....+.....+....+..... ...+....+.....| | +.. | 5e+08 |-+ | | | 4e+08 |-+ | | | 3e+08 |-+ | 2e+08 |-+ | | | 1e+08 |-+ O O O O | | | 0 +-------------------------------------------------------------------+ syscalls.sys_open.max 2.5e+09 +-----------------------------------------------------------------+ | | | | 2e+09 |-+ | | | | | 1.5e+09 |-+ | | | 1e+09 |-+ | | | | | 5e+08 |-+ | | | | | 0 +-----------------------------------------------------------------+ syscalls.sys_open.min 13000 +-------------------------------------------------------------------+ 12500 |-+ O | | | 12000 |-+ | 11500 |-+ | 11000 |-+ | 10500 |-+ O | | O O | 10000 |-+ | 9500 |-+ | 9000 |-+ | 8500 |-+ | | ...+.... ...+..... | 8000 |.. +.. +....+.....+.....+....+.....+.....+....+.....| 7500 +-------------------------------------------------------------------+ syscalls.sys_open.med 18000 +-------------------------------------------------------------------+ | O | 17000 |-+ O | | O O | 16000 |-+ | | | 15000 |-+ | | | 14000 |-+ | | | 13000 |-+ | | ...+..... | 12000 |.....+....+.. +....+.....+.....+....+.....+.....+....+.....| | | 11000 +-------------------------------------------------------------------+ syscalls.sys_open.noise.100_ 1.4e+12 +-----------------------------------------------------------------+ | .+.... ...+.... .. .. ... .| 1.2e+12 |-... +.. +.....+....+..... .. . ..+. | |. + +.. | 1e+12 |-+ | | | 8e+11 |-+ | | | 6e+11 |-+ | | | 4e+11 |-+ | | | 2e+11 |-+ | | | 0 +-----------------------------------------------------------------+ syscalls.sys_open.noise.75_ 1.4e+12 +-----------------------------------------------------------------+ | .+.... ...+.... .. .. ... .| 1.2e+12 |-... +.. +.....+....+..... .. . ..+. | |. + +.. | 1e+12 |-+ | | | 8e+11 |-+ | | | 6e+11 |-+ | | | 4e+11 |-+ | | | 2e+11 |-+ | | | 0 +-----------------------------------------------------------------+ syscalls.sys_open.noise.50_ 1.4e+12 +-----------------------------------------------------------------+ | .+.... ...+.... .. .. ... .| 1.2e+12 |-... +.. +.....+....+..... .. . ..+. | |. + +.. | 1e+12 |-+ | | | 8e+11 |-+ | | | 6e+11 |-+ | | | 4e+11 |-+ | | | 2e+11 |-+ | | | 0 +-----------------------------------------------------------------+ syscalls.sys_open.noise.25_ 1.4e+12 +-----------------------------------------------------------------+ | .+.... ...+.... .. .. ... .| 1.2e+12 |-... +.. +.....+....+..... .. . ..+. | |. + +.. | 1e+12 |-+ | | | 8e+11 |-+ | | | 6e+11 |-+ | | | 4e+11 |-+ | | | 2e+11 |-+ | | | 0 +-----------------------------------------------------------------+ syscalls.sys_open.noise.5_ 1.4e+12 +-----------------------------------------------------------------+ | .+.... ...+.... .. .. ... .| 1.2e+12 |-... +.. +.....+....+..... .. . ..+. | |. + +.. | 1e+12 |-+ | | | 8e+11 |-+ | | | 6e+11 |-+ | | | 4e+11 |-+ | | | 2e+11 |-+ | | | 0 +-----------------------------------------------------------------+ syscalls.sys_open.noise.2_ 1.4e+12 +-----------------------------------------------------------------+ | .+.... ...+.... .. .. ... .| 1.2e+12 |-... +.. +.....+....+..... .. . ..+. | |. + +.. | 1e+12 |-+ | | | 8e+11 |-+ | | | 6e+11 |-+ | | | 4e+11 |-+ | | | 2e+11 |-+ | | | 0 +-----------------------------------------------------------------+ syscalls.sys_read.max 3e+09 +-----------------------------------------------------------------+ |.....+....+.....+....+.....+.... ...+....+.....+....+.....+....| 2.5e+09 |-+ +.. | | | | | 2e+09 |-+ | | | 1.5e+09 |-+ | | | 1e+09 |-+ | | | | | 5e+08 |-+ | | | 0 +-----------------------------------------------------------------+ syscalls.sys_read.noise.100_ 9e+11 +-------------------------------------------------------------------+ | .. .. | 8e+11 |-+ ..+.....+..... . . .+.....| 7e+11 |.....+.. +.... .. +.... .. | | + +..... ...+. | 6e+11 |-+ +.. | 5e+11 |-+ | | | 4e+11 |-+ | 3e+11 |-+ | | | 2e+11 |-+ | 1e+11 |-+ O | | | 0 +-------------------------------------------------------------------+ syscalls.sys_read.noise.75_ 9e+11 +-------------------------------------------------------------------+ | .. .. | 8e+11 |-+ ..+.....+..... . . .+.....| 7e+11 |.....+.. +.... .. +.... .. | | + +..... ...+. | 6e+11 |-+ +.. | 5e+11 |-+ | | | 4e+11 |-+ | 3e+11 |-+ | | | 2e+11 |-+ | 1e+11 |-+ O | | | 0 +-------------------------------------------------------------------+ syscalls.sys_read.noise.50_ 9e+11 +-------------------------------------------------------------------+ | .. .. | 8e+11 |-+ ..+.....+..... . . .+.....| 7e+11 |.....+.. +.... .. +.... .. | | + +..... ...+. | 6e+11 |-+ +.. | 5e+11 |-+ | | | 4e+11 |-+ | 3e+11 |-+ | | | 2e+11 |-+ | 1e+11 |-+ O | | | 0 +-------------------------------------------------------------------+ syscalls.sys_read.noise.25_ 9e+11 +-------------------------------------------------------------------+ | .. .. | 8e+11 |-+ ..+.....+..... . . .+.....| 7e+11 |.....+.. +.... .. +.... .. | | + +..... ...+. | 6e+11 |-+ +.. | 5e+11 |-+ | | | 4e+11 |-+ | 3e+11 |-+ | | | 2e+11 |-+ | 1e+11 |-+ O | | | 0 +-------------------------------------------------------------------+ syscalls.sys_read.noise.5_ 9e+11 +-------------------------------------------------------------------+ | .. .. | 8e+11 |-+ ..+.....+..... . . .+.....| 7e+11 |.....+.. +.... .. +.... .. | | + +..... ...+. | 6e+11 |-+ +.. | 5e+11 |-+ | | | 4e+11 |-+ | 3e+11 |-+ | | | 2e+11 |-+ | 1e+11 |-+ O | | | 0 +-------------------------------------------------------------------+ syscalls.sys_read.noise.2_ 9e+11 +-------------------------------------------------------------------+ | .. .. | 8e+11 |-+ ..+.....+..... . . .+.....| 7e+11 |.....+.. +.... .. +.... .. | | + +..... ...+. | 6e+11 |-+ +.. | 5e+11 |-+ | | | 4e+11 |-+ | 3e+11 |-+ | | | 2e+11 |-+ | 1e+11 |-+ O | | | 0 +-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2sp4: 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-csl-2sp4/mmap2/will-it-scale/0x400002c commit: 591e8cfa8d ("percpu counter init debug") a530431919 ("new overcommit patch") 591e8cfa8d9bfbd7 a5304319194e6f6fdae62fca227 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :3 67% 2:2 kmsg.mm_compute_batch()]:vm_committed_as_batch= :3 67% 2:2 kmsg.mmap_init()]:totalram_pages 2:3 -67% :2 dmesg.WARNING:at#for_ip_interrupt_entry/0x :3 33% 1:2 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :3 33% 1:2 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 126186 +13.6% 143311 ± 10% softirqs.CPU66.TIMER 3427 ± 7% +26.3% 4328 ± 8% slabinfo.eventpoll_pwq.active_objs 3427 ± 7% +26.3% 4328 ± 8% slabinfo.eventpoll_pwq.num_objs 3986 ± 5% +18.3% 4714 slabinfo.kmalloc-rcl-64.active_objs 3986 ± 5% +18.3% 4714 slabinfo.kmalloc-rcl-64.num_objs 185978 ± 41% +45.7% 270997 numa-meminfo.node0.Active 182574 ± 42% +46.6% 267611 numa-meminfo.node0.Active(anon) 115609 ± 52% +54.1% 178126 numa-meminfo.node0.AnonHugePages 181137 ± 42% +47.1% 266439 numa-meminfo.node0.AnonPages 7748 ± 69% +39.3% 10793 ± 55% numa-meminfo.node0.Inactive 6824 ± 77% +44.4% 9855 ± 60% numa-meminfo.node0.Inactive(anon) 14537 ± 14% +19.4% 17356 ± 14% numa-meminfo.node0.Mapped 5301 ± 10% +20.8% 6404 ± 4% numa-meminfo.node0.PageTables 194727 ± 40% -43.7% 109696 numa-meminfo.node1.Active 194703 ± 40% -43.7% 109670 numa-meminfo.node1.Active(anon) 120244 ± 64% -71.4% 34348 ± 5% numa-meminfo.node1.AnonPages 6380 ± 8% -17.5% 5262 ± 4% numa-meminfo.node1.PageTables 45645 ± 42% +46.6% 66900 numa-vmstat.node0.nr_active_anon 45286 ± 42% +47.1% 66607 numa-vmstat.node0.nr_anon_pages 1705 ± 77% +44.4% 2463 ± 60% numa-vmstat.node0.nr_inactive_anon 3731 ± 13% +19.0% 4440 ± 14% numa-vmstat.node0.nr_mapped 1324 ± 10% +20.8% 1600 ± 4% numa-vmstat.node0.nr_page_table_pages 45645 ± 42% +46.6% 66900 numa-vmstat.node0.nr_zone_active_anon 1705 ± 77% +44.4% 2463 ± 60% numa-vmstat.node0.nr_zone_inactive_anon 48649 ± 40% -43.6% 27458 numa-vmstat.node1.nr_active_anon 30059 ± 64% -71.5% 8565 ± 5% numa-vmstat.node1.nr_anon_pages 57.00 ±141% +202.6% 172.50 numa-vmstat.node1.nr_mlock 1593 ± 8% -17.5% 1315 ± 4% numa-vmstat.node1.nr_page_table_pages 48649 ± 40% -43.6% 27458 numa-vmstat.node1.nr_zone_active_anon 11091 ± 9% +17.3% 13012 ± 9% sched_debug.cfs_rq:/.load.avg 14475 ± 58% +111.1% 30563 ± 34% sched_debug.cfs_rq:/.load.stddev 20.33 ± 8% +15.2% 23.42 ± 3% sched_debug.cfs_rq:/.load_avg.avg 40.32 ± 7% +10.2% 44.43 sched_debug.cfs_rq:/.load_avg.stddev 0.06 ± 5% +8.7% 0.07 ± 2% sched_debug.cfs_rq:/.nr_running.stddev 2.75 ± 2% -21.6% 2.15 ± 9% sched_debug.cfs_rq:/.nr_spread_over.avg 5.39 ± 15% +54.2% 8.31 ± 6% sched_debug.cfs_rq:/.removed.load_avg.avg 29.46 ± 7% +23.8% 36.49 ± 2% sched_debug.cfs_rq:/.removed.load_avg.stddev 5.39 ± 15% +54.2% 8.31 ± 6% sched_debug.cfs_rq:/.removed.runnable_avg.avg 29.46 ± 7% +23.8% 36.49 ± 2% sched_debug.cfs_rq:/.removed.runnable_avg.stddev 262.47 ± 4% +8.7% 285.26 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev 557532 ± 3% +11.8% 623180 ± 5% sched_debug.cpu.avg_idle.min -67.33 -42.1% -39.00 sched_debug.cpu.nr_uninterruptible.min 13.46 ± 5% -14.9% 11.46 ± 4% sched_debug.cpu.nr_uninterruptible.stddev 4.22 ± 11% -21.1% 3.33 sched_debug.cpu.sched_goidle.min 5.67 ±141% +4382.4% 254.00 ± 51% interrupts.59:PCI-MSI.31981592-edge.i40e-eth0-TxRx-23 1768 ± 10% +20.3% 2127 ± 6% interrupts.CPU1.RES:Rescheduling_interrupts 317.67 +8.6% 345.00 ± 6% interrupts.CPU15.RES:Rescheduling_interrupts 5.67 ±141% +4373.5% 253.50 ± 51% interrupts.CPU23.59:PCI-MSI.31981592-edge.i40e-eth0-TxRx-23 834.33 ± 21% +28.1% 1069 ± 10% interrupts.CPU27.RES:Rescheduling_interrupts 687.67 ± 14% -36.5% 437.00 ± 8% interrupts.CPU28.RES:Rescheduling_interrupts 879.00 ± 17% +23.6% 1086 ± 10% interrupts.CPU3.RES:Rescheduling_interrupts 331.33 ± 3% +8.5% 359.50 ± 3% interrupts.CPU35.RES:Rescheduling_interrupts 8754 -50.0% 4377 interrupts.CPU47.NMI:Non-maskable_interrupts 8754 -50.0% 4377 interrupts.CPU47.PMI:Performance_monitoring_interrupts 8743 -50.0% 4369 interrupts.CPU49.NMI:Non-maskable_interrupts 8743 -50.0% 4369 interrupts.CPU49.PMI:Performance_monitoring_interrupts 8742 -50.0% 4369 interrupts.CPU50.NMI:Non-maskable_interrupts 8742 -50.0% 4369 interrupts.CPU50.PMI:Performance_monitoring_interrupts 776.67 ± 8% +64.2% 1275 ± 10% interrupts.CPU50.RES:Rescheduling_interrupts 321.33 +17.8% 378.50 interrupts.CPU56.RES:Rescheduling_interrupts 422.00 ± 33% +56.6% 661.00 ± 50% interrupts.CPU64.RES:Rescheduling_interrupts 318.00 ± 2% +12.3% 357.00 ± 8% interrupts.CPU69.RES:Rescheduling_interrupts 331.00 +13.9% 377.00 ± 9% interrupts.CPU85.RES:Rescheduling_interrupts 375.00 ± 8% -12.8% 327.00 interrupts.CPU87.RES:Rescheduling_interrupts 326.67 +26.1% 412.00 ± 15% interrupts.CPU9.RES:Rescheduling_interrupts 337.67 ± 2% +51.6% 512.00 ± 31% interrupts.CPU90.RES:Rescheduling_interrupts 47.85 -0.2 47.68 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.percpu_counter_add_batch.__do_munmap.__vm_munmap.__x64_sys_munmap 1.01 +0.1 1.07 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap 1.10 +0.1 1.16 perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 1.40 +0.1 1.48 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 95.54 -0.2 95.39 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 95.84 -0.2 95.69 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 96.62 -0.1 96.49 perf-profile.children.cycles-pp.percpu_counter_add_batch 0.08 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.13 -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.20 ± 2% -0.0 0.19 perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.14 ± 3% -0.0 0.13 perf-profile.children.cycles-pp.get_unmapped_area 0.09 ± 5% -0.0 0.08 perf-profile.children.cycles-pp.tick_sched_timer 0.08 ± 5% -0.0 0.07 perf-profile.children.cycles-pp.update_process_times 0.31 -0.0 0.30 perf-profile.children.cycles-pp.vm_area_alloc 0.18 -0.0 0.17 perf-profile.children.cycles-pp.hrtimer_interrupt 0.12 -0.0 0.11 perf-profile.children.cycles-pp.shmem_get_unmapped_area 0.06 -0.0 0.05 perf-profile.children.cycles-pp.scheduler_tick 0.08 +0.0 0.09 perf-profile.children.cycles-pp.entry_SYSCALL_64 0.06 +0.0 0.07 perf-profile.children.cycles-pp.find_vma 0.09 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.vma_link 0.09 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.d_path 0.18 ± 2% +0.0 0.20 perf-profile.children.cycles-pp.free_pgd_range 0.17 ± 2% +0.0 0.19 perf-profile.children.cycles-pp.free_p4d_range 0.30 +0.0 0.33 perf-profile.children.cycles-pp.perf_event_mmap 0.50 +0.0 0.53 perf-profile.children.cycles-pp.___might_sleep 1.10 +0.1 1.16 perf-profile.children.cycles-pp.unmap_vmas 1.07 +0.1 1.13 perf-profile.children.cycles-pp.unmap_page_range 1.40 +0.1 1.48 perf-profile.children.cycles-pp.unmap_region 95.54 -0.2 95.39 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.27 -0.0 0.26 perf-profile.self.cycles-pp.kmem_cache_alloc 0.17 +0.0 0.19 perf-profile.self.cycles-pp.free_p4d_range 0.49 +0.0 0.52 perf-profile.self.cycles-pp.unmap_page_range 0.46 +0.0 0.49 perf-profile.self.cycles-pp.___might_sleep 0.46 +0.0 0.50 perf-profile.self.cycles-pp.percpu_counter_add_batch 0.00 +0.1 0.05 perf-profile.self.cycles-pp.shmem_get_unmapped_area *************************************************************************************************** lkp-csl-2ap3: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap3/mmap2/will-it-scale/0x500002c commit: 591e8cfa8d ("percpu counter init debug") a530431919 ("new overcommit patch") 591e8cfa8d9bfbd7 a5304319194e6f6fdae62fca227 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :1 100% 1:1 kmsg.mm_compute_batch()]:vm_committed_as_batch= :1 100% 1:1 kmsg.mmap_init()]:totalram_pages *************************************************************************************************** lkp-skl-2sp7: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-skl-2sp7/mmap2/will-it-scale/0x2000065 commit: 591e8cfa8d ("percpu counter init debug") a530431919 ("new overcommit patch") 591e8cfa8d9bfbd7 a5304319194e6f6fdae62fca227 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 100% 2:2 kmsg.mm_compute_batch()]:vm_committed_as_batch= :2 100% 2:2 kmsg.mmap_init()]:totalram_pages *************************************************************************************************** lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory *************************************************************************************************** lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 48G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-bdw-de1/mmap2/will-it-scale/0x7000019 commit: 591e8cfa8d ("percpu counter init debug") a530431919 ("new overcommit patch") 591e8cfa8d9bfbd7 a5304319194e6f6fdae62fca227 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 100% 2:2 kmsg.mm_compute_batch()]:vm_committed_as_batch= :2 100% 2:2 kmsg.mmap_init()]:totalram_pages Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

8 months, 3 weeks

1
0
0 / 0

[pipe] f2af7d90e2: aim7.jobs-per-min 1827.5% improvement

by kernel test robot

Greeting, FYI, we noticed a 1827.5% improvement of aim7.jobs-per-min due to commit: commit: f2af7d90e25ce1cf03318bf3290fcfc1e762d8f9 ("pipe: read/write_iter() handler should check for IOCB_NOWAIT") url: https://github.com/0day-ci/linux/commits/UPDATE-20200501-052027/Jens-Axbo... in testcase: aim7 on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory with following parameters: disk: 4BRD_12G md: RAID0 fs: xfs test: disk_src load: 3000 cpufreq_governor: performance ucode: 0x2000065 test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | fsmark: fsmark.files_per_sec 17.1% improvement | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory | | test parameters | cpufreq_governor=performance | | | disk=1BRD_32G | | | filesize=4K | | | fs2=nfsv4 | | | fs=xfs | | | iterations=1x | | | nr_files_per_directory=1fpd | | | nr_threads=1t | | | sync_method=fsyncBeforeClose | | | test_size=4G | | | ucode=0x500002c | +------------------+-----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops 7.7% improvement | | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=100% | | | test=pthread_mutex2 | | | ucode=0x11 | +------------------+-----------------------------------------------------------------------+ | testcase: change | reaim: reaim.jobs_per_min 4.6% improvement | | test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | nr_job=1500 | | | nr_task=100% | | | runtime=300s | | | test=five_sec | | | ucode=0x43 | +------------------+-----------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.access.ops_per_sec 13.4% improvement | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory | | test parameters | class=os | | | cpufreq_governor=performance | | | disk=1HDD | | | fs=ext4 | | | nr_threads=10% | | | testtime=1s | | | ucode=0x500002c | +------------------+-----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID0/debian-x86_64-20191114.cgz/lkp-skl-2sp7/disk_src/aim7/0x2000065 commit: 1d2cc5ac6f (" RISC-V Fixes for 5.7-rc4") f2af7d90e2 ("pipe: read/write_iter() handler should check for IOCB_NOWAIT") 1d2cc5ac6f6668cc f2af7d90e25ce1cf03318bf3290 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 2:4 -50% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 2:4 -50% :4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 87440 +1827.5% 1685394 aim7.jobs-per-min 206.09 -94.7% 10.85 aim7.time.elapsed_time 206.09 -94.7% 10.85 aim7.time.elapsed_time.max 1568 +105.6% 3224 ± 3% aim7.time.involuntary_context_switches 1326 -97.1% 38.98 ± 3% aim7.time.system_time 40.41 -86.1% 5.61 aim7.time.user_time 16313528 -99.9% 14270 ± 11% aim7.time.voluntary_context_switches 8.71 -3.8 4.89 ± 3% mpstat.cpu.all.sys% 0.33 +1.8 2.11 ± 5% mpstat.cpu.all.usr% 25.71 -27.3% 18.69 ± 2% boot-time.boot 16.24 -4.1% 15.58 ± 2% boot-time.dhcp 1528 -26.2% 1128 ± 2% boot-time.idle 91.03 -100.0% 0.00 iostat.cpu.idle 8.64 -100.0% 0.00 iostat.cpu.system 481.82 -100.0% 0.00 iostat.md0.w/s 12668 -100.0% 0.00 iostat.md0.wkB/s 91.00 +2.2% 93.00 vmstat.cpu.id 12060 -99.4% 77.00 vmstat.io.bo 6.00 -50.0% 3.00 vmstat.procs.r 158518 -97.6% 3876 ± 4% vmstat.system.cs 144151 -94.8% 7556 ± 4% vmstat.system.in 354433 ± 12% -79.0% 74416 ± 61% numa-numastat.node0.local_node 354489 ± 12% -79.0% 74419 ± 61% numa-numastat.node0.numa_hit 85.00 ± 85% -91.4% 7.33 ± 70% numa-numastat.node0.other_node 296329 ± 15% -58.5% 122834 ± 35% numa-numastat.node1.local_node 319669 ± 14% -61.6% 122839 ± 35% numa-numastat.node1.numa_hit 23340 -100.0% 8.33 ±116% numa-numastat.node1.other_node 9.967e+08 ± 2% -100.0% 448009 ± 8% cpuidle.C1.time 8275047 -99.9% 11657 ± 9% cpuidle.C1.usage 9.532e+09 ± 21% -100.0% 2812312 ± 7% cpuidle.C1E.time 31069386 ± 7% -100.0% 7981 ± 5% cpuidle.C1E.usage 2.883e+09 ± 69% -71.6% 8.174e+08 cpuidle.C6.time 4023285 ± 64% -99.2% 30952 ± 9% cpuidle.C6.usage 1080693 -99.1% 9548 ± 5% cpuidle.POLL.time 103408 -96.3% 3837 ± 6% cpuidle.POLL.usage 474900 -39.3% 288452 ± 2% meminfo.Active 474725 -39.3% 288278 ± 2% meminfo.Active(anon) 154654 -92.6% 11447 ± 19% meminfo.AnonHugePages 459216 -37.3% 288133 ± 2% meminfo.AnonPages 2799896 -17.6% 2306199 meminfo.Committed_AS 9060352 ± 6% -24.2% 6869149 ± 7% meminfo.DirectMap2M 580972 ± 7% -13.9% 500259 ± 4% meminfo.DirectMap4k 1395 -20.0% 1116 ± 5% meminfo.Inactive(file) 127199 -49.0% 64816 meminfo.KReclaimable 60367 -18.3% 49319 meminfo.KernelStack 26849 -24.5% 20279 meminfo.Mapped 2654090 -11.1% 2358259 meminfo.Memused 1359 -100.0% 0.00 meminfo.Mlocked 104337 ± 5% -21.8% 81613 ± 4% meminfo.PageTables 127199 -49.0% 64816 meminfo.SReclaimable 251585 -19.8% 201824 meminfo.SUnreclaim 33971 -46.3% 18253 meminfo.Shmem 378784 -29.6% 266641 meminfo.Slab 13407 +1319.0% 190257 meminfo.max_used_kB 236497 ± 25% -42.8% 135337 ± 5% numa-meminfo.node0.Active 236417 ± 25% -42.8% 135258 ± 5% numa-meminfo.node0.Active(anon) 231908 ± 26% -41.7% 135288 ± 5% numa-meminfo.node0.AnonPages 10173 ± 12% -71.2% 2933 ± 66% numa-meminfo.node0.Inactive 9479 ± 7% -73.5% 2513 ± 84% numa-meminfo.node0.Inactive(anon) 75684 ± 2% -62.6% 28295 ± 5% numa-meminfo.node0.KReclaimable 30878 ± 76% -39.9% 18570 ± 96% numa-meminfo.node0.KernelStack 13192 ± 18% -41.7% 7691 ± 2% numa-meminfo.node0.Mapped 1338915 ± 11% -22.1% 1042914 ± 12% numa-meminfo.node0.MemUsed 75684 ± 2% -62.6% 28295 ± 5% numa-meminfo.node0.SReclaimable 129054 ± 34% -32.2% 87508 ± 37% numa-meminfo.node0.SUnreclaim 14490 ± 4% -81.0% 2747 ± 82% numa-meminfo.node0.Shmem 204739 ± 22% -43.4% 115804 ± 29% numa-meminfo.node0.Slab 238359 ± 27% -35.7% 153233 ± 6% numa-meminfo.node1.Active 238263 ± 27% -35.7% 153137 ± 6% numa-meminfo.node1.Active(anon) 227239 ± 29% -32.7% 152964 ± 7% numa-meminfo.node1.AnonPages 8875 ± 13% +76.6% 15671 ± 11% numa-meminfo.node1.Inactive 8173 ± 8% +83.2% 14975 ± 13% numa-meminfo.node1.Inactive(anon) 51512 ± 3% -29.1% 36521 ± 4% numa-meminfo.node1.KReclaimable 51512 ± 3% -29.1% 36521 ± 4% numa-meminfo.node1.SReclaimable 19480 ± 4% -20.3% 15517 ± 14% numa-meminfo.node1.Shmem 59066 ± 25% -42.8% 33814 ± 5% numa-vmstat.node0.nr_active_anon 57939 ± 26% -41.6% 33821 ± 5% numa-vmstat.node0.nr_anon_pages 2369 ± 7% -73.5% 627.67 ± 84% numa-vmstat.node0.nr_inactive_anon 30837 ± 76% -39.8% 18570 ± 96% numa-vmstat.node0.nr_kernel_stack 3299 ± 18% -41.7% 1923 ± 2% numa-vmstat.node0.nr_mapped 196.50 -100.0% 0.00 numa-vmstat.node0.nr_mlock 3622 ± 4% -81.1% 686.33 ± 82% numa-vmstat.node0.nr_shmem 18903 ± 2% -62.6% 7073 ± 5% numa-vmstat.node0.nr_slab_reclaimable 32248 ± 34% -32.2% 21877 ± 37% numa-vmstat.node0.nr_slab_unreclaimable 59066 ± 25% -42.8% 33814 ± 5% numa-vmstat.node0.nr_zone_active_anon 2369 ± 7% -73.5% 627.67 ± 84% numa-vmstat.node0.nr_zone_inactive_anon 678622 ± 6% -32.4% 458573 ± 7% numa-vmstat.node0.numa_hit 606314 ± 19% -48.3% 313744 ± 10% numa-vmstat.node0.numa_local 59525 ± 27% -35.7% 38284 ± 6% numa-vmstat.node1.nr_active_anon 56764 ± 29% -32.6% 38237 ± 7% numa-vmstat.node1.nr_anon_pages 2042 ± 8% +83.3% 3743 ± 13% numa-vmstat.node1.nr_inactive_anon 142.50 -100.0% 0.00 numa-vmstat.node1.nr_mlock 4870 ± 4% -20.3% 3879 ± 14% numa-vmstat.node1.nr_shmem 12866 ± 3% -29.0% 9129 ± 4% numa-vmstat.node1.nr_slab_reclaimable 59525 ± 27% -35.7% 38284 ± 6% numa-vmstat.node1.nr_zone_active_anon 2042 ± 8% +83.3% 3743 ± 13% numa-vmstat.node1.nr_zone_inactive_anon 716586 ± 5% -21.6% 561911 ± 7% numa-vmstat.node1.numa_hit 110633 ± 65% -86.5% 14896 numa-vmstat.node1.numa_other 118655 -39.3% 72066 ± 2% proc-vmstat.nr_active_anon 114768 -37.2% 72037 ± 2% proc-vmstat.nr_anon_pages 314.00 -2.8% 305.33 proc-vmstat.nr_dirtied 284356 -2.4% 277610 proc-vmstat.nr_file_pages 4413 -0.9% 4371 proc-vmstat.nr_inactive_anon 347.50 -19.8% 278.67 ± 5% proc-vmstat.nr_inactive_file 60328 -18.2% 49319 proc-vmstat.nr_kernel_stack 6838 -25.9% 5069 proc-vmstat.nr_mapped 339.50 -100.0% 0.00 proc-vmstat.nr_mlock 26064 ± 5% -21.7% 20403 ± 4% proc-vmstat.nr_page_table_pages 8492 -46.3% 4563 proc-vmstat.nr_shmem 31795 -49.0% 16203 proc-vmstat.nr_slab_reclaimable 62885 -19.8% 50455 proc-vmstat.nr_slab_unreclaimable 275480 -1.0% 272716 proc-vmstat.nr_unevictable 313.00 -4.5% 299.00 proc-vmstat.nr_written 118655 -39.3% 72066 ± 2% proc-vmstat.nr_zone_active_anon 4413 -0.9% 4371 proc-vmstat.nr_zone_inactive_anon 347.50 -19.8% 278.67 ± 5% proc-vmstat.nr_zone_inactive_file 275480 -1.0% 272716 proc-vmstat.nr_zone_unevictable 690747 -67.4% 225063 proc-vmstat.numa_hit 667314 -66.3% 225040 proc-vmstat.numa_local 23432 -99.9% 23.67 ± 50% proc-vmstat.numa_other 344.00 ± 16% +609.0% 2439 ± 31% proc-vmstat.numa_pages_migrated 29944 -59.8% 12027 ± 96% proc-vmstat.numa_pte_updates 3082 ± 3% -99.7% 10.00 ± 32% proc-vmstat.pgactivate 859080 -73.6% 227001 proc-vmstat.pgalloc_normal 717981 -70.0% 215317 ± 2% proc-vmstat.pgfault 814095 -72.4% 224608 ± 5% proc-vmstat.pgfree 344.00 ± 16% +609.0% 2439 ± 31% proc-vmstat.pgmigrate_success 2526720 -100.0% 1068 proc-vmstat.pgpgout 1839 -19.0% 1490 proc-vmstat.unevictable_pgs_culled 46.88 -46.9 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64 46.05 -46.0 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 46.05 -46.0 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 46.03 -46.0 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 44.09 -44.1 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 43.92 -43.9 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 42.29 -42.3 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 28.38 -28.4 0.00 perf-profile.calltrace.cycles-pp.unlink 28.18 -28.2 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink 28.16 -28.2 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 27.95 -28.0 0.00 perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 22.09 ± 2% -22.1 0.00 perf-profile.calltrace.cycles-pp.creat 21.88 ± 3% -21.9 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat 21.87 ± 3% -21.9 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat 21.73 ± 3% -21.7 0.00 perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat 21.73 ± 3% -21.7 0.00 perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat 21.52 ± 3% -21.5 0.00 perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 21.50 ± 3% -21.5 0.00 perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64 13.54 -13.5 0.00 perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open 13.43 -13.4 0.00 perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2 11.44 -11.4 0.00 perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 11.34 -11.3 0.00 perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.31 -11.3 0.00 perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64 8.54 -8.5 0.00 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 7.79 -7.8 0.00 perf-profile.calltrace.cycles-pp.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat 7.65 -7.7 0.00 perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.96 -7.0 0.00 perf-profile.calltrace.cycles-pp.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 6.78 -6.8 0.00 perf-profile.calltrace.cycles-pp.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.72 -6.7 0.00 perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64 6.71 -6.7 0.00 perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat 6.60 ± 7% -6.6 0.00 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2.do_sys_open 6.46 -6.5 0.00 perf-profile.calltrace.cycles-pp.xfs_dir2_block_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink 5.67 ± 8% -5.7 0.00 perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2 5.17 -5.2 0.00 perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64 51.59 -51.6 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 51.56 -51.6 0.00 perf-profile.children.cycles-pp.do_syscall_64 46.89 -46.9 0.00 perf-profile.children.cycles-pp.do_idle 46.88 -46.9 0.00 perf-profile.children.cycles-pp.secondary_startup_64 46.88 -46.9 0.00 perf-profile.children.cycles-pp.cpu_startup_entry 46.05 -46.0 0.00 perf-profile.children.cycles-pp.start_secondary 44.88 -44.9 0.00 perf-profile.children.cycles-pp.cpuidle_enter 44.88 -44.9 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state 42.29 -42.3 0.00 perf-profile.children.cycles-pp.intel_idle 28.41 -28.4 0.00 perf-profile.children.cycles-pp.unlink 27.96 -28.0 0.00 perf-profile.children.cycles-pp.do_unlinkat 22.12 ± 2% -22.1 0.00 perf-profile.children.cycles-pp.creat 21.76 ± 3% -21.8 0.00 perf-profile.children.cycles-pp.do_sys_open 21.75 ± 3% -21.8 0.00 perf-profile.children.cycles-pp.do_sys_openat2 21.54 ± 3% -21.5 0.00 perf-profile.children.cycles-pp.do_filp_open 21.52 ± 3% -21.5 0.00 perf-profile.children.cycles-pp.path_openat 15.14 ± 3% -15.1 0.00 perf-profile.children.cycles-pp.rwsem_down_write_slowpath 13.54 -13.5 0.00 perf-profile.children.cycles-pp.xfs_generic_create 13.43 -13.4 0.00 perf-profile.children.cycles-pp.xfs_create 13.36 ± 3% -13.4 0.00 perf-profile.children.cycles-pp.rwsem_optimistic_spin 11.44 -11.4 0.00 perf-profile.children.cycles-pp.vfs_unlink 11.34 -11.3 0.00 perf-profile.children.cycles-pp.xfs_vn_unlink 11.32 -11.3 0.00 perf-profile.children.cycles-pp.xfs_remove 10.07 -10.1 0.00 perf-profile.children.cycles-pp.rwsem_spin_on_owner 9.54 ± 2% -9.5 0.00 perf-profile.children.cycles-pp.xfs_dir3_data_check 9.54 ± 2% -9.5 0.00 perf-profile.children.cycles-pp.__xfs_dir3_data_check 8.43 -8.4 0.00 perf-profile.children.cycles-pp.__xfs_trans_commit 8.12 -8.1 0.00 perf-profile.children.cycles-pp.xfs_log_commit_cil 7.79 -7.8 0.00 perf-profile.children.cycles-pp.xfs_dir_removename 6.96 -7.0 0.00 perf-profile.children.cycles-pp.destroy_inode 6.78 -6.8 0.00 perf-profile.children.cycles-pp.xfs_fs_destroy_inode 6.72 -6.7 0.00 perf-profile.children.cycles-pp.xfs_inactive 6.71 -6.7 0.00 perf-profile.children.cycles-pp.xfs_inactive_ifree 6.46 -6.5 0.00 perf-profile.children.cycles-pp.xfs_dir2_block_removename 42.28 -42.3 0.00 perf-profile.self.cycles-pp.intel_idle 10.02 -10.0 0.00 perf-profile.self.cycles-pp.rwsem_spin_on_owner 5.12 -5.1 0.00 perf-profile.self.cycles-pp.__xfs_dir3_data_check 8156 -100.0% 0.01 ±141% sched_debug.cfs_rq:/.exec_clock.avg 10624 -100.0% 0.15 ±141% sched_debug.cfs_rq:/.exec_clock.max 7473 -100.0% 0.00 sched_debug.cfs_rq:/.exec_clock.min 480.00 -100.0% 0.02 ±141% sched_debug.cfs_rq:/.exec_clock.stddev 32.59 ± 8% +209.4% 100.81 ± 17% sched_debug.cfs_rq:/.load_avg.avg 393.50 ± 7% +246.9% 1365 ± 31% sched_debug.cfs_rq:/.load_avg.max 84.75 ± 7% +245.2% 292.54 sched_debug.cfs_rq:/.load_avg.stddev 79414 -85.3% 11665 ± 18% sched_debug.cfs_rq:/.min_vruntime.avg 150905 ± 10% -85.1% 22451 ± 17% sched_debug.cfs_rq:/.min_vruntime.max 68449 -90.0% 6860 ± 11% sched_debug.cfs_rq:/.min_vruntime.min 12210 ± 7% -77.2% 2788 ± 21% sched_debug.cfs_rq:/.min_vruntime.stddev 0.13 ± 6% +55.0% 0.21 ± 5% sched_debug.cfs_rq:/.nr_running.avg 0.34 +20.7% 0.41 sched_debug.cfs_rq:/.nr_running.stddev 0.30 ± 15% -100.0% 0.00 sched_debug.cfs_rq:/.nr_spread_over.avg 1.50 -100.0% 0.00 sched_debug.cfs_rq:/.nr_spread_over.max 0.46 ± 3% -100.0% 0.00 sched_debug.cfs_rq:/.nr_spread_over.stddev 8.76 ± 19% +430.6% 46.48 ± 37% sched_debug.cfs_rq:/.removed.load_avg.avg 254.25 +302.2% 1022 sched_debug.cfs_rq:/.removed.load_avg.max 45.79 ± 10% +350.3% 206.19 ± 17% sched_debug.cfs_rq:/.removed.load_avg.stddev 8.76 ± 19% +430.5% 46.48 ± 37% sched_debug.cfs_rq:/.removed.runnable_avg.avg 254.25 +302.2% 1022 sched_debug.cfs_rq:/.removed.runnable_avg.max 45.79 ± 10% +350.3% 206.19 ± 17% sched_debug.cfs_rq:/.removed.runnable_avg.stddev 2.04 ± 7% +681.9% 15.99 ± 42% sched_debug.cfs_rq:/.removed.util_avg.avg 98.25 ± 25% +387.2% 478.67 ± 10% sched_debug.cfs_rq:/.removed.util_avg.max 12.61 ± 18% +511.2% 77.05 ± 27% sched_debug.cfs_rq:/.removed.util_avg.stddev 313.68 +194.9% 925.14 ± 4% sched_debug.cfs_rq:/.runnable_avg.avg 1923 ± 10% +155.2% 4908 ± 21% sched_debug.cfs_rq:/.runnable_avg.max 15.88 ± 19% -100.0% 0.00 sched_debug.cfs_rq:/.runnable_avg.min 349.97 ± 12% +196.8% 1038 ± 16% sched_debug.cfs_rq:/.runnable_avg.stddev 69980 ± 32% -83.5% 11549 ± 46% sched_debug.cfs_rq:/.spread0.max 12211 ± 7% -77.2% 2788 ± 21% sched_debug.cfs_rq:/.spread0.stddev 156.22 +107.6% 324.35 ± 6% sched_debug.cfs_rq:/.util_avg.avg 948.62 +69.3% 1605 ± 27% sched_debug.cfs_rq:/.util_avg.max 15.50 ± 12% -100.0% 0.00 sched_debug.cfs_rq:/.util_avg.min 152.10 +134.7% 357.06 ± 13% sched_debug.cfs_rq:/.util_avg.stddev 276.50 ± 12% +86.0% 514.33 sched_debug.cfs_rq:/.util_est_enqueued.max 43.73 ± 12% +98.2% 86.67 ± 16% sched_debug.cfs_rq:/.util_est_enqueued.stddev 534104 +21.0% 646023 ± 2% sched_debug.cpu.avg_idle.avg 119229 -81.3% 22279 ± 2% sched_debug.cpu.clock.avg 119252 -81.3% 22283 ± 2% sched_debug.cpu.clock.max 119205 -81.3% 22276 ± 2% sched_debug.cpu.clock.min 13.37 ± 13% -85.7% 1.91 ± 3% sched_debug.cpu.clock.stddev 119229 -81.3% 22279 ± 2% sched_debug.cpu.clock_task.avg 119252 -81.3% 22283 ± 2% sched_debug.cpu.clock_task.max 119205 -81.3% 22276 ± 2% sched_debug.cpu.clock_task.min 13.37 ± 13% -85.7% 1.91 ± 3% sched_debug.cpu.clock_task.stddev 451.28 -25.7% 335.13 ± 14% sched_debug.cpu.curr->pid.avg 6692 -71.2% 1927 sched_debug.cpu.curr->pid.max 1274 -46.9% 677.01 ± 8% sched_debug.cpu.curr->pid.stddev 0.14 ± 7% +53.8% 0.21 ± 10% sched_debug.cpu.nr_running.avg 0.33 +21.2% 0.41 ± 4% sched_debug.cpu.nr_running.stddev 194669 -99.3% 1434 sched_debug.cpu.nr_switches.avg 216006 -97.1% 6225 ± 14% sched_debug.cpu.nr_switches.max 179904 -99.7% 507.67 ± 27% sched_debug.cpu.nr_switches.min 6589 ± 6% -86.7% 878.98 ± 9% sched_debug.cpu.nr_switches.stddev 31.18 -99.9% 0.03 ± 20% sched_debug.cpu.nr_uninterruptible.avg 86.62 ± 11% -41.9% 50.33 ± 19% sched_debug.cpu.nr_uninterruptible.max -20.88 +155.5% -53.33 sched_debug.cpu.nr_uninterruptible.min 20.65 ± 10% -38.9% 12.61 ± 4% sched_debug.cpu.nr_uninterruptible.stddev 193037 -100.0% 2.11 ±141% sched_debug.cpu.sched_count.avg 212309 -100.0% 28.33 ±141% sched_debug.cpu.sched_count.max 178699 -100.0% 0.00 sched_debug.cpu.sched_count.min 6240 ± 5% -99.9% 7.03 ±141% sched_debug.cpu.sched_count.stddev 95616 -100.0% 1.11 ±141% sched_debug.cpu.sched_goidle.avg 105396 -100.0% 14.67 ±141% sched_debug.cpu.sched_goidle.max 88201 -100.0% 0.00 sched_debug.cpu.sched_goidle.min 3192 ± 5% -99.9% 3.63 ±141% sched_debug.cpu.sched_goidle.stddev 105622 -100.0% 0.09 ±141% sched_debug.cpu.ttwu_count.avg 115490 -100.0% 1.00 ±141% sched_debug.cpu.ttwu_count.max 98571 -100.0% 0.00 sched_debug.cpu.ttwu_count.min 3458 ± 5% -100.0% 0.26 ±141% sched_debug.cpu.ttwu_count.stddev 1758 -100.0% 0.04 ±141% sched_debug.cpu.ttwu_local.avg 2638 -100.0% 0.67 ±141% sched_debug.cpu.ttwu_local.max 1167 -100.0% 0.00 sched_debug.cpu.ttwu_local.min 338.61 -100.0% 0.17 ±141% sched_debug.cpu.ttwu_local.stddev 119205 -81.3% 22277 ± 2% sched_debug.cpu_clk 118607 -81.7% 21744 ± 2% sched_debug.ktime 119626 -81.0% 22677 ± 2% sched_debug.sched_clk 6266 ± 3% -38.1% 3876 ± 3% slabinfo.Acpi-State.active_objs 6266 ± 3% -38.1% 3876 ± 3% slabinfo.Acpi-State.num_objs 431.00 ± 14% -17.4% 356.00 ± 12% slabinfo.UDP.active_objs 431.00 ± 14% -17.4% 356.00 ± 12% slabinfo.UDP.num_objs 1550 ± 10% -11.3% 1374 ± 10% slabinfo.UNIX.active_objs 1550 ± 10% -11.3% 1374 ± 10% slabinfo.UNIX.num_objs 61966 -14.3% 53095 slabinfo.anon_vma.active_objs 1346 -14.3% 1153 slabinfo.anon_vma.active_slabs 61968 -14.3% 53103 slabinfo.anon_vma.num_objs 1346 -14.3% 1153 slabinfo.anon_vma.num_slabs 129767 -15.7% 109368 slabinfo.anon_vma_chain.active_objs 2028 -15.7% 1709 slabinfo.anon_vma_chain.active_slabs 129841 -15.7% 109429 slabinfo.anon_vma_chain.num_objs 2028 -15.7% 1709 slabinfo.anon_vma_chain.num_slabs 2687 -70.8% 785.00 slabinfo.blkdev_ioc.active_objs 2687 -70.8% 785.00 slabinfo.blkdev_ioc.num_objs 230175 -66.3% 77510 slabinfo.dentry.active_objs 5549 -66.7% 1846 slabinfo.dentry.active_slabs 233080 -66.7% 77546 slabinfo.dentry.num_objs 5549 -66.7% 1846 slabinfo.dentry.num_slabs 3640 -15.9% 3061 ± 2% slabinfo.eventpoll_pwq.active_objs 3640 -15.9% 3061 ± 2% slabinfo.eventpoll_pwq.num_objs 6396 -11.5% 5657 ± 2% slabinfo.files_cache.active_objs 6396 -11.5% 5657 ± 2% slabinfo.files_cache.num_objs 2368 ± 2% -17.1% 1962 ± 8% slabinfo.fsnotify_mark_connector.active_objs 2368 ± 2% -17.1% 1962 ± 8% slabinfo.fsnotify_mark_connector.num_objs 2993 -94.7% 160.00 ± 7% slabinfo.ip6-frags.active_objs 2993 -94.7% 160.00 ± 7% slabinfo.ip6-frags.num_objs 809.00 ± 2% -61.4% 312.00 ± 10% slabinfo.khugepaged_mm_slot.active_objs 809.00 ± 2% -61.4% 312.00 ± 10% slabinfo.khugepaged_mm_slot.num_objs 5249 ± 3% -32.5% 3545 ± 5% slabinfo.kmalloc-2k.active_objs 330.50 ± 2% -32.7% 222.33 ± 5% slabinfo.kmalloc-2k.active_slabs 5290 ± 2% -32.7% 3561 ± 5% slabinfo.kmalloc-2k.num_objs 330.50 ± 2% -32.7% 222.33 ± 5% slabinfo.kmalloc-2k.num_slabs 1166 -9.0% 1061 slabinfo.kmalloc-4k.active_objs 1205 -12.0% 1061 slabinfo.kmalloc-4k.num_objs 15704 -53.1% 7373 ± 6% slabinfo.kmalloc-512.active_objs 495.00 -53.5% 230.33 ± 6% slabinfo.kmalloc-512.active_slabs 15846 -53.4% 7391 ± 6% slabinfo.kmalloc-512.num_objs 495.00 -53.5% 230.33 ± 6% slabinfo.kmalloc-512.num_slabs 1517 -22.1% 1181 slabinfo.kmalloc-8k.active_objs 1538 -23.2% 1181 slabinfo.kmalloc-8k.num_objs 960.00 ± 3% -27.8% 693.33 ± 21% slabinfo.kmalloc-rcl-128.active_objs 960.00 ± 3% -27.8% 693.33 ± 21% slabinfo.kmalloc-rcl-128.num_objs 4014 ± 6% -12.3% 3520 ± 7% slabinfo.kmalloc-rcl-64.active_objs 4014 ± 6% -12.3% 3520 ± 7% slabinfo.kmalloc-rcl-64.num_objs 2415 -17.7% 1988 ± 14% slabinfo.kmalloc-rcl-96.active_objs 2415 -17.7% 1988 ± 14% slabinfo.kmalloc-rcl-96.num_objs 3234 -82.2% 576.00 slabinfo.names_cache.active_objs 408.00 -82.4% 72.00 slabinfo.names_cache.active_slabs 3264 -82.4% 576.00 slabinfo.names_cache.num_objs 408.00 -82.4% 72.00 slabinfo.names_cache.num_slabs 192.50 -65.7% 66.00 slabinfo.nfs_read_data.active_objs 192.50 -65.7% 66.00 slabinfo.nfs_read_data.num_objs 6146 -12.3% 5393 slabinfo.pid.active_objs 6146 -12.1% 5404 slabinfo.pid.num_objs 2253 -52.3% 1075 ± 7% slabinfo.pool_workqueue.active_objs 2253 -51.8% 1085 ± 8% slabinfo.pool_workqueue.num_objs 39773 -82.5% 6960 slabinfo.proc_inode_cache.active_objs 832.00 -82.6% 144.67 slabinfo.proc_inode_cache.active_slabs 39956 -82.5% 6974 slabinfo.proc_inode_cache.num_objs 832.00 -82.6% 144.67 slabinfo.proc_inode_cache.num_slabs 4722 -15.8% 3977 slabinfo.sighand_cache.active_objs 4733 -15.2% 4012 slabinfo.sighand_cache.num_objs 5774 -14.1% 4958 slabinfo.signal_cache.active_objs 5787 -14.1% 4968 slabinfo.signal_cache.num_objs 2935 -41.6% 1715 ± 9% slabinfo.skbuff_ext_cache.active_objs 2935 -41.6% 1715 ± 9% slabinfo.skbuff_ext_cache.num_objs 2481 ± 4% -7.8% 2289 ± 4% slabinfo.sock_inode_cache.active_objs 2481 ± 4% -7.8% 2289 ± 4% slabinfo.sock_inode_cache.num_objs 7179 -10.6% 6415 ± 2% slabinfo.task_delay_info.active_objs 7179 -10.4% 6433 ± 2% slabinfo.task_delay_info.num_objs 1223 ± 2% -25.9% 906.33 ± 21% slabinfo.task_group.active_objs 1223 ± 2% -25.9% 906.33 ± 21% slabinfo.task_group.num_objs 3967 -17.6% 3268 slabinfo.task_struct.active_objs 3972 -17.6% 3273 slabinfo.task_struct.active_slabs 3972 -17.6% 3273 slabinfo.task_struct.num_objs 3972 -17.6% 3273 slabinfo.task_struct.num_slabs 109513 -18.3% 89433 slabinfo.vm_area_struct.active_objs 2739 -18.3% 2239 slabinfo.vm_area_struct.active_slabs 109607 -18.3% 89586 slabinfo.vm_area_struct.num_objs 2739 -18.3% 2239 slabinfo.vm_area_struct.num_slabs 15602 -24.6% 11757 ± 2% slabinfo.vmap_area.active_objs 15609 -24.2% 11837 ± 2% slabinfo.vmap_area.num_objs 2451 -94.0% 146.00 ± 10% slabinfo.xfs_btree_cur.active_objs 2451 -94.0% 146.00 ± 10% slabinfo.xfs_btree_cur.num_objs 2980 -90.0% 299.00 ± 6% slabinfo.xfs_buf.active_objs 2980 -90.0% 299.00 ± 6% slabinfo.xfs_buf.num_objs 4197 -94.1% 246.00 ± 9% slabinfo.xfs_buf_item.active_objs 4197 -94.1% 246.00 ± 9% slabinfo.xfs_buf_item.num_objs 2306 -100.0% 0.00 slabinfo.xfs_da_state.active_objs 2306 -100.0% 0.00 slabinfo.xfs_da_state.num_objs 2507 -99.7% 8.00 slabinfo.xfs_efd_item.active_objs 2507 -99.7% 8.00 slabinfo.xfs_efd_item.num_objs 3802 ± 2% -98.1% 71.00 slabinfo.xfs_ili.active_objs 132.00 ± 3% -99.2% 1.00 slabinfo.xfs_ili.active_slabs 5574 ± 2% -98.7% 71.00 slabinfo.xfs_ili.num_objs 132.00 ± 3% -99.2% 1.00 slabinfo.xfs_ili.num_slabs 3030 ± 3% -97.2% 86.00 slabinfo.xfs_inode.active_objs 154.50 ± 6% -98.7% 2.00 slabinfo.xfs_inode.active_slabs 4955 ± 6% -98.3% 86.00 slabinfo.xfs_inode.num_objs 154.50 ± 6% -98.7% 2.00 slabinfo.xfs_inode.num_slabs 57460 -97.9% 1201 ± 5% softirqs.CPU0.RCU 37265 ± 2% -82.1% 6683 ± 13% softirqs.CPU0.SCHED 78585 ± 3% -90.0% 7897 ± 2% softirqs.CPU0.TIMER 58357 -98.2% 1029 ± 10% softirqs.CPU1.RCU 32437 -93.1% 2230 ± 7% softirqs.CPU1.SCHED 70585 ± 2% -90.5% 6677 ± 2% softirqs.CPU1.TIMER 58282 -98.1% 1093 ± 18% softirqs.CPU10.RCU 30456 -94.5% 1680 ± 2% softirqs.CPU10.SCHED 69680 ± 3% -90.4% 6682 ± 5% softirqs.CPU10.TIMER 58216 -97.6% 1393 ± 37% softirqs.CPU11.RCU 29848 ± 3% -94.1% 1746 ± 7% softirqs.CPU11.SCHED 69667 ± 3% -89.3% 7454 ± 19% softirqs.CPU11.TIMER 56837 -98.3% 950.33 ± 3% softirqs.CPU12.RCU 30525 -94.4% 1697 ± 4% softirqs.CPU12.SCHED 69352 ± 3% -90.2% 6792 ± 5% softirqs.CPU12.TIMER 57987 -98.2% 1021 ± 6% softirqs.CPU13.RCU 30863 -94.7% 1638 ± 3% softirqs.CPU13.SCHED 72585 ± 2% -90.9% 6633 ± 4% softirqs.CPU13.TIMER 57156 -98.3% 973.67 ± 7% softirqs.CPU14.RCU 30101 ± 3% -94.6% 1637 ± 6% softirqs.CPU14.SCHED 68865 ± 2% -90.6% 6486 ± 3% softirqs.CPU14.TIMER 53550 -98.3% 890.33 ± 6% softirqs.CPU15.RCU 30427 ± 2% -95.0% 1531 ± 3% softirqs.CPU15.SCHED 69398 ± 3% -91.1% 6156 ± 4% softirqs.CPU15.TIMER 54388 -98.0% 1070 ± 17% softirqs.CPU16.RCU 30856 -94.8% 1598 softirqs.CPU16.SCHED 71431 ± 4% -90.6% 6689 ± 9% softirqs.CPU16.TIMER 54631 -98.4% 859.33 ± 2% softirqs.CPU17.RCU 30070 ± 5% -94.4% 1683 ± 4% softirqs.CPU17.SCHED 71492 ± 4% -91.1% 6393 softirqs.CPU17.TIMER 60133 -97.1% 1721 ± 11% softirqs.CPU18.RCU 30915 ± 2% -92.3% 2369 ± 19% softirqs.CPU18.SCHED 72602 ± 4% -88.9% 8038 ± 7% softirqs.CPU18.TIMER 59867 -98.3% 1041 ± 11% softirqs.CPU19.RCU 30788 ± 2% -94.7% 1626 ± 2% softirqs.CPU19.SCHED 70410 ± 2% -91.0% 6353 softirqs.CPU19.TIMER 57519 -97.9% 1189 ± 26% softirqs.CPU2.RCU 30786 -94.3% 1745 ± 11% softirqs.CPU2.SCHED 70578 ± 3% -90.7% 6550 ± 3% softirqs.CPU2.TIMER 63275 ± 3% -95.4% 2900 ± 90% softirqs.CPU20.RCU 30631 ± 4% -95.1% 1503 ± 4% softirqs.CPU20.SCHED 74363 ± 4% -89.7% 7669 ± 24% softirqs.CPU20.TIMER 60808 -98.0% 1246 ± 24% softirqs.CPU21.RCU 30850 ± 2% -94.1% 1806 ± 6% softirqs.CPU21.SCHED 72485 ± 4% -90.7% 6776 ± 4% softirqs.CPU21.TIMER 60722 -98.3% 1061 ± 8% softirqs.CPU22.RCU 30814 -94.4% 1729 ± 3% softirqs.CPU22.SCHED 71788 ± 3% -90.6% 6770 ± 6% softirqs.CPU22.TIMER 58879 -98.2% 1064 ± 9% softirqs.CPU23.RCU 29549 ± 2% -94.3% 1686 ± 3% softirqs.CPU23.SCHED 68845 ± 3% -90.5% 6521 ± 4% softirqs.CPU23.TIMER 60792 -98.2% 1092 ± 18% softirqs.CPU24.RCU 29877 ± 5% -94.5% 1644 ± 4% softirqs.CPU24.SCHED 73313 ± 5% -91.3% 6391 ± 3% softirqs.CPU24.TIMER 60317 -98.0% 1194 ± 18% softirqs.CPU25.RCU 29960 ± 3% -94.3% 1709 ± 7% softirqs.CPU25.SCHED 70967 ± 4% -90.6% 6690 ± 3% softirqs.CPU25.TIMER 60792 -97.7% 1396 ± 9% softirqs.CPU26.RCU 30755 ± 2% -93.7% 1947 ± 4% softirqs.CPU26.SCHED 72943 ± 5% -89.0% 8052 ± 9% softirqs.CPU26.TIMER 60220 -97.4% 1558 ± 8% softirqs.CPU27.RCU 30819 ± 2% -93.9% 1894 ± 4% softirqs.CPU27.SCHED 70930 ± 3% -89.6% 7374 ± 8% softirqs.CPU27.TIMER 59908 -97.8% 1341 ± 14% softirqs.CPU28.RCU 30637 ± 3% -94.1% 1815 ± 8% softirqs.CPU28.SCHED 70661 ± 3% -90.3% 6864 ± 3% softirqs.CPU28.TIMER 60192 -97.8% 1335 ± 16% softirqs.CPU29.RCU 30511 ± 2% -94.2% 1775 ± 9% softirqs.CPU29.SCHED 70274 ± 4% -89.7% 7205 ± 6% softirqs.CPU29.TIMER 57876 -97.9% 1215 ± 20% softirqs.CPU3.RCU 30541 -94.2% 1772 ± 4% softirqs.CPU3.SCHED 72866 ± 2% -89.1% 7954 ± 19% softirqs.CPU3.TIMER 58189 -98.2% 1024 ± 20% softirqs.CPU30.RCU 30405 -94.2% 1754 ± 11% softirqs.CPU30.SCHED 71220 ± 3% -90.6% 6689 ± 12% softirqs.CPU30.TIMER 53534 ± 8% -97.4% 1386 ± 21% softirqs.CPU31.RCU 31001 ± 2% -93.8% 1918 ± 9% softirqs.CPU31.SCHED 74294 ± 7% -90.4% 7168 ± 9% softirqs.CPU31.TIMER 58824 -98.1% 1109 ± 11% softirqs.CPU32.RCU 30460 -94.2% 1764 ± 5% softirqs.CPU32.SCHED 70565 ± 3% -88.6% 8009 ± 8% softirqs.CPU32.TIMER 58496 -98.3% 1019 ± 10% softirqs.CPU33.RCU 30425 ± 2% -94.1% 1788 ± 7% softirqs.CPU33.SCHED 70479 ± 4% -89.6% 7331 ± 16% softirqs.CPU33.TIMER 58383 -98.2% 1078 ± 17% softirqs.CPU34.RCU 30768 ± 2% -93.9% 1884 ± 6% softirqs.CPU34.SCHED 71209 ± 4% -90.2% 6972 ± 6% softirqs.CPU34.TIMER 58308 -98.3% 967.67 ± 11% softirqs.CPU35.RCU 30339 ± 3% -94.2% 1746 ± 4% softirqs.CPU35.SCHED 70327 ± 4% -90.4% 6738 ± 4% softirqs.CPU35.TIMER 61408 -98.5% 898.67 ± 21% softirqs.CPU36.RCU 30130 -95.1% 1477 ± 7% softirqs.CPU36.SCHED 69424 ± 2% -91.0% 6246 ± 5% softirqs.CPU36.TIMER 59793 -98.5% 880.67 ± 7% softirqs.CPU37.RCU 29924 ± 2% -94.5% 1657 ± 4% softirqs.CPU37.SCHED 66607 ± 2% -90.5% 6298 ± 3% softirqs.CPU37.TIMER 59960 -98.7% 778.33 ± 4% softirqs.CPU38.RCU 29533 -94.7% 1578 softirqs.CPU38.SCHED 67446 ± 3% -91.0% 6063 ± 2% softirqs.CPU38.TIMER 61099 -98.6% 877.33 ± 8% softirqs.CPU39.RCU 30593 -94.4% 1704 ± 6% softirqs.CPU39.SCHED 69728 -90.5% 6602 ± 3% softirqs.CPU39.TIMER 57188 -98.1% 1069 ± 6% softirqs.CPU4.RCU 30620 -94.1% 1820 ± 5% softirqs.CPU4.SCHED 70913 ± 3% -90.4% 6837 ± 5% softirqs.CPU4.TIMER 61035 -98.5% 905.00 ± 2% softirqs.CPU40.RCU 29670 -94.7% 1580 ± 4% softirqs.CPU40.SCHED 66764 ± 2% -90.9% 6054 ± 5% softirqs.CPU40.TIMER 62438 ± 5% -98.6% 859.00 ± 10% softirqs.CPU41.RCU 29614 -94.1% 1733 ± 7% softirqs.CPU41.SCHED 67945 ± 6% -90.3% 6574 ± 8% softirqs.CPU41.TIMER 59208 ± 3% -98.6% 858.00 ± 12% softirqs.CPU42.RCU 31284 -94.7% 1670 ± 3% softirqs.CPU42.SCHED 90250 ± 21% -92.9% 6400 ± 4% softirqs.CPU42.TIMER 60717 -98.7% 812.00 ± 18% softirqs.CPU43.RCU 29627 -94.6% 1602 ± 9% softirqs.CPU43.SCHED 67872 ± 4% -91.0% 6082 ± 10% softirqs.CPU43.TIMER 60183 -98.0% 1200 ± 40% softirqs.CPU44.RCU 30168 -94.6% 1614 ± 9% softirqs.CPU44.SCHED 66725 ± 2% -88.8% 7459 ± 22% softirqs.CPU44.TIMER 59151 -98.7% 780.00 ± 19% softirqs.CPU45.RCU 27969 -94.0% 1674 ± 11% softirqs.CPU45.SCHED 65535 -90.5% 6216 ± 10% softirqs.CPU45.TIMER 60813 -98.6% 874.33 ± 17% softirqs.CPU46.RCU 29353 -94.3% 1669 ± 5% softirqs.CPU46.SCHED 65922 ± 2% -90.0% 6575 ± 5% softirqs.CPU46.TIMER 61284 -97.9% 1294 ± 46% softirqs.CPU47.RCU 30216 ± 2% -94.7% 1600 ± 15% softirqs.CPU47.SCHED 66953 ± 2% -88.4% 7762 ± 22% softirqs.CPU47.TIMER 60115 -98.6% 816.33 ± 19% softirqs.CPU48.RCU 29419 ± 2% -94.4% 1657 ± 8% softirqs.CPU48.SCHED 66952 -90.6% 6315 ± 9% softirqs.CPU48.TIMER 61630 -98.7% 788.00 ± 2% softirqs.CPU49.RCU 29663 ± 4% -94.3% 1684 ± 2% softirqs.CPU49.SCHED 67662 ± 3% -90.9% 6162 ± 2% softirqs.CPU49.TIMER 58167 -98.1% 1119 ± 8% softirqs.CPU5.RCU 30874 -94.3% 1758 ± 5% softirqs.CPU5.SCHED 68435 -90.0% 6861 ± 2% softirqs.CPU5.TIMER 59033 -98.6% 828.00 ± 17% softirqs.CPU50.RCU 28768 -93.9% 1745 ± 7% softirqs.CPU50.SCHED 65919 -90.2% 6441 ± 9% softirqs.CPU50.TIMER 60009 -98.6% 823.67 ± 12% softirqs.CPU51.RCU 29037 ± 4% -94.0% 1732 ± 5% softirqs.CPU51.SCHED 67172 -90.3% 6500 ± 7% softirqs.CPU51.TIMER 60561 -98.7% 790.67 ± 21% softirqs.CPU52.RCU 30549 ± 2% -94.3% 1730 ± 7% softirqs.CPU52.SCHED 67954 ± 3% -89.9% 6887 ± 19% softirqs.CPU52.TIMER 60862 -98.7% 802.00 ± 11% softirqs.CPU53.RCU 30286 -94.3% 1729 ± 6% softirqs.CPU53.SCHED 67590 ± 3% -90.5% 6421 ± 7% softirqs.CPU53.TIMER 58644 -98.6% 801.67 ± 8% softirqs.CPU54.RCU 29869 ± 4% -94.5% 1639 ± 3% softirqs.CPU54.SCHED 67441 -90.0% 6761 ± 12% softirqs.CPU54.TIMER 58235 -98.4% 946.33 ± 16% softirqs.CPU55.RCU 29046 ± 2% -93.8% 1812 ± 3% softirqs.CPU55.SCHED 66439 -90.1% 6599 ± 5% softirqs.CPU55.TIMER 59325 -98.6% 840.00 ± 5% softirqs.CPU56.RCU 30668 ± 2% -96.5% 1072 ± 14% softirqs.CPU56.SCHED 69572 ± 2% -93.6% 4459 ± 15% softirqs.CPU56.TIMER 59177 -98.3% 1024 ± 24% softirqs.CPU57.RCU 30605 ± 2% -94.1% 1809 ± 12% softirqs.CPU57.SCHED 68370 ± 2% -90.3% 6645 ± 11% softirqs.CPU57.TIMER 59256 -92.2% 4604 ± 51% softirqs.CPU58.RCU 28495 ± 6% -93.7% 1805 ± 3% softirqs.CPU58.SCHED 67352 -86.1% 9365 ± 19% softirqs.CPU58.TIMER 55801 -98.3% 973.00 ± 18% softirqs.CPU59.RCU 29044 -94.0% 1752 ± 5% softirqs.CPU59.SCHED 66198 -90.6% 6208 ± 4% softirqs.CPU59.TIMER 52149 ± 11% -97.9% 1107 ± 6% softirqs.CPU6.RCU 30518 -94.3% 1733 ± 5% softirqs.CPU6.SCHED 71874 ± 3% -90.2% 7043 ± 7% softirqs.CPU6.TIMER 54967 -98.5% 822.00 ± 5% softirqs.CPU60.RCU 30715 ± 3% -94.4% 1730 ± 5% softirqs.CPU60.SCHED 70145 ± 4% -91.0% 6342 ± 3% softirqs.CPU60.TIMER 54360 -98.2% 964.00 ± 6% softirqs.CPU61.RCU 29812 ± 3% -94.0% 1788 softirqs.CPU61.SCHED 69082 ± 3% -90.6% 6501 softirqs.CPU61.TIMER 54530 -98.3% 927.67 ± 8% softirqs.CPU62.RCU 29663 -94.1% 1757 ± 2% softirqs.CPU62.SCHED 69911 ± 4% -90.9% 6368 softirqs.CPU62.TIMER 54158 -98.6% 764.67 ± 5% softirqs.CPU63.RCU 29324 -94.1% 1737 ± 5% softirqs.CPU63.SCHED 66737 -90.6% 6261 ± 3% softirqs.CPU63.TIMER 54660 -98.6% 770.00 ± 10% softirqs.CPU64.RCU 29902 -94.3% 1707 ± 3% softirqs.CPU64.SCHED 67371 -91.0% 6047 ± 2% softirqs.CPU64.TIMER 54857 -98.6% 785.33 ± 10% softirqs.CPU65.RCU 30053 ± 2% -94.3% 1712 ± 3% softirqs.CPU65.SCHED 69637 ± 4% -91.1% 6186 ± 3% softirqs.CPU65.TIMER 53654 -98.5% 791.00 ± 4% softirqs.CPU66.RCU 29152 ± 4% -94.4% 1632 ± 7% softirqs.CPU66.SCHED 66889 -90.9% 6115 ± 3% softirqs.CPU66.TIMER 54597 -98.5% 845.00 ± 17% softirqs.CPU67.RCU 29559 ± 4% -94.1% 1736 ± 3% softirqs.CPU67.SCHED 88688 ± 24% -92.8% 6349 ± 5% softirqs.CPU67.TIMER 54667 -97.8% 1185 ± 38% softirqs.CPU68.RCU 29394 ± 4% -94.2% 1716 ± 13% softirqs.CPU68.SCHED 66570 -88.6% 7600 ± 15% softirqs.CPU68.TIMER 53100 -98.3% 904.00 ± 11% softirqs.CPU69.RCU 29945 ± 2% -93.8% 1850 ± 7% softirqs.CPU69.SCHED 66913 -90.1% 6598 ± 6% softirqs.CPU69.TIMER 58121 -98.2% 1035 ± 5% softirqs.CPU7.RCU 30595 -94.2% 1761 ± 6% softirqs.CPU7.SCHED 71318 ± 3% -90.5% 6764 ± 7% softirqs.CPU7.TIMER 54506 -98.6% 750.00 softirqs.CPU70.RCU 29397 ± 4% -94.0% 1754 ± 2% softirqs.CPU70.SCHED 68029 -90.9% 6186 ± 2% softirqs.CPU70.TIMER 54113 -98.5% 815.67 ± 16% softirqs.CPU71.RCU 30053 ± 2% -94.2% 1753 ± 2% softirqs.CPU71.SCHED 67132 -90.0% 6713 ± 7% softirqs.CPU71.TIMER 57533 -98.4% 937.33 ± 3% softirqs.CPU8.RCU 29636 ± 3% -94.5% 1615 ± 2% softirqs.CPU8.SCHED 70690 ± 3% -90.8% 6493 ± 2% softirqs.CPU8.TIMER 57877 -97.2% 1626 ± 48% softirqs.CPU9.RCU 30605 ± 2% -93.4% 2027 ± 22% softirqs.CPU9.SCHED 69228 ± 3% -89.2% 7489 ± 17% softirqs.CPU9.TIMER 4186875 -98.1% 78720 ± 2% softirqs.RCU 2177057 ± 2% -94.1% 129343 softirqs.SCHED 5042149 ± 2% -90.4% 485863 ± 3% softirqs.TIMER 206.50 ± 9% -94.8% 10.67 ± 4% interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 415.00 -94.1% 24.67 ± 3% interrupts.9:IO-APIC.9-fasteoi.acpi 23773 -40.3% 14189 ± 20% interrupts.CAL:Function_call_interrupts 415126 -99.7% 1261 ± 25% interrupts.CPU0.LOC:Local_timer_interrupts 1403 ± 6% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts 1403 ± 6% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts 415.00 -94.1% 24.67 ± 3% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi 329.50 -54.1% 151.33 ± 70% interrupts.CPU1.CAL:Function_call_interrupts 415504 -99.8% 805.33 ± 9% interrupts.CPU1.LOC:Local_timer_interrupts 1425 ± 5% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts 1425 ± 5% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts 3130 ± 21% -97.9% 66.33 ± 3% interrupts.CPU1.RES:Rescheduling_interrupts 328.00 -39.1% 199.67 ± 20% interrupts.CPU10.CAL:Function_call_interrupts 415516 -99.8% 986.00 ± 32% interrupts.CPU10.LOC:Local_timer_interrupts 1456 ± 3% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts 1456 ± 3% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts 1076 ± 3% -96.7% 35.33 ± 52% interrupts.CPU10.RES:Rescheduling_interrupts 332.00 -39.9% 199.67 ± 20% interrupts.CPU11.CAL:Function_call_interrupts 415516 -99.5% 2123 ± 71% interrupts.CPU11.LOC:Local_timer_interrupts 1490 ± 7% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts 1490 ± 7% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts 1037 ± 5% -97.0% 30.67 ± 50% interrupts.CPU11.RES:Rescheduling_interrupts 328.50 -39.2% 199.67 ± 20% interrupts.CPU12.CAL:Function_call_interrupts 415525 -99.8% 815.33 ± 12% interrupts.CPU12.LOC:Local_timer_interrupts 1422 ± 6% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts 1422 ± 6% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts 1077 ± 5% -97.6% 25.33 ± 49% interrupts.CPU12.RES:Rescheduling_interrupts 330.50 -39.6% 199.67 ± 20% interrupts.CPU13.CAL:Function_call_interrupts 415482 -99.8% 963.33 ± 22% interrupts.CPU13.LOC:Local_timer_interrupts 1076 ± 39% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts 1076 ± 39% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts 1070 ± 7% -97.7% 25.00 ± 16% interrupts.CPU13.RES:Rescheduling_interrupts 332.50 -39.8% 200.00 ± 20% interrupts.CPU14.CAL:Function_call_interrupts 415510 -99.8% 864.33 ± 20% interrupts.CPU14.LOC:Local_timer_interrupts 1072 ± 33% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts 1072 ± 33% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts 1086 -97.7% 24.67 ± 55% interrupts.CPU14.RES:Rescheduling_interrupts 328.50 -39.1% 200.00 ± 20% interrupts.CPU15.CAL:Function_call_interrupts 415468 -99.8% 963.00 ± 28% interrupts.CPU15.LOC:Local_timer_interrupts 988.00 ± 32% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts 988.00 ± 32% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts 1108 -96.7% 37.00 ± 42% interrupts.CPU15.RES:Rescheduling_interrupts 327.50 -39.0% 199.67 ± 20% interrupts.CPU16.CAL:Function_call_interrupts 415472 -99.7% 1269 ± 34% interrupts.CPU16.LOC:Local_timer_interrupts 943.50 ± 20% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts 943.50 ± 20% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts 1064 ± 4% -97.4% 27.33 ± 47% interrupts.CPU16.RES:Rescheduling_interrupts 327.50 -39.0% 199.67 ± 20% interrupts.CPU17.CAL:Function_call_interrupts 415491 -99.8% 978.33 ± 30% interrupts.CPU17.LOC:Local_timer_interrupts 982.50 ± 26% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts 982.50 ± 26% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts 1049 ± 4% -96.5% 36.33 ± 66% interrupts.CPU17.RES:Rescheduling_interrupts 326.50 -54.7% 148.00 ± 29% interrupts.CPU18.CAL:Function_call_interrupts 415073 -99.5% 2122 ± 48% interrupts.CPU18.LOC:Local_timer_interrupts 1356 ± 8% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts 1356 ± 8% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts 1165 ± 2% -97.0% 34.67 ± 21% interrupts.CPU18.RES:Rescheduling_interrupts 415483 -99.7% 1106 ± 26% interrupts.CPU19.LOC:Local_timer_interrupts 1029 ± 23% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts 1029 ± 23% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts 1124 ± 2% -97.9% 23.33 ± 11% interrupts.CPU19.RES:Rescheduling_interrupts 328.50 -39.1% 200.00 ± 20% interrupts.CPU2.CAL:Function_call_interrupts 415509 -99.8% 828.67 ± 7% interrupts.CPU2.LOC:Local_timer_interrupts 1382 -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 1382 -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 1082 -95.1% 53.00 ± 6% interrupts.CPU2.RES:Rescheduling_interrupts 415488 -99.8% 788.67 interrupts.CPU20.LOC:Local_timer_interrupts 1524 ± 7% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts 1524 ± 7% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts 1098 ± 3% -97.6% 26.33 ± 31% interrupts.CPU20.RES:Rescheduling_interrupts 330.50 -40.3% 197.33 ± 20% interrupts.CPU21.CAL:Function_call_interrupts 415499 -99.8% 947.33 ± 14% interrupts.CPU21.LOC:Local_timer_interrupts 1421 -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts 1421 -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts 1139 ± 5% -98.3% 19.67 ± 47% interrupts.CPU21.RES:Rescheduling_interrupts 330.00 -39.7% 199.00 ± 20% interrupts.CPU22.CAL:Function_call_interrupts 415471 -99.8% 953.67 ± 25% interrupts.CPU22.LOC:Local_timer_interrupts 1320 -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts 1320 -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts 1142 ± 4% -98.3% 19.33 ± 31% interrupts.CPU22.RES:Rescheduling_interrupts 329.00 -39.3% 199.67 ± 20% interrupts.CPU23.CAL:Function_call_interrupts 415453 -99.8% 897.00 ± 20% interrupts.CPU23.LOC:Local_timer_interrupts 915.50 ± 30% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts 915.50 ± 30% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts 1076 -98.0% 22.00 ± 19% interrupts.CPU23.RES:Rescheduling_interrupts 330.50 -39.6% 199.67 ± 20% interrupts.CPU24.CAL:Function_call_interrupts 415477 -99.8% 910.33 ± 21% interrupts.CPU24.LOC:Local_timer_interrupts 1045 ± 31% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts 1045 ± 31% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts 1063 -97.6% 25.00 ± 19% interrupts.CPU24.RES:Rescheduling_interrupts 328.00 -39.1% 199.67 ± 20% interrupts.CPU25.CAL:Function_call_interrupts 415475 -99.8% 896.00 ± 13% interrupts.CPU25.LOC:Local_timer_interrupts 993.00 ± 28% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts 993.00 ± 28% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts 1087 ± 2% -98.0% 21.33 ± 33% interrupts.CPU25.RES:Rescheduling_interrupts 329.00 -39.3% 199.67 ± 20% interrupts.CPU26.CAL:Function_call_interrupts 415489 -99.4% 2344 ± 55% interrupts.CPU26.LOC:Local_timer_interrupts 1073 ± 34% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts 1073 ± 34% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts 1071 ± 3% -98.1% 20.67 ± 12% interrupts.CPU26.RES:Rescheduling_interrupts 332.00 -39.8% 200.00 ± 20% interrupts.CPU27.CAL:Function_call_interrupts 415471 -99.7% 1234 ± 24% interrupts.CPU27.LOC:Local_timer_interrupts 715.50 -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts 715.50 -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts 1122 ± 5% -98.5% 17.00 ± 39% interrupts.CPU27.RES:Rescheduling_interrupts 329.50 -39.3% 200.00 ± 20% interrupts.CPU28.CAL:Function_call_interrupts 415463 -99.7% 1270 ± 41% interrupts.CPU28.LOC:Local_timer_interrupts 779.50 ± 2% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts 779.50 ± 2% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts 1067 ± 4% -98.1% 20.00 ± 26% interrupts.CPU28.RES:Rescheduling_interrupts 331.00 -39.7% 199.67 ± 20% interrupts.CPU29.CAL:Function_call_interrupts 415482 -99.7% 1118 ± 41% interrupts.CPU29.LOC:Local_timer_interrupts 715.00 ± 11% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts 715.00 ± 11% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts 1136 ± 4% -97.9% 23.67 ± 28% interrupts.CPU29.RES:Rescheduling_interrupts 328.50 -38.5% 202.00 ± 20% interrupts.CPU3.CAL:Function_call_interrupts 415480 -99.3% 2975 ±102% interrupts.CPU3.LOC:Local_timer_interrupts 1428 -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 1428 -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 1116 ± 7% -97.3% 29.67 ± 37% interrupts.CPU3.RES:Rescheduling_interrupts 330.00 -39.5% 199.67 ± 20% interrupts.CPU30.CAL:Function_call_interrupts 415477 -99.7% 1047 ± 20% interrupts.CPU30.LOC:Local_timer_interrupts 710.00 ± 2% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts 710.00 ± 2% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts 1076 ± 4% -97.2% 29.67 ± 20% interrupts.CPU30.RES:Rescheduling_interrupts 329.50 -39.4% 199.67 ± 20% interrupts.CPU31.CAL:Function_call_interrupts 415277 -99.7% 1171 ± 13% interrupts.CPU31.LOC:Local_timer_interrupts 713.00 ± 4% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts 713.00 ± 4% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts 1040 ± 3% -97.0% 31.00 ± 38% interrupts.CPU31.RES:Rescheduling_interrupts 329.50 -40.9% 194.67 ± 18% interrupts.CPU32.CAL:Function_call_interrupts 415476 -99.2% 3506 ± 41% interrupts.CPU32.LOC:Local_timer_interrupts 753.50 ± 2% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts 753.50 ± 2% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts 1117 ± 3% -98.4% 18.00 ± 36% interrupts.CPU32.RES:Rescheduling_interrupts 329.00 -53.9% 151.67 ± 51% interrupts.CPU33.CAL:Function_call_interrupts 415478 -99.4% 2507 ± 65% interrupts.CPU33.LOC:Local_timer_interrupts 1123 ± 32% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts 1123 ± 32% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts 1074 ± 8% -98.1% 20.00 ± 28% interrupts.CPU33.RES:Rescheduling_interrupts 329.50 -39.3% 200.00 ± 20% interrupts.CPU34.CAL:Function_call_interrupts 415452 -99.8% 888.67 ± 18% interrupts.CPU34.LOC:Local_timer_interrupts 1068 ± 30% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts 1068 ± 30% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts 1109 ± 9% -97.7% 25.33 ± 35% interrupts.CPU34.RES:Rescheduling_interrupts 329.00 -39.3% 199.67 ± 20% interrupts.CPU35.CAL:Function_call_interrupts 415485 -99.8% 932.33 ± 15% interrupts.CPU35.LOC:Local_timer_interrupts 1126 ± 37% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts 1126 ± 37% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts 1104 -98.5% 16.67 ± 26% interrupts.CPU35.RES:Rescheduling_interrupts 330.50 -39.6% 199.67 ± 20% interrupts.CPU36.CAL:Function_call_interrupts 415449 -99.8% 748.00 ± 3% interrupts.CPU36.LOC:Local_timer_interrupts 1413 ± 5% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts 1413 ± 5% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts 1179 ± 5% -97.1% 34.33 ± 25% interrupts.CPU36.RES:Rescheduling_interrupts 330.00 -39.5% 199.67 ± 20% interrupts.CPU37.CAL:Function_call_interrupts 415531 -99.8% 967.00 ± 13% interrupts.CPU37.LOC:Local_timer_interrupts 1098 ± 32% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts 1098 ± 32% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts 1140 -97.7% 25.67 ± 46% interrupts.CPU37.RES:Rescheduling_interrupts 331.50 -39.7% 200.00 ± 20% interrupts.CPU38.CAL:Function_call_interrupts 415451 -99.8% 821.67 ± 9% interrupts.CPU38.LOC:Local_timer_interrupts 1405 ± 2% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts 1405 ± 2% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts 1122 -97.8% 24.67 ± 43% interrupts.CPU38.RES:Rescheduling_interrupts 334.50 -40.3% 199.67 ± 20% interrupts.CPU39.CAL:Function_call_interrupts 415761 -99.8% 1002 ± 23% interrupts.CPU39.LOC:Local_timer_interrupts 1454 -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts 1454 -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts 1103 -97.4% 28.33 ± 39% interrupts.CPU39.RES:Rescheduling_interrupts 332.00 -39.9% 199.67 ± 20% interrupts.CPU4.CAL:Function_call_interrupts 415496 -99.8% 817.33 ± 8% interrupts.CPU4.LOC:Local_timer_interrupts 1532 ± 2% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts 1532 ± 2% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts 1094 ± 6% -97.3% 30.00 ± 30% interrupts.CPU4.RES:Rescheduling_interrupts 327.00 -38.9% 199.67 ± 20% interrupts.CPU40.CAL:Function_call_interrupts 415509 -99.8% 873.00 ± 5% interrupts.CPU40.LOC:Local_timer_interrupts 1496 ± 2% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts 1496 ± 2% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts 1099 ± 8% -97.2% 30.33 ± 63% interrupts.CPU40.RES:Rescheduling_interrupts 330.00 -39.8% 198.67 ± 20% interrupts.CPU41.CAL:Function_call_interrupts 415511 -99.8% 940.33 ± 19% interrupts.CPU41.LOC:Local_timer_interrupts 1535 ± 7% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts 1535 ± 7% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts 1095 ± 3% -97.4% 28.67 ± 38% interrupts.CPU41.RES:Rescheduling_interrupts 328.00 -39.1% 199.67 ± 20% interrupts.CPU42.CAL:Function_call_interrupts 415534 -99.7% 1064 ± 23% interrupts.CPU42.LOC:Local_timer_interrupts 1396 ± 6% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts 1396 ± 6% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts 1136 ± 6% -97.6% 27.67 ± 57% interrupts.CPU42.RES:Rescheduling_interrupts 330.00 -39.5% 199.67 ± 20% interrupts.CPU43.CAL:Function_call_interrupts 415508 -99.8% 869.67 ± 23% interrupts.CPU43.LOC:Local_timer_interrupts 1382 -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts 1382 -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts 1152 ± 2% -97.8% 25.33 ± 48% interrupts.CPU43.RES:Rescheduling_interrupts 330.00 -39.5% 199.67 ± 20% interrupts.CPU44.CAL:Function_call_interrupts 415515 -99.8% 939.67 ± 32% interrupts.CPU44.LOC:Local_timer_interrupts 1097 ± 41% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts 1097 ± 41% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts 1083 ± 3% -97.7% 25.00 ± 33% interrupts.CPU44.RES:Rescheduling_interrupts 330.00 -39.5% 199.67 ± 20% interrupts.CPU45.CAL:Function_call_interrupts 415522 -99.8% 932.00 ± 27% interrupts.CPU45.LOC:Local_timer_interrupts 1032 ± 40% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts 1032 ± 40% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts 1085 -97.2% 30.67 ± 46% interrupts.CPU45.RES:Rescheduling_interrupts 328.00 -39.1% 199.67 ± 20% interrupts.CPU46.CAL:Function_call_interrupts 415531 -99.7% 1055 ± 43% interrupts.CPU46.LOC:Local_timer_interrupts 1097 ± 39% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts 1097 ± 39% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts 1061 -97.3% 28.67 ± 44% interrupts.CPU46.RES:Rescheduling_interrupts 328.50 -39.2% 199.67 ± 20% interrupts.CPU47.CAL:Function_call_interrupts 415514 -99.8% 774.67 ± 4% interrupts.CPU47.LOC:Local_timer_interrupts 1117 ± 43% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts 1117 ± 43% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts 1098 ± 4% -97.0% 33.33 ± 42% interrupts.CPU47.RES:Rescheduling_interrupts 329.00 -39.3% 199.67 ± 20% interrupts.CPU48.CAL:Function_call_interrupts 415490 -99.8% 764.00 ± 9% interrupts.CPU48.LOC:Local_timer_interrupts 1092 ± 42% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts 1092 ± 42% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts 1073 -97.9% 23.00 ± 63% interrupts.CPU48.RES:Rescheduling_interrupts 328.00 -39.1% 199.67 ± 20% interrupts.CPU49.CAL:Function_call_interrupts 415533 -99.8% 776.00 ± 9% interrupts.CPU49.LOC:Local_timer_interrupts 1082 ± 40% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts 1082 ± 40% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts 1074 ± 3% -97.5% 27.00 ± 45% interrupts.CPU49.RES:Rescheduling_interrupts 328.00 -39.1% 199.67 ± 20% interrupts.CPU5.CAL:Function_call_interrupts 415524 -99.8% 1004 ± 14% interrupts.CPU5.LOC:Local_timer_interrupts 1554 ± 4% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts 1554 ± 4% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts 2483 ± 51% -98.7% 31.33 ± 44% interrupts.CPU5.RES:Rescheduling_interrupts 332.00 -39.8% 200.00 ± 20% interrupts.CPU50.CAL:Function_call_interrupts 415487 -99.8% 891.00 ± 22% interrupts.CPU50.LOC:Local_timer_interrupts 1162 ± 35% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts 1162 ± 35% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts 1135 -97.6% 27.33 ± 40% interrupts.CPU50.RES:Rescheduling_interrupts 327.50 -39.0% 199.67 ± 20% interrupts.CPU51.CAL:Function_call_interrupts 415463 -99.8% 769.33 ± 5% interrupts.CPU51.LOC:Local_timer_interrupts 694.00 -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts 694.00 -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts 1116 ± 3% -97.3% 30.00 ± 26% interrupts.CPU51.RES:Rescheduling_interrupts 327.00 -38.9% 199.67 ± 20% interrupts.CPU52.CAL:Function_call_interrupts 415428 -99.5% 1955 ± 86% interrupts.CPU52.LOC:Local_timer_interrupts 982.00 ± 24% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts 982.00 ± 24% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts 1094 ± 2% -97.8% 24.00 ± 50% interrupts.CPU52.RES:Rescheduling_interrupts 331.50 -39.8% 199.67 ± 20% interrupts.CPU53.CAL:Function_call_interrupts 415514 -99.7% 1054 ± 27% interrupts.CPU53.LOC:Local_timer_interrupts 998.00 ± 29% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts 998.00 ± 29% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts 1058 ± 2% -97.2% 30.00 ± 29% interrupts.CPU53.RES:Rescheduling_interrupts 332.00 -40.1% 199.00 ± 17% interrupts.CPU54.CAL:Function_call_interrupts 415476 -99.5% 2204 ± 72% interrupts.CPU54.LOC:Local_timer_interrupts 1018 ± 29% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts 1018 ± 29% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts 1143 ± 4% -97.3% 31.33 ± 19% interrupts.CPU54.RES:Rescheduling_interrupts 334.00 -40.2% 199.67 ± 20% interrupts.CPU55.CAL:Function_call_interrupts 415376 -99.7% 1163 ± 26% interrupts.CPU55.LOC:Local_timer_interrupts 1026 ± 25% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts 1026 ± 25% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts 1125 ± 11% -97.6% 26.67 ± 41% interrupts.CPU55.RES:Rescheduling_interrupts 329.00 -39.3% 199.67 ± 20% interrupts.CPU56.CAL:Function_call_interrupts 415464 -99.7% 1422 ± 43% interrupts.CPU56.LOC:Local_timer_interrupts 1069 ± 30% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 1069 ± 30% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 1159 -97.9% 24.33 ± 13% interrupts.CPU56.RES:Rescheduling_interrupts 340.50 ± 2% -41.4% 199.67 ± 20% interrupts.CPU57.CAL:Function_call_interrupts 415482 -99.6% 1655 ± 62% interrupts.CPU57.LOC:Local_timer_interrupts 682.00 -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts 682.00 -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts 1175 ± 6% -97.9% 25.00 ± 40% interrupts.CPU57.RES:Rescheduling_interrupts 340.00 ± 3% -41.3% 199.67 ± 20% interrupts.CPU58.CAL:Function_call_interrupts 415477 -99.7% 1198 ± 11% interrupts.CPU58.LOC:Local_timer_interrupts 654.00 -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts 654.00 -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts 1094 ± 6% -97.6% 26.00 ± 16% interrupts.CPU58.RES:Rescheduling_interrupts 349.50 -42.9% 199.67 ± 20% interrupts.CPU59.CAL:Function_call_interrupts 415488 -99.7% 1051 ± 24% interrupts.CPU59.LOC:Local_timer_interrupts 955.50 ± 37% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts 955.50 ± 37% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts 1141 ± 2% -98.3% 19.67 ± 6% interrupts.CPU59.RES:Rescheduling_interrupts 328.50 -39.1% 200.00 ± 20% interrupts.CPU6.CAL:Function_call_interrupts 415504 -99.7% 1126 ± 31% interrupts.CPU6.LOC:Local_timer_interrupts 1382 ± 8% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts 1382 ± 8% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts 1095 -96.8% 35.33 ± 50% interrupts.CPU6.RES:Rescheduling_interrupts 353.50 -43.4% 200.00 ± 20% interrupts.CPU60.CAL:Function_call_interrupts 415484 -99.7% 1226 ± 19% interrupts.CPU60.LOC:Local_timer_interrupts 1059 ± 33% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts 1059 ± 33% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts 1132 ± 8% -97.5% 28.33 ± 12% interrupts.CPU60.RES:Rescheduling_interrupts 355.50 -44.5% 197.33 ± 19% interrupts.CPU61.CAL:Function_call_interrupts 415486 -99.7% 1227 ± 8% interrupts.CPU61.LOC:Local_timer_interrupts 1072 ± 37% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts 1072 ± 37% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts 1133 ± 6% -97.5% 28.33 ± 35% interrupts.CPU61.RES:Rescheduling_interrupts 355.00 -43.7% 200.00 ± 20% interrupts.CPU62.CAL:Function_call_interrupts 415497 -99.8% 992.67 ± 12% interrupts.CPU62.LOC:Local_timer_interrupts 1032 ± 31% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts 1032 ± 31% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts 1182 ± 3% -97.7% 27.00 ± 16% interrupts.CPU62.RES:Rescheduling_interrupts 354.50 -44.0% 198.67 ± 20% interrupts.CPU63.CAL:Function_call_interrupts 415475 -99.8% 817.00 ± 11% interrupts.CPU63.LOC:Local_timer_interrupts 1041 ± 30% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts 1041 ± 30% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts 1155 -97.9% 24.33 ± 8% interrupts.CPU63.RES:Rescheduling_interrupts 358.00 -45.2% 196.33 ± 21% interrupts.CPU64.CAL:Function_call_interrupts 415217 -99.7% 1186 ± 27% interrupts.CPU64.LOC:Local_timer_interrupts 1157 ± 35% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts 1157 ± 35% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts 1124 -98.3% 19.67 ± 8% interrupts.CPU64.RES:Rescheduling_interrupts 354.00 -44.2% 197.67 ± 20% interrupts.CPU65.CAL:Function_call_interrupts 415475 -99.8% 1019 ± 24% interrupts.CPU65.LOC:Local_timer_interrupts 734.00 ± 4% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts 734.00 ± 4% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts 1127 ± 3% -97.8% 24.33 ± 7% interrupts.CPU65.RES:Rescheduling_interrupts 354.50 -44.2% 197.67 ± 20% interrupts.CPU66.CAL:Function_call_interrupts 415456 -99.7% 1100 ± 27% interrupts.CPU66.LOC:Local_timer_interrupts 731.00 ± 2% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts 731.00 ± 2% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts 1076 ± 5% -97.4% 27.67 ± 35% interrupts.CPU66.RES:Rescheduling_interrupts 357.50 -44.7% 197.67 ± 19% interrupts.CPU67.CAL:Function_call_interrupts 415465 -99.8% 902.67 ± 6% interrupts.CPU67.LOC:Local_timer_interrupts 1070 ± 29% -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts 1070 ± 29% -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts 1090 ± 4% -98.0% 21.67 ± 50% interrupts.CPU67.RES:Rescheduling_interrupts 353.00 -43.4% 199.67 ± 20% interrupts.CPU68.CAL:Function_call_interrupts 415410 -99.6% 1501 ± 15% interrupts.CPU68.LOC:Local_timer_interrupts 703.50 -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts 703.50 -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts 1150 ± 8% -98.7% 15.00 ± 46% interrupts.CPU68.RES:Rescheduling_interrupts 354.00 -44.9% 195.00 ± 18% interrupts.CPU69.CAL:Function_call_interrupts 415650 -99.7% 1102 ± 21% interrupts.CPU69.LOC:Local_timer_interrupts 1109 ± 34% -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts 1109 ± 34% -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts 1126 ± 4% -98.5% 17.00 ± 8% interrupts.CPU69.RES:Rescheduling_interrupts 206.50 ± 9% -94.8% 10.67 ± 4% interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 330.00 -39.5% 199.67 ± 20% interrupts.CPU7.CAL:Function_call_interrupts 415516 -99.8% 899.67 ± 13% interrupts.CPU7.LOC:Local_timer_interrupts 1357 -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts 1357 -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts 1037 -96.9% 31.67 ± 22% interrupts.CPU7.RES:Rescheduling_interrupts 353.00 -43.4% 199.67 ± 20% interrupts.CPU70.CAL:Function_call_interrupts 415390 -99.7% 1212 ± 30% interrupts.CPU70.LOC:Local_timer_interrupts 1037 ± 33% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts 1037 ± 33% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts 1058 ± 2% -97.6% 25.33 ± 41% interrupts.CPU70.RES:Rescheduling_interrupts 346.00 -45.2% 189.67 ± 15% interrupts.CPU71.CAL:Function_call_interrupts 415613 -99.6% 1726 ± 66% interrupts.CPU71.LOC:Local_timer_interrupts 1086 ± 36% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts 1086 ± 36% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts 1095 ± 2% -98.7% 14.33 ± 37% interrupts.CPU71.RES:Rescheduling_interrupts 105.50 -100.0% 0.00 interrupts.CPU71.TLB:TLB_shootdowns 327.50 -39.0% 199.67 ± 20% interrupts.CPU8.CAL:Function_call_interrupts 415506 -99.8% 963.33 ± 16% interrupts.CPU8.LOC:Local_timer_interrupts 1452 ± 7% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts 1452 ± 7% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts 1159 ± 6% -97.5% 29.33 ± 46% interrupts.CPU8.RES:Rescheduling_interrupts 330.00 -39.5% 199.67 ± 20% interrupts.CPU9.CAL:Function_call_interrupts 415519 -99.8% 818.33 ± 11% interrupts.CPU9.LOC:Local_timer_interrupts 1399 ± 7% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 1399 ± 7% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 1099 -96.7% 36.67 ± 18% interrupts.CPU9.RES:Rescheduling_interrupts 29914281 -99.7% 85011 ± 2% interrupts.LOC:Local_timer_interrupts 80907 ± 3% -100.0% 0.00 interrupts.NMI:Non-maskable_interrupts 80907 ± 3% -100.0% 0.00 interrupts.PMI:Performance_monitoring_interrupts 83727 -95.2% 4018 ± 19% interrupts.RES:Rescheduling_interrupts 2208 ± 11% -98.4% 34.67 ± 86% interrupts.TLB:TLB_shootdowns aim7.jobs-per-min 1.8e+06 +-----------------------------------------------------------------+ | OO OO O O OO O O O O | 1.6e+06 |-+ | 1.4e+06 |-+ | | | 1.2e+06 |-+ | 1e+06 |-+ | | | 800000 |-+ | 600000 |-+ | | | 400000 |-+ | 200000 |-+ | |.++.++.+.++.++. .++.+.++. +. .++.++.+.++.++.+.++.+.++.++.+.+ .+ .| 0 +-----------------------------------------------------------------+ aim7.time.user_time 45 +----------------------------------------------------------------------+ |.+ .++.+. .+ +.+.++. + +.++. +.+.+ .+ + | 40 |-++.+.+ + : : + : : +.+.+ +.+.+ +.+.+.++.+ : | 35 |-+ : : : : : : : | | : : : : : : : | 30 |-+ : : : :: : : :: | 25 |-+ : : : :: : : :: :| | : : : :: : : :: :| 20 |-+ :: :: : : : : ::| 15 |-+ :: :: : : : : ::| | :: :: : : : : ::| 10 |-+ : : : : ::| 5 |-OO O O OO O O: O OO : : : : | | : : : : : | 0 +----------------------------------------------------------------------+ aim7.time.system_time 1400 +--------------------------------------------------------------------+ |.++.+ +.+.++.+ : +.+: : +.++.+.+ +.++.+.+ +.+ + ++ + | 1200 |-+ : : : : : : : | | : : : : : : : | 1000 |-+ : : : :: : : : | | : : : :: : : :: :| 800 |-+ : : : :: : : :: :| | : : : : :: : :: :| 600 |-+ : : : : :: : : ::| | : : : : :: : : ::| 400 |-+ : : : : :: : : ::| | : : : : : | 200 |-+ : : : : : | | : : : : : | 0 +--------------------------------------------------------------------+ aim7.time.elapsed_time 250 +---------------------------------------------------------------------+ | | |. +.+.+.++.+. + ++.+.+.+ + .+. .++.+.++. .+.++.+.++.+.+ .+ + | 200 |-+ +: : : : + + + + : : | | : : : : : : : | | : : : :: : : : | 150 |-+ : : : : : : : :: :| | : : : : : : : :: :| 100 |-+ : : :: : : : : ::| | : : :: :: : : ::| | : : :: :: : : ::| 50 |-+ :: :: : :: ::| | : : : : : | | OO O O OO OO : O O O : : : : | 0 +---------------------------------------------------------------------+ aim7.time.elapsed_time.max 250 +---------------------------------------------------------------------+ | | |. +.+.+.++.+. + ++.+.+.+ + .+. .++.+.++. .+.++.+.++.+.+ .+ + | 200 |-+ +: : : : + + + + : : | | : : : : : : : | | : : : :: : : : | 150 |-+ : : : : : : : :: :| | : : : : : : : :: :| 100 |-+ : : :: : : : : ::| | : : :: :: : : ::| | : : :: :: : : ::| 50 |-+ :: :: : :: ::| | : : : : : | | OO O O OO OO : O O O : : : : | 0 +---------------------------------------------------------------------+ aim7.time.voluntary_context_switches 1.8e+07 +-----------------------------------------------------------------+ |. +.++.+.+ + ++.+.++ + +. +.+.++.+ .+.++.+.++.++. .+ + | 1.6e+07 |-+ +.+: : : : + + + + : : | 1.4e+07 |-+ : : : : : : : | | : : : : : : : | 1.2e+07 |-+ : : : :: : : :: :| 1e+07 |-+ : : : :: : : :: :| | : : : :: : : :: :| 8e+06 |-+ : : :: : : :: ::| 6e+06 |-+ : : :: : : :: ::| | : : :: : : :: ::| 4e+06 |-+ : : :: : : | 2e+06 |-+ : : : : : | | : : : : : | 0 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2sp5: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory ========================================================================================= compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode: gcc-7/performance/1BRD_32G/4K/nfsv4/xfs/1x/x86_64-rhel-7.6/1fpd/1t/debian-x86_64-20191114.cgz/fsyncBeforeClose/lkp-csl-2sp5/4G/fsmark/0x500002c commit: 1d2cc5ac6f (" RISC-V Fixes for 5.7-rc4") f2af7d90e2 ("pipe: read/write_iter() handler should check for IOCB_NOWAIT") 1d2cc5ac6f6668cc f2af7d90e25ce1cf03318bf3290 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 2:4 -50% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x 1:4 -25% :4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 92458734 -14.9% 78692337 fsmark.app_overhead 3044 +17.1% 3566 fsmark.files_per_sec 332.41 -14.6% 284.01 fsmark.time.elapsed_time 332.41 -14.6% 284.01 fsmark.time.elapsed_time.max 66.12 -14.2% 56.73 fsmark.time.system_time 1.00 +100.0% 2.00 nfsstat.Client.nfs.v3.fsstat.percent 20811 ± 70% -99.6% 89.33 ± 57% numa-numastat.node1.other_node 27.53 ± 2% -24.7% 20.72 ± 2% boot-time.boot 2256 -25.5% 1679 ± 2% boot-time.idle 98.83 -100.0% 0.00 iostat.cpu.idle 1.11 -100.0% 0.00 iostat.cpu.system 0.02 -0.0 0.01 ± 3% mpstat.cpu.all.iowait% 0.04 ± 2% +0.0 0.06 mpstat.cpu.all.usr% 27287 -23.8% 20798 meminfo.Mapped 26727 -15.8% 22501 meminfo.Shmem 69442 +17.3% 81429 meminfo.max_used_kB 93555 +18.9% 111258 vmstat.io.bo 222603 +16.9% 260285 vmstat.system.cs 191926 -96.0% 7591 vmstat.system.in 12253 ± 38% -58.0% 5150 ± 80% numa-meminfo.node0.Inactive(anon) 14383 ± 14% -33.5% 9571 ± 24% numa-meminfo.node0.Mapped 14061 ± 38% -53.8% 6495 ± 65% numa-meminfo.node0.Shmem 6251 ± 74% +114.2% 13390 ± 31% numa-meminfo.node1.Inactive(anon) 4.622e+08 -10.6% 4.131e+08 cpuidle.C1.time 2.542e+10 ± 30% -97.4% 6.715e+08 cpuidle.C1E.time 70486976 ± 15% -79.2% 14627655 cpuidle.C1E.usage 5.342e+09 ±140% +386.5% 2.599e+10 cpuidle.C6.time 2668395 -53.2% 1249020 cpuidle.POLL.time 465586 -28.4% 333516 cpuidle.POLL.usage 3063 ± 38% -58.0% 1287 ± 80% numa-vmstat.node0.nr_inactive_anon 3637 ± 12% -34.2% 2392 ± 24% numa-vmstat.node0.nr_mapped 3515 ± 38% -53.8% 1623 ± 65% numa-vmstat.node0.nr_shmem 3063 ± 38% -58.0% 1287 ± 80% numa-vmstat.node0.nr_zone_inactive_anon 1562 ± 74% +114.2% 3347 ± 31% numa-vmstat.node1.nr_inactive_anon 1562 ± 74% +114.2% 3347 ± 31% numa-vmstat.node1.nr_zone_inactive_anon 49242 ± 29% -42.0% 28557 numa-vmstat.node1.numa_other 1285 -1.4% 1268 proc-vmstat.nr_active_file 90505 +1.1% 91542 proc-vmstat.nr_anon_pages 16670 -1.7% 16381 proc-vmstat.nr_kernel_stack 6946 -25.1% 5201 proc-vmstat.nr_mapped 1220 -4.9% 1160 proc-vmstat.nr_page_table_pages 6681 -15.8% 5625 proc-vmstat.nr_shmem 230615 -1.2% 227899 proc-vmstat.nr_slab_unreclaimable 102.33 ± 11% -73.3% 27.33 ± 6% proc-vmstat.nr_writeback 1285 -1.4% 1268 proc-vmstat.nr_zone_active_file 91.00 ± 12% -57.1% 39.00 ± 2% proc-vmstat.nr_zone_write_pending 10869314 -2.2% 10627566 proc-vmstat.numa_hit 10838029 -1.9% 10627400 proc-vmstat.numa_local 31284 -99.5% 166.67 ± 24% proc-vmstat.numa_other 1036 ± 7% -86.0% 144.67 ± 25% proc-vmstat.pgactivate 15316449 -1.7% 15048707 proc-vmstat.pgalloc_normal 974973 -23.6% 744676 proc-vmstat.pgfault 11704295 -2.3% 11432266 proc-vmstat.pgfree 31557801 +1.3% 31967218 proc-vmstat.pgpgout 175.85 ± 50% -53.8% 81.19 ± 56% sched_debug.cfs_rq:/.MIN_vruntime.avg 18.25 ± 23% -73.6% 4.81 ± 56% sched_debug.cfs_rq:/.exec_clock.min 190.34 ± 10% +18.3% 225.12 ± 8% sched_debug.cfs_rq:/.load_avg.avg 5630 +23.8% 6970 sched_debug.cfs_rq:/.load_avg.max 816.20 ± 2% +26.8% 1034 ± 6% sched_debug.cfs_rq:/.load_avg.stddev 175.85 ± 50% -53.8% 81.19 ± 56% sched_debug.cfs_rq:/.max_vruntime.avg 1.29 ± 37% -60.8% 0.50 ± 57% sched_debug.cfs_rq:/.removed.util_avg.avg 89.38 ± 17% -47.6% 46.87 ± 61% sched_debug.cfs_rq:/.removed.util_avg.max 10.20 ± 21% -53.2% 4.77 ± 60% sched_debug.cfs_rq:/.removed.util_avg.stddev 569.01 ±484% -682.1% -3312 sched_debug.cfs_rq:/.spread0.avg 14048 ± 11% -22.2% 10928 ± 9% sched_debug.cfs_rq:/.spread0.max 387.56 ± 5% +12.3% 435.13 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.max 13348 ± 14% -19.8% 10704 ± 7% sched_debug.cpu.avg_idle.min 2.17 ± 2% +40.3% 3.05 sched_debug.cpu.clock.stddev 2.17 ± 2% +40.4% 3.05 sched_debug.cpu.clock_task.stddev 156.88 ± 8% -19.4% 126.52 ± 13% sched_debug.cpu.curr->pid.avg 800.69 ± 6% -15.7% 675.37 ± 2% sched_debug.cpu.curr->pid.stddev 850773 ± 11% +28.4% 1092362 ± 4% sched_debug.cpu.nr_switches.max 2982 ± 24% -63.1% 1099 ± 7% sched_debug.cpu.nr_switches.min 303121 ± 10% +14.0% 345473 sched_debug.cpu.nr_switches.stddev -42.78 +54.6% -66.13 sched_debug.cpu.nr_uninterruptible.min 847907 ± 11% +28.5% 1089815 ± 4% sched_debug.cpu.sched_count.max 1847 ± 53% -95.3% 86.40 ± 27% sched_debug.cpu.sched_count.min 302909 ± 10% +14.0% 345324 sched_debug.cpu.sched_count.stddev 379876 ± 10% +30.9% 497157 ± 7% sched_debug.cpu.sched_goidle.max 795.63 ± 56% -94.5% 43.40 ± 25% sched_debug.cpu.sched_goidle.min 131452 ± 10% +14.7% 150719 sched_debug.cpu.sched_goidle.stddev 458318 ± 9% +24.0% 568529 ± 4% sched_debug.cpu.ttwu_count.max 778.04 ± 62% -95.2% 37.07 ± 11% sched_debug.cpu.ttwu_count.min 109852 ± 12% +21.7% 133742 ± 8% sched_debug.cpu.ttwu_local.max 224.89 ± 22% -85.7% 32.27 ± 8% sched_debug.cpu.ttwu_local.min 68.44 -68.4 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64 67.51 ± 2% -67.5 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 67.50 ± 2% -67.5 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 67.46 ± 2% -67.5 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 58.52 ± 3% -58.5 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 57.29 ± 3% -57.3 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 44.56 -44.6 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 25.09 ± 3% -25.1 0.00 perf-profile.calltrace.cycles-pp.ret_from_fork 25.09 ± 3% -25.1 0.00 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 15.39 ± 5% -15.4 0.00 perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork 14.37 ± 5% -14.4 0.00 perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 10.70 ± 8% -10.7 0.00 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 10.37 ± 5% -10.4 0.00 perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork 10.36 ± 5% -10.4 0.00 perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork 9.87 ± 5% -9.9 0.00 perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread 9.50 ± 4% -9.5 0.00 perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd 9.18 -9.2 0.00 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 8.06 ± 2% -8.1 0.00 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 6.05 ± 8% -6.0 0.00 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 5.78 ± 6% -5.8 0.00 perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 68.48 -68.5 0.00 perf-profile.children.cycles-pp.do_idle 68.44 -68.4 0.00 perf-profile.children.cycles-pp.secondary_startup_64 68.44 -68.4 0.00 perf-profile.children.cycles-pp.cpu_startup_entry 67.51 ± 2% -67.5 0.00 perf-profile.children.cycles-pp.start_secondary 59.36 ± 2% -59.4 0.00 perf-profile.children.cycles-pp.cpuidle_enter 59.32 ± 2% -59.3 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state 44.61 -44.6 0.00 perf-profile.children.cycles-pp.intel_idle 25.10 ± 3% -25.1 0.00 perf-profile.children.cycles-pp.ret_from_fork 25.09 ± 3% -25.1 0.00 perf-profile.children.cycles-pp.kthread 15.39 ± 5% -15.4 0.00 perf-profile.children.cycles-pp.nfsd 13.05 ± 6% -13.1 0.00 perf-profile.children.cycles-pp.apic_timer_interrupt 11.13 ± 7% -11.1 0.00 perf-profile.children.cycles-pp.smp_apic_timer_interrupt 10.37 ± 5% -10.4 0.00 perf-profile.children.cycles-pp.svc_process 10.36 ± 5% -10.4 0.00 perf-profile.children.cycles-pp.svc_process_common 9.87 ± 5% -9.9 0.00 perf-profile.children.cycles-pp.nfsd_dispatch 9.50 ± 4% -9.5 0.00 perf-profile.children.cycles-pp.nfsd4_proc_compound 9.18 -9.2 0.00 perf-profile.children.cycles-pp.worker_thread 8.06 ± 2% -8.1 0.00 perf-profile.children.cycles-pp.process_one_work 6.30 ± 8% -6.3 0.00 perf-profile.children.cycles-pp.hrtimer_interrupt 5.87 ± 6% -5.9 0.00 perf-profile.children.cycles-pp.menu_select 5.38 ± 3% -5.4 0.00 perf-profile.children.cycles-pp.try_to_wake_up 5.27 ± 3% -5.3 0.00 perf-profile.children.cycles-pp.__softirqentry_text_start 44.60 -44.6 0.00 perf-profile.self.cycles-pp.intel_idle 14239 ± 3% +41.4% 20128 ± 3% slabinfo.Acpi-State.active_objs 283.33 ± 2% +40.2% 397.33 ± 3% slabinfo.Acpi-State.active_slabs 14459 ± 2% +40.4% 20295 ± 3% slabinfo.Acpi-State.num_objs 283.33 ± 2% +40.2% 397.33 ± 3% slabinfo.Acpi-State.num_slabs 664.67 ± 2% -25.0% 498.33 ± 9% slabinfo.UDP.active_objs 664.67 ± 2% -25.0% 498.33 ± 9% slabinfo.UDP.num_objs 2092 ± 4% -7.2% 1942 ± 4% slabinfo.UNIX.active_objs 2092 ± 4% -7.2% 1942 ± 4% slabinfo.UNIX.num_objs 18631 -16.6% 15540 ± 2% slabinfo.anon_vma.active_objs 18631 -16.6% 15540 ± 2% slabinfo.anon_vma.num_objs 40301 ± 2% -18.5% 32862 slabinfo.anon_vma_chain.active_objs 629.33 ± 2% -18.5% 512.67 slabinfo.anon_vma_chain.active_slabs 40302 ± 2% -18.5% 32862 slabinfo.anon_vma_chain.num_objs 629.33 ± 2% -18.5% 512.67 slabinfo.anon_vma_chain.num_slabs 57068 ± 5% -17.4% 47146 ± 6% slabinfo.cred_jar.active_objs 1387 ± 5% -15.8% 1167 ± 6% slabinfo.cred_jar.active_slabs 58269 ± 5% -15.8% 49064 ± 6% slabinfo.cred_jar.num_objs 1387 ± 5% -15.8% 1167 ± 6% slabinfo.cred_jar.num_slabs 4444 ± 5% -17.3% 3677 ± 10% slabinfo.eventpoll_pwq.active_objs 4444 ± 5% -17.3% 3677 ± 10% slabinfo.eventpoll_pwq.num_objs 1024 ± 6% -18.5% 835.00 ± 12% slabinfo.file_lock_cache.active_objs 1024 ± 6% -18.5% 835.00 ± 12% slabinfo.file_lock_cache.num_objs 65114 -9.6% 58889 slabinfo.inode_cache.active_objs 1210 -9.9% 1090 slabinfo.inode_cache.active_slabs 65387 -9.9% 58889 slabinfo.inode_cache.num_objs 1210 -9.9% 1090 slabinfo.inode_cache.num_slabs 4006 ± 3% -18.4% 3267 ± 6% slabinfo.ip6-frags.active_objs 4006 ± 3% -18.4% 3267 ± 6% slabinfo.ip6-frags.num_objs 731.00 ± 8% -55.8% 323.33 ± 8% slabinfo.khugepaged_mm_slot.active_objs 731.00 ± 8% -55.8% 323.33 ± 8% slabinfo.khugepaged_mm_slot.num_objs 6830 -23.0% 5256 ± 4% slabinfo.kmalloc-2k.active_objs 431.67 -23.8% 329.00 ± 5% slabinfo.kmalloc-2k.active_slabs 6912 -23.8% 5269 ± 5% slabinfo.kmalloc-2k.num_objs 431.67 -23.8% 329.00 ± 5% slabinfo.kmalloc-2k.num_slabs 1396 -10.1% 1254 slabinfo.kmalloc-4k.num_objs 1142 ± 2% -18.7% 929.00 ± 5% slabinfo.kmem_cache_node.active_objs 1173 ± 2% -18.2% 960.00 ± 5% slabinfo.kmem_cache_node.num_objs 2173 ± 3% -15.7% 1832 ± 2% slabinfo.nfs_read_data.active_objs 2173 ± 3% -15.7% 1832 ± 2% slabinfo.nfs_read_data.num_objs 12844 ± 4% +51.7% 19489 ± 7% slabinfo.nfsd_file.active_objs 245.00 ± 4% +50.9% 369.67 ± 7% slabinfo.nfsd_file.active_slabs 13011 ± 4% +50.8% 19625 ± 8% slabinfo.nfsd_file.num_objs 245.00 ± 4% +50.9% 369.67 ± 7% slabinfo.nfsd_file.num_slabs 640.67 ± 4% +29.0% 826.67 ± 12% slabinfo.numa_policy.active_objs 640.67 ± 4% +29.0% 826.67 ± 12% slabinfo.numa_policy.num_objs 3006 -15.2% 2549 ± 4% slabinfo.pool_workqueue.active_objs 3006 -15.2% 2549 ± 4% slabinfo.pool_workqueue.num_objs 15026 -35.5% 9688 ± 3% slabinfo.proc_inode_cache.active_objs 313.00 -35.7% 201.33 ± 3% slabinfo.proc_inode_cache.active_slabs 15047 -35.6% 9689 ± 3% slabinfo.proc_inode_cache.num_objs 313.00 -35.7% 201.33 ± 3% slabinfo.proc_inode_cache.num_slabs 2878 -8.9% 2622 ± 2% slabinfo.skbuff_ext_cache.active_objs 2878 -8.9% 2622 ± 2% slabinfo.skbuff_ext_cache.num_objs 2947 ± 2% -17.4% 2435 ± 5% slabinfo.skbuff_fclone_cache.active_objs 2947 ± 2% -17.4% 2435 ± 5% slabinfo.skbuff_fclone_cache.num_objs 36819 -12.9% 32082 ± 2% slabinfo.vm_area_struct.active_objs 921.00 -12.9% 802.33 ± 2% slabinfo.vm_area_struct.active_slabs 36860 -12.9% 32111 ± 2% slabinfo.vm_area_struct.num_objs 921.00 -12.9% 802.33 ± 2% slabinfo.vm_area_struct.num_slabs 17176 -14.0% 14769 ± 2% slabinfo.vmap_area.active_objs 17181 -14.0% 14773 ± 2% slabinfo.vmap_area.num_objs 3222 ± 3% -19.2% 2604 ± 6% slabinfo.xfs_btree_cur.active_objs 3222 ± 3% -19.2% 2604 ± 6% slabinfo.xfs_btree_cur.num_objs 4924 ± 3% -14.9% 4188 ± 6% slabinfo.xfs_buf_item.active_objs 4924 ± 3% -14.9% 4188 ± 6% slabinfo.xfs_buf_item.num_objs 2466 ± 5% -15.5% 2084 ± 5% slabinfo.xfs_da_state.active_objs 2466 ± 5% -15.5% 2084 ± 5% slabinfo.xfs_da_state.num_objs 129558 ± 2% -86.7% 17262 ± 51% softirqs.CPU0.RCU 44027 ± 2% +174.6% 120904 softirqs.CPU0.SCHED 145672 ± 19% -89.2% 15679 ± 8% softirqs.CPU0.TIMER 130370 ± 5% -84.8% 19822 ± 53% softirqs.CPU1.RCU 43639 ± 10% -75.9% 10520 ± 19% softirqs.CPU1.SCHED 144764 ± 19% -86.8% 19169 ± 12% softirqs.CPU1.TIMER 127644 ± 7% -85.5% 18560 ± 64% softirqs.CPU10.RCU 40587 ± 9% -89.9% 4106 ± 41% softirqs.CPU10.SCHED 144760 ± 19% -89.5% 15252 ± 37% softirqs.CPU10.TIMER 129136 ± 3% -85.8% 18386 ± 65% softirqs.CPU11.RCU 40093 ± 10% -90.9% 3659 ± 43% softirqs.CPU11.SCHED 144832 ± 19% -89.3% 15450 ± 40% softirqs.CPU11.TIMER 128347 ± 5% -86.1% 17808 ± 67% softirqs.CPU12.RCU 41215 ± 7% -89.4% 4363 ± 52% softirqs.CPU12.SCHED 145372 ± 18% -88.7% 16404 ± 45% softirqs.CPU12.TIMER 124392 -81.6% 22865 ± 65% softirqs.CPU13.RCU 42461 ± 11% -89.7% 4355 ± 39% softirqs.CPU13.SCHED 145027 ± 20% -88.0% 17400 ± 37% softirqs.CPU13.TIMER 131480 ± 4% -83.5% 21712 ± 63% softirqs.CPU14.RCU 37626 ± 15% -87.8% 4594 ± 35% softirqs.CPU14.SCHED 144223 ± 19% -88.7% 16298 ± 31% softirqs.CPU14.TIMER 134269 ± 6% -85.5% 19486 ± 60% softirqs.CPU15.RCU 40312 ± 13% -88.2% 4762 ± 30% softirqs.CPU15.SCHED 144302 ± 18% -88.1% 17147 ± 27% softirqs.CPU15.TIMER 118147 ± 4% -81.6% 21754 ± 67% softirqs.CPU16.RCU 40934 ± 7% -88.6% 4658 ± 46% softirqs.CPU16.SCHED 145660 ± 18% -89.0% 16071 ± 41% softirqs.CPU16.TIMER 116966 ± 3% -83.9% 18835 ± 66% softirqs.CPU17.RCU 40113 ± 10% -91.1% 3585 ± 39% softirqs.CPU17.SCHED 144409 ± 19% -90.5% 13747 ± 35% softirqs.CPU17.TIMER 118483 ± 4% -82.7% 20514 ± 67% softirqs.CPU18.RCU 41764 ± 6% -90.5% 3961 ± 55% softirqs.CPU18.SCHED 144284 ± 19% -88.6% 16386 ± 42% softirqs.CPU18.TIMER 115838 ± 2% -82.5% 20283 ± 68% softirqs.CPU19.RCU 33697 ± 18% -87.5% 4229 ± 46% softirqs.CPU19.SCHED 145363 ± 20% -89.1% 15884 ± 43% softirqs.CPU19.TIMER 126950 ± 2% -84.9% 19164 ± 60% softirqs.CPU2.RCU 41017 ± 9% -85.6% 5891 ± 15% softirqs.CPU2.SCHED 146833 ± 19% -88.8% 16437 ± 28% softirqs.CPU2.TIMER 117523 ± 3% -80.8% 22549 ± 66% softirqs.CPU20.RCU 40456 ± 8% -87.5% 5052 ± 43% softirqs.CPU20.SCHED 145515 ± 19% -87.9% 17670 ± 39% softirqs.CPU20.TIMER 117252 ± 4% -79.9% 23519 ± 51% softirqs.CPU21.RCU 33894 ± 36% -78.4% 7316 ± 18% softirqs.CPU21.SCHED 144583 ± 19% -86.2% 19976 ± 13% softirqs.CPU21.TIMER 117672 ± 3% -81.4% 21888 ± 66% softirqs.CPU22.RCU 41423 ± 8% -88.2% 4875 ± 47% softirqs.CPU22.SCHED 144922 ± 19% -88.7% 16332 ± 41% softirqs.CPU22.TIMER 117482 ± 2% -80.0% 23508 ± 69% softirqs.CPU23.RCU 39447 ± 14% -86.4% 5370 ± 50% softirqs.CPU23.SCHED 145047 ± 20% -88.0% 17448 ± 45% softirqs.CPU23.TIMER 108547 ± 8% -90.6% 10186 ±108% softirqs.CPU24.RCU 42818 ± 7% -92.2% 3358 ± 64% softirqs.CPU24.SCHED 127697 ± 20% -91.3% 11075 ± 38% softirqs.CPU24.TIMER 106607 ± 8% -90.4% 10274 ±118% softirqs.CPU25.RCU 43408 ± 10% -91.4% 3748 ± 70% softirqs.CPU25.SCHED 124072 ± 23% -89.5% 13014 ± 56% softirqs.CPU25.TIMER 107077 ± 8% -86.7% 14211 ± 90% softirqs.CPU26.RCU 43321 ± 5% -94.3% 2490 ± 87% softirqs.CPU26.SCHED 124684 ± 22% -90.7% 11633 ± 53% softirqs.CPU26.TIMER 110934 ± 12% -90.8% 10161 ±115% softirqs.CPU27.RCU 40868 ± 9% -91.9% 3313 ± 59% softirqs.CPU27.SCHED 125133 ± 20% -90.6% 11750 ± 49% softirqs.CPU27.TIMER 108689 ± 10% -91.6% 9126 ±117% softirqs.CPU28.RCU 41369 ± 9% -92.9% 2939 ± 58% softirqs.CPU28.SCHED 124554 ± 22% -91.2% 10953 ± 47% softirqs.CPU28.TIMER 108255 ± 9% -90.3% 10541 ±124% softirqs.CPU29.RCU 41043 ± 9% -92.7% 3011 ± 58% softirqs.CPU29.SCHED 124562 ± 22% -91.5% 10539 ± 46% softirqs.CPU29.TIMER 129694 ± 3% -84.5% 20124 ± 62% softirqs.CPU3.RCU 40654 ± 10% -87.3% 5175 ± 20% softirqs.CPU3.SCHED 145402 ± 18% -88.8% 16224 ± 27% softirqs.CPU3.TIMER 108416 ± 8% -90.6% 10179 ±126% softirqs.CPU30.RCU 40454 ± 7% -92.5% 3054 ± 70% softirqs.CPU30.SCHED 124502 ± 23% -91.4% 10745 ± 55% softirqs.CPU30.TIMER 108899 ± 9% -90.0% 10845 ±120% softirqs.CPU31.RCU 41965 ± 7% -91.6% 3532 ± 70% softirqs.CPU31.SCHED 123898 ± 23% -90.2% 12113 ± 54% softirqs.CPU31.TIMER 118481 ± 5% -90.0% 11900 ±114% softirqs.CPU32.RCU 43647 -92.2% 3388 ± 58% softirqs.CPU32.SCHED 124804 ± 22% -90.3% 12106 ± 45% softirqs.CPU32.TIMER 121522 ± 6% -91.6% 10217 ±109% softirqs.CPU33.RCU 40217 ± 6% -92.3% 3094 ± 49% softirqs.CPU33.SCHED 124861 ± 22% -90.6% 11795 ± 37% softirqs.CPU33.TIMER 123703 ± 7% -92.7% 8984 ±121% softirqs.CPU34.RCU 40926 ± 11% -92.5% 3060 ± 65% softirqs.CPU34.SCHED 125043 ± 22% -91.5% 10678 ± 52% softirqs.CPU34.TIMER 125291 ± 8% -90.1% 12413 ±122% softirqs.CPU35.RCU 42116 ± 6% -92.5% 3153 ± 63% softirqs.CPU35.SCHED 125324 ± 22% -90.9% 11389 ± 48% softirqs.CPU35.TIMER 122785 ± 7% -91.1% 10885 ±127% softirqs.CPU36.RCU 41551 ± 8% -92.1% 3296 ± 69% softirqs.CPU36.SCHED 123750 ± 22% -91.3% 10768 ± 52% softirqs.CPU36.TIMER 117884 ± 11% -91.9% 9532 ±125% softirqs.CPU37.RCU 41300 ± 6% -93.9% 2513 ± 48% softirqs.CPU37.SCHED 124307 ± 24% -92.1% 9835 ± 44% softirqs.CPU37.TIMER 122727 ± 8% -90.9% 11120 ±123% softirqs.CPU38.RCU 41960 ± 6% -92.0% 3336 ± 66% softirqs.CPU38.SCHED 125038 ± 22% -90.4% 11951 ± 55% softirqs.CPU38.TIMER 124565 ± 8% -91.9% 10150 ±123% softirqs.CPU39.RCU 41361 ± 6% -91.8% 3380 ± 73% softirqs.CPU39.SCHED 124458 ± 22% -90.5% 11818 ± 62% softirqs.CPU39.TIMER 129928 ± 3% -85.5% 18877 ± 58% softirqs.CPU4.RCU 38601 ± 12% -88.3% 4532 ± 6% softirqs.CPU4.SCHED 145316 ± 18% -89.6% 15075 ± 16% softirqs.CPU4.TIMER 124403 ± 8% -92.9% 8841 ±122% softirqs.CPU40.RCU 42396 ± 4% -92.7% 3079 ± 68% softirqs.CPU40.SCHED 124073 ± 22% -91.6% 10384 ± 54% softirqs.CPU40.TIMER 121808 ± 7% -91.0% 10960 ±125% softirqs.CPU41.RCU 42839 ± 4% -91.9% 3474 ± 75% softirqs.CPU41.SCHED 124429 ± 23% -90.4% 11992 ± 63% softirqs.CPU41.TIMER 123150 ± 8% -90.9% 11203 ±113% softirqs.CPU42.RCU 41175 ± 9% -92.0% 3302 ± 57% softirqs.CPU42.SCHED 123737 ± 22% -90.7% 11452 ± 42% softirqs.CPU42.TIMER 123312 ± 8% -90.8% 11397 ±121% softirqs.CPU43.RCU 40843 ± 9% -92.6% 3005 ± 62% softirqs.CPU43.SCHED 123738 ± 22% -91.5% 10523 ± 49% softirqs.CPU43.TIMER 122130 ± 8% -91.9% 9896 ±128% softirqs.CPU44.RCU 42237 ± 6% -92.2% 3281 ± 72% softirqs.CPU44.SCHED 125045 ± 22% -91.2% 10996 ± 58% softirqs.CPU44.TIMER 125539 ± 10% -92.5% 9469 ±104% softirqs.CPU45.RCU 40783 ± 12% -91.5% 3466 ± 51% softirqs.CPU45.SCHED 124918 ± 24% -90.0% 12473 ± 38% softirqs.CPU45.TIMER 124834 ± 9% -92.5% 9319 ±118% softirqs.CPU46.RCU 42622 ± 5% -92.9% 3010 ± 59% softirqs.CPU46.SCHED 125566 ± 23% -91.2% 11019 ± 49% softirqs.CPU46.TIMER 122750 ± 7% -92.0% 9793 ±111% softirqs.CPU47.RCU 38070 ± 10% -91.8% 3112 ± 52% softirqs.CPU47.SCHED 124618 ± 23% -90.9% 11283 ± 38% softirqs.CPU47.TIMER 130092 -87.2% 16600 ± 59% softirqs.CPU48.RCU 41675 ± 7% -90.8% 3834 ± 26% softirqs.CPU48.SCHED 145714 ± 19% -90.2% 14333 ± 26% softirqs.CPU48.TIMER 131255 ± 3% -85.2% 19362 ± 65% softirqs.CPU49.RCU 40699 ± 10% -88.3% 4781 ± 43% softirqs.CPU49.SCHED 146107 ± 17% -89.4% 15466 ± 39% softirqs.CPU49.TIMER 123981 ± 6% -82.9% 21160 ± 65% softirqs.CPU5.RCU 40399 ± 9% -89.2% 4359 ± 40% softirqs.CPU5.SCHED 168168 ± 2% -90.9% 15245 ± 36% softirqs.CPU5.TIMER 129277 -85.9% 18189 ± 63% softirqs.CPU50.RCU 42586 ± 5% -89.3% 4542 ± 43% softirqs.CPU50.SCHED 147516 ± 18% -90.7% 13785 ± 37% softirqs.CPU50.TIMER 130976 ± 2% -84.1% 20797 ± 67% softirqs.CPU51.RCU 41146 ± 7% -87.6% 5117 ± 51% softirqs.CPU51.SCHED 144352 ± 17% -88.4% 16707 ± 47% softirqs.CPU51.TIMER 132196 -84.8% 20157 ± 67% softirqs.CPU52.RCU 41508 ± 6% -88.4% 4810 ± 47% softirqs.CPU52.SCHED 145354 ± 17% -88.8% 16263 ± 41% softirqs.CPU52.TIMER 129252 ± 6% -83.4% 21513 ± 69% softirqs.CPU53.RCU 42460 ± 4% -87.2% 5436 ± 49% softirqs.CPU53.SCHED 145096 ± 18% -88.7% 16453 ± 43% softirqs.CPU53.TIMER 128690 -85.4% 18780 ± 66% softirqs.CPU54.RCU 41371 ± 8% -88.4% 4814 ± 44% softirqs.CPU54.SCHED 146528 ± 17% -89.7% 15091 ± 39% softirqs.CPU54.TIMER 130216 -84.4% 20277 ± 61% softirqs.CPU55.RCU 42062 ± 5% -88.4% 4858 ± 32% softirqs.CPU55.SCHED 144703 ± 19% -88.3% 16889 ± 34% softirqs.CPU55.TIMER 130288 ± 2% -84.2% 20595 ± 66% softirqs.CPU56.RCU 39897 ± 10% -88.4% 4614 ± 47% softirqs.CPU56.SCHED 144468 ± 18% -89.6% 14980 ± 38% softirqs.CPU56.TIMER 129500 ± 2% -87.3% 16442 ± 67% softirqs.CPU57.RCU 34368 ± 20% -88.1% 4085 ± 44% softirqs.CPU57.SCHED 146523 ± 17% -90.7% 13608 ± 42% softirqs.CPU57.TIMER 128066 -84.7% 19633 ± 64% softirqs.CPU58.RCU 39533 ± 10% -87.5% 4951 ± 45% softirqs.CPU58.SCHED 146108 ± 18% -89.0% 16089 ± 40% softirqs.CPU58.TIMER 127515 -85.2% 18843 ± 68% softirqs.CPU59.RCU 38242 ± 7% -87.7% 4686 ± 49% softirqs.CPU59.SCHED 145535 ± 18% -90.0% 14564 ± 44% softirqs.CPU59.TIMER 130081 ± 2% -83.8% 21026 ± 60% softirqs.CPU6.RCU 40883 ± 10% -87.8% 4999 ± 31% softirqs.CPU6.SCHED 145340 ± 19% -87.8% 17737 ± 26% softirqs.CPU6.TIMER 128412 ± 2% -85.7% 18398 ± 66% softirqs.CPU60.RCU 40839 ± 8% -88.6% 4653 ± 45% softirqs.CPU60.SCHED 146058 ± 17% -90.1% 14465 ± 39% softirqs.CPU60.TIMER 123637 ± 3% -86.3% 16971 ± 67% softirqs.CPU61.RCU 39657 ± 10% -87.7% 4895 ± 50% softirqs.CPU61.SCHED 145869 ± 19% -89.7% 15080 ± 45% softirqs.CPU61.TIMER 129321 ± 4% -85.9% 18200 ± 52% softirqs.CPU62.RCU 40715 ± 9% -85.7% 5834 ± 13% softirqs.CPU62.SCHED 144125 ± 19% -88.0% 17354 ± 15% softirqs.CPU62.TIMER 129748 ± 4% -85.8% 18440 ± 68% softirqs.CPU63.RCU 35168 ± 9% -86.3% 4801 ± 43% softirqs.CPU63.SCHED 145027 ± 18% -90.0% 14553 ± 36% softirqs.CPU63.TIMER 118969 ± 3% -83.5% 19641 ± 65% softirqs.CPU64.RCU 41706 ± 6% -87.7% 5137 ± 41% softirqs.CPU64.SCHED 145820 ± 16% -89.2% 15720 ± 35% softirqs.CPU64.TIMER 115373 -84.5% 17864 ± 59% softirqs.CPU65.RCU 40569 ± 9% -89.0% 4479 ± 27% softirqs.CPU65.SCHED 145217 ± 17% -89.9% 14638 ± 18% softirqs.CPU65.TIMER 120019 ± 3% -85.7% 17187 ± 62% softirqs.CPU66.RCU 40995 ± 8% -88.6% 4663 ± 35% softirqs.CPU66.SCHED 145363 ± 17% -89.8% 14801 ± 30% softirqs.CPU66.TIMER 117154 ± 3% -86.4% 15920 ± 67% softirqs.CPU67.RCU 41413 ± 7% -90.3% 3998 ± 44% softirqs.CPU67.SCHED 146697 ± 19% -90.8% 13493 ± 38% softirqs.CPU67.TIMER 113076 -85.3% 16648 ± 68% softirqs.CPU68.RCU 41993 ± 6% -89.6% 4358 ± 44% softirqs.CPU68.SCHED 146202 ± 18% -90.1% 14411 ± 41% softirqs.CPU68.TIMER 113226 ± 2% -84.7% 17373 ± 68% softirqs.CPU69.RCU 46191 ± 25% -90.4% 4418 ± 46% softirqs.CPU69.SCHED 144604 ± 18% -90.3% 13970 ± 41% softirqs.CPU69.TIMER 130032 ± 3% -85.2% 19299 ± 62% softirqs.CPU7.RCU 37394 ± 18% -90.2% 3663 ± 31% softirqs.CPU7.SCHED 144866 ± 19% -90.1% 14327 ± 29% softirqs.CPU7.TIMER 117576 ± 3% -85.9% 16562 ± 67% softirqs.CPU70.RCU 39953 ± 11% -86.0% 5574 ± 56% softirqs.CPU70.SCHED 145159 ± 18% -88.4% 16876 ± 50% softirqs.CPU70.TIMER 115774 ± 3% -84.7% 17659 ± 66% softirqs.CPU71.RCU 40438 ± 11% -87.5% 5035 ± 44% softirqs.CPU71.SCHED 146453 ± 19% -89.3% 15742 ± 39% softirqs.CPU71.TIMER 109444 ± 8% -92.7% 8041 ±112% softirqs.CPU72.RCU 43454 ± 4% -93.1% 3017 ± 58% softirqs.CPU72.SCHED 127964 ± 20% -91.2% 11254 ± 39% softirqs.CPU72.TIMER 107811 ± 7% -92.8% 7813 ±122% softirqs.CPU73.RCU 32279 ± 34% -92.1% 2553 ± 48% softirqs.CPU73.SCHED 123487 ± 22% -92.1% 9793 ± 49% softirqs.CPU73.TIMER 109593 ± 9% -91.3% 9555 ±114% softirqs.CPU74.RCU 43164 ± 3% -94.0% 2600 ± 56% softirqs.CPU74.SCHED 124080 ± 22% -90.7% 11486 ± 36% softirqs.CPU74.TIMER 114309 ± 7% -89.9% 11517 ± 84% softirqs.CPU75.RCU 37943 ± 21% -92.6% 2792 ± 32% softirqs.CPU75.SCHED 129139 ± 21% -88.4% 15004 ± 42% softirqs.CPU75.TIMER 109421 ± 8% -90.6% 10286 ±112% softirqs.CPU76.RCU 43201 ± 5% -91.6% 3645 ± 57% softirqs.CPU76.SCHED 125002 ± 21% -90.4% 11944 ± 43% softirqs.CPU76.TIMER 108868 ± 9% -93.0% 7653 ±109% softirqs.CPU77.RCU 43599 ± 5% -92.9% 3085 ± 53% softirqs.CPU77.SCHED 125041 ± 21% -91.8% 10243 ± 41% softirqs.CPU77.TIMER 108056 ± 8% -94.4% 6044 ±108% softirqs.CPU78.RCU 41609 ± 9% -93.3% 2795 ± 51% softirqs.CPU78.SCHED 124219 ± 22% -92.3% 9623 ± 39% softirqs.CPU78.TIMER 107752 ± 8% -92.2% 8456 ±110% softirqs.CPU79.RCU 42007 ± 8% -92.8% 3004 ± 46% softirqs.CPU79.SCHED 125166 ± 21% -91.7% 10367 ± 35% softirqs.CPU79.TIMER 129779 ± 3% -84.9% 19646 ± 62% softirqs.CPU8.RCU 41355 ± 7% -89.1% 4498 ± 45% softirqs.CPU8.SCHED 145182 ± 20% -87.6% 17993 ± 37% softirqs.CPU8.TIMER 115269 ± 6% -91.1% 10226 ±115% softirqs.CPU80.RCU 43084 ± 4% -91.7% 3593 ± 59% softirqs.CPU80.SCHED 147652 ± 20% -90.9% 13430 ± 59% softirqs.CPU80.TIMER 121994 ± 8% -91.8% 9987 ±111% softirqs.CPU81.RCU 38755 ± 12% -91.3% 3367 ± 55% softirqs.CPU81.SCHED 125237 ± 21% -90.7% 11627 ± 45% softirqs.CPU81.TIMER 121325 ± 6% -92.7% 8826 ± 97% softirqs.CPU82.RCU 42853 ± 3% -92.8% 3071 ± 34% softirqs.CPU82.SCHED 127194 ± 20% -91.3% 11112 ± 24% softirqs.CPU82.TIMER 123822 ± 8% -92.5% 9321 ±114% softirqs.CPU83.RCU 41338 ± 11% -93.4% 2723 ± 38% softirqs.CPU83.SCHED 125880 ± 21% -91.2% 11079 ± 43% softirqs.CPU83.TIMER 122309 ± 7% -92.7% 8987 ±106% softirqs.CPU84.RCU 41554 ± 8% -90.9% 3788 ± 56% softirqs.CPU84.SCHED 123759 ± 22% -90.2% 12126 ± 41% softirqs.CPU84.TIMER 117837 ± 10% -92.4% 8917 ±115% softirqs.CPU85.RCU 39912 ± 10% -92.8% 2871 ± 43% softirqs.CPU85.SCHED 148973 ± 20% -93.2% 10068 ± 38% softirqs.CPU85.TIMER 120884 ± 8% -91.4% 10355 ± 96% softirqs.CPU86.RCU 46293 ± 16% -93.5% 3006 ± 40% softirqs.CPU86.SCHED 125358 ± 21% -91.6% 10533 ± 32% softirqs.CPU86.TIMER 121885 ± 7% -93.3% 8226 ±106% softirqs.CPU87.RCU 42860 ± 4% -93.6% 2751 ± 42% softirqs.CPU87.SCHED 125707 ± 22% -92.5% 9396 ± 34% softirqs.CPU87.TIMER 121188 ± 6% -91.8% 9889 ±121% softirqs.CPU88.RCU 42696 ± 5% -92.8% 3055 ± 53% softirqs.CPU88.SCHED 124819 ± 21% -90.8% 11532 ± 56% softirqs.CPU88.TIMER 121801 ± 6% -93.5% 7910 ±117% softirqs.CPU89.RCU 39818 ± 10% -92.5% 2985 ± 57% softirqs.CPU89.SCHED 124832 ± 22% -92.1% 9833 ± 46% softirqs.CPU89.TIMER 128275 ± 3% -84.9% 19346 ± 64% softirqs.CPU9.RCU 37552 ± 8% -90.2% 3674 ± 53% softirqs.CPU9.SCHED 146343 ± 18% -89.2% 15862 ± 36% softirqs.CPU9.TIMER 121823 ± 6% -93.8% 7592 ±121% softirqs.CPU90.RCU 42372 ± 6% -94.2% 2459 ± 45% softirqs.CPU90.SCHED 123964 ± 22% -92.6% 9139 ± 47% softirqs.CPU90.TIMER 122019 ± 6% -93.8% 7553 ±104% softirqs.CPU91.RCU 44761 ± 17% -92.9% 3171 ± 46% softirqs.CPU91.SCHED 124510 ± 22% -91.5% 10612 ± 38% softirqs.CPU91.TIMER 121389 ± 7% -92.8% 8750 ±100% softirqs.CPU92.RCU 41735 ± 9% -91.4% 3587 ± 32% softirqs.CPU92.SCHED 125706 ± 21% -90.6% 11843 ± 37% softirqs.CPU92.TIMER 119468 ± 6% -93.1% 8265 ± 98% softirqs.CPU93.RCU 36218 ± 20% -91.1% 3227 ± 47% softirqs.CPU93.SCHED 124400 ± 22% -90.6% 11700 ± 40% softirqs.CPU93.TIMER 120903 ± 6% -92.9% 8593 ±100% softirqs.CPU94.RCU 42764 ± 5% -91.6% 3581 ± 47% softirqs.CPU94.SCHED 125636 ± 21% -89.6% 13039 ± 42% softirqs.CPU94.TIMER 120419 ± 6% -90.8% 11084 ±109% softirqs.CPU95.RCU 41261 ± 10% -92.9% 2939 ± 29% softirqs.CPU95.SCHED 124885 ± 22% -89.5% 13090 ± 48% softirqs.CPU95.TIMER 11644837 ± 2% -88.0% 1394948 ± 4% softirqs.RCU 3929601 -87.3% 497586 softirqs.SCHED 13046259 -90.0% 1302756 softirqs.TIMER 87.67 ±137% -100.0% 0.00 interrupts.122:PCI-MSI.31981655-edge.i40e-eth0-TxRx-86 313.00 ± 8% -19.0% 253.67 ± 4% interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 1.33 ±141% +3850.0% 52.67 ±119% interrupts.98:PCI-MSI.31981631-edge.i40e-eth0-TxRx-62 41042 -53.1% 19260 interrupts.CAL:Function_call_interrupts 231.00 -12.0% 203.33 interrupts.CPU0.CAL:Function_call_interrupts 671654 -97.0% 20309 ± 30% interrupts.CPU0.LOC:Local_timer_interrupts 311.33 ± 54% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts 311.33 ± 54% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts 2265 ± 63% +9649.1% 220881 interrupts.CPU0.RES:Rescheduling_interrupts 363.67 ± 25% -44.4% 202.33 interrupts.CPU1.CAL:Function_call_interrupts 671668 -95.2% 32309 ± 30% interrupts.CPU1.LOC:Local_timer_interrupts 266.67 ± 57% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts 266.67 ± 57% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts 429.00 -52.9% 202.00 interrupts.CPU10.CAL:Function_call_interrupts 671618 -96.6% 22954 ± 65% interrupts.CPU10.LOC:Local_timer_interrupts 325.33 ± 60% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts 325.33 ± 60% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts 428.33 -52.8% 202.00 interrupts.CPU11.CAL:Function_call_interrupts 671682 -96.6% 23063 ± 68% interrupts.CPU11.LOC:Local_timer_interrupts 405.33 ± 59% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts 405.33 ± 59% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts 429.33 -52.4% 204.33 interrupts.CPU12.CAL:Function_call_interrupts 671601 -96.1% 25872 ± 69% interrupts.CPU12.LOC:Local_timer_interrupts 398.33 ± 64% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts 398.33 ± 64% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.33 interrupts.CPU13.CAL:Function_call_interrupts 671604 -95.6% 29551 ± 60% interrupts.CPU13.LOC:Local_timer_interrupts 278.00 ± 64% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts 278.00 ± 64% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts 3519 ± 57% -73.1% 947.33 ± 70% interrupts.CPU13.RES:Rescheduling_interrupts 428.67 -52.1% 205.33 interrupts.CPU14.CAL:Function_call_interrupts 671749 -96.2% 25712 ± 53% interrupts.CPU14.LOC:Local_timer_interrupts 373.67 ± 68% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts 373.67 ± 68% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.67 interrupts.CPU15.CAL:Function_call_interrupts 671719 -96.2% 25353 ± 44% interrupts.CPU15.LOC:Local_timer_interrupts 355.00 ± 59% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts 355.00 ± 59% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts 2943 ± 24% -74.4% 752.67 ± 70% interrupts.CPU15.RES:Rescheduling_interrupts 429.33 -52.9% 202.33 interrupts.CPU16.CAL:Function_call_interrupts 671626 -96.2% 25389 ± 67% interrupts.CPU16.LOC:Local_timer_interrupts 398.67 ± 58% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts 398.67 ± 58% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.67 interrupts.CPU17.CAL:Function_call_interrupts 671589 -97.1% 19347 ± 65% interrupts.CPU17.LOC:Local_timer_interrupts 416.67 ± 58% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts 416.67 ± 58% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts 429.00 -52.6% 203.33 interrupts.CPU18.CAL:Function_call_interrupts 671595 -96.1% 26229 ± 67% interrupts.CPU18.LOC:Local_timer_interrupts 243.33 ± 52% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts 243.33 ± 52% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts 429.67 -52.8% 203.00 interrupts.CPU19.CAL:Function_call_interrupts 671616 -96.2% 25718 ± 69% interrupts.CPU19.LOC:Local_timer_interrupts 281.67 ± 66% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts 281.67 ± 66% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts 429.33 -52.6% 203.67 interrupts.CPU2.CAL:Function_call_interrupts 671599 -96.4% 23988 ± 49% interrupts.CPU2.LOC:Local_timer_interrupts 265.00 ± 59% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 265.00 ± 59% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.67 interrupts.CPU20.CAL:Function_call_interrupts 671567 -95.6% 29381 ± 68% interrupts.CPU20.LOC:Local_timer_interrupts 180.67 ± 54% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts 180.67 ± 54% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts 429.00 -52.7% 203.00 interrupts.CPU21.CAL:Function_call_interrupts 671595 -95.2% 31973 ± 27% interrupts.CPU21.LOC:Local_timer_interrupts 244.33 ± 71% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts 244.33 ± 71% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.67 interrupts.CPU22.CAL:Function_call_interrupts 671605 -96.2% 25401 ± 64% interrupts.CPU22.LOC:Local_timer_interrupts 315.00 ± 57% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts 315.00 ± 57% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts 429.67 -52.9% 202.33 interrupts.CPU23.CAL:Function_call_interrupts 671582 -95.5% 30159 ± 68% interrupts.CPU23.LOC:Local_timer_interrupts 248.00 ± 55% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts 248.00 ± 55% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts 432.67 -53.2% 202.33 interrupts.CPU24.CAL:Function_call_interrupts 671597 -98.3% 11281 ± 88% interrupts.CPU24.LOC:Local_timer_interrupts 185.67 ± 90% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts 185.67 ± 90% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts 429.33 -82.8% 73.67 ±124% interrupts.CPU25.CAL:Function_call_interrupts 671542 -97.2% 18712 ±127% interrupts.CPU25.LOC:Local_timer_interrupts 220.33 ± 86% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts 220.33 ± 86% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts 3103 ±101% -84.3% 487.00 ±137% interrupts.CPU25.RES:Rescheduling_interrupts 429.33 -53.3% 200.33 interrupts.CPU26.CAL:Function_call_interrupts 671563 -98.0% 13575 ±103% interrupts.CPU26.LOC:Local_timer_interrupts 245.33 ± 86% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts 245.33 ± 86% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts 429.00 -53.0% 201.67 interrupts.CPU27.CAL:Function_call_interrupts 671791 -97.8% 14786 ±118% interrupts.CPU27.LOC:Local_timer_interrupts 243.67 ±116% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts 243.67 ±116% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts 429.33 -53.0% 202.00 interrupts.CPU28.CAL:Function_call_interrupts 671644 -98.1% 12676 ±114% interrupts.CPU28.LOC:Local_timer_interrupts 224.33 ±103% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts 224.33 ±103% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts 429.33 -53.0% 202.00 interrupts.CPU29.CAL:Function_call_interrupts 671636 -98.3% 11093 ±122% interrupts.CPU29.LOC:Local_timer_interrupts 233.00 ± 88% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts 233.00 ± 88% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.67 interrupts.CPU3.CAL:Function_call_interrupts 671603 -96.2% 25466 ± 48% interrupts.CPU3.LOC:Local_timer_interrupts 270.67 ± 64% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 270.67 ± 64% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 428.00 -52.8% 202.00 interrupts.CPU30.CAL:Function_call_interrupts 671747 -98.1% 12709 ±129% interrupts.CPU30.LOC:Local_timer_interrupts 183.33 ± 91% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts 183.33 ± 91% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts 428.33 -52.6% 203.00 interrupts.CPU31.CAL:Function_call_interrupts 671630 -97.5% 16787 ±118% interrupts.CPU31.LOC:Local_timer_interrupts 168.67 ± 88% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts 168.67 ± 88% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.33 interrupts.CPU32.CAL:Function_call_interrupts 671667 -97.7% 15468 ±106% interrupts.CPU32.LOC:Local_timer_interrupts 160.00 ± 73% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts 160.00 ± 73% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts 428.33 -52.8% 202.33 interrupts.CPU33.CAL:Function_call_interrupts 671607 -97.9% 14334 ± 93% interrupts.CPU33.LOC:Local_timer_interrupts 114.33 ± 65% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts 114.33 ± 65% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts 2404 ± 81% -84.0% 385.33 ±134% interrupts.CPU33.RES:Rescheduling_interrupts 429.00 -52.9% 202.00 interrupts.CPU34.CAL:Function_call_interrupts 671631 -98.1% 12994 ±123% interrupts.CPU34.LOC:Local_timer_interrupts 184.67 ± 91% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts 184.67 ± 91% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts 424.33 -52.2% 202.67 interrupts.CPU35.CAL:Function_call_interrupts 671622 -98.1% 12719 ± 97% interrupts.CPU35.LOC:Local_timer_interrupts 187.67 ± 93% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts 187.67 ± 93% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts 429.00 -52.9% 202.00 interrupts.CPU36.CAL:Function_call_interrupts 671667 -98.3% 11249 ±126% interrupts.CPU36.LOC:Local_timer_interrupts 282.00 ± 90% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts 282.00 ± 90% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts 429.00 -52.9% 202.00 interrupts.CPU37.CAL:Function_call_interrupts 671615 -98.5% 10073 ±129% interrupts.CPU37.LOC:Local_timer_interrupts 312.00 ±102% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts 312.00 ±102% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts 431.00 -53.1% 202.00 interrupts.CPU38.CAL:Function_call_interrupts 671587 -97.6% 16107 ±120% interrupts.CPU38.LOC:Local_timer_interrupts 180.67 ± 74% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts 180.67 ± 74% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts 427.00 -52.5% 202.67 interrupts.CPU39.CAL:Function_call_interrupts 671592 -97.6% 16251 ±132% interrupts.CPU39.LOC:Local_timer_interrupts 156.67 ± 81% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts 156.67 ± 81% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts 428.67 -52.7% 202.67 interrupts.CPU4.CAL:Function_call_interrupts 671514 -96.7% 22263 ± 35% interrupts.CPU4.LOC:Local_timer_interrupts 175.00 ± 43% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts 175.00 ± 43% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts 429.00 -52.9% 202.00 interrupts.CPU40.CAL:Function_call_interrupts 671675 -98.2% 12350 ±129% interrupts.CPU40.LOC:Local_timer_interrupts 229.67 ± 88% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts 229.67 ± 88% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts 428.67 -52.7% 202.67 interrupts.CPU41.CAL:Function_call_interrupts 671645 -97.6% 15908 ±131% interrupts.CPU41.LOC:Local_timer_interrupts 204.67 ± 80% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts 204.67 ± 80% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts 428.67 -52.7% 202.67 interrupts.CPU42.CAL:Function_call_interrupts 671649 -97.9% 14397 ± 96% interrupts.CPU42.LOC:Local_timer_interrupts 235.33 ± 81% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts 235.33 ± 81% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts 428.67 -52.8% 202.33 interrupts.CPU43.CAL:Function_call_interrupts 671688 -98.2% 12358 ±116% interrupts.CPU43.LOC:Local_timer_interrupts 149.67 ± 48% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts 149.67 ± 48% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts 428.67 -52.7% 202.67 interrupts.CPU44.CAL:Function_call_interrupts 671754 -98.0% 13743 ±135% interrupts.CPU44.LOC:Local_timer_interrupts 139.67 ± 47% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts 139.67 ± 47% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts 429.33 -53.0% 202.00 interrupts.CPU45.CAL:Function_call_interrupts 671732 -97.9% 14425 ± 79% interrupts.CPU45.LOC:Local_timer_interrupts 183.67 ± 66% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts 183.67 ± 66% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts 428.33 -52.8% 202.00 interrupts.CPU46.CAL:Function_call_interrupts 671661 -98.0% 13495 ±112% interrupts.CPU46.LOC:Local_timer_interrupts 235.00 ± 98% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts 235.00 ± 98% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts 429.67 -52.8% 202.67 interrupts.CPU47.CAL:Function_call_interrupts 671670 -98.1% 13067 ± 96% interrupts.CPU47.LOC:Local_timer_interrupts 125.00 ± 71% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts 125.00 ± 71% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts 427.67 -52.6% 202.67 interrupts.CPU48.CAL:Function_call_interrupts 671600 -96.9% 21091 ± 52% interrupts.CPU48.LOC:Local_timer_interrupts 252.67 ± 72% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts 252.67 ± 72% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts 429.00 -53.4% 200.00 interrupts.CPU49.CAL:Function_call_interrupts 671595 -96.3% 25130 ± 64% interrupts.CPU49.LOC:Local_timer_interrupts 309.33 ± 77% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts 309.33 ± 77% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts 429.33 -52.9% 202.33 interrupts.CPU5.CAL:Function_call_interrupts 671622 -96.6% 22836 ± 59% interrupts.CPU5.LOC:Local_timer_interrupts 270.67 ± 61% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts 270.67 ± 61% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts 426.67 -52.3% 203.67 interrupts.CPU50.CAL:Function_call_interrupts 671607 -97.1% 19597 ± 63% interrupts.CPU50.LOC:Local_timer_interrupts 281.67 ± 82% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts 281.67 ± 82% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts 424.00 -52.3% 202.33 interrupts.CPU51.CAL:Function_call_interrupts 671620 -95.6% 29301 ± 72% interrupts.CPU51.LOC:Local_timer_interrupts 292.33 ± 70% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts 292.33 ± 70% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts 428.67 -52.6% 203.00 interrupts.CPU52.CAL:Function_call_interrupts 671563 -95.8% 27989 ± 64% interrupts.CPU52.LOC:Local_timer_interrupts 344.00 ± 56% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts 344.00 ± 56% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts 428.67 -52.7% 202.67 interrupts.CPU53.CAL:Function_call_interrupts 671584 -95.9% 27417 ± 67% interrupts.CPU53.LOC:Local_timer_interrupts 342.67 ± 57% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts 342.67 ± 57% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts 429.33 -52.4% 204.33 interrupts.CPU54.CAL:Function_call_interrupts 671640 -96.4% 24040 ± 61% interrupts.CPU54.LOC:Local_timer_interrupts 346.00 ± 54% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts 346.00 ± 54% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts 428.33 -52.5% 203.33 interrupts.CPU55.CAL:Function_call_interrupts 671660 -96.2% 25743 ± 49% interrupts.CPU55.LOC:Local_timer_interrupts 261.00 ± 57% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts 261.00 ± 57% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts 428.33 -52.6% 203.00 interrupts.CPU56.CAL:Function_call_interrupts 671694 -96.3% 24525 ± 63% interrupts.CPU56.LOC:Local_timer_interrupts 340.00 ± 56% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 340.00 ± 56% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 429.33 -52.3% 205.00 interrupts.CPU57.CAL:Function_call_interrupts 671654 -96.9% 20603 ± 70% interrupts.CPU57.LOC:Local_timer_interrupts 260.00 ± 69% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts 260.00 ± 69% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts 430.00 -52.9% 202.67 interrupts.CPU58.CAL:Function_call_interrupts 671696 -96.1% 26382 ± 64% interrupts.CPU58.LOC:Local_timer_interrupts 319.67 ± 60% -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts 319.67 ± 60% -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts 430.67 -52.9% 202.67 interrupts.CPU59.CAL:Function_call_interrupts 671648 -96.4% 23962 ± 74% interrupts.CPU59.LOC:Local_timer_interrupts 378.00 ± 57% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts 378.00 ± 57% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.67 interrupts.CPU6.CAL:Function_call_interrupts 671725 -95.8% 28541 ± 43% interrupts.CPU6.LOC:Local_timer_interrupts 204.33 ± 36% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts 204.33 ± 36% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts 430.33 -53.0% 202.33 interrupts.CPU60.CAL:Function_call_interrupts 671638 -96.6% 22642 ± 66% interrupts.CPU60.LOC:Local_timer_interrupts 342.67 ± 59% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts 342.67 ± 59% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts 429.00 -52.7% 203.00 interrupts.CPU61.CAL:Function_call_interrupts 671637 -96.4% 24286 ± 76% interrupts.CPU61.LOC:Local_timer_interrupts 242.67 ± 56% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts 242.67 ± 56% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts 1.00 ±141% +5133.3% 52.33 ±120% interrupts.CPU62.98:PCI-MSI.31981631-edge.i40e-eth0-TxRx-62 429.33 -53.0% 201.67 interrupts.CPU62.CAL:Function_call_interrupts 671749 -95.7% 29169 ± 32% interrupts.CPU62.LOC:Local_timer_interrupts 334.33 ± 67% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts 334.33 ± 67% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts 429.67 -52.8% 202.67 interrupts.CPU63.CAL:Function_call_interrupts 671641 -96.5% 23449 ± 61% interrupts.CPU63.LOC:Local_timer_interrupts 299.67 ± 58% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts 299.67 ± 58% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts 429.67 -52.7% 203.33 interrupts.CPU64.CAL:Function_call_interrupts 671739 -96.4% 23924 ± 55% interrupts.CPU64.LOC:Local_timer_interrupts 315.67 ± 55% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts 315.67 ± 55% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts 429.00 -52.9% 202.00 interrupts.CPU65.CAL:Function_call_interrupts 671668 -96.8% 21507 ± 39% interrupts.CPU65.LOC:Local_timer_interrupts 377.33 ± 56% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts 377.33 ± 56% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts 429.00 -52.5% 203.67 interrupts.CPU66.CAL:Function_call_interrupts 671618 -96.5% 23287 ± 50% interrupts.CPU66.LOC:Local_timer_interrupts 277.67 ± 65% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts 277.67 ± 65% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts 430.00 -52.9% 202.67 interrupts.CPU67.CAL:Function_call_interrupts 671676 -97.2% 18827 ± 66% interrupts.CPU67.LOC:Local_timer_interrupts 294.67 ± 62% -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts 294.67 ± 62% -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts 429.33 -52.8% 202.67 interrupts.CPU68.CAL:Function_call_interrupts 671886 -96.4% 24274 ± 68% interrupts.CPU68.LOC:Local_timer_interrupts 204.67 ± 59% -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts 204.67 ± 59% -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.33 interrupts.CPU69.CAL:Function_call_interrupts 671649 -96.9% 20720 ± 71% interrupts.CPU69.LOC:Local_timer_interrupts 249.67 ± 62% -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts 249.67 ± 62% -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts 12702 ±119% -94.8% 662.67 ± 70% interrupts.CPU69.RES:Rescheduling_interrupts 313.00 ± 8% -19.0% 253.67 ± 4% interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 430.33 -52.9% 202.67 interrupts.CPU7.CAL:Function_call_interrupts 671602 -96.9% 20637 ± 51% interrupts.CPU7.LOC:Local_timer_interrupts 184.00 ± 40% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts 184.00 ± 40% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts 429.00 -52.5% 203.67 interrupts.CPU70.CAL:Function_call_interrupts 671596 -95.3% 31776 ± 80% interrupts.CPU70.LOC:Local_timer_interrupts 389.33 ± 70% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts 389.33 ± 70% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts 429.33 -52.8% 202.67 interrupts.CPU71.CAL:Function_call_interrupts 671618 -96.3% 25046 ± 68% interrupts.CPU71.LOC:Local_timer_interrupts 217.33 ± 61% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts 217.33 ± 61% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts 430.00 -53.0% 202.00 interrupts.CPU72.CAL:Function_call_interrupts 671690 -98.1% 12427 ±112% interrupts.CPU72.LOC:Local_timer_interrupts 131.67 ± 40% -100.0% 0.00 interrupts.CPU72.NMI:Non-maskable_interrupts 131.67 ± 40% -100.0% 0.00 interrupts.CPU72.PMI:Performance_monitoring_interrupts 429.33 -52.9% 202.33 interrupts.CPU73.CAL:Function_call_interrupts 671674 -98.4% 10879 ±124% interrupts.CPU73.LOC:Local_timer_interrupts 142.00 ± 56% -100.0% 0.00 interrupts.CPU73.NMI:Non-maskable_interrupts 142.00 ± 56% -100.0% 0.00 interrupts.CPU73.PMI:Performance_monitoring_interrupts 428.33 -52.8% 202.00 interrupts.CPU74.CAL:Function_call_interrupts 671811 -98.2% 11934 ±124% interrupts.CPU74.LOC:Local_timer_interrupts 156.67 ± 56% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts 156.67 ± 56% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts 429.67 -52.9% 202.33 interrupts.CPU75.CAL:Function_call_interrupts 671583 -96.7% 21874 ±103% interrupts.CPU75.LOC:Local_timer_interrupts 191.67 ± 76% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts 191.67 ± 76% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts 429.67 -52.5% 204.00 interrupts.CPU76.CAL:Function_call_interrupts 671776 -97.5% 16801 ± 92% interrupts.CPU76.LOC:Local_timer_interrupts 166.33 ± 69% -100.0% 0.00 interrupts.CPU76.NMI:Non-maskable_interrupts 166.33 ± 69% -100.0% 0.00 interrupts.CPU76.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.33 interrupts.CPU77.CAL:Function_call_interrupts 671632 -98.3% 11652 ±101% interrupts.CPU77.LOC:Local_timer_interrupts 168.33 ± 69% -100.0% 0.00 interrupts.CPU77.NMI:Non-maskable_interrupts 168.33 ± 69% -100.0% 0.00 interrupts.CPU77.PMI:Performance_monitoring_interrupts 364.33 ± 24% -44.0% 204.00 interrupts.CPU78.CAL:Function_call_interrupts 671732 -98.6% 9538 ± 99% interrupts.CPU78.LOC:Local_timer_interrupts 147.00 ± 47% -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts 147.00 ± 47% -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts 429.00 -52.8% 202.67 interrupts.CPU79.CAL:Function_call_interrupts 671732 -98.2% 12311 ± 86% interrupts.CPU79.LOC:Local_timer_interrupts 187.00 ± 73% -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts 187.00 ± 73% -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts 430.00 -52.5% 204.33 ± 2% interrupts.CPU8.CAL:Function_call_interrupts 671691 -95.1% 32931 ± 60% interrupts.CPU8.LOC:Local_timer_interrupts 241.67 ± 54% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts 241.67 ± 54% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts 428.67 -67.7% 138.33 ± 65% interrupts.CPU80.CAL:Function_call_interrupts 671707 -96.7% 21946 ±114% interrupts.CPU80.LOC:Local_timer_interrupts 177.00 ± 63% -100.0% 0.00 interrupts.CPU80.NMI:Non-maskable_interrupts 177.00 ± 63% -100.0% 0.00 interrupts.CPU80.PMI:Performance_monitoring_interrupts 364.33 ± 24% -44.4% 202.67 interrupts.CPU81.CAL:Function_call_interrupts 671681 -97.8% 15078 ±105% interrupts.CPU81.LOC:Local_timer_interrupts 211.67 ± 83% -100.0% 0.00 interrupts.CPU81.NMI:Non-maskable_interrupts 211.67 ± 83% -100.0% 0.00 interrupts.CPU81.PMI:Performance_monitoring_interrupts 428.33 -52.6% 203.00 interrupts.CPU82.CAL:Function_call_interrupts 671681 -98.1% 12798 ± 59% interrupts.CPU82.LOC:Local_timer_interrupts 227.67 ± 84% -100.0% 0.00 interrupts.CPU82.NMI:Non-maskable_interrupts 227.67 ± 84% -100.0% 0.00 interrupts.CPU82.PMI:Performance_monitoring_interrupts 436.67 ± 2% -53.7% 202.33 interrupts.CPU83.CAL:Function_call_interrupts 671679 -98.0% 13645 ± 99% interrupts.CPU83.LOC:Local_timer_interrupts 233.33 ± 85% -100.0% 0.00 interrupts.CPU83.NMI:Non-maskable_interrupts 233.33 ± 85% -100.0% 0.00 interrupts.CPU83.PMI:Performance_monitoring_interrupts 445.33 ± 2% -54.6% 202.33 interrupts.CPU84.CAL:Function_call_interrupts 671720 -97.5% 16803 ± 99% interrupts.CPU84.LOC:Local_timer_interrupts 283.00 ± 92% -100.0% 0.00 interrupts.CPU84.NMI:Non-maskable_interrupts 283.00 ± 92% -100.0% 0.00 interrupts.CPU84.PMI:Performance_monitoring_interrupts 443.33 ± 2% -54.4% 202.33 interrupts.CPU85.CAL:Function_call_interrupts 671639 -98.3% 11606 ± 96% interrupts.CPU85.LOC:Local_timer_interrupts 303.67 ±100% -100.0% 0.00 interrupts.CPU85.NMI:Non-maskable_interrupts 303.67 ±100% -100.0% 0.00 interrupts.CPU85.PMI:Performance_monitoring_interrupts 444.33 ± 2% -54.5% 202.33 interrupts.CPU86.CAL:Function_call_interrupts 671715 -98.1% 12534 ± 84% interrupts.CPU86.LOC:Local_timer_interrupts 190.67 ± 78% -100.0% 0.00 interrupts.CPU86.NMI:Non-maskable_interrupts 190.67 ± 78% -100.0% 0.00 interrupts.CPU86.PMI:Performance_monitoring_interrupts 9545 ±126% -96.5% 331.67 ±127% interrupts.CPU86.RES:Rescheduling_interrupts 449.67 -55.0% 202.33 interrupts.CPU87.CAL:Function_call_interrupts 671619 -98.5% 9856 ± 93% interrupts.CPU87.LOC:Local_timer_interrupts 191.67 ± 77% -100.0% 0.00 interrupts.CPU87.NMI:Non-maskable_interrupts 191.67 ± 77% -100.0% 0.00 interrupts.CPU87.PMI:Performance_monitoring_interrupts 453.00 -55.3% 202.33 interrupts.CPU88.CAL:Function_call_interrupts 671721 -97.6% 15976 ±115% interrupts.CPU88.LOC:Local_timer_interrupts 256.00 ± 94% -100.0% 0.00 interrupts.CPU88.NMI:Non-maskable_interrupts 256.00 ± 94% -100.0% 0.00 interrupts.CPU88.PMI:Performance_monitoring_interrupts 453.00 -55.2% 203.00 interrupts.CPU89.CAL:Function_call_interrupts 671633 -98.3% 11675 ±112% interrupts.CPU89.LOC:Local_timer_interrupts 198.33 ± 79% -100.0% 0.00 interrupts.CPU89.NMI:Non-maskable_interrupts 198.33 ± 79% -100.0% 0.00 interrupts.CPU89.PMI:Performance_monitoring_interrupts 428.00 -52.8% 202.00 interrupts.CPU9.CAL:Function_call_interrupts 671644 -96.4% 23863 ± 63% interrupts.CPU9.LOC:Local_timer_interrupts 381.33 ± 58% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 381.33 ± 58% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 453.00 -55.4% 202.00 interrupts.CPU90.CAL:Function_call_interrupts 671651 -98.5% 10284 ±117% interrupts.CPU90.LOC:Local_timer_interrupts 148.67 ± 46% -100.0% 0.00 interrupts.CPU90.NMI:Non-maskable_interrupts 148.67 ± 46% -100.0% 0.00 interrupts.CPU90.PMI:Performance_monitoring_interrupts 453.00 -55.2% 203.00 interrupts.CPU91.CAL:Function_call_interrupts 671583 -98.0% 13311 ± 84% interrupts.CPU91.LOC:Local_timer_interrupts 105.33 ± 21% -100.0% 0.00 interrupts.CPU91.NMI:Non-maskable_interrupts 105.33 ± 21% -100.0% 0.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts 7765 ±121% -95.9% 316.67 ±129% interrupts.CPU91.RES:Rescheduling_interrupts 452.67 -55.3% 202.33 interrupts.CPU92.CAL:Function_call_interrupts 671608 -98.2% 12301 ± 90% interrupts.CPU92.LOC:Local_timer_interrupts 145.00 ± 54% -100.0% 0.00 interrupts.CPU92.NMI:Non-maskable_interrupts 145.00 ± 54% -100.0% 0.00 interrupts.CPU92.PMI:Performance_monitoring_interrupts 451.67 -55.1% 203.00 interrupts.CPU93.CAL:Function_call_interrupts 671716 -97.6% 16120 ± 92% interrupts.CPU93.LOC:Local_timer_interrupts 183.00 ± 63% -100.0% 0.00 interrupts.CPU93.NMI:Non-maskable_interrupts 183.00 ± 63% -100.0% 0.00 interrupts.CPU93.PMI:Performance_monitoring_interrupts 452.00 -55.0% 203.33 interrupts.CPU94.CAL:Function_call_interrupts 671696 -97.7% 15481 ± 76% interrupts.CPU94.LOC:Local_timer_interrupts 213.67 ± 70% -100.0% 0.00 interrupts.CPU94.NMI:Non-maskable_interrupts 213.67 ± 70% -100.0% 0.00 interrupts.CPU94.PMI:Performance_monitoring_interrupts 442.33 -54.3% 202.33 interrupts.CPU95.CAL:Function_call_interrupts 672259 -97.2% 19120 ±100% interrupts.CPU95.LOC:Local_timer_interrupts 141.33 ± 51% -100.0% 0.00 interrupts.CPU95.NMI:Non-maskable_interrupts 141.33 ± 51% -100.0% 0.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts 106.67 ± 7% -99.7% 0.33 ±141% interrupts.CPU95.TLB:TLB_shootdowns 64479328 -97.1% 1870473 interrupts.LOC:Local_timer_interrupts 23593 -100.0% 0.00 interrupts.NMI:Non-maskable_interrupts 23593 -100.0% 0.00 interrupts.PMI:Performance_monitoring_interrupts 148959 ± 5% +93.1% 287634 interrupts.RES:Rescheduling_interrupts 805.33 ± 22% -70.1% 241.00 ± 62% interrupts.TLB:TLB_shootdowns *************************************************************************************************** lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/pthread_mutex2/will-it-scale/0x11 commit: 1d2cc5ac6f (" RISC-V Fixes for 5.7-rc4") f2af7d90e2 ("pipe: read/write_iter() handler should check for IOCB_NOWAIT") 1d2cc5ac6f6668cc f2af7d90e25ce1cf03318bf3290 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 4:4 -100% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 4:4 -100% :4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 15862078 +7.7% 17089722 will-it-scale.per_process_ops 4.568e+09 +7.7% 4.922e+09 will-it-scale.workload 47.41 -22.3% 36.86 ± 2% boot-time.boot 11801 ± 2% -23.0% 9091 ± 2% boot-time.idle 1.32 ± 9% -0.2 1.08 mpstat.cpu.all.idle% 0.41 -0.2 0.26 ± 2% mpstat.cpu.all.sys% 923632 -27.5% 669626 numa-numastat.node0.local_node 923601 -27.5% 669602 numa-numastat.node0.numa_hit 2240 -8.2% 2055 ± 3% vmstat.system.cs 583862 -50.5% 289122 vmstat.system.in 1.179e+09 ± 10% -10.0% 1.061e+09 ± 4% cpuidle.C1.time 2487664 ± 11% -79.5% 510669 ± 2% cpuidle.C1.usage 4718 ± 14% -82.4% 829.75 ± 29% cpuidle.POLL.time 392.50 ± 7% -89.0% 43.00 ± 12% cpuidle.POLL.usage 183281 ± 3% -23.4% 140482 ± 19% meminfo.AnonHugePages 444902 +10.7% 492610 meminfo.AnonPages 120082 -14.1% 103204 meminfo.KReclaimable 35693 -34.6% 23361 meminfo.Mapped 120082 -14.1% 103204 meminfo.SReclaimable 99701 ± 2% -75.4% 24509 meminfo.Shmem 10811 +14.4% 12371 meminfo.max_used_kB 183276 ± 3% -23.4% 140441 ± 18% numa-meminfo.node0.AnonHugePages 444818 +10.6% 491786 numa-meminfo.node0.AnonPages 664770 -14.4% 569229 numa-meminfo.node0.FilePages 101106 -16.6% 84323 numa-meminfo.node0.KReclaimable 24869 -42.9% 14189 numa-meminfo.node0.Mapped 101106 -16.6% 84323 numa-meminfo.node0.SReclaimable 99712 ± 2% -75.4% 24507 numa-meminfo.node0.Shmem 10843 -15.4% 9178 numa-meminfo.node1.Mapped 111181 +10.6% 122946 numa-vmstat.node0.nr_anon_pages 166181 -14.4% 142307 numa-vmstat.node0.nr_file_pages 6301 -43.7% 3546 numa-vmstat.node0.nr_mapped 24917 ± 2% -75.4% 6127 numa-vmstat.node0.nr_shmem 25272 -16.6% 21080 numa-vmstat.node0.nr_slab_reclaimable 2095207 -15.6% 1769021 ± 3% numa-vmstat.node0.numa_hit 2095208 -15.6% 1769021 ± 3% numa-vmstat.node0.numa_local 2709 -15.3% 2294 numa-vmstat.node1.nr_mapped 130250 -5.3% 123384 proc-vmstat.nr_active_anon 111216 +10.6% 122975 proc-vmstat.nr_anon_pages 298885 -7.0% 278012 proc-vmstat.nr_file_pages 40374 +1.5% 40989 proc-vmstat.nr_kernel_stack 8982 -35.0% 5839 proc-vmstat.nr_mapped 15276 +4.8% 16006 proc-vmstat.nr_page_table_pages 24927 ± 2% -75.4% 6127 proc-vmstat.nr_shmem 30019 -14.1% 25801 proc-vmstat.nr_slab_reclaimable 79108 -3.1% 76670 proc-vmstat.nr_slab_unreclaimable 130250 -5.3% 123384 proc-vmstat.nr_zone_active_anon 935680 -27.0% 683089 ± 2% proc-vmstat.numa_hit 935680 -27.0% 683089 ± 2% proc-vmstat.numa_local 77696 ± 3% +27.8% 99328 ± 11% proc-vmstat.numa_pte_updates 2061 ± 2% -87.8% 252.25 ± 9% proc-vmstat.pgactivate 1025671 -27.9% 739546 proc-vmstat.pgalloc_normal 886034 -11.8% 781478 proc-vmstat.pgfault 47.40 ± 3% -47.4 0.00 perf-profile.calltrace.cycles-pp.pthread_mutex_unlock 47.40 ± 2% -47.4 0.00 perf-profile.calltrace.cycles-pp.pthread_mutex_lock 8.30 ± 24% -8.3 0.00 perf-profile.calltrace.cycles-pp.apic_timer_interrupt 7.70 ± 23% -7.7 0.00 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt 6.28 ± 23% -6.3 0.00 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 44.26 ± 3% -44.3 0.00 perf-profile.children.cycles-pp.pthread_mutex_unlock 43.55 ± 2% -43.6 0.00 perf-profile.children.cycles-pp.pthread_mutex_lock 13.88 ± 22% -13.9 0.00 perf-profile.children.cycles-pp.apic_timer_interrupt 12.78 ± 22% -12.8 0.00 perf-profile.children.cycles-pp.smp_apic_timer_interrupt 10.47 ± 22% -10.5 0.00 perf-profile.children.cycles-pp.hrtimer_interrupt 9.50 ± 2% -9.5 0.00 perf-profile.children.cycles-pp.pthread_mutex_unlock@plt 9.41 ± 2% -9.4 0.00 perf-profile.children.cycles-pp.pthread_mutex_lock@plt 7.90 ± 21% -7.9 0.00 perf-profile.children.cycles-pp.__hrtimer_run_queues 33.08 ± 4% -33.1 0.00 perf-profile.self.cycles-pp.pthread_mutex_unlock 32.31 ± 4% -32.3 0.00 perf-profile.self.cycles-pp.pthread_mutex_lock 8.85 ± 4% -8.8 0.00 perf-profile.self.cycles-pp.pthread_mutex_lock@plt 8.76 ± 4% -8.8 0.00 perf-profile.self.cycles-pp.pthread_mutex_unlock@plt 875.00 -61.4% 337.32 ± 2% sched_debug.cfs_rq:/.exec_clock.stddev 3534 +13.7% 4020 ± 7% sched_debug.cfs_rq:/.load.avg 459444 ± 5% -10.4% 411793 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev 0.02 ± 8% +34.2% 0.03 ± 14% sched_debug.cfs_rq:/.nr_running.stddev 1.04 ± 8% +15324.7% 159.69 ± 8% sched_debug.cfs_rq:/.nr_spread_over.avg 95.25 ± 19% +720.5% 781.50 ± 18% sched_debug.cfs_rq:/.nr_spread_over.max 6.91 ± 11% +2322.5% 167.45 ± 5% sched_debug.cfs_rq:/.nr_spread_over.stddev 397594 ± 14% -17.2% 329399 ± 3% sched_debug.cfs_rq:/.spread0.max -4988450 -48.6% -2566104 sched_debug.cfs_rq:/.spread0.min 451583 ± 5% -12.0% 397413 ± 6% sched_debug.cfs_rq:/.spread0.stddev 96.75 ± 2% -23.6% 73.91 sched_debug.cfs_rq:/.util_est_enqueued.stddev 2057215 ± 5% +46.9% 3021159 ± 21% sched_debug.cpu.max_idle_balance_cost.max 4075 -22.7% 3148 ± 2% sched_debug.cpu.nr_switches.avg 1303 +17.1% 1526 ± 3% sched_debug.cpu.sched_count.avg 454.83 -51.1% 222.46 ± 30% sched_debug.cpu.sched_count.min 46.63 ± 3% +286.4% 180.21 ± 4% sched_debug.cpu.sched_goidle.avg 518.33 +30.0% 673.96 ± 3% sched_debug.cpu.ttwu_count.avg 169.71 -52.0% 81.42 ± 10% sched_debug.cpu.ttwu_count.min 578.53 ± 6% +13.3% 655.19 ± 2% sched_debug.cpu.ttwu_count.stddev 345.35 -39.8% 207.94 ± 5% sched_debug.cpu.ttwu_local.avg 162.83 -72.1% 45.50 sched_debug.cpu.ttwu_local.min 2.03 -100.0% 0.00 sched_debug.rt_rq:/.rt_runtime.stddev 536.00 ± 9% -21.8% 419.25 ± 5% slabinfo.UDP.active_objs 536.00 ± 9% -21.8% 419.25 ± 5% slabinfo.UDP.num_objs 134086 -18.7% 109073 slabinfo.dentry.active_objs 3251 -19.8% 2605 slabinfo.dentry.active_slabs 136556 -19.8% 109475 slabinfo.dentry.num_objs 3251 -19.8% 2605 slabinfo.dentry.num_slabs 1471 ± 7% -29.3% 1040 ± 4% slabinfo.file_lock_cache.active_objs 1471 ± 7% -29.3% 1040 ± 4% slabinfo.file_lock_cache.num_objs 49742 -22.6% 38517 slabinfo.filp.active_objs 777.00 ± 2% -22.6% 601.25 slabinfo.filp.active_slabs 49756 -22.6% 38518 slabinfo.filp.num_objs 777.00 ± 2% -22.6% 601.25 slabinfo.filp.num_slabs 9456 ± 2% -21.0% 7471 slabinfo.kmalloc-2k.active_objs 592.75 ± 2% -20.0% 474.00 slabinfo.kmalloc-2k.active_slabs 9490 ± 2% -20.0% 7590 slabinfo.kmalloc-2k.num_objs 592.75 ± 2% -20.0% 474.00 slabinfo.kmalloc-2k.num_slabs 3453 -11.8% 3045 slabinfo.kmalloc-4k.active_objs 3483 -12.5% 3048 slabinfo.kmalloc-4k.num_objs 34071 -33.7% 22581 slabinfo.proc_inode_cache.active_objs 709.00 -33.5% 471.25 slabinfo.proc_inode_cache.active_slabs 34077 -33.6% 22634 slabinfo.proc_inode_cache.num_objs 709.00 -33.5% 471.25 slabinfo.proc_inode_cache.num_slabs 1075 ± 11% +19.9% 1289 ± 5% slabinfo.task_group.active_objs 1075 ± 11% +19.9% 1289 ± 5% slabinfo.task_group.num_objs 344817 +297.7% 1371460 ± 2% interrupts.CAL:Function_call_interrupts 614.00 +678.0% 4777 ± 2% interrupts.CPU0.CAL:Function_call_interrupts 615083 -50.6% 303696 interrupts.CPU0.LOC:Local_timer_interrupts 4734 ± 34% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts 4734 ± 34% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts 1745 ± 32% +40.9% 2459 ± 17% interrupts.CPU0.RES:Rescheduling_interrupts 1197 +298.2% 4770 ± 2% interrupts.CPU1.CAL:Function_call_interrupts 614874 -50.6% 303669 interrupts.CPU1.LOC:Local_timer_interrupts 3787 -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts 3787 -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts 1195 +298.8% 4766 ± 2% interrupts.CPU10.CAL:Function_call_interrupts 614721 -50.5% 304157 interrupts.CPU10.LOC:Local_timer_interrupts 4682 ± 34% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts 4682 ± 34% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts 569.25 ± 23% -53.7% 263.50 ± 38% interrupts.CPU10.RES:Rescheduling_interrupts 1197 +297.8% 4761 ± 2% interrupts.CPU100.CAL:Function_call_interrupts 614758 -50.6% 303436 interrupts.CPU100.LOC:Local_timer_interrupts 4670 ± 34% -100.0% 0.00 interrupts.CPU100.NMI:Non-maskable_interrupts 4670 ± 34% -100.0% 0.00 interrupts.CPU100.PMI:Performance_monitoring_interrupts 1197 +296.7% 4751 ± 2% interrupts.CPU101.CAL:Function_call_interrupts 614580 -50.6% 303356 interrupts.CPU101.LOC:Local_timer_interrupts 3734 -100.0% 0.00 interrupts.CPU101.NMI:Non-maskable_interrupts 3734 -100.0% 0.00 interrupts.CPU101.PMI:Performance_monitoring_interrupts 120.75 ± 39% +354.7% 549.00 ± 49% interrupts.CPU101.RES:Rescheduling_interrupts 1053 ± 23% +350.9% 4749 ± 3% interrupts.CPU102.CAL:Function_call_interrupts 614977 -50.7% 303260 interrupts.CPU102.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU102.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU102.PMI:Performance_monitoring_interrupts 1197 +298.1% 4767 ± 2% interrupts.CPU103.CAL:Function_call_interrupts 614862 -50.6% 303692 interrupts.CPU103.LOC:Local_timer_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU103.NMI:Non-maskable_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU103.PMI:Performance_monitoring_interrupts 91.75 ± 92% +132.4% 213.25 ± 36% interrupts.CPU103.RES:Rescheduling_interrupts 1196 +298.5% 4769 ± 2% interrupts.CPU104.CAL:Function_call_interrupts 614745 -50.7% 303195 interrupts.CPU104.LOC:Local_timer_interrupts 4649 ± 34% -100.0% 0.00 interrupts.CPU104.NMI:Non-maskable_interrupts 4649 ± 34% -100.0% 0.00 interrupts.CPU104.PMI:Performance_monitoring_interrupts 1198 +298.0% 4770 ± 2% interrupts.CPU105.CAL:Function_call_interrupts 614688 -50.7% 303160 interrupts.CPU105.LOC:Local_timer_interrupts 4666 ± 35% -100.0% 0.00 interrupts.CPU105.NMI:Non-maskable_interrupts 4666 ± 35% -100.0% 0.00 interrupts.CPU105.PMI:Performance_monitoring_interrupts 1197 +298.4% 4770 ± 2% interrupts.CPU106.CAL:Function_call_interrupts 614881 -50.7% 303321 interrupts.CPU106.LOC:Local_timer_interrupts 4666 ± 34% -100.0% 0.00 interrupts.CPU106.NMI:Non-maskable_interrupts 4666 ± 34% -100.0% 0.00 interrupts.CPU106.PMI:Performance_monitoring_interrupts 1197 +298.7% 4774 ± 2% interrupts.CPU107.CAL:Function_call_interrupts 614865 -50.6% 303486 interrupts.CPU107.LOC:Local_timer_interrupts 4680 ± 34% -100.0% 0.00 interrupts.CPU107.NMI:Non-maskable_interrupts 4680 ± 34% -100.0% 0.00 interrupts.CPU107.PMI:Performance_monitoring_interrupts 1198 +298.2% 4770 ± 2% interrupts.CPU108.CAL:Function_call_interrupts 614907 -50.7% 303374 interrupts.CPU108.LOC:Local_timer_interrupts 4663 ± 34% -100.0% 0.00 interrupts.CPU108.NMI:Non-maskable_interrupts 4663 ± 34% -100.0% 0.00 interrupts.CPU108.PMI:Performance_monitoring_interrupts 1197 +298.1% 4767 ± 2% interrupts.CPU109.CAL:Function_call_interrupts 614704 -50.5% 303988 interrupts.CPU109.LOC:Local_timer_interrupts 4665 ± 34% -100.0% 0.00 interrupts.CPU109.NMI:Non-maskable_interrupts 4665 ± 34% -100.0% 0.00 interrupts.CPU109.PMI:Performance_monitoring_interrupts 1194 +296.8% 4738 ± 2% interrupts.CPU11.CAL:Function_call_interrupts 614834 -50.5% 304057 interrupts.CPU11.LOC:Local_timer_interrupts 3758 -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts 3758 -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts 1197 +297.7% 4762 ± 2% interrupts.CPU110.CAL:Function_call_interrupts 614794 -50.7% 303323 interrupts.CPU110.LOC:Local_timer_interrupts 4664 ± 34% -100.0% 0.00 interrupts.CPU110.NMI:Non-maskable_interrupts 4664 ± 34% -100.0% 0.00 interrupts.CPU110.PMI:Performance_monitoring_interrupts 1198 +297.4% 4762 ± 2% interrupts.CPU111.CAL:Function_call_interrupts 614740 -50.6% 303509 interrupts.CPU111.LOC:Local_timer_interrupts 4653 ± 34% -100.0% 0.00 interrupts.CPU111.NMI:Non-maskable_interrupts 4653 ± 34% -100.0% 0.00 interrupts.CPU111.PMI:Performance_monitoring_interrupts 1199 +297.9% 4770 ± 2% interrupts.CPU112.CAL:Function_call_interrupts 614522 -50.6% 303421 interrupts.CPU112.LOC:Local_timer_interrupts 4688 ± 34% -100.0% 0.00 interrupts.CPU112.NMI:Non-maskable_interrupts 4688 ± 34% -100.0% 0.00 interrupts.CPU112.PMI:Performance_monitoring_interrupts 1054 ± 24% +350.9% 4752 ± 2% interrupts.CPU113.CAL:Function_call_interrupts 614759 -50.7% 303302 interrupts.CPU113.LOC:Local_timer_interrupts 4678 ± 34% -100.0% 0.00 interrupts.CPU113.NMI:Non-maskable_interrupts 4678 ± 34% -100.0% 0.00 interrupts.CPU113.PMI:Performance_monitoring_interrupts 185.25 ± 31% +195.1% 546.75 ± 35% interrupts.CPU113.RES:Rescheduling_interrupts 1198 +297.3% 4762 ± 2% interrupts.CPU114.CAL:Function_call_interrupts 614619 -50.7% 303299 interrupts.CPU114.LOC:Local_timer_interrupts 5593 ± 33% -100.0% 0.00 interrupts.CPU114.NMI:Non-maskable_interrupts 5593 ± 33% -100.0% 0.00 interrupts.CPU114.PMI:Performance_monitoring_interrupts 1199 +297.5% 4765 ± 2% interrupts.CPU115.CAL:Function_call_interrupts 614867 -50.7% 303423 interrupts.CPU115.LOC:Local_timer_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU115.NMI:Non-maskable_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU115.PMI:Performance_monitoring_interrupts 1198 +297.2% 4761 ± 2% interrupts.CPU116.CAL:Function_call_interrupts 614588 -50.7% 302996 interrupts.CPU116.LOC:Local_timer_interrupts 4675 ± 34% -100.0% 0.00 interrupts.CPU116.NMI:Non-maskable_interrupts 4675 ± 34% -100.0% 0.00 interrupts.CPU116.PMI:Performance_monitoring_interrupts 1198 +297.4% 4762 ± 3% interrupts.CPU117.CAL:Function_call_interrupts 614834 -50.7% 303238 interrupts.CPU117.LOC:Local_timer_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU117.NMI:Non-maskable_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU117.PMI:Performance_monitoring_interrupts 1198 +297.9% 4768 ± 2% interrupts.CPU118.CAL:Function_call_interrupts 614788 -50.7% 303390 interrupts.CPU118.LOC:Local_timer_interrupts 3737 -100.0% 0.00 interrupts.CPU118.NMI:Non-maskable_interrupts 3737 -100.0% 0.00 interrupts.CPU118.PMI:Performance_monitoring_interrupts 1198 +297.7% 4766 ± 2% interrupts.CPU119.CAL:Function_call_interrupts 614778 -50.6% 303461 interrupts.CPU119.LOC:Local_timer_interrupts 3732 -100.0% 0.00 interrupts.CPU119.NMI:Non-maskable_interrupts 3732 -100.0% 0.00 interrupts.CPU119.PMI:Performance_monitoring_interrupts 87.25 ± 65% +253.6% 308.50 ± 68% interrupts.CPU119.RES:Rescheduling_interrupts 1197 +296.6% 4747 ± 2% interrupts.CPU12.CAL:Function_call_interrupts 614999 -50.7% 303292 interrupts.CPU12.LOC:Local_timer_interrupts 4653 ± 35% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts 4653 ± 35% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts 1198 +296.0% 4744 ± 2% interrupts.CPU120.CAL:Function_call_interrupts 614651 -50.6% 303349 interrupts.CPU120.LOC:Local_timer_interrupts 4661 ± 34% -100.0% 0.00 interrupts.CPU120.NMI:Non-maskable_interrupts 4661 ± 34% -100.0% 0.00 interrupts.CPU120.PMI:Performance_monitoring_interrupts 2349 ± 53% -64.8% 827.25 ± 56% interrupts.CPU120.RES:Rescheduling_interrupts 1199 +297.0% 4763 ± 2% interrupts.CPU121.CAL:Function_call_interrupts 614875 -50.6% 303556 interrupts.CPU121.LOC:Local_timer_interrupts 5612 ± 33% -100.0% 0.00 interrupts.CPU121.NMI:Non-maskable_interrupts 5612 ± 33% -100.0% 0.00 interrupts.CPU121.PMI:Performance_monitoring_interrupts 149.50 ± 67% +228.4% 491.00 ± 49% interrupts.CPU121.RES:Rescheduling_interrupts 1198 +297.6% 4766 ± 2% interrupts.CPU122.CAL:Function_call_interrupts 614738 -50.7% 303241 interrupts.CPU122.LOC:Local_timer_interrupts 5622 ± 33% -100.0% 0.00 interrupts.CPU122.NMI:Non-maskable_interrupts 5622 ± 33% -100.0% 0.00 interrupts.CPU122.PMI:Performance_monitoring_interrupts 1198 +297.9% 4767 ± 2% interrupts.CPU123.CAL:Function_call_interrupts 614696 -50.6% 303607 interrupts.CPU123.LOC:Local_timer_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU123.NMI:Non-maskable_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU123.PMI:Performance_monitoring_interrupts 38.50 ± 15% +380.5% 185.00 ± 31% interrupts.CPU123.RES:Rescheduling_interrupts 1199 +296.5% 4755 ± 2% interrupts.CPU124.CAL:Function_call_interrupts 614786 -50.6% 303480 interrupts.CPU124.LOC:Local_timer_interrupts 5604 ± 33% -100.0% 0.00 interrupts.CPU124.NMI:Non-maskable_interrupts 5604 ± 33% -100.0% 0.00 interrupts.CPU124.PMI:Performance_monitoring_interrupts 1198 +298.1% 4769 ± 2% interrupts.CPU125.CAL:Function_call_interrupts 614811 -50.7% 303394 interrupts.CPU125.LOC:Local_timer_interrupts 3730 -100.0% 0.00 interrupts.CPU125.NMI:Non-maskable_interrupts 3730 -100.0% 0.00 interrupts.CPU125.PMI:Performance_monitoring_interrupts 1198 +296.5% 4751 ± 2% interrupts.CPU126.CAL:Function_call_interrupts 614867 -50.6% 303564 interrupts.CPU126.LOC:Local_timer_interrupts 4648 ± 34% -100.0% 0.00 interrupts.CPU126.NMI:Non-maskable_interrupts 4648 ± 34% -100.0% 0.00 interrupts.CPU126.PMI:Performance_monitoring_interrupts 1198 +298.0% 4769 ± 2% interrupts.CPU127.CAL:Function_call_interrupts 614838 -50.6% 303515 interrupts.CPU127.LOC:Local_timer_interrupts 4669 ± 34% -100.0% 0.00 interrupts.CPU127.NMI:Non-maskable_interrupts 4669 ± 34% -100.0% 0.00 interrupts.CPU127.PMI:Performance_monitoring_interrupts 1198 +297.2% 4758 ± 2% interrupts.CPU128.CAL:Function_call_interrupts 614841 -50.7% 303178 interrupts.CPU128.LOC:Local_timer_interrupts 4662 ± 34% -100.0% 0.00 interrupts.CPU128.NMI:Non-maskable_interrupts 4662 ± 34% -100.0% 0.00 interrupts.CPU128.PMI:Performance_monitoring_interrupts 1198 +297.6% 4764 ± 2% interrupts.CPU129.CAL:Function_call_interrupts 614741 -50.7% 303336 interrupts.CPU129.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU129.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU129.PMI:Performance_monitoring_interrupts 1191 +298.1% 4741 ± 2% interrupts.CPU13.CAL:Function_call_interrupts 615123 -50.7% 303272 interrupts.CPU13.LOC:Local_timer_interrupts 4656 ± 35% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts 4656 ± 35% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts 1198 +298.1% 4770 ± 2% interrupts.CPU130.CAL:Function_call_interrupts 614690 -50.7% 303272 interrupts.CPU130.LOC:Local_timer_interrupts 5594 ± 33% -100.0% 0.00 interrupts.CPU130.NMI:Non-maskable_interrupts 5594 ± 33% -100.0% 0.00 interrupts.CPU130.PMI:Performance_monitoring_interrupts 1878 ± 55% -54.4% 856.00 ± 65% interrupts.CPU130.RES:Rescheduling_interrupts 1198 +297.8% 4767 ± 2% interrupts.CPU131.CAL:Function_call_interrupts 614400 -50.6% 303462 interrupts.CPU131.LOC:Local_timer_interrupts 3740 -100.0% 0.00 interrupts.CPU131.NMI:Non-maskable_interrupts 3740 -100.0% 0.00 interrupts.CPU131.PMI:Performance_monitoring_interrupts 1198 +298.0% 4769 ± 2% interrupts.CPU132.CAL:Function_call_interrupts 614713 -50.7% 303319 interrupts.CPU132.LOC:Local_timer_interrupts 3732 -100.0% 0.00 interrupts.CPU132.NMI:Non-maskable_interrupts 3732 -100.0% 0.00 interrupts.CPU132.PMI:Performance_monitoring_interrupts 1118 ± 55% -51.0% 547.50 ± 42% interrupts.CPU132.RES:Rescheduling_interrupts 1198 +298.0% 4768 ± 2% interrupts.CPU133.CAL:Function_call_interrupts 614719 -50.7% 303325 interrupts.CPU133.LOC:Local_timer_interrupts 3721 -100.0% 0.00 interrupts.CPU133.NMI:Non-maskable_interrupts 3721 -100.0% 0.00 interrupts.CPU133.PMI:Performance_monitoring_interrupts 1198 +297.6% 4762 ± 2% interrupts.CPU134.CAL:Function_call_interrupts 614506 -50.6% 303307 interrupts.CPU134.LOC:Local_timer_interrupts 3724 -100.0% 0.00 interrupts.CPU134.NMI:Non-maskable_interrupts 3724 -100.0% 0.00 interrupts.CPU134.PMI:Performance_monitoring_interrupts 1200 +297.7% 4772 ± 2% interrupts.CPU135.CAL:Function_call_interrupts 614606 -50.6% 303788 interrupts.CPU135.LOC:Local_timer_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU135.NMI:Non-maskable_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU135.PMI:Performance_monitoring_interrupts 1199 +296.1% 4749 ± 3% interrupts.CPU136.CAL:Function_call_interrupts 614853 -50.7% 303384 interrupts.CPU136.LOC:Local_timer_interrupts 4670 ± 34% -100.0% 0.00 interrupts.CPU136.NMI:Non-maskable_interrupts 4670 ± 34% -100.0% 0.00 interrupts.CPU136.PMI:Performance_monitoring_interrupts 1197 +297.2% 4756 ± 3% interrupts.CPU137.CAL:Function_call_interrupts 614794 -50.7% 303220 interrupts.CPU137.LOC:Local_timer_interrupts 3740 -100.0% 0.00 interrupts.CPU137.NMI:Non-maskable_interrupts 3740 -100.0% 0.00 interrupts.CPU137.PMI:Performance_monitoring_interrupts 146.00 ±105% +441.6% 790.75 ± 55% interrupts.CPU137.RES:Rescheduling_interrupts 1197 +298.1% 4768 ± 2% interrupts.CPU138.CAL:Function_call_interrupts 615103 -50.7% 303372 interrupts.CPU138.LOC:Local_timer_interrupts 3748 -100.0% 0.00 interrupts.CPU138.NMI:Non-maskable_interrupts 3748 -100.0% 0.00 interrupts.CPU138.PMI:Performance_monitoring_interrupts 366.00 ± 25% +207.4% 1125 ± 16% interrupts.CPU138.RES:Rescheduling_interrupts 1198 +297.4% 4762 ± 2% interrupts.CPU139.CAL:Function_call_interrupts 614783 -50.6% 303420 interrupts.CPU139.LOC:Local_timer_interrupts 3753 -100.0% 0.00 interrupts.CPU139.NMI:Non-maskable_interrupts 3753 -100.0% 0.00 interrupts.CPU139.PMI:Performance_monitoring_interrupts 1196 +297.9% 4761 ± 2% interrupts.CPU14.CAL:Function_call_interrupts 614805 -50.5% 304146 interrupts.CPU14.LOC:Local_timer_interrupts 4700 ± 35% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts 4700 ± 35% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts 1198 +296.2% 4749 ± 2% interrupts.CPU140.CAL:Function_call_interrupts 614580 -50.7% 303093 interrupts.CPU140.LOC:Local_timer_interrupts 3732 -100.0% 0.00 interrupts.CPU140.NMI:Non-maskable_interrupts 3732 -100.0% 0.00 interrupts.CPU140.PMI:Performance_monitoring_interrupts 1199 +297.5% 4768 ± 2% interrupts.CPU141.CAL:Function_call_interrupts 614712 -50.7% 303219 interrupts.CPU141.LOC:Local_timer_interrupts 3732 -100.0% 0.00 interrupts.CPU141.NMI:Non-maskable_interrupts 3732 -100.0% 0.00 interrupts.CPU141.PMI:Performance_monitoring_interrupts 1198 +296.6% 4752 ± 2% interrupts.CPU142.CAL:Function_call_interrupts 614693 -50.7% 303166 interrupts.CPU142.LOC:Local_timer_interrupts 3686 -100.0% 0.00 interrupts.CPU142.NMI:Non-maskable_interrupts 3686 -100.0% 0.00 interrupts.CPU142.PMI:Performance_monitoring_interrupts 1198 +298.1% 4771 ± 2% interrupts.CPU143.CAL:Function_call_interrupts 614760 -50.6% 303669 interrupts.CPU143.LOC:Local_timer_interrupts 5511 ± 33% -100.0% 0.00 interrupts.CPU143.NMI:Non-maskable_interrupts 5511 ± 33% -100.0% 0.00 interrupts.CPU143.PMI:Performance_monitoring_interrupts 1198 +298.8% 4778 ± 2% interrupts.CPU144.CAL:Function_call_interrupts 614713 -50.6% 303894 interrupts.CPU144.LOC:Local_timer_interrupts 4679 ± 34% -100.0% 0.00 interrupts.CPU144.NMI:Non-maskable_interrupts 4679 ± 34% -100.0% 0.00 interrupts.CPU144.PMI:Performance_monitoring_interrupts 1198 +298.7% 4778 ± 2% interrupts.CPU145.CAL:Function_call_interrupts 614737 -50.7% 303219 interrupts.CPU145.LOC:Local_timer_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU145.NMI:Non-maskable_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU145.PMI:Performance_monitoring_interrupts 1198 +297.2% 4760 ± 2% interrupts.CPU146.CAL:Function_call_interrupts 614654 -50.7% 303047 interrupts.CPU146.LOC:Local_timer_interrupts 5594 ± 33% -100.0% 0.00 interrupts.CPU146.NMI:Non-maskable_interrupts 5594 ± 33% -100.0% 0.00 interrupts.CPU146.PMI:Performance_monitoring_interrupts 1198 +298.4% 4773 ± 2% interrupts.CPU147.CAL:Function_call_interrupts 614763 -50.7% 303335 interrupts.CPU147.LOC:Local_timer_interrupts 3742 -100.0% 0.00 interrupts.CPU147.NMI:Non-maskable_interrupts 3742 -100.0% 0.00 interrupts.CPU147.PMI:Performance_monitoring_interrupts 1199 +297.9% 4770 ± 2% interrupts.CPU148.CAL:Function_call_interrupts 614813 -50.6% 303864 interrupts.CPU148.LOC:Local_timer_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU148.NMI:Non-maskable_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU148.PMI:Performance_monitoring_interrupts 451.00 ± 49% -54.6% 204.75 ± 87% interrupts.CPU148.RES:Rescheduling_interrupts 1198 +297.9% 4768 ± 2% interrupts.CPU149.CAL:Function_call_interrupts 614778 -50.7% 303237 interrupts.CPU149.LOC:Local_timer_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU149.NMI:Non-maskable_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU149.PMI:Performance_monitoring_interrupts 1196 +297.6% 4755 ± 2% interrupts.CPU15.CAL:Function_call_interrupts 614997 -50.7% 303235 interrupts.CPU15.LOC:Local_timer_interrupts 4709 ± 34% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts 4709 ± 34% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts 1198 +297.9% 4767 ± 2% interrupts.CPU150.CAL:Function_call_interrupts 614812 -50.6% 303495 interrupts.CPU150.LOC:Local_timer_interrupts 5583 ± 33% -100.0% 0.00 interrupts.CPU150.NMI:Non-maskable_interrupts 5583 ± 33% -100.0% 0.00 interrupts.CPU150.PMI:Performance_monitoring_interrupts 1198 +297.7% 4764 ± 2% interrupts.CPU151.CAL:Function_call_interrupts 614787 -50.6% 303478 interrupts.CPU151.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU151.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU151.PMI:Performance_monitoring_interrupts 35.25 ± 95% +665.2% 269.75 ±108% interrupts.CPU151.RES:Rescheduling_interrupts 1202 +296.2% 4763 ± 2% interrupts.CPU152.CAL:Function_call_interrupts 614242 -50.6% 303709 interrupts.CPU152.LOC:Local_timer_interrupts 4681 ± 34% -100.0% 0.00 interrupts.CPU152.NMI:Non-maskable_interrupts 4681 ± 34% -100.0% 0.00 interrupts.CPU152.PMI:Performance_monitoring_interrupts 1201 +296.9% 4769 ± 2% interrupts.CPU153.CAL:Function_call_interrupts 614230 -50.6% 303312 interrupts.CPU153.LOC:Local_timer_interrupts 3753 -100.0% 0.00 interrupts.CPU153.NMI:Non-maskable_interrupts 3753 -100.0% 0.00 interrupts.CPU153.PMI:Performance_monitoring_interrupts 1198 +296.9% 4757 ± 2% interrupts.CPU154.CAL:Function_call_interrupts 614740 -50.7% 303359 interrupts.CPU154.LOC:Local_timer_interrupts 3754 -100.0% 0.00 interrupts.CPU154.NMI:Non-maskable_interrupts 3754 -100.0% 0.00 interrupts.CPU154.PMI:Performance_monitoring_interrupts 1198 +297.5% 4762 ± 2% interrupts.CPU155.CAL:Function_call_interrupts 614586 -50.6% 303325 interrupts.CPU155.LOC:Local_timer_interrupts 3749 -100.0% 0.00 interrupts.CPU155.NMI:Non-maskable_interrupts 3749 -100.0% 0.00 interrupts.CPU155.PMI:Performance_monitoring_interrupts 1198 +298.2% 4773 ± 2% interrupts.CPU156.CAL:Function_call_interrupts 614805 -50.6% 303700 interrupts.CPU156.LOC:Local_timer_interrupts 4678 ± 34% -100.0% 0.00 interrupts.CPU156.NMI:Non-maskable_interrupts 4678 ± 34% -100.0% 0.00 interrupts.CPU156.PMI:Performance_monitoring_interrupts 1197 +297.3% 4758 ± 3% interrupts.CPU157.CAL:Function_call_interrupts 614931 -50.7% 303102 interrupts.CPU157.LOC:Local_timer_interrupts 5608 ± 33% -100.0% 0.00 interrupts.CPU157.NMI:Non-maskable_interrupts 5608 ± 33% -100.0% 0.00 interrupts.CPU157.PMI:Performance_monitoring_interrupts 1198 +297.5% 4763 ± 2% interrupts.CPU158.CAL:Function_call_interrupts 614737 -50.7% 303116 interrupts.CPU158.LOC:Local_timer_interrupts 5631 ± 33% -100.0% 0.00 interrupts.CPU158.NMI:Non-maskable_interrupts 5631 ± 33% -100.0% 0.00 interrupts.CPU158.PMI:Performance_monitoring_interrupts 1198 +298.7% 4776 ± 2% interrupts.CPU159.CAL:Function_call_interrupts 614781 -50.7% 303240 interrupts.CPU159.LOC:Local_timer_interrupts 5635 ± 33% -100.0% 0.00 interrupts.CPU159.NMI:Non-maskable_interrupts 5635 ± 33% -100.0% 0.00 interrupts.CPU159.PMI:Performance_monitoring_interrupts 1195 +297.7% 4754 ± 2% interrupts.CPU16.CAL:Function_call_interrupts 614965 -50.7% 303382 interrupts.CPU16.LOC:Local_timer_interrupts 4697 ± 34% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts 4697 ± 34% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts 1199 +298.0% 4772 ± 2% interrupts.CPU160.CAL:Function_call_interrupts 614841 -50.6% 303910 interrupts.CPU160.LOC:Local_timer_interrupts 6574 ± 24% -100.0% 0.00 interrupts.CPU160.NMI:Non-maskable_interrupts 6574 ± 24% -100.0% 0.00 interrupts.CPU160.PMI:Performance_monitoring_interrupts 1198 +298.0% 4769 ± 2% interrupts.CPU161.CAL:Function_call_interrupts 614722 -50.7% 303200 interrupts.CPU161.LOC:Local_timer_interrupts 4688 ± 34% -100.0% 0.00 interrupts.CPU161.NMI:Non-maskable_interrupts 4688 ± 34% -100.0% 0.00 interrupts.CPU161.PMI:Performance_monitoring_interrupts 1198 +297.7% 4765 ± 2% interrupts.CPU162.CAL:Function_call_interrupts 614809 -50.7% 303206 interrupts.CPU162.LOC:Local_timer_interrupts 4690 ± 35% -100.0% 0.00 interrupts.CPU162.NMI:Non-maskable_interrupts 4690 ± 35% -100.0% 0.00 interrupts.CPU162.PMI:Performance_monitoring_interrupts 199.25 ± 49% +115.3% 429.00 ± 35% interrupts.CPU162.RES:Rescheduling_interrupts 1198 +298.2% 4773 ± 2% interrupts.CPU163.CAL:Function_call_interrupts 615060 -50.7% 303411 interrupts.CPU163.LOC:Local_timer_interrupts 5623 ± 33% -100.0% 0.00 interrupts.CPU163.NMI:Non-maskable_interrupts 5623 ± 33% -100.0% 0.00 interrupts.CPU163.PMI:Performance_monitoring_interrupts 1198 +298.3% 4774 ± 2% interrupts.CPU164.CAL:Function_call_interrupts 614696 -50.5% 304057 interrupts.CPU164.LOC:Local_timer_interrupts 5601 ± 32% -100.0% 0.00 interrupts.CPU164.NMI:Non-maskable_interrupts 5601 ± 32% -100.0% 0.00 interrupts.CPU164.PMI:Performance_monitoring_interrupts 1197 +299.1% 4779 ± 2% interrupts.CPU165.CAL:Function_call_interrupts 614926 -50.7% 303425 interrupts.CPU165.LOC:Local_timer_interrupts 5618 ± 33% -100.0% 0.00 interrupts.CPU165.NMI:Non-maskable_interrupts 5618 ± 33% -100.0% 0.00 interrupts.CPU165.PMI:Performance_monitoring_interrupts 1198 +297.3% 4761 ± 2% interrupts.CPU166.CAL:Function_call_interrupts 614657 -50.6% 303346 interrupts.CPU166.LOC:Local_timer_interrupts 5622 ± 33% -100.0% 0.00 interrupts.CPU166.NMI:Non-maskable_interrupts 5622 ± 33% -100.0% 0.00 interrupts.CPU166.PMI:Performance_monitoring_interrupts 1199 +296.3% 4752 ± 3% interrupts.CPU167.CAL:Function_call_interrupts 614773 -50.7% 303215 interrupts.CPU167.LOC:Local_timer_interrupts 5614 ± 33% -100.0% 0.00 interrupts.CPU167.NMI:Non-maskable_interrupts 5614 ± 33% -100.0% 0.00 interrupts.CPU167.PMI:Performance_monitoring_interrupts 1197 +298.2% 4767 ± 2% interrupts.CPU168.CAL:Function_call_interrupts 614531 -50.6% 303665 interrupts.CPU168.LOC:Local_timer_interrupts 4662 ± 34% -100.0% 0.00 interrupts.CPU168.NMI:Non-maskable_interrupts 4662 ± 34% -100.0% 0.00 interrupts.CPU168.PMI:Performance_monitoring_interrupts 1197 +298.1% 4766 ± 2% interrupts.CPU169.CAL:Function_call_interrupts 614914 -50.7% 303198 interrupts.CPU169.LOC:Local_timer_interrupts 5595 ± 33% -100.0% 0.00 interrupts.CPU169.NMI:Non-maskable_interrupts 5595 ± 33% -100.0% 0.00 interrupts.CPU169.PMI:Performance_monitoring_interrupts 23.00 ±115% +515.2% 141.50 ± 80% interrupts.CPU169.RES:Rescheduling_interrupts 1197 +297.1% 4753 ± 2% interrupts.CPU17.CAL:Function_call_interrupts 614868 -50.6% 303854 interrupts.CPU17.LOC:Local_timer_interrupts 4715 ± 34% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts 4715 ± 34% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts 1198 +297.6% 4764 ± 2% interrupts.CPU170.CAL:Function_call_interrupts 614764 -50.7% 303325 interrupts.CPU170.LOC:Local_timer_interrupts 5625 ± 33% -100.0% 0.00 interrupts.CPU170.NMI:Non-maskable_interrupts 5625 ± 33% -100.0% 0.00 interrupts.CPU170.PMI:Performance_monitoring_interrupts 1198 +298.3% 4771 ± 2% interrupts.CPU171.CAL:Function_call_interrupts 614933 -50.7% 303280 interrupts.CPU171.LOC:Local_timer_interrupts 5631 ± 33% -100.0% 0.00 interrupts.CPU171.NMI:Non-maskable_interrupts 5631 ± 33% -100.0% 0.00 interrupts.CPU171.PMI:Performance_monitoring_interrupts 40.50 ± 42% +261.1% 146.25 ± 46% interrupts.CPU171.RES:Rescheduling_interrupts 1199 +298.1% 4773 ± 2% interrupts.CPU172.CAL:Function_call_interrupts 614752 -50.6% 303711 interrupts.CPU172.LOC:Local_timer_interrupts 6521 ± 24% -100.0% 0.00 interrupts.CPU172.NMI:Non-maskable_interrupts 6521 ± 24% -100.0% 0.00 interrupts.CPU172.PMI:Performance_monitoring_interrupts 1199 +298.1% 4773 ± 2% interrupts.CPU173.CAL:Function_call_interrupts 614617 -50.7% 303109 interrupts.CPU173.LOC:Local_timer_interrupts 5610 ± 33% -100.0% 0.00 interrupts.CPU173.NMI:Non-maskable_interrupts 5610 ± 33% -100.0% 0.00 interrupts.CPU173.PMI:Performance_monitoring_interrupts 1199 +296.7% 4757 ± 2% interrupts.CPU174.CAL:Function_call_interrupts 614890 -50.7% 303135 interrupts.CPU174.LOC:Local_timer_interrupts 4681 ± 34% -100.0% 0.00 interrupts.CPU174.NMI:Non-maskable_interrupts 4681 ± 34% -100.0% 0.00 interrupts.CPU174.PMI:Performance_monitoring_interrupts 1197 +298.2% 4768 ± 2% interrupts.CPU175.CAL:Function_call_interrupts 614695 -50.6% 303360 interrupts.CPU175.LOC:Local_timer_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU175.NMI:Non-maskable_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU175.PMI:Performance_monitoring_interrupts 1197 +297.9% 4765 ± 2% interrupts.CPU176.CAL:Function_call_interrupts 614143 -50.5% 303713 interrupts.CPU176.LOC:Local_timer_interrupts 5613 ± 33% -100.0% 0.00 interrupts.CPU176.NMI:Non-maskable_interrupts 5613 ± 33% -100.0% 0.00 interrupts.CPU176.PMI:Performance_monitoring_interrupts 148.75 ± 29% +95.1% 290.25 ± 18% interrupts.CPU176.RES:Rescheduling_interrupts 1198 +297.5% 4763 ± 2% interrupts.CPU177.CAL:Function_call_interrupts 614866 -50.7% 303322 interrupts.CPU177.LOC:Local_timer_interrupts 5636 ± 33% -100.0% 0.00 interrupts.CPU177.NMI:Non-maskable_interrupts 5636 ± 33% -100.0% 0.00 interrupts.CPU177.PMI:Performance_monitoring_interrupts 1199 +297.4% 4765 ± 2% interrupts.CPU178.CAL:Function_call_interrupts 614882 -50.7% 303276 interrupts.CPU178.LOC:Local_timer_interrupts 4685 ± 34% -100.0% 0.00 interrupts.CPU178.NMI:Non-maskable_interrupts 4685 ± 34% -100.0% 0.00 interrupts.CPU178.PMI:Performance_monitoring_interrupts 1198 +298.6% 4777 ± 2% interrupts.CPU179.CAL:Function_call_interrupts 614795 -50.7% 303258 interrupts.CPU179.LOC:Local_timer_interrupts 5638 ± 33% -100.0% 0.00 interrupts.CPU179.NMI:Non-maskable_interrupts 5638 ± 33% -100.0% 0.00 interrupts.CPU179.PMI:Performance_monitoring_interrupts 1196 +298.4% 4767 ± 2% interrupts.CPU18.CAL:Function_call_interrupts 614912 -50.6% 303581 interrupts.CPU18.LOC:Local_timer_interrupts 5661 ± 32% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts 5661 ± 32% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts 508.75 ± 37% -59.8% 204.75 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 1198 +298.9% 4779 ± 2% interrupts.CPU180.CAL:Function_call_interrupts 614467 -50.6% 303526 interrupts.CPU180.LOC:Local_timer_interrupts 5615 ± 33% -100.0% 0.00 interrupts.CPU180.NMI:Non-maskable_interrupts 5615 ± 33% -100.0% 0.00 interrupts.CPU180.PMI:Performance_monitoring_interrupts 1198 +297.4% 4762 ± 2% interrupts.CPU181.CAL:Function_call_interrupts 614725 -50.7% 303141 interrupts.CPU181.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU181.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU181.PMI:Performance_monitoring_interrupts 1198 +297.6% 4764 ± 2% interrupts.CPU182.CAL:Function_call_interrupts 614687 -50.7% 303128 interrupts.CPU182.LOC:Local_timer_interrupts 5615 ± 33% -100.0% 0.00 interrupts.CPU182.NMI:Non-maskable_interrupts 5615 ± 33% -100.0% 0.00 interrupts.CPU182.PMI:Performance_monitoring_interrupts 1197 +297.6% 4762 ± 2% interrupts.CPU183.CAL:Function_call_interrupts 614964 -50.7% 303218 interrupts.CPU183.LOC:Local_timer_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU183.NMI:Non-maskable_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU183.PMI:Performance_monitoring_interrupts 1199 +297.9% 4772 ± 2% interrupts.CPU184.CAL:Function_call_interrupts 614785 -50.6% 303774 interrupts.CPU184.LOC:Local_timer_interrupts 4687 ± 34% -100.0% 0.00 interrupts.CPU184.NMI:Non-maskable_interrupts 4687 ± 34% -100.0% 0.00 interrupts.CPU184.PMI:Performance_monitoring_interrupts 1198 +296.4% 4751 ± 3% interrupts.CPU185.CAL:Function_call_interrupts 614686 -50.7% 303305 interrupts.CPU185.LOC:Local_timer_interrupts 4690 ± 34% -100.0% 0.00 interrupts.CPU185.NMI:Non-maskable_interrupts 4690 ± 34% -100.0% 0.00 interrupts.CPU185.PMI:Performance_monitoring_interrupts 1198 +296.7% 4755 ± 2% interrupts.CPU186.CAL:Function_call_interrupts 614783 -50.7% 303109 interrupts.CPU186.LOC:Local_timer_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU186.NMI:Non-maskable_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU186.PMI:Performance_monitoring_interrupts 1198 +297.7% 4765 ± 2% interrupts.CPU187.CAL:Function_call_interrupts 614872 -50.7% 303160 interrupts.CPU187.LOC:Local_timer_interrupts 3733 -100.0% 0.00 interrupts.CPU187.NMI:Non-maskable_interrupts 3733 -100.0% 0.00 interrupts.CPU187.PMI:Performance_monitoring_interrupts 33.50 ±133% +724.6% 276.25 ±116% interrupts.CPU187.RES:Rescheduling_interrupts 1198 +297.8% 4767 ± 2% interrupts.CPU188.CAL:Function_call_interrupts 614850 -50.7% 303370 interrupts.CPU188.LOC:Local_timer_interrupts 3748 -100.0% 0.00 interrupts.CPU188.NMI:Non-maskable_interrupts 3748 -100.0% 0.00 interrupts.CPU188.PMI:Performance_monitoring_interrupts 1198 +298.4% 4773 ± 2% interrupts.CPU189.CAL:Function_call_interrupts 614885 -50.6% 303806 interrupts.CPU189.LOC:Local_timer_interrupts 4673 ± 34% -100.0% 0.25 ±173% interrupts.CPU189.NMI:Non-maskable_interrupts 4673 ± 34% -100.0% 0.25 ±173% interrupts.CPU189.PMI:Performance_monitoring_interrupts 55.75 ± 93% +362.8% 258.00 ± 53% interrupts.CPU189.RES:Rescheduling_interrupts 1196 +298.5% 4766 ± 2% interrupts.CPU19.CAL:Function_call_interrupts 614733 -50.7% 303104 interrupts.CPU19.LOC:Local_timer_interrupts 4698 ± 34% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts 4698 ± 34% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts 1198 +297.1% 4758 ± 2% interrupts.CPU190.CAL:Function_call_interrupts 614488 -50.7% 303169 interrupts.CPU190.LOC:Local_timer_interrupts 4681 ± 34% -100.0% 0.00 interrupts.CPU190.NMI:Non-maskable_interrupts 4681 ± 34% -100.0% 0.00 interrupts.CPU190.PMI:Performance_monitoring_interrupts 1198 +297.9% 4767 ± 2% interrupts.CPU191.CAL:Function_call_interrupts 614804 -50.7% 303341 interrupts.CPU191.LOC:Local_timer_interrupts 4690 ± 34% -100.0% 0.00 interrupts.CPU191.NMI:Non-maskable_interrupts 4690 ± 34% -100.0% 0.00 interrupts.CPU191.PMI:Performance_monitoring_interrupts 1199 +297.7% 4771 ± 2% interrupts.CPU192.CAL:Function_call_interrupts 614888 -50.6% 303563 interrupts.CPU192.LOC:Local_timer_interrupts 4659 ± 34% -100.0% 0.00 interrupts.CPU192.NMI:Non-maskable_interrupts 4659 ± 34% -100.0% 0.00 interrupts.CPU192.PMI:Performance_monitoring_interrupts 1198 +296.5% 4752 ± 2% interrupts.CPU193.CAL:Function_call_interrupts 614816 -50.7% 303205 interrupts.CPU193.LOC:Local_timer_interrupts 5635 ± 33% -100.0% 0.00 interrupts.CPU193.NMI:Non-maskable_interrupts 5635 ± 33% -100.0% 0.00 interrupts.CPU193.PMI:Performance_monitoring_interrupts 1198 +297.2% 4759 ± 2% interrupts.CPU194.CAL:Function_call_interrupts 614756 -50.7% 303337 interrupts.CPU194.LOC:Local_timer_interrupts 4655 ± 34% -100.0% 0.00 interrupts.CPU194.NMI:Non-maskable_interrupts 4655 ± 34% -100.0% 0.00 interrupts.CPU194.PMI:Performance_monitoring_interrupts 1198 +298.2% 4771 ± 2% interrupts.CPU195.CAL:Function_call_interrupts 614932 -50.7% 303236 interrupts.CPU195.LOC:Local_timer_interrupts 3731 -100.0% 0.00 interrupts.CPU195.NMI:Non-maskable_interrupts 3731 -100.0% 0.00 interrupts.CPU195.PMI:Performance_monitoring_interrupts 1198 +297.6% 4765 ± 2% interrupts.CPU196.CAL:Function_call_interrupts 614683 -50.6% 303375 interrupts.CPU196.LOC:Local_timer_interrupts 3741 -100.0% 0.00 interrupts.CPU196.NMI:Non-maskable_interrupts 3741 -100.0% 0.00 interrupts.CPU196.PMI:Performance_monitoring_interrupts 1198 +297.9% 4767 ± 3% interrupts.CPU197.CAL:Function_call_interrupts 614866 -50.7% 303076 interrupts.CPU197.LOC:Local_timer_interrupts 3749 -100.0% 0.00 interrupts.CPU197.NMI:Non-maskable_interrupts 3749 -100.0% 0.00 interrupts.CPU197.PMI:Performance_monitoring_interrupts 1198 +297.3% 4759 ± 2% interrupts.CPU198.CAL:Function_call_interrupts 614733 -50.6% 303418 interrupts.CPU198.LOC:Local_timer_interrupts 3720 -100.0% 0.00 interrupts.CPU198.NMI:Non-maskable_interrupts 3720 -100.0% 0.00 interrupts.CPU198.PMI:Performance_monitoring_interrupts 1198 +298.7% 4776 ± 2% interrupts.CPU199.CAL:Function_call_interrupts 614765 -50.7% 303289 interrupts.CPU199.LOC:Local_timer_interrupts 4658 ± 34% -100.0% 0.00 interrupts.CPU199.NMI:Non-maskable_interrupts 4658 ± 34% -100.0% 0.00 interrupts.CPU199.PMI:Performance_monitoring_interrupts 1196 +298.5% 4769 ± 2% interrupts.CPU2.CAL:Function_call_interrupts 614899 -50.6% 303730 interrupts.CPU2.LOC:Local_timer_interrupts 4749 ± 34% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 4749 ± 34% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 1197 +297.6% 4758 ± 2% interrupts.CPU20.CAL:Function_call_interrupts 614508 -50.7% 303248 interrupts.CPU20.LOC:Local_timer_interrupts 5669 ± 33% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts 5669 ± 33% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts 1198 +298.5% 4775 ± 2% interrupts.CPU200.CAL:Function_call_interrupts 614888 -50.6% 303653 interrupts.CPU200.LOC:Local_timer_interrupts 5612 ± 33% -100.0% 0.00 interrupts.CPU200.NMI:Non-maskable_interrupts 5612 ± 33% -100.0% 0.00 interrupts.CPU200.PMI:Performance_monitoring_interrupts 1198 +298.1% 4770 ± 2% interrupts.CPU201.CAL:Function_call_interrupts 614830 -50.7% 303121 interrupts.CPU201.LOC:Local_timer_interrupts 5624 ± 33% -100.0% 0.00 interrupts.CPU201.NMI:Non-maskable_interrupts 5624 ± 33% -100.0% 0.00 interrupts.CPU201.PMI:Performance_monitoring_interrupts 1198 +298.0% 4769 ± 3% interrupts.CPU202.CAL:Function_call_interrupts 614715 -50.7% 303162 interrupts.CPU202.LOC:Local_timer_interrupts 4675 ± 35% -100.0% 0.00 interrupts.CPU202.NMI:Non-maskable_interrupts 4675 ± 35% -100.0% 0.00 interrupts.CPU202.PMI:Performance_monitoring_interrupts 1197 +296.7% 4749 ± 2% interrupts.CPU203.CAL:Function_call_interrupts 614734 -50.7% 303033 interrupts.CPU203.LOC:Local_timer_interrupts 3741 -100.0% 0.00 interrupts.CPU203.NMI:Non-maskable_interrupts 3741 -100.0% 0.00 interrupts.CPU203.PMI:Performance_monitoring_interrupts 46.50 ± 44% +776.9% 407.75 ± 29% interrupts.CPU203.RES:Rescheduling_interrupts 1198 +296.6% 4752 ± 2% interrupts.CPU204.CAL:Function_call_interrupts 614751 -50.6% 303739 interrupts.CPU204.LOC:Local_timer_interrupts 3723 -100.0% 0.00 interrupts.CPU204.NMI:Non-maskable_interrupts 3723 -100.0% 0.00 interrupts.CPU204.PMI:Performance_monitoring_interrupts 1198 +297.5% 4765 ± 2% interrupts.CPU205.CAL:Function_call_interrupts 614931 -50.6% 303659 interrupts.CPU205.LOC:Local_timer_interrupts 3721 -100.0% 0.00 interrupts.CPU205.NMI:Non-maskable_interrupts 3721 -100.0% 0.00 interrupts.CPU205.PMI:Performance_monitoring_interrupts 73.00 ± 98% +674.3% 565.25 ± 62% interrupts.CPU205.RES:Rescheduling_interrupts 1200 +296.5% 4757 ± 2% interrupts.CPU206.CAL:Function_call_interrupts 614333 -50.6% 303423 interrupts.CPU206.LOC:Local_timer_interrupts 4684 ± 35% -100.0% 0.00 interrupts.CPU206.NMI:Non-maskable_interrupts 4684 ± 35% -100.0% 0.00 interrupts.CPU206.PMI:Performance_monitoring_interrupts 1200 +297.3% 4768 ± 2% interrupts.CPU207.CAL:Function_call_interrupts 614821 -50.7% 303311 interrupts.CPU207.LOC:Local_timer_interrupts 5622 ± 33% -100.0% 0.00 interrupts.CPU207.NMI:Non-maskable_interrupts 5622 ± 33% -100.0% 0.00 interrupts.CPU207.PMI:Performance_monitoring_interrupts 1198 +296.5% 4749 ± 2% interrupts.CPU208.CAL:Function_call_interrupts 614865 -50.7% 303354 interrupts.CPU208.LOC:Local_timer_interrupts 4679 ± 34% -100.0% 0.00 interrupts.CPU208.NMI:Non-maskable_interrupts 4679 ± 34% -100.0% 0.00 interrupts.CPU208.PMI:Performance_monitoring_interrupts 1198 +297.5% 4762 ± 2% interrupts.CPU209.CAL:Function_call_interrupts 614915 -50.6% 303688 interrupts.CPU209.LOC:Local_timer_interrupts 4672 ± 34% -100.0% 0.00 interrupts.CPU209.NMI:Non-maskable_interrupts 4672 ± 34% -100.0% 0.00 interrupts.CPU209.PMI:Performance_monitoring_interrupts 1198 +297.8% 4767 ± 2% interrupts.CPU21.CAL:Function_call_interrupts 614771 -50.6% 303429 interrupts.CPU21.LOC:Local_timer_interrupts 5596 ± 32% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts 5596 ± 32% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts 1197 +298.7% 4775 ± 2% interrupts.CPU210.CAL:Function_call_interrupts 614699 -50.7% 303192 interrupts.CPU210.LOC:Local_timer_interrupts 4670 ± 34% -100.0% 0.00 interrupts.CPU210.NMI:Non-maskable_interrupts 4670 ± 34% -100.0% 0.00 interrupts.CPU210.PMI:Performance_monitoring_interrupts 1198 +297.6% 4763 ± 3% interrupts.CPU211.CAL:Function_call_interrupts 614908 -50.7% 303356 interrupts.CPU211.LOC:Local_timer_interrupts 4664 ± 34% -100.0% 0.00 interrupts.CPU211.NMI:Non-maskable_interrupts 4664 ± 34% -100.0% 0.00 interrupts.CPU211.PMI:Performance_monitoring_interrupts 1198 +298.9% 4780 ± 2% interrupts.CPU212.CAL:Function_call_interrupts 614702 -50.6% 303558 interrupts.CPU212.LOC:Local_timer_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU212.NMI:Non-maskable_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU212.PMI:Performance_monitoring_interrupts 1198 +297.2% 4761 ± 2% interrupts.CPU213.CAL:Function_call_interrupts 614755 -50.7% 303028 interrupts.CPU213.LOC:Local_timer_interrupts 4658 ± 34% -100.0% 0.00 interrupts.CPU213.NMI:Non-maskable_interrupts 4658 ± 34% -100.0% 0.00 interrupts.CPU213.PMI:Performance_monitoring_interrupts 1197 +298.7% 4774 ± 2% interrupts.CPU214.CAL:Function_call_interrupts 614809 -50.7% 303313 interrupts.CPU214.LOC:Local_timer_interrupts 4586 ± 34% -100.0% 0.00 interrupts.CPU214.NMI:Non-maskable_interrupts 4586 ± 34% -100.0% 0.00 interrupts.CPU214.PMI:Performance_monitoring_interrupts 726.50 ± 62% -72.6% 198.75 ± 77% interrupts.CPU214.RES:Rescheduling_interrupts 1198 +296.8% 4754 ± 2% interrupts.CPU215.CAL:Function_call_interrupts 614797 -50.7% 303309 interrupts.CPU215.LOC:Local_timer_interrupts 4546 ± 34% -100.0% 0.00 interrupts.CPU215.NMI:Non-maskable_interrupts 4546 ± 34% -100.0% 0.00 interrupts.CPU215.PMI:Performance_monitoring_interrupts 1198 +298.6% 4776 ± 2% interrupts.CPU216.CAL:Function_call_interrupts 614679 -50.6% 303442 interrupts.CPU216.LOC:Local_timer_interrupts 4658 ± 34% -100.0% 0.00 interrupts.CPU216.NMI:Non-maskable_interrupts 4658 ± 34% -100.0% 0.00 interrupts.CPU216.PMI:Performance_monitoring_interrupts 1198 +298.9% 4780 ± 2% interrupts.CPU217.CAL:Function_call_interrupts 614721 -50.5% 303993 interrupts.CPU217.LOC:Local_timer_interrupts 3728 -100.0% 0.00 interrupts.CPU217.NMI:Non-maskable_interrupts 3728 -100.0% 0.00 interrupts.CPU217.PMI:Performance_monitoring_interrupts 1197 +298.3% 4770 ± 2% interrupts.CPU218.CAL:Function_call_interrupts 614531 -50.6% 303422 interrupts.CPU218.LOC:Local_timer_interrupts 3749 -100.0% 0.00 interrupts.CPU218.NMI:Non-maskable_interrupts 3749 -100.0% 0.00 interrupts.CPU218.PMI:Performance_monitoring_interrupts 1197 +298.7% 4775 ± 2% interrupts.CPU219.CAL:Function_call_interrupts 614940 -50.7% 303463 interrupts.CPU219.LOC:Local_timer_interrupts 4686 ± 34% -100.0% 0.00 interrupts.CPU219.NMI:Non-maskable_interrupts 4686 ± 34% -100.0% 0.00 interrupts.CPU219.PMI:Performance_monitoring_interrupts 1196 +297.6% 4757 ± 2% interrupts.CPU22.CAL:Function_call_interrupts 614806 -50.6% 303693 interrupts.CPU22.LOC:Local_timer_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts 4674 ± 34% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts 1197 +297.8% 4763 ± 2% interrupts.CPU220.CAL:Function_call_interrupts 614840 -50.6% 303537 interrupts.CPU220.LOC:Local_timer_interrupts 4671 ± 34% -100.0% 0.00 interrupts.CPU220.NMI:Non-maskable_interrupts 4671 ± 34% -100.0% 0.00 interrupts.CPU220.PMI:Performance_monitoring_interrupts 1198 +298.8% 4780 ± 2% interrupts.CPU221.CAL:Function_call_interrupts 614781 -50.6% 303683 interrupts.CPU221.LOC:Local_timer_interrupts 4679 ± 34% -100.0% 0.00 interrupts.CPU221.NMI:Non-maskable_interrupts 4679 ± 34% -100.0% 0.00 interrupts.CPU221.PMI:Performance_monitoring_interrupts 1202 +296.2% 4762 ± 2% interrupts.CPU222.CAL:Function_call_interrupts 614763 -50.6% 303386 interrupts.CPU222.LOC:Local_timer_interrupts 6518 ± 24% -100.0% 0.00 interrupts.CPU222.NMI:Non-maskable_interrupts 6518 ± 24% -100.0% 0.00 interrupts.CPU222.PMI:Performance_monitoring_interrupts 74.75 ± 15% +427.8% 394.50 ± 53% interrupts.CPU222.RES:Rescheduling_interrupts 1202 +294.7% 4744 ± 2% interrupts.CPU223.CAL:Function_call_interrupts 614874 -50.7% 303147 interrupts.CPU223.LOC:Local_timer_interrupts 6525 ± 24% -100.0% 0.00 interrupts.CPU223.NMI:Non-maskable_interrupts 6525 ± 24% -100.0% 0.00 interrupts.CPU223.PMI:Performance_monitoring_interrupts 48.50 ± 75% +665.5% 371.25 ± 62% interrupts.CPU223.RES:Rescheduling_interrupts 1209 +293.5% 4760 ± 2% interrupts.CPU224.CAL:Function_call_interrupts 614589 -50.6% 303425 interrupts.CPU224.LOC:Local_timer_interrupts 5612 ± 33% -100.0% 0.00 interrupts.CPU224.NMI:Non-maskable_interrupts 5612 ± 33% -100.0% 0.00 interrupts.CPU224.PMI:Performance_monitoring_interrupts 1213 +293.8% 4776 ± 2% interrupts.CPU225.CAL:Function_call_interrupts 614597 -50.6% 303606 interrupts.CPU225.LOC:Local_timer_interrupts 4661 ± 34% -100.0% 0.00 interrupts.CPU225.NMI:Non-maskable_interrupts 4661 ± 34% -100.0% 0.00 interrupts.CPU225.PMI:Performance_monitoring_interrupts 1212 +292.6% 4761 ± 2% interrupts.CPU226.CAL:Function_call_interrupts 614666 -50.7% 303255 interrupts.CPU226.LOC:Local_timer_interrupts 4687 ± 34% -100.0% 0.00 interrupts.CPU226.NMI:Non-maskable_interrupts 4687 ± 34% -100.0% 0.00 interrupts.CPU226.PMI:Performance_monitoring_interrupts 1213 +291.7% 4752 ± 2% interrupts.CPU227.CAL:Function_call_interrupts 614642 -50.7% 303227 interrupts.CPU227.LOC:Local_timer_interrupts 4683 ± 35% -100.0% 0.00 interrupts.CPU227.NMI:Non-maskable_interrupts 4683 ± 35% -100.0% 0.00 interrupts.CPU227.PMI:Performance_monitoring_interrupts 1212 +293.7% 4773 ± 2% interrupts.CPU228.CAL:Function_call_interrupts 614868 -50.7% 303339 interrupts.CPU228.LOC:Local_timer_interrupts 6529 ± 24% -100.0% 0.00 interrupts.CPU228.NMI:Non-maskable_interrupts 6529 ± 24% -100.0% 0.00 interrupts.CPU228.PMI:Performance_monitoring_interrupts 1213 +293.1% 4770 ± 2% interrupts.CPU229.CAL:Function_call_interrupts 614887 -50.6% 303718 interrupts.CPU229.LOC:Local_timer_interrupts 6545 ± 24% -100.0% 0.00 interrupts.CPU229.NMI:Non-maskable_interrupts 6545 ± 24% -100.0% 0.00 interrupts.CPU229.PMI:Performance_monitoring_interrupts 1196 +297.3% 4752 ± 2% interrupts.CPU23.CAL:Function_call_interrupts 614963 -50.7% 303232 interrupts.CPU23.LOC:Local_timer_interrupts 4671 ± 34% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts 4671 ± 34% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts 1213 +293.1% 4768 ± 2% interrupts.CPU230.CAL:Function_call_interrupts 614771 -50.7% 303356 interrupts.CPU230.LOC:Local_timer_interrupts 5595 ± 33% -100.0% 0.00 interrupts.CPU230.NMI:Non-maskable_interrupts 5595 ± 33% -100.0% 0.00 interrupts.CPU230.PMI:Performance_monitoring_interrupts 1214 +292.9% 4770 ± 2% interrupts.CPU231.CAL:Function_call_interrupts 614840 -50.7% 303097 interrupts.CPU231.LOC:Local_timer_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU231.NMI:Non-maskable_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU231.PMI:Performance_monitoring_interrupts 1213 +292.7% 4765 ± 2% interrupts.CPU232.CAL:Function_call_interrupts 614779 -50.7% 303287 interrupts.CPU232.LOC:Local_timer_interrupts 3747 -100.0% 0.00 interrupts.CPU232.NMI:Non-maskable_interrupts 3747 -100.0% 0.00 interrupts.CPU232.PMI:Performance_monitoring_interrupts 1213 +293.5% 4773 ± 2% interrupts.CPU233.CAL:Function_call_interrupts 614741 -50.6% 303626 interrupts.CPU233.LOC:Local_timer_interrupts 4689 ± 34% -100.0% 0.00 interrupts.CPU233.NMI:Non-maskable_interrupts 4689 ± 34% -100.0% 0.00 interrupts.CPU233.PMI:Performance_monitoring_interrupts 1213 +294.1% 4780 ± 2% interrupts.CPU234.CAL:Function_call_interrupts 614837 -50.7% 303232 interrupts.CPU234.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU234.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU234.PMI:Performance_monitoring_interrupts 1213 +292.2% 4759 ± 2% interrupts.CPU235.CAL:Function_call_interrupts 614812 -50.7% 303382 interrupts.CPU235.LOC:Local_timer_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU235.NMI:Non-maskable_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU235.PMI:Performance_monitoring_interrupts 22.00 ± 92% +947.7% 230.50 ± 40% interrupts.CPU235.RES:Rescheduling_interrupts 1213 +292.9% 4766 ± 2% interrupts.CPU236.CAL:Function_call_interrupts 614602 -50.6% 303366 interrupts.CPU236.LOC:Local_timer_interrupts 4672 ± 34% -100.0% 0.00 interrupts.CPU236.NMI:Non-maskable_interrupts 4672 ± 34% -100.0% 0.00 interrupts.CPU236.PMI:Performance_monitoring_interrupts 1216 +292.9% 4777 ± 2% interrupts.CPU237.CAL:Function_call_interrupts 614747 -50.6% 303780 interrupts.CPU237.LOC:Local_timer_interrupts 5619 ± 33% -100.0% 0.00 interrupts.CPU237.NMI:Non-maskable_interrupts 5619 ± 33% -100.0% 0.00 interrupts.CPU237.PMI:Performance_monitoring_interrupts 1213 +292.9% 4766 ± 2% interrupts.CPU238.CAL:Function_call_interrupts 614697 -50.7% 303171 interrupts.CPU238.LOC:Local_timer_interrupts 5591 ± 33% -100.0% 0.00 interrupts.CPU238.NMI:Non-maskable_interrupts 5591 ± 33% -100.0% 0.00 interrupts.CPU238.PMI:Performance_monitoring_interrupts 1213 +292.3% 4759 ± 2% interrupts.CPU239.CAL:Function_call_interrupts 614738 -50.7% 303301 interrupts.CPU239.LOC:Local_timer_interrupts 5576 ± 33% -100.0% 0.00 interrupts.CPU239.NMI:Non-maskable_interrupts 5576 ± 33% -100.0% 0.00 interrupts.CPU239.PMI:Performance_monitoring_interrupts 109.00 ± 73% +343.3% 483.25 ± 50% interrupts.CPU239.RES:Rescheduling_interrupts 1197 +296.6% 4746 ± 2% interrupts.CPU24.CAL:Function_call_interrupts 614796 -50.7% 303278 interrupts.CPU24.LOC:Local_timer_interrupts 5661 ± 33% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts 5661 ± 33% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts 1213 +292.7% 4767 ± 2% interrupts.CPU240.CAL:Function_call_interrupts 614841 -50.7% 303318 interrupts.CPU240.LOC:Local_timer_interrupts 3733 -100.0% 0.00 interrupts.CPU240.NMI:Non-maskable_interrupts 3733 -100.0% 0.00 interrupts.CPU240.PMI:Performance_monitoring_interrupts 1214 +293.2% 4776 ± 2% interrupts.CPU241.CAL:Function_call_interrupts 614889 -50.6% 303775 interrupts.CPU241.LOC:Local_timer_interrupts 5598 ± 33% -100.0% 0.00 interrupts.CPU241.NMI:Non-maskable_interrupts 5598 ± 33% -100.0% 0.00 interrupts.CPU241.PMI:Performance_monitoring_interrupts 11.50 ± 25% +1087.0% 136.50 ±135% interrupts.CPU241.RES:Rescheduling_interrupts 1213 +293.1% 4770 ± 2% interrupts.CPU242.CAL:Function_call_interrupts 614580 -50.6% 303664 interrupts.CPU242.LOC:Local_timer_interrupts 4685 ± 34% -100.0% 0.00 interrupts.CPU242.NMI:Non-maskable_interrupts 4685 ± 34% -100.0% 0.00 interrupts.CPU242.PMI:Performance_monitoring_interrupts 1214 +291.5% 4753 ± 2% interrupts.CPU243.CAL:Function_call_interrupts 614986 -50.7% 303153 interrupts.CPU243.LOC:Local_timer_interrupts 6554 ± 24% -100.0% 0.00 interrupts.CPU243.NMI:Non-maskable_interrupts 6554 ± 24% -100.0% 0.00 interrupts.CPU243.PMI:Performance_monitoring_interrupts 99.50 ± 74% +275.4% 373.50 ± 26% interrupts.CPU243.RES:Rescheduling_interrupts 1213 +292.8% 4766 ± 2% interrupts.CPU244.CAL:Function_call_interrupts 614811 -50.7% 303345 interrupts.CPU244.LOC:Local_timer_interrupts 5599 ± 33% -100.0% 0.00 interrupts.CPU244.NMI:Non-maskable_interrupts 5599 ± 33% -100.0% 0.00 interrupts.CPU244.PMI:Performance_monitoring_interrupts 1216 +293.1% 4781 ± 2% interrupts.CPU245.CAL:Function_call_interrupts 614574 -50.6% 303583 interrupts.CPU245.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU245.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU245.PMI:Performance_monitoring_interrupts 1213 +293.7% 4777 ± 2% interrupts.CPU246.CAL:Function_call_interrupts 614729 -50.7% 303166 interrupts.CPU246.LOC:Local_timer_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU246.NMI:Non-maskable_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU246.PMI:Performance_monitoring_interrupts 1214 +291.0% 4747 ± 2% interrupts.CPU247.CAL:Function_call_interrupts 614686 -50.6% 303433 interrupts.CPU247.LOC:Local_timer_interrupts 4699 ± 34% -100.0% 0.00 interrupts.CPU247.NMI:Non-maskable_interrupts 4699 ± 34% -100.0% 0.00 interrupts.CPU247.PMI:Performance_monitoring_interrupts 1213 +293.2% 4772 ± 2% interrupts.CPU248.CAL:Function_call_interrupts 614677 -50.6% 303380 interrupts.CPU248.LOC:Local_timer_interrupts 5613 ± 33% -100.0% 0.00 interrupts.CPU248.NMI:Non-maskable_interrupts 5613 ± 33% -100.0% 0.00 interrupts.CPU248.PMI:Performance_monitoring_interrupts 1213 +294.0% 4782 ± 2% interrupts.CPU249.CAL:Function_call_interrupts 614844 -50.6% 303810 interrupts.CPU249.LOC:Local_timer_interrupts 5610 ± 33% -100.0% 0.00 interrupts.CPU249.NMI:Non-maskable_interrupts 5610 ± 33% -100.0% 0.00 interrupts.CPU249.PMI:Performance_monitoring_interrupts 1197 +298.2% 4767 ± 2% interrupts.CPU25.CAL:Function_call_interrupts 614817 -50.7% 303394 interrupts.CPU25.LOC:Local_timer_interrupts 4712 ± 34% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts 4712 ± 34% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts 1213 +292.8% 4766 ± 2% interrupts.CPU250.CAL:Function_call_interrupts 614817 -50.7% 303323 interrupts.CPU250.LOC:Local_timer_interrupts 6527 ± 24% -100.0% 0.00 interrupts.CPU250.NMI:Non-maskable_interrupts 6527 ± 24% -100.0% 0.00 interrupts.CPU250.PMI:Performance_monitoring_interrupts 1213 +293.6% 4776 ± 2% interrupts.CPU251.CAL:Function_call_interrupts 614858 -50.7% 303310 interrupts.CPU251.LOC:Local_timer_interrupts 5589 ± 33% -100.0% 0.00 interrupts.CPU251.NMI:Non-maskable_interrupts 5589 ± 33% -100.0% 0.00 interrupts.CPU251.PMI:Performance_monitoring_interrupts 1213 +293.6% 4774 ± 2% interrupts.CPU252.CAL:Function_call_interrupts 614687 -50.7% 303161 interrupts.CPU252.LOC:Local_timer_interrupts 5619 ± 33% -100.0% 0.00 interrupts.CPU252.NMI:Non-maskable_interrupts 5619 ± 33% -100.0% 0.00 interrupts.CPU252.PMI:Performance_monitoring_interrupts 1213 +292.7% 4766 ± 2% interrupts.CPU253.CAL:Function_call_interrupts 614733 -50.6% 303622 interrupts.CPU253.LOC:Local_timer_interrupts 4699 ± 35% -100.0% 0.00 interrupts.CPU253.NMI:Non-maskable_interrupts 4699 ± 35% -100.0% 0.00 interrupts.CPU253.PMI:Performance_monitoring_interrupts 1213 +292.5% 4761 ± 2% interrupts.CPU254.CAL:Function_call_interrupts 614812 -50.7% 303347 interrupts.CPU254.LOC:Local_timer_interrupts 3741 -100.0% 0.00 interrupts.CPU254.NMI:Non-maskable_interrupts 3741 -100.0% 0.00 interrupts.CPU254.PMI:Performance_monitoring_interrupts 94.00 ± 70% +236.7% 316.50 ± 73% interrupts.CPU254.RES:Rescheduling_interrupts 1213 +291.6% 4752 ± 2% interrupts.CPU255.CAL:Function_call_interrupts 614747 -50.7% 303179 interrupts.CPU255.LOC:Local_timer_interrupts 3731 -100.0% 0.00 interrupts.CPU255.NMI:Non-maskable_interrupts 3731 -100.0% 0.00 interrupts.CPU255.PMI:Performance_monitoring_interrupts 80.25 ±133% +555.8% 526.25 ± 24% interrupts.CPU255.RES:Rescheduling_interrupts 1213 +293.3% 4771 ± 2% interrupts.CPU256.CAL:Function_call_interrupts 614403 -50.7% 303166 interrupts.CPU256.LOC:Local_timer_interrupts 3728 -100.0% 0.00 interrupts.CPU256.NMI:Non-maskable_interrupts 3728 -100.0% 0.00 interrupts.CPU256.PMI:Performance_monitoring_interrupts 1214 +293.0% 4771 ± 2% interrupts.CPU257.CAL:Function_call_interrupts 614656 -50.6% 303685 interrupts.CPU257.LOC:Local_timer_interrupts 3722 -100.0% 0.00 interrupts.CPU257.NMI:Non-maskable_interrupts 3722 -100.0% 0.00 interrupts.CPU257.PMI:Performance_monitoring_interrupts 14.75 ± 75% +435.6% 79.00 ± 51% interrupts.CPU257.RES:Rescheduling_interrupts 1213 +293.5% 4776 ± 2% interrupts.CPU258.CAL:Function_call_interrupts 614721 -50.7% 303200 interrupts.CPU258.LOC:Local_timer_interrupts 5598 ± 33% -100.0% 0.00 interrupts.CPU258.NMI:Non-maskable_interrupts 5598 ± 33% -100.0% 0.00 interrupts.CPU258.PMI:Performance_monitoring_interrupts 1214 +292.4% 4765 ± 2% interrupts.CPU259.CAL:Function_call_interrupts 614876 -50.7% 303368 interrupts.CPU259.LOC:Local_timer_interrupts 5616 ± 33% -100.0% 0.00 interrupts.CPU259.NMI:Non-maskable_interrupts 5616 ± 33% -100.0% 0.00 interrupts.CPU259.PMI:Performance_monitoring_interrupts 1197 +298.6% 4773 ± 2% interrupts.CPU26.CAL:Function_call_interrupts 614857 -50.6% 303581 interrupts.CPU26.LOC:Local_timer_interrupts 4674 ± 33% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts 4674 ± 33% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts 1212 +293.7% 4774 ± 2% interrupts.CPU260.CAL:Function_call_interrupts 614818 -50.7% 303128 interrupts.CPU260.LOC:Local_timer_interrupts 4660 ± 34% -100.0% 0.00 interrupts.CPU260.NMI:Non-maskable_interrupts 4660 ± 34% -100.0% 0.00 interrupts.CPU260.PMI:Performance_monitoring_interrupts 1213 +293.3% 4773 ± 2% interrupts.CPU261.CAL:Function_call_interrupts 614795 -50.6% 303456 interrupts.CPU261.LOC:Local_timer_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU261.NMI:Non-maskable_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU261.PMI:Performance_monitoring_interrupts 1212 +292.7% 4762 ± 2% interrupts.CPU262.CAL:Function_call_interrupts 614781 -50.7% 303360 interrupts.CPU262.LOC:Local_timer_interrupts 3732 -100.0% 0.00 interrupts.CPU262.NMI:Non-maskable_interrupts 3732 -100.0% 0.00 interrupts.CPU262.PMI:Performance_monitoring_interrupts 153.00 ± 91% +342.5% 677.00 ± 31% interrupts.CPU262.RES:Rescheduling_interrupts 1214 +292.4% 4765 ± 2% interrupts.CPU263.CAL:Function_call_interrupts 614723 -50.7% 303308 interrupts.CPU263.LOC:Local_timer_interrupts 4675 ± 34% -100.0% 0.00 interrupts.CPU263.NMI:Non-maskable_interrupts 4675 ± 34% -100.0% 0.00 interrupts.CPU263.PMI:Performance_monitoring_interrupts 136.25 ± 79% +190.6% 396.00 ± 43% interrupts.CPU263.RES:Rescheduling_interrupts 1214 +293.1% 4772 ± 2% interrupts.CPU264.CAL:Function_call_interrupts 614696 -50.7% 303122 interrupts.CPU264.LOC:Local_timer_interrupts 4668 ± 34% -100.0% 0.00 interrupts.CPU264.NMI:Non-maskable_interrupts 4668 ± 34% -100.0% 0.00 interrupts.CPU264.PMI:Performance_monitoring_interrupts 1213 +292.8% 4767 ± 2% interrupts.CPU265.CAL:Function_call_interrupts 614801 -50.6% 303699 interrupts.CPU265.LOC:Local_timer_interrupts 4659 ± 34% -100.0% 0.00 interrupts.CPU265.NMI:Non-maskable_interrupts 4659 ± 34% -100.0% 0.00 interrupts.CPU265.PMI:Performance_monitoring_interrupts 39.25 ±133% +404.5% 198.00 ± 82% interrupts.CPU265.RES:Rescheduling_interrupts 1214 +291.8% 4757 ± 3% interrupts.CPU266.CAL:Function_call_interrupts 614650 -50.7% 303136 interrupts.CPU266.LOC:Local_timer_interrupts 4656 ± 34% -100.0% 0.00 interrupts.CPU266.NMI:Non-maskable_interrupts 4656 ± 34% -100.0% 0.00 interrupts.CPU266.PMI:Performance_monitoring_interrupts 1214 +292.0% 4759 ± 3% interrupts.CPU267.CAL:Function_call_interrupts 614679 -50.7% 303334 interrupts.CPU267.LOC:Local_timer_interrupts 5588 ± 33% -100.0% 0.00 interrupts.CPU267.NMI:Non-maskable_interrupts 5588 ± 33% -100.0% 0.00 interrupts.CPU267.PMI:Performance_monitoring_interrupts 1213 +293.4% 4773 ± 2% interrupts.CPU268.CAL:Function_call_interrupts 615201 -50.7% 303140 interrupts.CPU268.LOC:Local_timer_interrupts 3733 -100.0% 0.00 interrupts.CPU268.NMI:Non-maskable_interrupts 3733 -100.0% 0.00 interrupts.CPU268.PMI:Performance_monitoring_interrupts 1214 +293.7% 4779 ± 2% interrupts.CPU269.CAL:Function_call_interrupts 614781 -50.7% 303366 interrupts.CPU269.LOC:Local_timer_interrupts 3730 -100.0% 0.00 interrupts.CPU269.NMI:Non-maskable_interrupts 3730 -100.0% 0.00 interrupts.CPU269.PMI:Performance_monitoring_interrupts 1197 +297.7% 4761 ± 2% interrupts.CPU27.CAL:Function_call_interrupts 614728 -50.7% 303074 interrupts.CPU27.LOC:Local_timer_interrupts 4676 ± 34% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts 4676 ± 34% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts 1214 +291.7% 4756 ± 2% interrupts.CPU270.CAL:Function_call_interrupts 614752 -50.7% 303261 interrupts.CPU270.LOC:Local_timer_interrupts 3702 -100.0% 0.00 interrupts.CPU270.NMI:Non-maskable_interrupts 3702 -100.0% 0.00 interrupts.CPU270.PMI:Performance_monitoring_interrupts 100.50 ± 36% +398.0% 500.50 ± 54% interrupts.CPU270.RES:Rescheduling_interrupts 1213 +293.6% 4775 ± 2% interrupts.CPU271.CAL:Function_call_interrupts 614649 -50.7% 303232 interrupts.CPU271.LOC:Local_timer_interrupts 3684 -100.0% 0.00 interrupts.CPU271.NMI:Non-maskable_interrupts 3684 -100.0% 0.00 interrupts.CPU271.PMI:Performance_monitoring_interrupts 1213 +294.0% 4782 ± 2% interrupts.CPU272.CAL:Function_call_interrupts 614825 -50.7% 303137 interrupts.CPU272.LOC:Local_timer_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU272.NMI:Non-maskable_interrupts 4657 ± 34% -100.0% 0.00 interrupts.CPU272.PMI:Performance_monitoring_interrupts 1214 +293.4% 4776 ± 2% interrupts.CPU273.CAL:Function_call_interrupts 614712 -50.5% 303992 interrupts.CPU273.LOC:Local_timer_interrupts 5577 ± 33% -100.0% 0.25 ±173% interrupts.CPU273.NMI:Non-maskable_interrupts 5577 ± 33% -100.0% 0.25 ±173% interrupts.CPU273.PMI:Performance_monitoring_interrupts 1213 +292.3% 4760 ± 2% interrupts.CPU274.CAL:Function_call_interrupts 614828 -50.7% 303156 interrupts.CPU274.LOC:Local_timer_interrupts 5588 ± 33% -100.0% 0.00 interrupts.CPU274.NMI:Non-maskable_interrupts 5588 ± 33% -100.0% 0.00 interrupts.CPU274.PMI:Performance_monitoring_interrupts 1212 +293.4% 4769 ± 2% interrupts.CPU275.CAL:Function_call_interrupts 614350 -50.6% 303189 interrupts.CPU275.LOC:Local_timer_interrupts 4646 ± 34% -100.0% 0.00 interrupts.CPU275.NMI:Non-maskable_interrupts 4646 ± 34% -100.0% 0.00 interrupts.CPU275.PMI:Performance_monitoring_interrupts 1214 +292.6% 4766 ± 2% interrupts.CPU276.CAL:Function_call_interrupts 614608 -50.7% 303143 interrupts.CPU276.LOC:Local_timer_interrupts 5576 ± 32% -100.0% 0.00 interrupts.CPU276.NMI:Non-maskable_interrupts 5576 ± 32% -100.0% 0.00 interrupts.CPU276.PMI:Performance_monitoring_interrupts 1213 +294.1% 4782 ± 2% interrupts.CPU277.CAL:Function_call_interrupts 614901 -50.6% 303547 interrupts.CPU277.LOC:Local_timer_interrupts 3697 -100.0% 0.00 interrupts.CPU277.NMI:Non-maskable_interrupts 3697 -100.0% 0.00 interrupts.CPU277.PMI:Performance_monitoring_interrupts 1214 +292.7% 4768 ± 3% interrupts.CPU278.CAL:Function_call_interrupts 614604 -50.6% 303419 interrupts.CPU278.LOC:Local_timer_interrupts 5596 ± 33% -100.0% 0.00 interrupts.CPU278.NMI:Non-maskable_interrupts 5596 ± 33% -100.0% 0.00 interrupts.CPU278.PMI:Performance_monitoring_interrupts 1213 +293.4% 4773 ± 2% interrupts.CPU279.CAL:Function_call_interrupts 614675 -50.6% 303451 interrupts.CPU279.LOC:Local_timer_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU279.NMI:Non-maskable_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU279.PMI:Performance_monitoring_interrupts 30.50 ± 59% +1282.8% 421.75 ± 66% interrupts.CPU279.RES:Rescheduling_interrupts 1197 +296.1% 4743 ± 2% interrupts.CPU28.CAL:Function_call_interrupts 614786 -50.7% 303112 interrupts.CPU28.LOC:Local_timer_interrupts 4682 ± 35% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts 4682 ± 35% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts 1213 +293.5% 4776 ± 2% interrupts.CPU280.CAL:Function_call_interrupts 614750 -50.6% 303705 interrupts.CPU280.LOC:Local_timer_interrupts 6521 ± 24% -100.0% 0.00 interrupts.CPU280.NMI:Non-maskable_interrupts 6521 ± 24% -100.0% 0.00 interrupts.CPU280.PMI:Performance_monitoring_interrupts 1214 +293.6% 4778 ± 2% interrupts.CPU281.CAL:Function_call_interrupts 614643 -50.6% 303819 interrupts.CPU281.LOC:Local_timer_interrupts 6556 ± 24% -100.0% 0.00 interrupts.CPU281.NMI:Non-maskable_interrupts 6556 ± 24% -100.0% 0.00 interrupts.CPU281.PMI:Performance_monitoring_interrupts 22.25 ± 33% +756.2% 190.50 ± 46% interrupts.CPU281.RES:Rescheduling_interrupts 1213 +292.7% 4764 ± 2% interrupts.CPU282.CAL:Function_call_interrupts 614572 -50.6% 303437 interrupts.CPU282.LOC:Local_timer_interrupts 6538 ± 24% -100.0% 0.00 interrupts.CPU282.NMI:Non-maskable_interrupts 6538 ± 24% -100.0% 0.00 interrupts.CPU282.PMI:Performance_monitoring_interrupts 1213 +293.6% 4774 ± 2% interrupts.CPU283.CAL:Function_call_interrupts 614833 -50.7% 303105 interrupts.CPU283.LOC:Local_timer_interrupts 6538 ± 24% -100.0% 0.00 interrupts.CPU283.NMI:Non-maskable_interrupts 6538 ± 24% -100.0% 0.00 interrupts.CPU283.PMI:Performance_monitoring_interrupts 1213 +293.2% 4771 ± 2% interrupts.CPU284.CAL:Function_call_interrupts 614858 -50.7% 303115 interrupts.CPU284.LOC:Local_timer_interrupts 3734 -100.0% 0.00 interrupts.CPU284.NMI:Non-maskable_interrupts 3734 -100.0% 0.00 interrupts.CPU284.PMI:Performance_monitoring_interrupts 1214 +292.1% 4762 ± 2% interrupts.CPU285.CAL:Function_call_interrupts 614746 -50.6% 303562 interrupts.CPU285.LOC:Local_timer_interrupts 4675 ± 34% -100.0% 0.00 interrupts.CPU285.NMI:Non-maskable_interrupts 4675 ± 34% -100.0% 0.00 interrupts.CPU285.PMI:Performance_monitoring_interrupts 38.00 ±107% +638.8% 280.75 ± 80% interrupts.CPU285.RES:Rescheduling_interrupts 1213 +293.0% 4767 ± 2% interrupts.CPU286.CAL:Function_call_interrupts 614860 -50.7% 303381 interrupts.CPU286.LOC:Local_timer_interrupts 5513 ± 33% -100.0% 0.00 interrupts.CPU286.NMI:Non-maskable_interrupts 5513 ± 33% -100.0% 0.00 interrupts.CPU286.PMI:Performance_monitoring_interrupts 58.75 ± 36% +571.9% 394.75 ± 67% interrupts.CPU286.RES:Rescheduling_interrupts 918.25 ± 31% +388.0% 4481 ± 6% interrupts.CPU287.CAL:Function_call_interrupts 614984 -50.6% 303630 interrupts.CPU287.LOC:Local_timer_interrupts 6338 ± 24% -100.0% 0.00 interrupts.CPU287.NMI:Non-maskable_interrupts 6338 ± 24% -100.0% 0.00 interrupts.CPU287.PMI:Performance_monitoring_interrupts 1198 +295.9% 4746 ± 2% interrupts.CPU29.CAL:Function_call_interrupts 614744 -50.6% 303394 interrupts.CPU29.LOC:Local_timer_interrupts 3758 -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts 3758 -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts 1196 +299.1% 4775 ± 2% interrupts.CPU3.CAL:Function_call_interrupts 614825 -50.6% 303781 interrupts.CPU3.LOC:Local_timer_interrupts 3798 -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 3798 -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 1196 +297.3% 4754 ± 2% interrupts.CPU30.CAL:Function_call_interrupts 614821 -50.6% 303648 interrupts.CPU30.LOC:Local_timer_interrupts 5643 ± 33% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts 5643 ± 33% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts 1197 +295.4% 4733 ± 2% interrupts.CPU31.CAL:Function_call_interrupts 614814 -50.7% 303265 interrupts.CPU31.LOC:Local_timer_interrupts 5606 ± 32% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts 5606 ± 32% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts 1197 +296.6% 4750 ± 2% interrupts.CPU32.CAL:Function_call_interrupts 614856 -50.7% 303305 interrupts.CPU32.LOC:Local_timer_interrupts 4678 ± 34% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts 4678 ± 34% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts 1197 +298.3% 4767 ± 2% interrupts.CPU33.CAL:Function_call_interrupts 615099 -50.7% 303410 interrupts.CPU33.LOC:Local_timer_interrupts 4689 ± 34% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts 4689 ± 34% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts 1197 +299.2% 4778 ± 2% interrupts.CPU34.CAL:Function_call_interrupts 614606 -50.6% 303804 interrupts.CPU34.LOC:Local_timer_interrupts 5658 ± 33% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts 5658 ± 33% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts 455.25 ± 22% -48.7% 233.50 ± 43% interrupts.CPU34.RES:Rescheduling_interrupts 1197 +297.8% 4762 ± 2% interrupts.CPU35.CAL:Function_call_interrupts 614926 -50.7% 303407 interrupts.CPU35.LOC:Local_timer_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts 397.25 ± 11% -66.8% 132.00 ± 10% interrupts.CPU35.RES:Rescheduling_interrupts 1197 +296.4% 4747 ± 3% interrupts.CPU36.CAL:Function_call_interrupts 614780 -50.7% 303268 interrupts.CPU36.LOC:Local_timer_interrupts 5600 ± 32% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts 5600 ± 32% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts 1196 +295.6% 4732 ± 2% interrupts.CPU37.CAL:Function_call_interrupts 615083 -50.7% 303252 interrupts.CPU37.LOC:Local_timer_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts 5617 ± 33% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts 1197 +297.0% 4753 ± 3% interrupts.CPU38.CAL:Function_call_interrupts 614557 -50.6% 303853 interrupts.CPU38.LOC:Local_timer_interrupts 5686 ± 32% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts 5686 ± 32% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts 1197 +295.8% 4740 ± 2% interrupts.CPU39.CAL:Function_call_interrupts 614810 -50.7% 303301 interrupts.CPU39.LOC:Local_timer_interrupts 5675 ± 33% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts 5675 ± 33% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts 1196 +296.8% 4747 ± 2% interrupts.CPU4.CAL:Function_call_interrupts 615330 -50.7% 303220 interrupts.CPU4.LOC:Local_timer_interrupts 3807 -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts 3807 -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts 1197 +298.0% 4767 ± 2% interrupts.CPU40.CAL:Function_call_interrupts 614586 -50.7% 303263 interrupts.CPU40.LOC:Local_timer_interrupts 3729 -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts 3729 -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts 1197 +296.5% 4746 ± 2% interrupts.CPU41.CAL:Function_call_interrupts 614370 -50.6% 303486 interrupts.CPU41.LOC:Local_timer_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts 1197 +297.5% 4761 ± 2% interrupts.CPU42.CAL:Function_call_interrupts 614812 -50.6% 303585 interrupts.CPU42.LOC:Local_timer_interrupts 4673 ± 34% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts 4673 ± 34% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts 1197 +297.1% 4755 ± 2% interrupts.CPU43.CAL:Function_call_interrupts 614937 -50.7% 303344 interrupts.CPU43.LOC:Local_timer_interrupts 4655 ± 34% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts 4655 ± 34% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts 503.25 ± 33% -55.7% 223.00 ± 92% interrupts.CPU43.RES:Rescheduling_interrupts 1197 +296.7% 4751 ± 3% interrupts.CPU44.CAL:Function_call_interrupts 614799 -50.7% 303380 interrupts.CPU44.LOC:Local_timer_interrupts 4646 ± 34% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts 4646 ± 34% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts 1197 +297.5% 4758 ± 2% interrupts.CPU45.CAL:Function_call_interrupts 614754 -50.7% 303096 interrupts.CPU45.LOC:Local_timer_interrupts 4646 ± 34% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts 4646 ± 34% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts 413.50 ± 43% -52.0% 198.50 ± 44% interrupts.CPU45.RES:Rescheduling_interrupts 1196 +297.6% 4758 ± 2% interrupts.CPU46.CAL:Function_call_interrupts 614909 -50.6% 303549 interrupts.CPU46.LOC:Local_timer_interrupts 4647 ± 34% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts 4647 ± 34% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts 405.75 ± 24% -58.2% 169.50 ± 60% interrupts.CPU46.RES:Rescheduling_interrupts 1198 +297.7% 4764 ± 2% interrupts.CPU47.CAL:Function_call_interrupts 614918 -50.7% 303397 interrupts.CPU47.LOC:Local_timer_interrupts 3742 -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts 3742 -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts 1196 +296.4% 4744 ± 2% interrupts.CPU48.CAL:Function_call_interrupts 615077 -50.7% 303237 interrupts.CPU48.LOC:Local_timer_interrupts 3739 -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts 3739 -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts 1197 +297.0% 4753 ± 2% interrupts.CPU49.CAL:Function_call_interrupts 614885 -50.6% 303915 interrupts.CPU49.LOC:Local_timer_interrupts 4660 ± 34% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts 4660 ± 34% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts 1197 +298.2% 4768 ± 2% interrupts.CPU5.CAL:Function_call_interrupts 614813 -50.7% 303385 interrupts.CPU5.LOC:Local_timer_interrupts 3793 -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts 3793 -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts 1196 +297.7% 4759 ± 2% interrupts.CPU50.CAL:Function_call_interrupts 614773 -50.6% 303663 interrupts.CPU50.LOC:Local_timer_interrupts 5573 ± 33% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts 5573 ± 33% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts 1197 +297.2% 4755 ± 2% interrupts.CPU51.CAL:Function_call_interrupts 614816 -50.7% 303177 interrupts.CPU51.LOC:Local_timer_interrupts 5562 ± 33% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts 5562 ± 33% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts 1196 +297.7% 4760 ± 2% interrupts.CPU52.CAL:Function_call_interrupts 615276 -50.6% 304085 interrupts.CPU52.LOC:Local_timer_interrupts 5608 ± 33% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts 5608 ± 33% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts 1197 +297.4% 4759 ± 2% interrupts.CPU53.CAL:Function_call_interrupts 614863 -50.7% 303356 interrupts.CPU53.LOC:Local_timer_interrupts 4663 ± 35% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts 4663 ± 35% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts 1196 +298.0% 4762 ± 2% interrupts.CPU54.CAL:Function_call_interrupts 614853 -50.6% 303452 interrupts.CPU54.LOC:Local_timer_interrupts 4609 ± 33% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts 4609 ± 33% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts 1197 +297.2% 4755 ± 2% interrupts.CPU55.CAL:Function_call_interrupts 615003 -50.7% 303230 interrupts.CPU55.LOC:Local_timer_interrupts 3709 -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts 3709 -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts 1197 +297.4% 4760 ± 2% interrupts.CPU56.CAL:Function_call_interrupts 614775 -50.7% 303238 interrupts.CPU56.LOC:Local_timer_interrupts 4665 ± 34% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 4665 ± 34% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 1197 +298.0% 4765 ± 3% interrupts.CPU57.CAL:Function_call_interrupts 614835 -50.6% 303595 interrupts.CPU57.LOC:Local_timer_interrupts 4685 ± 34% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts 4685 ± 34% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts 1197 +298.4% 4771 ± 2% interrupts.CPU58.CAL:Function_call_interrupts 614792 -50.7% 303266 interrupts.CPU58.LOC:Local_timer_interrupts 6542 ± 25% -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts 6542 ± 25% -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts 1197 +298.1% 4767 ± 2% interrupts.CPU59.CAL:Function_call_interrupts 614386 -50.6% 303328 interrupts.CPU59.LOC:Local_timer_interrupts 5581 ± 33% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts 5581 ± 33% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts 1196 +298.0% 4760 ± 2% interrupts.CPU6.CAL:Function_call_interrupts 614839 -50.6% 303705 interrupts.CPU6.LOC:Local_timer_interrupts 4666 ± 35% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts 4666 ± 35% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts 1196 +297.5% 4755 ± 2% interrupts.CPU60.CAL:Function_call_interrupts 614736 -50.6% 303397 interrupts.CPU60.LOC:Local_timer_interrupts 5577 ± 33% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts 5577 ± 33% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts 1197 +296.5% 4747 ± 2% interrupts.CPU61.CAL:Function_call_interrupts 614780 -50.7% 303308 interrupts.CPU61.LOC:Local_timer_interrupts 4627 ± 34% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts 4627 ± 34% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts 1198 +297.3% 4759 ± 2% interrupts.CPU62.CAL:Function_call_interrupts 614721 -50.6% 303418 interrupts.CPU62.LOC:Local_timer_interrupts 4694 ± 34% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts 4694 ± 34% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts 1198 +297.4% 4761 ± 2% interrupts.CPU63.CAL:Function_call_interrupts 614595 -50.6% 303406 interrupts.CPU63.LOC:Local_timer_interrupts 4686 ± 34% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts 4686 ± 34% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts 1193 +298.8% 4760 ± 2% interrupts.CPU64.CAL:Function_call_interrupts 614738 -50.7% 303289 interrupts.CPU64.LOC:Local_timer_interrupts 5605 ± 33% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts 5605 ± 33% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts 129.75 ± 18% +257.6% 464.00 ± 37% interrupts.CPU64.RES:Rescheduling_interrupts 1197 +296.6% 4748 ± 2% interrupts.CPU65.CAL:Function_call_interrupts 614757 -50.7% 302989 interrupts.CPU65.LOC:Local_timer_interrupts 6535 ± 24% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts 6535 ± 24% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts 187.00 ± 96% +145.6% 459.25 ± 47% interrupts.CPU65.RES:Rescheduling_interrupts 1198 +297.5% 4761 ± 2% interrupts.CPU66.CAL:Function_call_interrupts 614573 -50.6% 303394 interrupts.CPU66.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts 1197 +296.9% 4752 ± 2% interrupts.CPU67.CAL:Function_call_interrupts 615019 -50.7% 303330 interrupts.CPU67.LOC:Local_timer_interrupts 5594 ± 33% -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts 5594 ± 33% -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts 64.25 ± 68% +479.8% 372.50 ± 28% interrupts.CPU67.RES:Rescheduling_interrupts 1196 +299.1% 4775 ± 2% interrupts.CPU68.CAL:Function_call_interrupts 614853 -50.7% 303275 interrupts.CPU68.LOC:Local_timer_interrupts 4671 ± 34% -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts 4671 ± 34% -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts 1197 +297.8% 4762 ± 2% interrupts.CPU69.CAL:Function_call_interrupts 614963 -50.7% 303313 interrupts.CPU69.LOC:Local_timer_interrupts 5620 ± 33% -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts 5620 ± 33% -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts 131.75 ± 81% +182.2% 371.75 ± 53% interrupts.CPU69.RES:Rescheduling_interrupts 1196 +297.8% 4760 ± 2% interrupts.CPU7.CAL:Function_call_interrupts 614942 -50.7% 303142 interrupts.CPU7.LOC:Local_timer_interrupts 4682 ± 34% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts 4682 ± 34% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts 1198 +273.5% 4475 ± 7% interrupts.CPU70.CAL:Function_call_interrupts 614789 -50.6% 303602 interrupts.CPU70.LOC:Local_timer_interrupts 6437 ± 24% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts 6437 ± 24% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts 1197 +296.0% 4740 ± 2% interrupts.CPU71.CAL:Function_call_interrupts 614800 -50.7% 303277 interrupts.CPU71.LOC:Local_timer_interrupts 5475 ± 33% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts 5475 ± 33% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts 1198 +298.4% 4772 ± 2% interrupts.CPU72.CAL:Function_call_interrupts 614895 -50.7% 303270 interrupts.CPU72.LOC:Local_timer_interrupts 6548 ± 24% -100.0% 0.00 interrupts.CPU72.NMI:Non-maskable_interrupts 6548 ± 24% -100.0% 0.00 interrupts.CPU72.PMI:Performance_monitoring_interrupts 469.75 ± 27% -58.2% 196.50 ± 33% interrupts.CPU72.RES:Rescheduling_interrupts 1197 +298.0% 4764 ± 2% interrupts.CPU73.CAL:Function_call_interrupts 614827 -50.7% 303182 interrupts.CPU73.LOC:Local_timer_interrupts 5633 ± 33% -100.0% 0.00 interrupts.CPU73.NMI:Non-maskable_interrupts 5633 ± 33% -100.0% 0.00 interrupts.CPU73.PMI:Performance_monitoring_interrupts 1196 +297.7% 4759 ± 2% interrupts.CPU74.CAL:Function_call_interrupts 614692 -50.6% 303444 interrupts.CPU74.LOC:Local_timer_interrupts 4663 ± 34% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts 4663 ± 34% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts 1197 +298.7% 4772 ± 2% interrupts.CPU75.CAL:Function_call_interrupts 614710 -50.6% 303716 interrupts.CPU75.LOC:Local_timer_interrupts 4661 ± 34% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts 4661 ± 34% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts 1197 +297.2% 4756 ± 2% interrupts.CPU76.CAL:Function_call_interrupts 614799 -50.6% 303424 interrupts.CPU76.LOC:Local_timer_interrupts 5604 ± 33% -100.0% 0.00 interrupts.CPU76.NMI:Non-maskable_interrupts 5604 ± 33% -100.0% 0.00 interrupts.CPU76.PMI:Performance_monitoring_interrupts 721.00 ± 21% +89.2% 1364 ± 33% interrupts.CPU76.RES:Rescheduling_interrupts 1197 +296.5% 4747 ± 2% interrupts.CPU77.CAL:Function_call_interrupts 614710 -50.7% 303313 interrupts.CPU77.LOC:Local_timer_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU77.NMI:Non-maskable_interrupts 5609 ± 33% -100.0% 0.00 interrupts.CPU77.PMI:Performance_monitoring_interrupts 205.00 ± 60% +122.8% 456.75 ± 14% interrupts.CPU77.RES:Rescheduling_interrupts 1197 +297.3% 4758 ± 2% interrupts.CPU78.CAL:Function_call_interrupts 614824 -50.6% 303460 interrupts.CPU78.LOC:Local_timer_interrupts 6540 ± 24% -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts 6540 ± 24% -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts 1197 +297.6% 4760 ± 2% interrupts.CPU79.CAL:Function_call_interrupts 614761 -50.6% 303796 interrupts.CPU79.LOC:Local_timer_interrupts 5614 ± 33% -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts 5614 ± 33% -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts 1199 +295.7% 4746 ± 2% interrupts.CPU8.CAL:Function_call_interrupts 614948 -50.7% 303338 interrupts.CPU8.LOC:Local_timer_interrupts 3792 -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts 3792 -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts 1196 +297.8% 4760 ± 2% interrupts.CPU80.CAL:Function_call_interrupts 614393 -50.6% 303372 interrupts.CPU80.LOC:Local_timer_interrupts 4667 ± 34% -100.0% 0.00 interrupts.CPU80.NMI:Non-maskable_interrupts 4667 ± 34% -100.0% 0.00 interrupts.CPU80.PMI:Performance_monitoring_interrupts 1197 +297.9% 4763 ± 2% interrupts.CPU81.CAL:Function_call_interrupts 614527 -50.7% 303238 interrupts.CPU81.LOC:Local_timer_interrupts 3725 -100.0% 0.00 interrupts.CPU81.NMI:Non-maskable_interrupts 3725 -100.0% 0.00 interrupts.CPU81.PMI:Performance_monitoring_interrupts 1197 +297.6% 4759 ± 2% interrupts.CPU82.CAL:Function_call_interrupts 614837 -50.6% 303523 interrupts.CPU82.LOC:Local_timer_interrupts 4687 ± 34% -100.0% 0.00 interrupts.CPU82.NMI:Non-maskable_interrupts 4687 ± 34% -100.0% 0.00 interrupts.CPU82.PMI:Performance_monitoring_interrupts 442.50 ± 73% +171.0% 1199 ± 62% interrupts.CPU82.RES:Rescheduling_interrupts 1199 +297.3% 4763 ± 2% interrupts.CPU83.CAL:Function_call_interrupts 614764 -50.6% 303652 interrupts.CPU83.LOC:Local_timer_interrupts 5619 ± 33% -100.0% 0.00 interrupts.CPU83.NMI:Non-maskable_interrupts 5619 ± 33% -100.0% 0.00 interrupts.CPU83.PMI:Performance_monitoring_interrupts 1196 +297.0% 4751 ± 2% interrupts.CPU84.CAL:Function_call_interrupts 614913 -50.7% 303390 interrupts.CPU84.LOC:Local_timer_interrupts 6527 ± 24% -100.0% 0.00 interrupts.CPU84.NMI:Non-maskable_interrupts 6527 ± 24% -100.0% 0.00 interrupts.CPU84.PMI:Performance_monitoring_interrupts 1197 +297.1% 4755 ± 2% interrupts.CPU85.CAL:Function_call_interrupts 614783 -50.7% 303151 interrupts.CPU85.LOC:Local_timer_interrupts 5579 ± 33% -100.0% 0.00 interrupts.CPU85.NMI:Non-maskable_interrupts 5579 ± 33% -100.0% 0.00 interrupts.CPU85.PMI:Performance_monitoring_interrupts 133.00 ± 53% +256.0% 473.50 ± 34% interrupts.CPU85.RES:Rescheduling_interrupts 1196 +298.3% 4767 ± 2% interrupts.CPU86.CAL:Function_call_interrupts 614665 -50.6% 303666 interrupts.CPU86.LOC:Local_timer_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU86.NMI:Non-maskable_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU86.PMI:Performance_monitoring_interrupts 1197 +298.4% 4770 ± 2% interrupts.CPU87.CAL:Function_call_interrupts 614821 -50.6% 303804 interrupts.CPU87.LOC:Local_timer_interrupts 5613 ± 33% -100.0% 0.00 interrupts.CPU87.NMI:Non-maskable_interrupts 5613 ± 33% -100.0% 0.00 interrupts.CPU87.PMI:Performance_monitoring_interrupts 1196 +297.1% 4751 ± 2% interrupts.CPU88.CAL:Function_call_interrupts 615063 -50.7% 303252 interrupts.CPU88.LOC:Local_timer_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU88.NMI:Non-maskable_interrupts 4693 ± 34% -100.0% 0.00 interrupts.CPU88.PMI:Performance_monitoring_interrupts 1196 +297.7% 4759 ± 2% interrupts.CPU89.CAL:Function_call_interrupts 614906 -50.7% 303298 interrupts.CPU89.LOC:Local_timer_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU89.NMI:Non-maskable_interrupts 4677 ± 34% -100.0% 0.00 interrupts.CPU89.PMI:Performance_monitoring_interrupts 1198 +297.8% 4767 ± 2% interrupts.CPU9.CAL:Function_call_interrupts 614788 -50.6% 303675 interrupts.CPU9.LOC:Local_timer_interrupts 3797 -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 3797 -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 1196 +299.2% 4776 ± 2% interrupts.CPU90.CAL:Function_call_interrupts 614868 -50.7% 303262 interrupts.CPU90.LOC:Local_timer_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU90.NMI:Non-maskable_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU90.PMI:Performance_monitoring_interrupts 1197 +298.5% 4769 ± 2% interrupts.CPU91.CAL:Function_call_interrupts 614800 -50.6% 303694 interrupts.CPU91.LOC:Local_timer_interrupts 5621 ± 33% -100.0% 0.00 interrupts.CPU91.NMI:Non-maskable_interrupts 5621 ± 33% -100.0% 0.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts 1198 +296.0% 4745 ± 2% interrupts.CPU92.CAL:Function_call_interrupts 614593 -50.6% 303305 interrupts.CPU92.LOC:Local_timer_interrupts 5607 ± 33% -100.0% 0.00 interrupts.CPU92.NMI:Non-maskable_interrupts 5607 ± 33% -100.0% 0.00 interrupts.CPU92.PMI:Performance_monitoring_interrupts 1197 +297.3% 4756 ± 2% interrupts.CPU93.CAL:Function_call_interrupts 614683 -50.7% 303239 interrupts.CPU93.LOC:Local_timer_interrupts 4662 ± 34% -100.0% 0.00 interrupts.CPU93.NMI:Non-maskable_interrupts 4662 ± 34% -100.0% 0.00 interrupts.CPU93.PMI:Performance_monitoring_interrupts 1197 +299.0% 4777 ± 2% interrupts.CPU94.CAL:Function_call_interrupts 614864 -50.6% 303523 interrupts.CPU94.LOC:Local_timer_interrupts 4643 ± 34% -100.0% 0.00 interrupts.CPU94.NMI:Non-maskable_interrupts 4643 ± 34% -100.0% 0.00 interrupts.CPU94.PMI:Performance_monitoring_interrupts 1197 +298.1% 4765 ± 2% interrupts.CPU95.CAL:Function_call_interrupts 614701 -50.6% 303684 interrupts.CPU95.LOC:Local_timer_interrupts 4659 ± 34% -100.0% 0.00 interrupts.CPU95.NMI:Non-maskable_interrupts 4659 ± 34% -100.0% 0.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts 1196 +298.2% 4765 ± 2% interrupts.CPU96.CAL:Function_call_interrupts 614842 -50.6% 303508 interrupts.CPU96.LOC:Local_timer_interrupts 4651 ± 34% -100.0% 0.00 interrupts.CPU96.NMI:Non-maskable_interrupts 4651 ± 34% -100.0% 0.00 interrupts.CPU96.PMI:Performance_monitoring_interrupts 994.75 ± 50% +90.0% 1890 ± 18% interrupts.CPU96.RES:Rescheduling_interrupts 1197 +297.3% 4757 ± 2% interrupts.CPU97.CAL:Function_call_interrupts 614862 -50.7% 303277 interrupts.CPU97.LOC:Local_timer_interrupts 3717 -100.0% 0.00 interrupts.CPU97.NMI:Non-maskable_interrupts 3717 -100.0% 0.00 interrupts.CPU97.PMI:Performance_monitoring_interrupts 1197 +298.7% 4774 ± 2% interrupts.CPU98.CAL:Function_call_interrupts 614436 -50.6% 303236 interrupts.CPU98.LOC:Local_timer_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU98.NMI:Non-maskable_interrupts 4683 ± 34% -100.0% 0.00 interrupts.CPU98.PMI:Performance_monitoring_interrupts 1197 +298.0% 4765 ± 2% interrupts.CPU99.CAL:Function_call_interrupts 614729 -50.6% 303833 interrupts.CPU99.LOC:Local_timer_interrupts 4691 ± 34% -100.0% 0.00 interrupts.CPU99.NMI:Non-maskable_interrupts 4691 ± 34% -100.0% 0.00 interrupts.CPU99.PMI:Performance_monitoring_interrupts 96.75 ± 8% -100.0% 0.00 interrupts.IWI:IRQ_work_interrupts 1.771e+08 -50.6% 87381059 interrupts.LOC:Local_timer_interrupts 1410954 ± 2% -100.0% 0.50 ±100% interrupts.NMI:Non-maskable_interrupts 1410954 ± 2% -100.0% 0.50 ±100% interrupts.PMI:Performance_monitoring_interrupts 102963 ± 3% +32.1% 136019 ± 4% interrupts.RES:Rescheduling_interrupts 1481 ± 11% -58.3% 618.00 ± 17% interrupts.TLB:TLB_shootdowns *************************************************************************************************** lkp-hsw-ep4: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_job/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/1500/100%/debian-x86_64-20191114.cgz/300s/lkp-hsw-ep4/five_sec/reaim/0x43 commit: 1d2cc5ac6f (" RISC-V Fixes for 5.7-rc4") f2af7d90e2 ("pipe: read/write_iter() handler should check for IOCB_NOWAIT") 1d2cc5ac6f6668cc f2af7d90e25ce1cf03318bf3290 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 50% 1:4 kmsg.blk_update_request:critical_medium_error,dev_nvme#n1,sector#op#:(READ)flags#phys_seg#prio_class 2:2 -100% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 2:2 -100% :4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 2372 -4.7% 2261 reaim.child_systime 164.68 -1.2% 162.70 reaim.child_utime 175971 +4.6% 183978 reaim.jobs_per_min 2444 +4.6% 2555 reaim.jobs_per_min_child 176821 +4.5% 184702 reaim.max_jobs_per_min 36.73 -4.4% 35.13 reaim.parent_time 0.07 ± 4% -4.3% 0.07 ± 4% reaim.std_dev_time 310.12 +7.8% 334.41 reaim.time.elapsed_time 310.12 +7.8% 334.41 reaim.time.elapsed_time.max 723613 +6.2% 768160 reaim.time.involuntary_context_switches 3050 +2.3% 3118 reaim.time.maximum_resident_set_size 2.598e+08 +12.3% 2.918e+08 reaim.time.minor_page_faults 18976 +7.2% 20350 reaim.time.system_time 1317 +11.1% 1464 reaim.time.user_time 3336290 +13.0% 3771271 reaim.time.voluntary_context_switches 864000 +12.5% 972000 reaim.workload 0.00 ± 17% -0.0 0.00 ± 35% mpstat.cpu.all.soft% 40.94 ± 8% -15.3% 34.69 ± 10% boot-time.boot 2579 ± 10% -14.4% 2209 ± 11% boot-time.idle 26827 +3.2% 27688 vmstat.system.cs 147298 -53.0% 69198 vmstat.system.in 252479 ± 2% -27.1% 183999 cpuidle.C1.usage 2.013e+09 +12.5% 2.266e+09 cpuidle.C1E.time 6792951 -41.0% 4008898 cpuidle.C1E.usage 230414 ± 6% -55.2% 103251 ± 4% cpuidle.POLL.time 29806 ± 4% -40.7% 17672 ± 7% cpuidle.POLL.usage 1.241e+08 +12.3% 1.393e+08 numa-numastat.node0.local_node 1.241e+08 +12.3% 1.393e+08 numa-numastat.node0.numa_hit 1.245e+08 +12.4% 1.4e+08 numa-numastat.node1.local_node 1.245e+08 +12.4% 1.4e+08 numa-numastat.node1.numa_hit 23323 -99.6% 100.33 ± 26% numa-numastat.node1.other_node 318729 -11.3% 282661 meminfo.Active 318485 -11.3% 282433 meminfo.Active(anon) 434904 -11.5% 384796 meminfo.Committed_AS 78897 -9.4% 71502 meminfo.KReclaimable 29773 -31.9% 20263 meminfo.Mapped 78897 -9.4% 71502 meminfo.SReclaimable 60102 ± 3% -65.5% 20752 meminfo.Shmem 8117 ± 2% -18.3% 6635 meminfo.max_used_kB 93397 ± 3% -6.3% 87470 ± 2% numa-meminfo.node0.SUnreclaim 133032 ± 4% -7.5% 123037 ± 3% numa-meminfo.node0.Slab 584979 -8.6% 534408 numa-meminfo.node1.FilePages 39245 ± 8% -8.4% 35954 ± 9% numa-meminfo.node1.KReclaimable 15276 ± 17% -39.1% 9302 ± 24% numa-meminfo.node1.Mapped 1260488 ± 6% -7.1% 1171110 ± 3% numa-meminfo.node1.MemUsed 39245 ± 8% -8.4% 35954 ± 9% numa-meminfo.node1.SReclaimable 90737 ± 3% -11.1% 80696 ± 3% numa-meminfo.node1.SUnreclaim 49662 -84.3% 7799 ± 78% numa-meminfo.node1.Shmem 129983 ± 4% -10.3% 116651 ± 4% numa-meminfo.node1.Slab 23322 ± 3% -6.3% 21857 ± 2% numa-vmstat.node0.nr_slab_unreclaimable 62206867 +12.0% 69679807 numa-vmstat.node0.numa_hit 62068210 +12.1% 69587119 numa-vmstat.node0.numa_local 146247 -8.6% 133600 numa-vmstat.node1.nr_file_pages 3933 ± 18% -40.9% 2326 ± 24% numa-vmstat.node1.nr_mapped 12418 -84.3% 1948 ± 78% numa-vmstat.node1.nr_shmem 9809 ± 8% -8.4% 8986 ± 9% numa-vmstat.node1.nr_slab_reclaimable 22651 ± 3% -11.0% 20166 ± 3% numa-vmstat.node1.nr_slab_unreclaimable 62615263 +12.2% 70229344 numa-vmstat.node1.numa_hit 62564575 +12.1% 70155936 numa-vmstat.node1.numa_local 79587 -11.3% 70594 proc-vmstat.nr_active_anon 280484 -4.2% 268607 proc-vmstat.nr_file_pages 4767 -7.4% 4412 proc-vmstat.nr_inactive_anon 7604 -33.4% 5063 proc-vmstat.nr_mapped 15015 ± 3% -65.5% 5187 proc-vmstat.nr_shmem 19725 -9.4% 17877 proc-vmstat.nr_slab_reclaimable 46089 -8.7% 42072 proc-vmstat.nr_slab_unreclaimable 79587 -11.3% 70594 proc-vmstat.nr_zone_active_anon 4767 -7.4% 4412 proc-vmstat.nr_zone_inactive_anon 2.481e+08 +12.6% 2.794e+08 proc-vmstat.numa_hit 2.481e+08 +12.6% 2.794e+08 proc-vmstat.numa_local 23432 -99.1% 200.00 ± 11% proc-vmstat.numa_other 57733 -16.7% 48099 proc-vmstat.pgactivate 2.549e+08 +12.2% 2.861e+08 proc-vmstat.pgalloc_normal 2.608e+08 +12.2% 2.926e+08 proc-vmstat.pgfault 2.548e+08 +12.3% 2.861e+08 proc-vmstat.pgfree 90.12 -90.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 90.09 -90.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 86.41 -86.4 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_getdents.do_syscall_64.entry_SYSCALL_64_after_hwframe 86.40 -86.4 0.00 perf-profile.calltrace.cycles-pp.iterate_dir.__x64_sys_getdents.do_syscall_64.entry_SYSCALL_64_after_hwframe 86.38 -86.4 0.00 perf-profile.calltrace.cycles-pp.dcache_readdir.iterate_dir.__x64_sys_getdents.do_syscall_64.entry_SYSCALL_64_after_hwframe 84.93 -84.9 0.00 perf-profile.calltrace.cycles-pp.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents.do_syscall_64 84.28 -84.3 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents 83.45 -83.5 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir 93.22 -93.2 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 93.18 -93.2 0.00 perf-profile.children.cycles-pp.do_syscall_64 86.48 -86.5 0.00 perf-profile.children.cycles-pp._raw_spin_lock 86.41 -86.4 0.00 perf-profile.children.cycles-pp.__x64_sys_getdents 86.41 -86.4 0.00 perf-profile.children.cycles-pp.iterate_dir 86.38 -86.4 0.00 perf-profile.children.cycles-pp.dcache_readdir 85.58 -85.6 0.00 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 84.94 -84.9 0.00 perf-profile.children.cycles-pp.scan_positives 85.19 -85.2 0.00 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 2328 -100.0% 0.00 sched_debug.cfs_rq:/.load.min 37.90 ± 4% -15.8% 31.92 ± 19% sched_debug.cfs_rq:/.load_avg.avg 869.75 -32.5% 586.94 ± 12% sched_debug.cfs_rq:/.load_avg.max 126.64 -30.8% 87.57 ± 16% sched_debug.cfs_rq:/.load_avg.stddev 42489 ± 5% -14.1% 36511 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev 0.17 -100.0% 0.00 sched_debug.cfs_rq:/.nr_running.min 764.42 ± 11% -41.4% 448.06 ± 17% sched_debug.cfs_rq:/.removed.load_avg.max 106.78 -35.1% 69.26 ± 30% sched_debug.cfs_rq:/.removed.load_avg.stddev 764.42 ± 11% -41.4% 448.06 ± 17% sched_debug.cfs_rq:/.removed.runnable_avg.max 106.78 -35.1% 69.26 ± 30% sched_debug.cfs_rq:/.removed.runnable_avg.stddev 442.83 ± 23% -25.9% 328.33 ± 2% sched_debug.cfs_rq:/.runnable_avg.min 42496 ± 5% -14.1% 36496 ± 7% sched_debug.cfs_rq:/.spread0.stddev 926.08 ± 10% +28.6% 1190 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.max 0.17 -100.0% 0.00 sched_debug.cfs_rq:/.util_est_enqueued.min 245.68 ± 4% +16.4% 286.07 ± 5% sched_debug.cfs_rq:/.util_est_enqueued.stddev 1.83 +15.2% 2.11 ± 3% sched_debug.cpu.nr_running.max 0.00 ±100% +5100.0% 0.06 ± 65% sched_debug.cpu.nr_uninterruptible.avg 49.00 ± 10% -30.7% 33.94 ± 40% sched_debug.cpu.nr_uninterruptible.max 3.30 -100.0% 0.00 sched_debug.rt_rq:/.rt_runtime.stddev 1569 ± 2% -16.3% 1314 ± 7% slabinfo.Acpi-Parse.active_objs 1569 ± 2% -16.3% 1314 ± 7% slabinfo.Acpi-Parse.num_objs 652.50 ± 4% -14.2% 559.67 ± 13% slabinfo.UDP.active_objs 652.50 ± 4% -14.2% 559.67 ± 13% slabinfo.UDP.num_objs 778.50 ± 9% -37.3% 488.00 ± 4% slabinfo.file_lock_cache.active_objs 778.50 ± 9% -37.3% 488.00 ± 4% slabinfo.file_lock_cache.num_objs 1856 ± 3% -17.2% 1536 ± 6% slabinfo.fsnotify_mark_connector.active_objs 1856 ± 3% -17.2% 1536 ± 6% slabinfo.fsnotify_mark_connector.num_objs 593.00 ± 3% -53.5% 276.00 ± 12% slabinfo.khugepaged_mm_slot.active_objs 593.00 ± 3% -53.5% 276.00 ± 12% slabinfo.khugepaged_mm_slot.num_objs 5114 ± 2% -24.0% 3885 slabinfo.kmalloc-2k.active_objs 5146 ± 2% -24.1% 3906 slabinfo.kmalloc-2k.num_objs 1285 -9.3% 1165 slabinfo.kmalloc-4k.active_objs 1307 -10.8% 1165 slabinfo.kmalloc-4k.num_objs 378.00 ± 14% -20.6% 300.00 ± 14% slabinfo.kmem_cache.active_objs 378.00 ± 14% -20.6% 300.00 ± 14% slabinfo.kmem_cache.num_objs 749.00 ± 12% -18.5% 610.33 ± 13% slabinfo.kmem_cache_node.active_objs 800.00 ± 12% -17.3% 661.33 ± 12% slabinfo.kmem_cache_node.num_objs 11474 -34.2% 7553 slabinfo.proc_inode_cache.active_objs 11478 -34.1% 7565 slabinfo.proc_inode_cache.num_objs 26853 -66.7% 8936 slabinfo.shmem_inode_cache.active_objs 586.00 -66.9% 194.00 slabinfo.shmem_inode_cache.active_slabs 26988 -66.9% 8936 slabinfo.shmem_inode_cache.num_objs 586.00 -66.9% 194.00 slabinfo.shmem_inode_cache.num_slabs 112027 +16.2% 130205 softirqs.CPU0.TIMER 16688 ± 10% -12.9% 14532 ± 2% softirqs.CPU1.SCHED 136401 ± 18% -19.5% 109770 softirqs.CPU10.TIMER 14520 ± 2% -10.6% 12985 ± 3% softirqs.CPU11.SCHED 14458 ± 2% -9.0% 13163 softirqs.CPU13.SCHED 14474 -10.4% 12965 softirqs.CPU14.SCHED 14892 ± 2% -11.6% 13168 softirqs.CPU15.SCHED 14641 ± 3% -10.8% 13064 softirqs.CPU16.SCHED 14626 ± 3% -10.0% 13166 softirqs.CPU18.SCHED 15634 ± 7% -7.9% 14404 ± 7% softirqs.CPU19.SCHED 14469 ± 8% -11.0% 12877 softirqs.CPU20.SCHED 14543 -7.6% 13439 ± 3% softirqs.CPU23.SCHED 14316 ± 2% -10.1% 12866 softirqs.CPU25.SCHED 14274 ± 4% -9.2% 12963 ± 3% softirqs.CPU26.SCHED 14487 -8.1% 13318 ± 2% softirqs.CPU28.SCHED 14427 -9.4% 13074 ± 2% softirqs.CPU30.SCHED 14459 -10.4% 12951 ± 2% softirqs.CPU32.SCHED 14177 -10.1% 12748 softirqs.CPU33.SCHED 14769 -10.1% 13284 ± 2% softirqs.CPU34.SCHED 14370 ± 2% -7.9% 13230 softirqs.CPU35.SCHED 61243 ± 2% -8.2% 56243 softirqs.CPU36.RCU 14862 -12.2% 13049 softirqs.CPU36.SCHED 14186 -8.8% 12940 ± 2% softirqs.CPU38.SCHED 14661 ± 4% -12.0% 12900 softirqs.CPU39.SCHED 14320 ± 4% -8.7% 13074 softirqs.CPU41.SCHED 14603 -10.9% 13017 softirqs.CPU43.SCHED 14641 ± 5% -10.2% 13147 ± 2% softirqs.CPU44.SCHED 14692 ± 2% -10.4% 13160 softirqs.CPU46.SCHED 14602 -10.0% 13139 softirqs.CPU48.SCHED 14370 -10.6% 12852 ± 2% softirqs.CPU49.SCHED 14538 -9.3% 13186 softirqs.CPU5.SCHED 14423 -9.9% 12998 ± 2% softirqs.CPU50.SCHED 14622 ± 2% -11.1% 13005 ± 2% softirqs.CPU51.SCHED 14717 ± 2% -9.7% 13288 softirqs.CPU52.SCHED 14383 ± 2% -7.8% 13260 softirqs.CPU54.SCHED 14433 ± 2% -7.1% 13413 softirqs.CPU57.SCHED 14159 -6.6% 13231 ± 2% softirqs.CPU59.SCHED 14460 -10.5% 12945 ± 4% softirqs.CPU60.SCHED 14290 -9.4% 12943 softirqs.CPU61.SCHED 14418 ± 2% -11.0% 12831 ± 2% softirqs.CPU62.SCHED 14077 ± 3% -9.7% 12716 ± 3% softirqs.CPU63.SCHED 14967 -12.0% 13168 ± 2% softirqs.CPU64.SCHED 14520 ± 4% -9.8% 13091 softirqs.CPU65.SCHED 14387 ± 2% -8.1% 13219 softirqs.CPU66.SCHED 14190 ± 3% -5.5% 13404 ± 6% softirqs.CPU67.SCHED 14075 ± 3% -6.1% 13221 softirqs.CPU68.SCHED 14545 ± 3% -8.1% 13366 ± 2% softirqs.CPU69.SCHED 15029 ± 3% -12.8% 13101 ± 2% softirqs.CPU70.SCHED 14495 -10.1% 13031 ± 2% softirqs.CPU71.SCHED 14791 -11.7% 13067 ± 2% softirqs.CPU9.SCHED 17572 ± 17% -27.4% 12758 softirqs.NET_RX 1047297 -8.5% 957909 softirqs.SCHED 163.50 +21.3% 198.33 ± 9% interrupts.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3 2097 ± 91% -91.6% 177.00 ± 8% interrupts.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7 169.00 ± 5% +47.3% 249.00 ± 14% interrupts.48:IR-PCI-MSI.1572873-edge.eth0-TxRx-9 512.50 ± 66% -67.4% 167.00 interrupts.53:IR-PCI-MSI.1572878-edge.eth0-TxRx-14 96400 +23.2% 118795 ± 6% interrupts.CAL:Function_call_interrupts 1171 ± 6% +46.2% 1712 ± 2% interrupts.CPU0.CAL:Function_call_interrupts 623255 -49.2% 316890 interrupts.CPU0.LOC:Local_timer_interrupts 4609 ± 32% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts 4609 ± 32% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts 1340 ± 3% +25.6% 1683 ± 7% interrupts.CPU1.CAL:Function_call_interrupts 623227 -49.3% 316149 interrupts.CPU1.LOC:Local_timer_interrupts 4609 ± 33% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts 4609 ± 33% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts 5558 +12.9% 6277 interrupts.CPU1.RES:Rescheduling_interrupts 1332 ± 4% +23.4% 1644 ± 5% interrupts.CPU10.CAL:Function_call_interrupts 623262 -49.2% 316807 interrupts.CPU10.LOC:Local_timer_interrupts 4565 ± 33% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts 4565 ± 33% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts 1357 ± 5% +19.3% 1619 ± 6% interrupts.CPU11.CAL:Function_call_interrupts 623237 -49.2% 316340 interrupts.CPU11.LOC:Local_timer_interrupts 4310 ± 29% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts 4310 ± 29% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts 623194 -49.2% 316806 interrupts.CPU12.LOC:Local_timer_interrupts 4589 ± 33% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts 4589 ± 33% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts 1360 ± 5% +19.4% 1624 ± 7% interrupts.CPU13.CAL:Function_call_interrupts 623188 -49.3% 316180 interrupts.CPU13.LOC:Local_timer_interrupts 3060 -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts 3060 -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts 512.50 ± 66% -67.4% 167.00 interrupts.CPU14.53:IR-PCI-MSI.1572878-edge.eth0-TxRx-14 623231 -49.3% 316280 interrupts.CPU14.LOC:Local_timer_interrupts 4068 ± 24% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts 4068 ± 24% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts 1393 +18.1% 1645 ± 6% interrupts.CPU15.CAL:Function_call_interrupts 623165 -49.2% 316552 interrupts.CPU15.LOC:Local_timer_interrupts 4681 ± 7% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts 4681 ± 7% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts 1378 ± 2% +17.7% 1623 ± 7% interrupts.CPU16.CAL:Function_call_interrupts 623345 -49.3% 316185 interrupts.CPU16.LOC:Local_timer_interrupts 4027 ± 23% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts 4027 ± 23% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts 1361 +20.3% 1637 ± 7% interrupts.CPU17.CAL:Function_call_interrupts 623576 -49.3% 316353 interrupts.CPU17.LOC:Local_timer_interrupts 3130 ± 2% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts 3130 ± 2% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts 1352 ± 3% +22.1% 1650 ± 7% interrupts.CPU18.CAL:Function_call_interrupts 622825 -49.2% 316239 interrupts.CPU18.LOC:Local_timer_interrupts 4598 ± 32% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts 4598 ± 32% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts 1318 +27.1% 1675 ± 4% interrupts.CPU19.CAL:Function_call_interrupts 623146 -49.1% 316925 interrupts.CPU19.LOC:Local_timer_interrupts 3983 ± 23% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts 3983 ± 23% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts 1339 ± 4% +23.6% 1655 ± 5% interrupts.CPU2.CAL:Function_call_interrupts 622896 -49.3% 316097 interrupts.CPU2.LOC:Local_timer_interrupts 4619 ± 33% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 4619 ± 33% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 1314 +26.9% 1668 ± 7% interrupts.CPU20.CAL:Function_call_interrupts 622929 -49.2% 316738 interrupts.CPU20.LOC:Local_timer_interrupts 5511 ± 7% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts 5511 ± 7% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts 1245 ± 4% +32.2% 1646 ± 6% interrupts.CPU21.CAL:Function_call_interrupts 623187 -49.2% 316287 interrupts.CPU21.LOC:Local_timer_interrupts 5635 ± 5% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts 5635 ± 5% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts 961.50 +12.3% 1079 ± 3% interrupts.CPU21.TLB:TLB_shootdowns 1345 ± 3% +21.9% 1639 ± 7% interrupts.CPU22.CAL:Function_call_interrupts 623174 -49.3% 316238 interrupts.CPU22.LOC:Local_timer_interrupts 6058 -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts 6058 -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts 1342 ± 3% +22.1% 1639 ± 5% interrupts.CPU23.CAL:Function_call_interrupts 623178 -49.3% 316213 interrupts.CPU23.LOC:Local_timer_interrupts 4129 ± 25% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts 4129 ± 25% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts 1323 +24.5% 1647 ± 5% interrupts.CPU24.CAL:Function_call_interrupts 623073 -49.3% 316136 interrupts.CPU24.LOC:Local_timer_interrupts 6117 -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts 6117 -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts 1327 +24.9% 1657 ± 8% interrupts.CPU25.CAL:Function_call_interrupts 623115 -49.1% 316877 interrupts.CPU25.LOC:Local_timer_interrupts 4101 ± 25% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts 4101 ± 25% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts 1331 +22.4% 1630 ± 10% interrupts.CPU26.CAL:Function_call_interrupts 623150 -49.3% 316068 interrupts.CPU26.LOC:Local_timer_interrupts 4559 ± 32% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts 4559 ± 32% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts 1290 +32.2% 1706 ± 7% interrupts.CPU27.CAL:Function_call_interrupts 623166 -49.2% 316271 interrupts.CPU27.LOC:Local_timer_interrupts 3967 ± 22% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts 3967 ± 22% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts 943.00 ± 2% +20.7% 1138 ± 4% interrupts.CPU27.TLB:TLB_shootdowns 1429 ± 2% +17.7% 1682 ± 6% interrupts.CPU28.CAL:Function_call_interrupts 623166 -49.2% 316485 interrupts.CPU28.LOC:Local_timer_interrupts 4375 ± 28% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts 4375 ± 28% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts 1304 ± 2% +25.0% 1630 ± 3% interrupts.CPU29.CAL:Function_call_interrupts 623163 -49.1% 316954 interrupts.CPU29.LOC:Local_timer_interrupts 4619 ± 33% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts 4619 ± 33% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts 163.50 +21.3% 198.33 ± 9% interrupts.CPU3.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3 1365 +23.5% 1686 ± 4% interrupts.CPU3.CAL:Function_call_interrupts 623222 -49.2% 316774 interrupts.CPU3.LOC:Local_timer_interrupts 6046 -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 6046 -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 1316 +22.6% 1613 ± 8% interrupts.CPU30.CAL:Function_call_interrupts 623162 -49.3% 316122 interrupts.CPU30.LOC:Local_timer_interrupts 4574 ± 32% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts 4574 ± 32% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts 1356 +20.9% 1639 ± 6% interrupts.CPU31.CAL:Function_call_interrupts 623168 -49.3% 316256 interrupts.CPU31.LOC:Local_timer_interrupts 6066 -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts 6066 -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts 1292 ± 10% +26.8% 1638 ± 5% interrupts.CPU32.CAL:Function_call_interrupts 623135 -49.2% 316283 interrupts.CPU32.LOC:Local_timer_interrupts 3072 -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts 3072 -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts 1333 ± 3% +21.9% 1625 ± 6% interrupts.CPU33.CAL:Function_call_interrupts 623147 -49.2% 316447 interrupts.CPU33.LOC:Local_timer_interrupts 3030 -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts 3030 -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts 1355 +21.0% 1639 ± 6% interrupts.CPU34.CAL:Function_call_interrupts 623104 -49.2% 316387 interrupts.CPU34.LOC:Local_timer_interrupts 4570 ± 33% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts 4570 ± 33% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts 1315 +26.0% 1657 ± 7% interrupts.CPU35.CAL:Function_call_interrupts 623175 -49.2% 316826 interrupts.CPU35.LOC:Local_timer_interrupts 5861 ± 4% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts 5861 ± 4% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts 969.00 +11.4% 1079 ± 3% interrupts.CPU35.TLB:TLB_shootdowns 1361 ± 2% +20.5% 1640 ± 6% interrupts.CPU36.CAL:Function_call_interrupts 622836 -49.2% 316285 interrupts.CPU36.LOC:Local_timer_interrupts 3065 -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts 3065 -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts 1352 ± 2% +24.9% 1688 ± 7% interrupts.CPU37.CAL:Function_call_interrupts 623217 -49.2% 316705 interrupts.CPU37.LOC:Local_timer_interrupts 3074 -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts 3074 -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts 1329 +22.7% 1631 ± 5% interrupts.CPU38.CAL:Function_call_interrupts 623237 -49.1% 317057 interrupts.CPU38.LOC:Local_timer_interrupts 6047 -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts 6047 -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts 1369 +19.6% 1637 ± 4% interrupts.CPU39.CAL:Function_call_interrupts 623231 -49.2% 316805 interrupts.CPU39.LOC:Local_timer_interrupts 4512 ± 32% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts 4512 ± 32% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts 1349 +24.2% 1676 ± 4% interrupts.CPU4.CAL:Function_call_interrupts 623071 -49.2% 316429 interrupts.CPU4.LOC:Local_timer_interrupts 3090 -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts 3090 -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts 990.00 +11.3% 1101 ± 6% interrupts.CPU4.TLB:TLB_shootdowns 1307 ± 3% +27.4% 1665 ± 8% interrupts.CPU40.CAL:Function_call_interrupts 623146 -49.2% 316812 interrupts.CPU40.LOC:Local_timer_interrupts 3156 -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts 3156 -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts 1354 +21.9% 1651 ± 4% interrupts.CPU41.CAL:Function_call_interrupts 623207 -49.2% 316298 interrupts.CPU41.LOC:Local_timer_interrupts 4601 ± 33% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts 4601 ± 33% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts 1321 +25.0% 1651 ± 7% interrupts.CPU42.CAL:Function_call_interrupts 623206 -49.1% 317089 interrupts.CPU42.LOC:Local_timer_interrupts 3059 -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts 3059 -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts 972.50 +10.4% 1074 ± 3% interrupts.CPU42.TLB:TLB_shootdowns 1325 ± 4% +22.2% 1619 ± 9% interrupts.CPU43.CAL:Function_call_interrupts 623216 -49.1% 316953 interrupts.CPU43.LOC:Local_timer_interrupts 4569 ± 32% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts 4569 ± 32% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts 1365 ± 2% +19.4% 1630 ± 8% interrupts.CPU44.CAL:Function_call_interrupts 623217 -49.1% 317012 interrupts.CPU44.LOC:Local_timer_interrupts 3059 -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts 3059 -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts 1305 +27.6% 1665 ± 10% interrupts.CPU45.CAL:Function_call_interrupts 623072 -49.1% 316939 interrupts.CPU45.LOC:Local_timer_interrupts 3088 -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts 3088 -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts 949.50 +15.7% 1099 ± 7% interrupts.CPU45.TLB:TLB_shootdowns 1396 ± 2% +18.7% 1658 ± 7% interrupts.CPU46.CAL:Function_call_interrupts 623240 -49.2% 316441 interrupts.CPU46.LOC:Local_timer_interrupts 3042 -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts 3042 -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts 1308 ± 2% +27.5% 1668 ± 4% interrupts.CPU47.CAL:Function_call_interrupts 623216 -49.2% 316580 interrupts.CPU47.LOC:Local_timer_interrupts 4567 ± 33% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts 4567 ± 33% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts 1337 ± 2% +26.7% 1693 ± 5% interrupts.CPU48.CAL:Function_call_interrupts 623183 -49.3% 316244 interrupts.CPU48.LOC:Local_timer_interrupts 3068 -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts 3068 -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts 623164 -49.3% 316116 interrupts.CPU49.LOC:Local_timer_interrupts 6113 -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts 6113 -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts 1343 +22.9% 1651 ± 5% interrupts.CPU5.CAL:Function_call_interrupts 623219 -49.1% 317074 interrupts.CPU5.LOC:Local_timer_interrupts 3199 ± 4% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts 3199 ± 4% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts 1355 +20.1% 1627 ± 6% interrupts.CPU50.CAL:Function_call_interrupts 623205 -49.2% 316463 interrupts.CPU50.LOC:Local_timer_interrupts 6120 -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts 6120 -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts 1375 +20.9% 1663 ± 8% interrupts.CPU51.CAL:Function_call_interrupts 623203 -49.3% 316166 interrupts.CPU51.LOC:Local_timer_interrupts 6118 -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts 6118 -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts 1363 ± 4% +19.8% 1633 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 623222 -49.2% 316461 interrupts.CPU52.LOC:Local_timer_interrupts 6115 -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts 6115 -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts 1308 +28.2% 1677 ± 3% interrupts.CPU53.CAL:Function_call_interrupts 623219 -49.2% 316436 interrupts.CPU53.LOC:Local_timer_interrupts 4571 ± 33% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts 4571 ± 33% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts 943.50 +17.8% 1111 ± 2% interrupts.CPU53.TLB:TLB_shootdowns 1314 +26.0% 1655 ± 3% interrupts.CPU54.CAL:Function_call_interrupts 623005 -49.2% 316348 interrupts.CPU54.LOC:Local_timer_interrupts 6078 -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts 6078 -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts 952.50 +13.0% 1076 ± 3% interrupts.CPU54.TLB:TLB_shootdowns 1299 ± 2% +28.5% 1668 ± 5% interrupts.CPU55.CAL:Function_call_interrupts 623167 -49.3% 316132 interrupts.CPU55.LOC:Local_timer_interrupts 6068 -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts 6068 -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts 950.50 ± 2% +15.9% 1101 ± 2% interrupts.CPU55.TLB:TLB_shootdowns 1339 ± 6% +21.2% 1622 ± 5% interrupts.CPU56.CAL:Function_call_interrupts 623020 -49.2% 316378 interrupts.CPU56.LOC:Local_timer_interrupts 6014 -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 6014 -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 1322 +21.6% 1608 ± 4% interrupts.CPU57.CAL:Function_call_interrupts 623197 -49.2% 316596 interrupts.CPU57.LOC:Local_timer_interrupts 6102 -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts 6102 -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts 945.00 +14.9% 1085 interrupts.CPU57.TLB:TLB_shootdowns 1416 ± 5% +18.4% 1677 ± 4% interrupts.CPU58.CAL:Function_call_interrupts 623200 -49.1% 317024 interrupts.CPU58.LOC:Local_timer_interrupts 3029 -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts 3029 -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts 1303 +28.4% 1673 ± 7% interrupts.CPU59.CAL:Function_call_interrupts 623180 -49.2% 316311 interrupts.CPU59.LOC:Local_timer_interrupts 6105 -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts 6105 -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts 927.00 +19.5% 1107 ± 5% interrupts.CPU59.TLB:TLB_shootdowns 1318 ± 4% +26.4% 1665 ± 7% interrupts.CPU6.CAL:Function_call_interrupts 623208 -49.2% 316453 interrupts.CPU6.LOC:Local_timer_interrupts 6116 -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts 6116 -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts 623180 -49.2% 316722 interrupts.CPU60.LOC:Local_timer_interrupts 4574 ± 34% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts 4574 ± 34% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts 1314 +24.0% 1630 ± 7% interrupts.CPU61.CAL:Function_call_interrupts 623154 -49.1% 316903 interrupts.CPU61.LOC:Local_timer_interrupts 4567 ± 33% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts 4567 ± 33% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts 950.50 ± 2% +11.2% 1057 ± 4% interrupts.CPU61.TLB:TLB_shootdowns 1315 ± 2% +24.3% 1634 ± 4% interrupts.CPU62.CAL:Function_call_interrupts 623209 -49.2% 316651 interrupts.CPU62.LOC:Local_timer_interrupts 6073 -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts 6073 -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts 1318 ± 2% +24.8% 1644 ± 10% interrupts.CPU63.CAL:Function_call_interrupts 623192 -49.2% 316499 interrupts.CPU63.LOC:Local_timer_interrupts 6125 -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts 6125 -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts 1350 ± 6% +25.2% 1690 ± 4% interrupts.CPU64.CAL:Function_call_interrupts 623084 -49.2% 316398 interrupts.CPU64.LOC:Local_timer_interrupts 6092 -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts 6092 -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts 1377 ± 4% +22.0% 1680 ± 8% interrupts.CPU65.CAL:Function_call_interrupts 623703 -49.3% 316407 interrupts.CPU65.LOC:Local_timer_interrupts 4563 ± 33% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts 4563 ± 33% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts 1356 ± 3% +20.3% 1631 ± 4% interrupts.CPU66.CAL:Function_call_interrupts 623088 -49.2% 316400 interrupts.CPU66.LOC:Local_timer_interrupts 4576 ± 33% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts 4576 ± 33% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts 1306 ± 4% +24.7% 1628 ± 8% interrupts.CPU67.CAL:Function_call_interrupts 623161 -49.3% 315979 interrupts.CPU67.LOC:Local_timer_interrupts 6130 -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts 6130 -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts 1381 ± 2% +21.8% 1683 ± 5% interrupts.CPU68.CAL:Function_call_interrupts 623150 -49.3% 316185 interrupts.CPU68.LOC:Local_timer_interrupts 5966 ± 2% -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts 5966 ± 2% -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts 1293 +30.6% 1689 ± 7% interrupts.CPU69.CAL:Function_call_interrupts 622885 -49.2% 316563 interrupts.CPU69.LOC:Local_timer_interrupts 6092 -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts 6092 -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts 928.50 ± 2% +24.0% 1151 ± 3% interrupts.CPU69.TLB:TLB_shootdowns 2097 ± 91% -91.6% 177.00 ± 8% interrupts.CPU7.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7 1367 ± 2% +22.5% 1674 ± 7% interrupts.CPU7.CAL:Function_call_interrupts 623155 -49.2% 316746 interrupts.CPU7.LOC:Local_timer_interrupts 4585 ± 33% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts 4585 ± 33% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts 623011 -49.2% 316247 interrupts.CPU70.LOC:Local_timer_interrupts 6087 -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts 6087 -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts 1355 ± 2% +20.0% 1625 ± 6% interrupts.CPU71.CAL:Function_call_interrupts 623435 -49.3% 316226 interrupts.CPU71.LOC:Local_timer_interrupts 4554 ± 32% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts 4554 ± 32% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts 1295 +28.2% 1660 ± 7% interrupts.CPU8.CAL:Function_call_interrupts 623199 -49.2% 316419 interrupts.CPU8.LOC:Local_timer_interrupts 4594 ± 33% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts 4594 ± 33% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts 946.00 +14.9% 1086 ± 3% interrupts.CPU8.TLB:TLB_shootdowns 169.00 ± 5% +47.3% 249.00 ± 14% interrupts.CPU9.48:IR-PCI-MSI.1572873-edge.eth0-TxRx-9 1398 ± 2% +19.4% 1669 ± 2% interrupts.CPU9.CAL:Function_call_interrupts 623093 -49.2% 316231 interrupts.CPU9.LOC:Local_timer_interrupts 4644 ± 34% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 4644 ± 34% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 118.50 ± 5% -100.0% 0.00 interrupts.IWI:IRQ_work_interrupts 44868185 -49.2% 22787372 interrupts.LOC:Local_timer_interrupts 340125 ± 3% -100.0% 0.00 interrupts.NMI:Non-maskable_interrupts 340125 ± 3% -100.0% 0.00 interrupts.PMI:Performance_monitoring_interrupts 71173 +18.2% 84116 ± 3% interrupts.TLB:TLB_shootdowns *************************************************************************************************** lkp-csl-2sp5: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode: os/gcc-7/performance/1HDD/ext4/x86_64-rhel-7.6/10%/debian-x86_64-20191114.cgz/lkp-csl-2sp5/stress-ng/1s/0x500002c commit: 1d2cc5ac6f (" RISC-V Fixes for 5.7-rc4") f2af7d90e2 ("pipe: read/write_iter() handler should check for IOCB_NOWAIT") 1d2cc5ac6f6668cc f2af7d90e25ce1cf03318bf3290 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 3:4 -75% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 3:4 -75% :4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 93063 ± 12% +13.4% 105538 ± 3% stress-ng.access.ops 93058 ± 12% +13.4% 105533 ± 3% stress-ng.access.ops_per_sec 19572 +14.7% 22444 ± 3% stress-ng.af-alg.ops 18441 +14.5% 21124 ± 4% stress-ng.af-alg.ops_per_sec 566944 ± 14% +35.3% 766958 stress-ng.aio.ops 566928 ± 14% +35.3% 766930 stress-ng.aio.ops_per_sec 4134998 +11.7% 4620283 stress-ng.bigheap.ops 3390893 +11.6% 3784181 stress-ng.bigheap.ops_per_sec 4828426 +20.3% 5809892 stress-ng.brk.ops 4003576 +19.6% 4786441 stress-ng.brk.ops_per_sec 52215 ± 7% +23.4% 64429 ± 4% stress-ng.chown.ops 52227 ± 7% +23.4% 64441 ± 4% stress-ng.chown.ops_per_sec 385355 ± 3% +17.6% 453022 ± 7% stress-ng.clock.ops 385369 ± 3% +17.6% 453028 ± 7% stress-ng.clock.ops_per_sec 2852 ± 2% -11.5% 2524 ± 2% stress-ng.clone.ops 2850 ± 2% -11.5% 2522 ± 2% stress-ng.clone.ops_per_sec 20655 +20.9% 24981 ± 3% stress-ng.dccp.ops 20646 +21.0% 24974 ± 3% stress-ng.dccp.ops_per_sec 23637 ± 6% +21.7% 28755 ± 7% stress-ng.dir.ops_per_sec 13197249 +22.7% 16189574 stress-ng.dup.ops 13197792 +22.7% 16189931 stress-ng.dup.ops_per_sec 11381 +19.1% 13552 stress-ng.dynlib.ops 11377 +19.1% 13547 stress-ng.dynlib.ops_per_sec 606225 +9.1% 661487 ± 3% stress-ng.epoll.ops 606134 +9.1% 661381 ± 3% stress-ng.epoll.ops_per_sec 2556936 ± 16% +14.3% 2923774 ± 4% stress-ng.fanotify.ops 2547308 ± 16% +14.4% 2914089 ± 4% stress-ng.fanotify.ops_per_sec 154696 ± 2% +13.0% 174811 ± 7% stress-ng.fault.ops 154692 ± 2% +13.0% 174799 ± 7% stress-ng.fault.ops_per_sec 26869 ± 8% +10.0% 29545 ± 5% stress-ng.fiemap.ops 26769 ± 8% +9.4% 29298 ± 5% stress-ng.fiemap.ops_per_sec 2802774 ± 7% +27.8% 3581224 ± 2% stress-ng.file-ioctl.ops 2802731 ± 7% +27.1% 3562377 ± 2% stress-ng.file-ioctl.ops_per_sec 2943421 ± 2% +17.5% 3457054 ± 2% stress-ng.icmp-flood.ops 2943502 ± 2% +17.4% 3457122 ± 2% stress-ng.icmp-flood.ops_per_sec 82594 ± 5% +14.4% 94464 ± 4% stress-ng.idle-page.ops 82595 ± 5% +14.2% 94362 ± 4% stress-ng.idle-page.ops_per_sec 5.54 ± 10% +16.4% 6.44 ± 10% stress-ng.inode-flags.ops_per_sec 135.67 ± 4% +57.4% 213.50 ± 8% stress-ng.inotify.ops 135.14 ± 4% +57.5% 212.83 ± 7% stress-ng.inotify.ops_per_sec 5143 ± 7% +14.9% 5911 ± 5% stress-ng.io.ops 5073 ± 5% +16.5% 5909 ± 5% stress-ng.io.ops_per_sec 14.40 ± 2% +14.1% 16.43 ± 10% stress-ng.iomix.ops_per_sec 199047 +16.1% 231054 ± 4% stress-ng.kcmp.ops 199029 +16.1% 231025 ± 4% stress-ng.kcmp.ops_per_sec 499.33 +14.4% 571.00 ± 4% stress-ng.klog.ops 497.11 +14.5% 569.08 ± 4% stress-ng.klog.ops_per_sec 24949702 +8.7% 27114798 ± 3% stress-ng.locka.ops 24942095 +8.7% 27106245 ± 3% stress-ng.locka.ops_per_sec 397807 ± 8% +18.4% 471066 ± 2% stress-ng.lockf.ops 397654 ± 8% +18.4% 470855 ± 2% stress-ng.lockf.ops_per_sec 76131695 +22.7% 93427669 stress-ng.malloc.ops 76124426 +22.7% 93416659 stress-ng.malloc.ops_per_sec 376026 +23.5% 464331 stress-ng.mincore.ops 376045 +23.5% 464349 stress-ng.mincore.ops_per_sec 66418 ± 7% -16.4% 55525 ± 3% stress-ng.mknod.ops_per_sec 539675 +13.6% 612962 ± 2% stress-ng.mmap.ops 539618 +13.6% 612868 ± 2% stress-ng.mmap.ops_per_sec 1419987 +22.1% 1734160 stress-ng.mmapaddr.ops 1419814 +22.1% 1733897 stress-ng.mmapaddr.ops_per_sec 564713 ± 3% +9.4% 617769 ± 3% stress-ng.mmapfixed.ops 564635 ± 3% +9.4% 617646 ± 3% stress-ng.mmapfixed.ops_per_sec 2356344 +19.7% 2820689 stress-ng.mmapmany.ops 2333240 +19.5% 2787220 stress-ng.mmapmany.ops_per_sec 1263811 ± 6% +14.8% 1450563 ± 7% stress-ng.mremap.ops 1263670 ± 6% +14.8% 1450370 ± 7% stress-ng.mremap.ops_per_sec 4109012 ± 5% +9.6% 4504477 ± 4% stress-ng.msg.ops 4108646 ± 5% +9.6% 4504067 ± 4% stress-ng.msg.ops_per_sec 83.67 ± 3% +8.2% 90.50 ± 2% stress-ng.msync.ops 80.84 ± 3% +7.4% 86.79 ± 2% stress-ng.msync.ops_per_sec 3700 ± 3% -13.6% 3195 ± 5% stress-ng.netlink-proc.ops 3700 ± 3% -13.6% 3195 ± 5% stress-ng.netlink-proc.ops_per_sec 2860159 +22.2% 3495855 stress-ng.personality.ops 2860286 +22.2% 3495934 stress-ng.personality.ops_per_sec 9846409 +10.3% 10859929 ± 3% stress-ng.pipe.ops 9845415 +10.3% 10858890 ± 3% stress-ng.pipe.ops_per_sec 77461 ± 2% +11.6% 86449 ± 3% stress-ng.ramfs.ops 38731 ± 2% +11.6% 43223 ± 3% stress-ng.ramfs.ops_per_sec 2386891 ± 7% +15.4% 2753423 ± 2% stress-ng.rawsock.ops 2383770 ± 7% +15.4% 2750541 ± 2% stress-ng.rawsock.ops_per_sec 10713402 +21.4% 13006737 stress-ng.readahead.ops 10575897 +22.3% 12929775 stress-ng.readahead.ops_per_sec 2524 +17.8% 2974 stress-ng.remap.ops 2519 +17.8% 2969 stress-ng.remap.ops_per_sec 2655867 ± 5% +16.6% 3095430 ± 12% stress-ng.schedpolicy.ops 2655970 ± 5% +16.5% 3095483 ± 12% stress-ng.schedpolicy.ops_per_sec 801907 +14.6% 918619 ± 2% stress-ng.seal.ops 801937 +14.6% 918622 ± 2% stress-ng.seal.ops_per_sec 140574 ± 65% +78.8% 251305 ± 2% stress-ng.sem-sysv.ops 140554 ± 65% +78.8% 251262 ± 2% stress-ng.sem-sysv.ops_per_sec 16452629 +16.9% 19230253 stress-ng.sendfile.ops 16452526 +16.9% 19230079 stress-ng.sendfile.ops_per_sec 212166 ± 3% +12.2% 238004 ± 5% stress-ng.set.ops 212142 ± 3% +12.2% 238007 ± 5% stress-ng.set.ops_per_sec 23418 +16.3% 27231 stress-ng.shm-sysv.ops 23409 +16.3% 27222 stress-ng.shm-sysv.ops_per_sec 9284 +9.3% 10149 stress-ng.shm.ops 9281 +9.3% 10146 stress-ng.shm.ops_per_sec 5141246 ± 6% +12.5% 5785294 ± 2% stress-ng.sigfd.ops 5140731 ± 6% +12.5% 5784743 ± 2% stress-ng.sigfd.ops_per_sec 3153249 +21.1% 3819393 stress-ng.sigfpe.ops 3153383 +21.1% 3819512 stress-ng.sigfpe.ops_per_sec 4012485 +18.2% 4743159 ± 3% stress-ng.sigpending.ops 4012701 +18.2% 4743369 ± 3% stress-ng.sigpending.ops_per_sec 4066908 +20.1% 4884546 stress-ng.sigsegv.ops 4067049 +20.1% 4884679 stress-ng.sigsegv.ops_per_sec 2350870 +14.1% 2681187 ± 4% stress-ng.sockdiag.ops 2350839 +14.0% 2681103 ± 4% stress-ng.sockdiag.ops_per_sec 259944 +10.0% 285982 stress-ng.sockmany.ops 7447693 ± 3% +11.0% 8266733 ± 8% stress-ng.sockpair.ops 7377541 ± 2% +11.0% 8189913 ± 8% stress-ng.sockpair.ops_per_sec 11005218 +17.2% 12897762 ± 6% stress-ng.splice.ops 11005686 +17.2% 12898182 ± 6% stress-ng.splice.ops_per_sec 206.24 -1.6% 202.89 stress-ng.time.elapsed_time 206.24 -1.6% 202.89 stress-ng.time.elapsed_time.max 40270916 +18.9% 47881328 ± 8% stress-ng.time.involuntary_context_switches 154782 ± 2% +13.0% 174904 ± 7% stress-ng.time.major_page_faults 1989732 +9.4% 2177217 stress-ng.time.maximum_resident_set_size 69846758 +8.8% 76007980 stress-ng.time.minor_page_faults 284.04 -2.0% 278.34 stress-ng.time.user_time 34309654 +9.0% 37393435 stress-ng.time.voluntary_context_switches 235.67 ± 3% +32.4% 312.00 ± 3% stress-ng.tlb-shootdown.ops 235.31 ± 3% +32.3% 311.36 ± 3% stress-ng.tlb-shootdown.ops_per_sec 2653948 ± 3% +15.8% 3071950 ± 4% stress-ng.udp-flood.ops 2654036 ± 3% +15.7% 3072013 ± 4% stress-ng.udp-flood.ops_per_sec 2888897 +13.7% 3284032 stress-ng.udp.ops 2888524 +13.7% 3283640 stress-ng.udp.ops_per_sec 5.683e+08 +22.9% 6.982e+08 stress-ng.vdso.ops 5.15e+08 +22.8% 6.326e+08 stress-ng.vdso.ops_per_sec 3298257 ± 6% +9.7% 3618657 ± 4% stress-ng.vm-splice.ops 3298379 ± 6% +9.7% 3618755 ± 4% stress-ng.vm-splice.ops_per_sec 67264188 +22.4% 82357359 stress-ng.x86syscall.ops 61072922 +22.5% 74825411 stress-ng.x86syscall.ops_per_sec 3448331 +23.3% 4252436 stress-ng.zero.ops 3448476 +23.3% 4252561 stress-ng.zero.ops_per_sec 0.51 ± 7% -0.2 0.32 ± 22% mpstat.cpu.all.iowait% 27.58 ± 3% -24.4% 20.86 boot-time.boot 2258 -25.0% 1694 ± 2% boot-time.idle 697443 ± 9% -32.0% 474041 ± 3% meminfo.AnonHugePages 27079 ± 2% -13.6% 23407 ± 5% meminfo.PageTables 354801 +22.8% 435717 ± 11% meminfo.max_used_kB 10703 ± 5% -36.4% 6812 ± 27% vmstat.io.bo 557344 +13.9% 634899 ± 3% vmstat.system.cs 269404 -61.5% 103853 ± 2% vmstat.system.in 3.292e+08 -27.9% 2.372e+08 cpuidle.C1.time 20066110 ± 2% -23.6% 15337632 ± 2% cpuidle.C1.usage 1.343e+10 ± 28% -96.5% 4.763e+08 cpuidle.C1E.time 34267750 ± 16% -83.7% 5585924 ± 4% cpuidle.C1E.usage 3.089e+09 ±123% +426.6% 1.627e+10 cpuidle.C6.time 18987358 ± 3% +14.4% 21721580 ± 2% cpuidle.POLL.usage 0.21 ± 10% -26.9% 0.16 ± 13% sched_debug.cfs_rq:/.nr_running.avg 473.97 ± 33% +43.7% 680.94 ± 9% sched_debug.cfs_rq:/.nr_spread_over.max 451.86 ± 17% -38.7% 276.90 ± 13% sched_debug.cfs_rq:/.removed.load_avg.max 96.50 ± 38% -49.1% 49.07 ± 37% sched_debug.cfs_rq:/.removed.load_avg.stddev 451.86 ± 17% -38.7% 276.90 ± 13% sched_debug.cfs_rq:/.removed.runnable_avg.max 96.50 ± 38% -49.1% 49.07 ± 37% sched_debug.cfs_rq:/.removed.runnable_avg.stddev 216.53 ± 26% -58.1% 90.65 ± 56% sched_debug.cfs_rq:/.removed.util_avg.max 1394652 ± 29% -27.8% 1006781 sched_debug.cpu.avg_idle.max 39544 ±105% -92.5% 2947 ±147% sched_debug.cpu.max_idle_balance_cost.stddev 31.74 ± 16% +16.0% 36.82 ± 12% sched_debug.cpu.nr_uninterruptible.stddev 78426 ± 4% +16.8% 91594 ± 8% sched_debug.cpu.ttwu_local.max 1765 ± 20% -41.0% 1041 ± 17% sched_debug.cpu.yld_count.min 86.51 -100.0% 0.00 iostat.cpu.idle 11.71 -100.0% 0.00 iostat.cpu.system 1.26 -100.0% 0.00 iostat.cpu.user 20.02 ±141% -100.0% 0.00 iostat.sda.avgqu-sz.max 147.85 ±141% -100.0% 0.00 iostat.sda.await.max 530.33 ±141% -100.0% 0.00 iostat.sda.r_await.max 3.02 ±141% -100.0% 0.00 iostat.sda.svctm.max 147.85 ±141% -100.0% 0.00 iostat.sda.w_await.max 39.02 ± 70% -100.0% 0.00 iostat.sdb.avgqu-sz.max 235.65 ± 74% -100.0% 0.00 iostat.sdb.await.max 660.73 ± 96% -100.0% 0.00 iostat.sdb.r_await.max 5.89 ± 70% -100.0% 0.00 iostat.sdb.svctm.max 237.85 ± 74% -100.0% 0.00 iostat.sdb.w_await.max 369695 ± 2% -7.6% 341444 ± 3% proc-vmstat.nr_active_anon 224169 ± 5% -13.2% 194519 ± 4% proc-vmstat.nr_anon_pages 337.00 ± 13% -30.3% 235.00 ± 3% proc-vmstat.nr_anon_transparent_hugepages 6947 -15.4% 5878 ± 3% proc-vmstat.nr_page_table_pages 38177 -5.5% 36070 proc-vmstat.nr_slab_reclaimable 86748 -3.0% 84136 proc-vmstat.nr_slab_unreclaimable 369695 ± 2% -7.6% 341445 ± 3% proc-vmstat.nr_zone_active_anon 1.569e+08 +15.0% 1.804e+08 ± 2% proc-vmstat.numa_hit 1.569e+08 +15.0% 1.804e+08 ± 2% proc-vmstat.numa_local 61944 -48.2% 32092 ± 5% proc-vmstat.numa_other 25986257 +1.6% 26400635 proc-vmstat.pgactivate 2.326e+08 ± 3% +12.1% 2.607e+08 proc-vmstat.pgalloc_normal 84243726 +7.9% 90871301 proc-vmstat.pgfault 2.365e+08 ± 3% +13.3% 2.679e+08 ± 2% proc-vmstat.pgfree 154782 ± 2% +13.0% 174905 ± 7% proc-vmstat.pgmajfault 2230056 ± 5% -36.3% 1420257 ± 29% proc-vmstat.pgpgout 1371532 +4.7% 1435795 proc-vmstat.unevictable_pgs_culled 1412453 +4.1% 1470689 ± 2% proc-vmstat.unevictable_pgs_mlocked 1376286 +4.7% 1440479 proc-vmstat.unevictable_pgs_munlocked 1371386 +4.7% 1435673 proc-vmstat.unevictable_pgs_rescued 4008263 +16.2% 4659375 proc-vmstat.unevictable_pgs_scanned 1269 ± 4% -9.2% 1153 ± 3% slabinfo.anon_vma.active_slabs 58429 ± 4% -9.2% 53072 ± 3% slabinfo.anon_vma.num_objs 1269 ± 4% -9.2% 1153 ± 3% slabinfo.anon_vma.num_slabs 1396 ± 2% -10.0% 1257 ± 6% slabinfo.filp.active_slabs 44708 ± 2% -10.0% 40244 ± 6% slabinfo.filp.num_objs 1396 ± 2% -10.0% 1257 ± 6% slabinfo.filp.num_slabs 4061 ± 8% +38.4% 5618 ± 16% slabinfo.fsnotify_mark_connector.active_objs 4269 ± 6% +36.9% 5842 ± 16% slabinfo.fsnotify_mark_connector.num_objs 7587 ± 2% -21.9% 5929 ± 2% slabinfo.kmalloc-2k.active_objs 491.00 ± 2% -21.5% 385.50 ± 2% slabinfo.kmalloc-2k.active_slabs 7867 ± 2% -21.5% 6172 ± 2% slabinfo.kmalloc-2k.num_objs 491.00 ± 2% -21.5% 385.50 ± 2% slabinfo.kmalloc-2k.num_slabs 1885 -12.5% 1650 ± 5% slabinfo.kmalloc-4k.active_objs 1964 -13.3% 1703 ± 6% slabinfo.kmalloc-4k.num_objs 582.00 ± 4% +11.5% 649.00 ± 6% slabinfo.numa_policy.active_objs 3770 ± 4% +26.6% 4772 ± 4% slabinfo.pool_workqueue.active_objs 3918 ± 4% +24.8% 4890 ± 4% slabinfo.pool_workqueue.num_objs 28839 -21.6% 22602 ± 5% slabinfo.proc_inode_cache.active_objs 606.67 -21.1% 478.75 ± 5% slabinfo.proc_inode_cache.active_slabs 29144 -21.1% 23005 ± 5% slabinfo.proc_inode_cache.num_objs 606.67 -21.1% 478.75 ± 5% slabinfo.proc_inode_cache.num_slabs 499.33 ± 9% -26.4% 367.50 ± 9% slabinfo.request_sock_TCP.active_objs 499.33 ± 9% -26.4% 367.50 ± 9% slabinfo.request_sock_TCP.num_objs 1138 ± 5% -12.4% 996.50 ± 8% slabinfo.uts_namespace.active_objs 1138 ± 5% -12.4% 996.50 ± 8% slabinfo.uts_namespace.num_objs 46.80 -46.8 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64 46.59 -46.6 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 46.58 -46.6 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 46.57 -46.6 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 42.14 -42.1 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 41.53 -41.5 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 36.56 -36.6 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 11.68 ± 3% -11.7 0.00 perf-profile.calltrace.cycles-pp.page_fault 11.47 ± 3% -11.5 0.00 perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 10.64 ± 2% -10.6 0.00 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault 10.35 ± 2% -10.3 0.00 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 9.99 ± 2% -10.0 0.00 perf-profile.calltrace.cycles-pp.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 7.37 ± 4% -7.4 0.00 perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.do_signal.exit_to_usermode_loop 7.11 ± 4% -7.1 0.00 perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.get_signal.do_signal 7.10 ± 4% -7.1 0.00 perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.get_signal 6.84 ± 3% -6.8 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 6.83 ± 3% -6.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.95 ± 7% -6.0 0.00 perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode 5.90 ± 7% -5.9 0.00 perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 5.87 ± 7% -5.9 0.00 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 5.87 ± 7% -5.9 0.00 perf-profile.calltrace.cycles-pp.do_signal.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 5.82 ± 7% -5.8 0.00 perf-profile.calltrace.cycles-pp.get_signal.do_signal.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 5.79 ± 7% -5.8 0.00 perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.do_signal.exit_to_usermode_loop.prepare_exit_to_usermode 5.36 ± 4% -5.4 0.00 perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit 5.32 ± 4% -5.3 0.00 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit 46.82 -46.8 0.00 perf-profile.children.cycles-pp.do_idle 46.80 -46.8 0.00 perf-profile.children.cycles-pp.secondary_startup_64 46.80 -46.8 0.00 perf-profile.children.cycles-pp.cpu_startup_entry 46.59 -46.6 0.00 perf-profile.children.cycles-pp.start_secondary 42.31 -42.3 0.00 perf-profile.children.cycles-pp.cpuidle_enter 42.30 -42.3 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state 36.68 -36.7 0.00 perf-profile.children.cycles-pp.intel_idle 27.85 ± 2% -27.8 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 27.68 ± 2% -27.7 0.00 perf-profile.children.cycles-pp.do_syscall_64 12.78 ± 3% -12.8 0.00 perf-profile.children.cycles-pp.page_fault 12.31 ± 2% -12.3 0.00 perf-profile.children.cycles-pp.do_page_fault 11.42 ± 2% -11.4 0.00 perf-profile.children.cycles-pp.handle_mm_fault 11.10 ± 2% -11.1 0.00 perf-profile.children.cycles-pp.__handle_mm_fault 10.73 ± 2% -10.7 0.00 perf-profile.children.cycles-pp.handle_pte_fault 8.08 ± 3% -8.1 0.00 perf-profile.children.cycles-pp.exit_to_usermode_loop 8.02 ± 3% -8.0 0.00 perf-profile.children.cycles-pp.do_signal 7.59 ± 3% -7.6 0.00 perf-profile.children.cycles-pp.get_signal 7.59 ± 4% -7.6 0.00 perf-profile.children.cycles-pp.do_group_exit 7.59 ± 4% -7.6 0.00 perf-profile.children.cycles-pp.do_exit 7.29 ± 4% -7.3 0.00 perf-profile.children.cycles-pp.mmput 7.28 ± 4% -7.3 0.00 perf-profile.children.cycles-pp.exit_mmap 5.96 ± 7% -6.0 0.00 perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 5.91 ± 7% -5.9 0.00 perf-profile.children.cycles-pp.prepare_exit_to_usermode 5.56 ± 4% -5.6 0.00 perf-profile.children.cycles-pp.unmap_vmas 5.53 ± 4% -5.5 0.00 perf-profile.children.cycles-pp.unmap_page_range 5.22 ± 5% -5.2 0.00 perf-profile.children.cycles-pp.tlb_flush_mmu 36.68 -36.7 0.00 perf-profile.self.cycles-pp.intel_idle 37041 ± 21% -39.3% 22466 ± 15% softirqs.CPU0.RCU 30290 ± 3% +87.0% 56641 ± 6% softirqs.CPU0.SCHED 80216 ± 6% -57.2% 34304 ± 7% softirqs.CPU0.TIMER 32120 ± 12% -37.8% 19968 ± 17% softirqs.CPU1.RCU 26280 ± 5% -18.5% 21422 ± 11% softirqs.CPU1.SCHED 76845 ± 8% -64.6% 27199 ± 4% softirqs.CPU1.TIMER 29568 ± 11% -60.1% 11790 ± 50% softirqs.CPU10.RCU 26023 ± 3% -76.5% 6127 ± 23% softirqs.CPU10.SCHED 79560 ± 10% -70.8% 23224 ± 13% softirqs.CPU10.TIMER 28786 ± 6% -58.1% 12053 ± 50% softirqs.CPU11.RCU 24329 ± 6% -74.9% 6109 ± 20% softirqs.CPU11.SCHED 78246 ± 8% -71.0% 22678 ± 9% softirqs.CPU11.TIMER 30351 ± 11% -58.4% 12629 ± 41% softirqs.CPU12.RCU 25522 ± 2% -77.5% 5747 ± 10% softirqs.CPU12.SCHED 78244 ± 8% -69.8% 23618 ± 18% softirqs.CPU12.TIMER 30441 ± 17% -57.7% 12890 ± 55% softirqs.CPU13.RCU 25017 ± 5% -76.5% 5890 ± 20% softirqs.CPU13.SCHED 78076 ± 9% -72.7% 21319 ± 13% softirqs.CPU13.TIMER 30947 ± 15% -65.9% 10559 ± 65% softirqs.CPU14.RCU 26201 ± 3% -79.3% 5429 ± 17% softirqs.CPU14.SCHED 90285 ± 17% -76.1% 21554 ± 15% softirqs.CPU14.TIMER 29313 ± 8% -67.3% 9583 ± 46% softirqs.CPU15.RCU 25239 ± 2% -78.4% 5443 ± 20% softirqs.CPU15.SCHED 77034 ± 10% -73.8% 20177 ± 15% softirqs.CPU15.TIMER 31400 ± 21% -63.4% 11493 ± 43% softirqs.CPU16.RCU 25027 -78.0% 5501 ± 13% softirqs.CPU16.SCHED 76239 ± 10% -72.9% 20641 ± 9% softirqs.CPU16.TIMER 29614 ± 10% -64.6% 10479 ± 60% softirqs.CPU17.RCU 25761 ± 2% -77.8% 5718 ± 15% softirqs.CPU17.SCHED 76932 ± 9% -72.0% 21504 ± 17% softirqs.CPU17.TIMER 30828 ± 19% -66.5% 10336 ± 37% softirqs.CPU18.RCU 26141 ± 3% -78.5% 5620 ± 11% softirqs.CPU18.SCHED 77356 ± 10% -73.4% 20567 ± 5% softirqs.CPU18.TIMER 32302 ± 20% -72.9% 8751 ± 28% softirqs.CPU19.RCU 25542 ± 3% -77.0% 5873 ± 16% softirqs.CPU19.SCHED 76556 ± 10% -71.1% 22112 ± 9% softirqs.CPU19.TIMER 27239 ± 4% -43.8% 15302 ± 31% softirqs.CPU2.RCU 26464 -55.9% 11667 ± 11% softirqs.CPU2.SCHED 77363 ± 9% -69.6% 23486 ± 6% softirqs.CPU2.TIMER 34485 ± 16% -64.5% 12253 ± 37% softirqs.CPU20.RCU 25253 ± 2% -76.1% 6044 ± 16% softirqs.CPU20.SCHED 76621 ± 11% -71.4% 21893 ± 9% softirqs.CPU20.TIMER 31202 ± 14% -67.6% 10113 ± 39% softirqs.CPU21.RCU 25995 ± 3% -78.6% 5555 ± 22% softirqs.CPU21.SCHED 77002 ± 10% -72.6% 21077 ± 12% softirqs.CPU21.TIMER 29780 ± 11% -59.4% 12089 ± 44% softirqs.CPU22.RCU 26213 ± 2% -77.7% 5836 ± 28% softirqs.CPU22.SCHED 89650 ± 17% -75.9% 21604 ± 17% softirqs.CPU22.TIMER 28953 ± 9% -62.2% 10953 ± 57% softirqs.CPU23.RCU 25889 ± 3% -79.1% 5421 ± 15% softirqs.CPU23.SCHED 76827 ± 10% -72.6% 21025 ± 13% softirqs.CPU23.TIMER 37966 ± 9% -40.8% 22466 ± 16% softirqs.CPU24.RCU 25148 ± 5% -66.3% 8482 ± 10% softirqs.CPU24.SCHED 84220 ± 6% -58.8% 34682 ± 11% softirqs.CPU24.TIMER 33985 ± 21% -53.7% 15735 ± 20% softirqs.CPU25.RCU 24545 ± 6% -66.6% 8194 ± 8% softirqs.CPU25.SCHED 80216 ± 6% -64.2% 28713 ± 5% softirqs.CPU25.TIMER 29663 ± 16% -45.4% 16189 ± 41% softirqs.CPU26.RCU 25146 ± 7% -76.1% 6018 ± 27% softirqs.CPU26.SCHED 80701 ± 6% -70.5% 23788 ± 17% softirqs.CPU26.TIMER 33652 ± 8% -60.1% 13429 ± 44% softirqs.CPU27.RCU 24505 ± 4% -75.4% 6037 ± 16% softirqs.CPU27.SCHED 82738 ± 10% -73.2% 22170 ± 13% softirqs.CPU27.TIMER 27764 ± 10% -53.9% 12801 ± 26% softirqs.CPU28.RCU 26315 -76.5% 6171 ± 16% softirqs.CPU28.SCHED 82501 ± 5% -73.5% 21895 ± 16% softirqs.CPU28.TIMER 28011 ± 14% -57.8% 11819 ± 52% softirqs.CPU29.RCU 26079 ± 2% -77.4% 5889 ± 13% softirqs.CPU29.SCHED 81904 ± 5% -73.3% 21830 ± 19% softirqs.CPU29.TIMER 32328 ± 14% -64.0% 11653 ± 42% softirqs.CPU3.RCU 27099 ± 3% -73.8% 7110 ± 2% softirqs.CPU3.SCHED 78727 ± 11% -70.5% 23260 ± 7% softirqs.CPU3.TIMER 28987 ± 22% -48.3% 14981 ± 45% softirqs.CPU30.RCU 25770 -76.0% 6190 ± 24% softirqs.CPU30.SCHED 82585 ± 6% -72.9% 22363 ± 20% softirqs.CPU30.TIMER 30650 ± 26% -61.7% 11742 ± 29% softirqs.CPU31.RCU 26466 ± 2% -78.6% 5651 ± 14% softirqs.CPU31.SCHED 81153 ± 5% -73.9% 21141 ± 17% softirqs.CPU31.TIMER 29454 ± 23% -68.0% 9437 ± 28% softirqs.CPU32.RCU 25058 ± 6% -77.7% 5588 ± 9% softirqs.CPU32.SCHED 81798 ± 5% -73.8% 21399 ± 19% softirqs.CPU32.TIMER 29263 ± 9% -62.8% 10888 ± 29% softirqs.CPU33.RCU 26911 -78.1% 5886 ± 6% softirqs.CPU33.SCHED 81472 ± 5% -73.9% 21300 ± 15% softirqs.CPU33.TIMER 29500 ± 14% -58.5% 12256 ± 31% softirqs.CPU34.RCU 25611 ± 4% -76.0% 6158 ± 14% softirqs.CPU34.SCHED 81739 ± 5% -72.5% 22479 ± 20% softirqs.CPU34.TIMER 28830 ± 8% -62.7% 10746 ± 30% softirqs.CPU35.RCU 26526 -76.9% 6131 ± 19% softirqs.CPU35.SCHED 83475 ± 7% -73.6% 22027 ± 17% softirqs.CPU35.TIMER 29523 ± 11% -65.6% 10154 ± 20% softirqs.CPU36.RCU 26117 -78.5% 5602 ± 16% softirqs.CPU36.SCHED 80803 ± 6% -74.8% 20336 ± 17% softirqs.CPU36.TIMER 29646 ± 22% -55.7% 13136 ± 36% softirqs.CPU37.RCU 25650 ± 2% -73.6% 6770 ± 12% softirqs.CPU37.SCHED 80683 ± 5% -72.5% 22190 ± 5% softirqs.CPU37.TIMER 25869 ± 9% -61.1% 10060 ± 34% softirqs.CPU38.RCU 26205 -76.0% 6277 ± 20% softirqs.CPU38.SCHED 81305 ± 6% -73.5% 21584 ± 14% softirqs.CPU38.TIMER 24658 -55.4% 10995 ± 37% softirqs.CPU39.RCU 26549 -78.6% 5688 ± 23% softirqs.CPU39.SCHED 80929 ± 6% -74.8% 20401 ± 13% softirqs.CPU39.TIMER 27350 ± 6% -53.4% 12756 ± 41% softirqs.CPU4.RCU 26369 ± 2% -74.0% 6850 ± 6% softirqs.CPU4.SCHED 77769 ± 9% -67.7% 25082 ± 12% softirqs.CPU4.TIMER 26646 ± 8% -64.2% 9530 ± 23% softirqs.CPU40.RCU 26438 ± 2% -80.0% 5291 ± 9% softirqs.CPU40.SCHED 81659 ± 5% -76.3% 19365 ± 10% softirqs.CPU40.TIMER 25629 ± 2% -66.3% 8635 ± 12% softirqs.CPU41.RCU 25627 ± 3% -79.0% 5387 ± 10% softirqs.CPU41.SCHED 81684 ± 5% -75.7% 19880 ± 10% softirqs.CPU41.TIMER 29480 ± 18% -58.6% 12215 ± 15% softirqs.CPU42.RCU 25185 -76.2% 6002 ± 10% softirqs.CPU42.SCHED 82540 ± 2% -73.5% 21835 ± 12% softirqs.CPU42.TIMER 25479 ± 9% -54.0% 11721 ± 33% softirqs.CPU43.RCU 24642 ± 5% -79.2% 5116 ± 21% softirqs.CPU43.SCHED 80912 ± 5% -77.1% 18563 ± 13% softirqs.CPU43.TIMER 27589 ± 11% -56.7% 11941 ± 29% softirqs.CPU44.RCU 26820 -81.0% 5100 ± 17% softirqs.CPU44.SCHED 81577 ± 5% -77.0% 18731 ± 9% softirqs.CPU44.TIMER 25727 ± 6% -48.5% 13247 ± 37% softirqs.CPU45.RCU 24637 ± 5% -77.8% 5465 ± 15% softirqs.CPU45.SCHED 80387 ± 6% -75.3% 19868 ± 10% softirqs.CPU45.TIMER 27904 ± 5% -61.0% 10876 ± 31% softirqs.CPU46.RCU 26077 ± 3% -78.2% 5672 ± 14% softirqs.CPU46.SCHED 81858 ± 6% -76.9% 18883 ± 12% softirqs.CPU46.TIMER 27164 ± 15% -65.7% 9307 ± 29% softirqs.CPU47.RCU 24189 ± 8% -79.1% 5060 ± 14% softirqs.CPU47.SCHED 81398 ± 5% -77.6% 18253 ± 10% softirqs.CPU47.TIMER 29943 ± 11% -67.5% 9741 ± 27% softirqs.CPU48.RCU 25502 ± 2% -79.9% 5138 ± 15% softirqs.CPU48.SCHED 78856 ± 7% -72.1% 21997 ± 12% softirqs.CPU48.TIMER 29986 ± 11% -73.3% 8002 ± 40% softirqs.CPU49.RCU 24277 ± 2% -77.0% 5593 ± 21% softirqs.CPU49.SCHED 77547 ± 7% -72.5% 21363 ± 15% softirqs.CPU49.TIMER 30266 ± 10% -50.9% 14871 ± 38% softirqs.CPU5.RCU 26625 -76.2% 6327 ± 9% softirqs.CPU5.SCHED 78397 ± 9% -71.2% 22615 ± 5% softirqs.CPU5.TIMER 28505 ± 9% -57.9% 12009 ± 33% softirqs.CPU50.RCU 24239 ± 8% -75.9% 5834 ± 20% softirqs.CPU50.SCHED 77866 ± 8% -69.4% 23857 ± 20% softirqs.CPU50.TIMER 28405 ± 8% -64.3% 10150 ± 39% softirqs.CPU51.RCU 25902 ± 4% -77.0% 5957 ± 11% softirqs.CPU51.SCHED 79370 ± 9% -69.2% 24409 ± 15% softirqs.CPU51.TIMER 30269 ±107% +575.3% 204402 ±102% softirqs.CPU52.NET_RX 29874 ± 10% -54.6% 13576 ± 42% softirqs.CPU52.RCU 25440 ± 2% -75.4% 6252 ± 6% softirqs.CPU52.SCHED 78978 ± 8% -69.2% 24301 ± 8% softirqs.CPU52.TIMER 30926 ± 9% -56.6% 13414 ± 46% softirqs.CPU53.RCU 24586 -75.0% 6142 ± 14% softirqs.CPU53.SCHED 79301 ± 8% -65.8% 27148 ± 24% softirqs.CPU53.TIMER 29040 ± 8% -60.4% 11511 ± 25% softirqs.CPU54.RCU 24818 ± 5% -73.0% 6690 ± 10% softirqs.CPU54.SCHED 79599 ± 9% -67.8% 25644 ± 10% softirqs.CPU54.TIMER 28243 ± 7% -56.8% 12199 ± 35% softirqs.CPU55.RCU 25583 ± 4% -76.2% 6086 ± 9% softirqs.CPU55.SCHED 78317 ± 6% -68.7% 24531 ± 19% softirqs.CPU55.TIMER 29376 ± 12% -53.3% 13715 ± 38% softirqs.CPU56.RCU 25101 ± 6% -74.3% 6443 ± 14% softirqs.CPU56.SCHED 78169 ± 9% -67.5% 25422 ± 15% softirqs.CPU56.TIMER 29968 ± 14% -59.6% 12108 ± 55% softirqs.CPU57.RCU 26084 ± 3% -74.6% 6612 ± 19% softirqs.CPU57.SCHED 79402 ± 9% -67.2% 26038 ± 15% softirqs.CPU57.TIMER 28223 ± 9% -52.9% 13293 ± 32% softirqs.CPU58.RCU 25120 -77.1% 5762 ± 12% softirqs.CPU58.SCHED 79058 ± 8% -68.1% 25219 ± 9% softirqs.CPU58.TIMER 26338 ± 3% -74.9% 6618 ± 16% softirqs.CPU59.SCHED 79685 ± 8% -66.1% 27020 ± 16% softirqs.CPU59.TIMER 30080 ± 9% -68.0% 9629 ± 30% softirqs.CPU6.RCU 27167 ± 4% -74.3% 6986 ± 18% softirqs.CPU6.SCHED 79342 ± 11% -71.9% 22312 ± 8% softirqs.CPU6.TIMER 23634 ± 8% -73.5% 6257 ± 22% softirqs.CPU60.SCHED 79167 ± 8% -66.5% 26495 ± 19% softirqs.CPU60.TIMER 31863 ± 19% -65.8% 10905 ± 29% softirqs.CPU61.RCU 25439 ± 5% -76.1% 6077 ± 9% softirqs.CPU61.SCHED 80669 ± 6% -69.7% 24444 ± 17% softirqs.CPU61.TIMER 26510 ± 9% -53.0% 12464 ± 56% softirqs.CPU62.RCU 25898 ± 4% -76.0% 6212 ± 12% softirqs.CPU62.SCHED 79015 ± 8% -69.6% 24024 ± 17% softirqs.CPU62.TIMER 30113 ± 8% -62.4% 11332 ± 58% softirqs.CPU63.RCU 25416 ± 3% -75.2% 6312 ± 11% softirqs.CPU63.SCHED 78810 ± 9% -67.4% 25704 ± 17% softirqs.CPU63.TIMER 25192 ± 5% -53.1% 11816 ± 47% softirqs.CPU64.RCU 25730 ± 3% -76.6% 6031 ± 12% softirqs.CPU64.SCHED 79336 ± 6% -69.2% 24459 ± 14% softirqs.CPU64.TIMER 28862 ± 10% -62.9% 10703 ± 40% softirqs.CPU65.RCU 25564 ± 5% -75.9% 6155 ± 9% softirqs.CPU65.SCHED 78817 ± 8% -67.7% 25452 ± 14% softirqs.CPU65.TIMER 27736 ± 8% -62.5% 10404 ± 36% softirqs.CPU66.RCU 25700 ± 5% -76.4% 6059 ± 8% softirqs.CPU66.SCHED 79642 ± 8% -69.6% 24189 ± 13% softirqs.CPU66.TIMER 27331 ± 11% -70.3% 8124 ± 17% softirqs.CPU67.RCU 23323 ± 7% -76.3% 5535 ± 12% softirqs.CPU67.SCHED 77933 ± 7% -71.2% 22417 ± 13% softirqs.CPU67.TIMER 25665 ± 3% -63.7% 9304 ± 14% softirqs.CPU68.RCU 25718 ± 3% -77.1% 5880 ± 8% softirqs.CPU68.SCHED 78102 ± 8% -68.1% 24935 ± 6% softirqs.CPU68.TIMER 25657 ± 6% -62.4% 9651 ± 46% softirqs.CPU69.RCU 24966 ± 6% -77.2% 5697 ± 8% softirqs.CPU69.SCHED 77780 ± 9% -71.1% 22505 ± 12% softirqs.CPU69.TIMER 33234 ± 21% -68.6% 10432 ± 26% softirqs.CPU7.RCU 25113 ± 6% -75.4% 6181 ± 21% softirqs.CPU7.SCHED 77832 ± 10% -69.1% 24046 ± 19% softirqs.CPU7.TIMER 25450 ± 4% -63.6% 9276 ± 29% softirqs.CPU70.RCU 25763 ± 5% -78.8% 5459 ± 9% softirqs.CPU70.SCHED 79089 ± 9% -71.5% 22503 ± 11% softirqs.CPU70.TIMER 27842 ± 9% -60.7% 10945 ± 43% softirqs.CPU71.RCU 25407 ± 5% -75.9% 6133 ± 13% softirqs.CPU71.SCHED 78082 ± 8% -70.3% 23214 ± 10% softirqs.CPU71.TIMER 27183 ± 21% -53.8% 12569 ± 37% softirqs.CPU72.RCU 24955 ± 5% -77.1% 5704 ± 20% softirqs.CPU72.SCHED 82004 ± 6% -74.9% 20594 ± 15% softirqs.CPU72.TIMER 24757 ± 10% -48.3% 12797 ± 37% softirqs.CPU73.RCU 27092 ± 9% -78.8% 5748 ± 13% softirqs.CPU73.SCHED 92680 ± 12% -77.6% 20716 ± 15% softirqs.CPU73.TIMER 26136 ± 16% -56.7% 11305 ± 47% softirqs.CPU74.RCU 25447 ± 2% -75.9% 6145 ± 15% softirqs.CPU74.SCHED 81341 ± 6% -72.9% 22051 ± 18% softirqs.CPU74.TIMER 30031 ± 26% -58.9% 12356 ± 31% softirqs.CPU75.RCU 26365 -77.4% 5949 ± 15% softirqs.CPU75.SCHED 81986 ± 4% -73.6% 21661 ± 20% softirqs.CPU75.TIMER 27104 ± 10% -51.7% 13095 ± 53% softirqs.CPU76.RCU 25780 ± 3% -75.2% 6393 ± 17% softirqs.CPU76.SCHED 81892 ± 6% -71.4% 23408 ± 22% softirqs.CPU76.TIMER 2297 ±139% +3798.3% 89544 ±118% softirqs.CPU77.NET_RX 26047 ± 3% -77.8% 5770 ± 22% softirqs.CPU77.SCHED 82358 ± 5% -71.9% 23154 ± 26% softirqs.CPU77.TIMER 26342 ± 17% -53.8% 12173 ± 44% softirqs.CPU78.RCU 25464 ± 6% -75.7% 6198 ± 15% softirqs.CPU78.SCHED 81152 ± 6% -70.6% 23826 ± 20% softirqs.CPU78.TIMER 26021 ± 2% -75.0% 6515 ± 14% softirqs.CPU79.SCHED 81148 ± 5% -70.9% 23650 ± 20% softirqs.CPU79.TIMER 31835 ± 9% -67.7% 10268 ± 33% softirqs.CPU8.RCU 26469 ± 3% -77.1% 6065 ± 10% softirqs.CPU8.SCHED 77721 ± 9% -70.6% 22881 ± 12% softirqs.CPU8.TIMER 27239 -77.6% 6103 ± 13% softirqs.CPU80.SCHED 82692 ± 5% -72.1% 23094 ± 19% softirqs.CPU80.TIMER 25151 ± 2% -75.3% 6220 ± 18% softirqs.CPU81.SCHED 81812 ± 6% -71.0% 23720 ± 17% softirqs.CPU81.TIMER 24993 ± 5% -47.4% 13149 ± 45% softirqs.CPU82.RCU 26136 -76.8% 6070 ± 10% softirqs.CPU82.SCHED 81989 ± 5% -72.2% 22770 ± 20% softirqs.CPU82.TIMER 26869 ± 13% -57.8% 11350 ± 36% softirqs.CPU83.RCU 26199 -75.4% 6457 ± 16% softirqs.CPU83.SCHED 81829 ± 5% -71.6% 23274 ± 20% softirqs.CPU83.TIMER 24846 ± 5% -57.9% 10449 ± 21% softirqs.CPU84.RCU 24989 ± 3% -73.2% 6688 ± 10% softirqs.CPU84.SCHED 80462 ± 7% -70.7% 23587 ± 12% softirqs.CPU84.TIMER 25620 ± 3% -49.5% 12931 ± 35% softirqs.CPU85.RCU 24880 ± 5% -74.3% 6389 ± 12% softirqs.CPU85.SCHED 80348 ± 7% -70.8% 23459 ± 15% softirqs.CPU85.TIMER 25088 ± 7% -50.2% 12497 ± 38% softirqs.CPU86.RCU 25294 ± 4% -75.8% 6113 ± 21% softirqs.CPU86.SCHED 80452 ± 7% -72.3% 22282 ± 20% softirqs.CPU86.TIMER 30601 ± 17% -62.2% 11580 ± 28% softirqs.CPU87.RCU 25740 ± 3% -74.3% 6622 ± 17% softirqs.CPU87.SCHED 81353 ± 7% -70.2% 24219 ± 17% softirqs.CPU87.TIMER 27274 ± 9% -41.5% 15956 ± 45% softirqs.CPU88.RCU 25800 -74.0% 6699 ± 14% softirqs.CPU88.SCHED 82474 ± 5% -70.3% 24477 ± 21% softirqs.CPU88.TIMER 27943 ± 9% -56.5% 12159 ± 33% softirqs.CPU89.RCU 25657 -75.3% 6325 ± 18% softirqs.CPU89.SCHED 81896 ± 6% -72.1% 22830 ± 17% softirqs.CPU89.TIMER 28337 ± 11% -57.4% 12076 ± 21% softirqs.CPU9.RCU 25443 -76.2% 6066 ± 16% softirqs.CPU9.SCHED 79222 ± 11% -70.5% 23373 ± 12% softirqs.CPU9.TIMER 26839 ± 6% -54.0% 12343 ± 33% softirqs.CPU90.RCU 25447 ± 5% -75.0% 6358 ± 17% softirqs.CPU90.SCHED 81277 ± 6% -72.8% 22109 ± 18% softirqs.CPU90.TIMER 28779 ± 5% -59.6% 11640 ± 27% softirqs.CPU91.RCU 26417 -75.4% 6487 ± 14% softirqs.CPU91.SCHED 81963 ± 5% -71.2% 23622 ± 16% softirqs.CPU91.TIMER 29945 ± 10% -65.4% 10367 ± 22% softirqs.CPU92.RCU 25256 ± 4% -77.4% 5705 ± 6% softirqs.CPU92.SCHED 82130 ± 6% -74.2% 21230 ± 5% softirqs.CPU92.TIMER 28155 ± 6% -67.5% 9157 ± 28% softirqs.CPU93.RCU 25929 ± 5% -75.6% 6320 ± 13% softirqs.CPU93.SCHED 81815 ± 6% -73.7% 21497 ± 7% softirqs.CPU93.TIMER 27214 ± 5% -61.8% 10401 ± 36% softirqs.CPU94.RCU 25874 ± 5% -75.8% 6263 ± 13% softirqs.CPU94.SCHED 82748 ± 6% -73.3% 22069 ± 10% softirqs.CPU94.TIMER 26670 ± 5% -55.5% 11863 ± 48% softirqs.CPU95.RCU 25663 ± 5% -75.8% 6208 ± 24% softirqs.CPU95.SCHED 82145 ± 5% -73.5% 21797 ± 21% softirqs.CPU95.TIMER 6676766 +14.4% 7639170 softirqs.NET_RX 2764689 -58.0% 1162152 ± 2% softirqs.RCU 2466136 -73.5% 653899 softirqs.SCHED 7716889 -71.4% 2207670 ± 2% softirqs.TIMER 178.67 ± 7% +16.0% 207.25 ± 3% interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 2.33 ±141% +22550.0% 528.50 ±170% interrupts.42:PCI-MSI.31981575-edge.i40e-eth0-TxRx-6 0.67 ± 70% +75800.0% 506.00 ±171% interrupts.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25 1427 ± 85% -100.0% 0.00 interrupts.64:PCI-MSI.31981597-edge.i40e-eth0-TxRx-28 484.67 ± 71% -99.9% 0.50 ±173% interrupts.69:PCI-MSI.31981602-edge.i40e-eth0-TxRx-33 482106 -73.9% 125910 ± 13% interrupts.CPU0.LOC:Local_timer_interrupts 2775 ± 20% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts 2775 ± 20% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts 422063 -83.9% 67902 ± 7% interrupts.CPU1.LOC:Local_timer_interrupts 1143 ± 29% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts 1143 ± 29% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts 13109 ± 47% +461.8% 73651 ± 45% interrupts.CPU1.RES:Rescheduling_interrupts 419980 -85.1% 62391 ± 40% interrupts.CPU10.LOC:Local_timer_interrupts 1355 ± 97% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts 1355 ± 97% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts 5237 ± 5% +53.7% 8050 ± 19% interrupts.CPU10.RES:Rescheduling_interrupts 421767 -85.5% 61287 ± 36% interrupts.CPU11.LOC:Local_timer_interrupts 206.67 ± 9% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts 206.67 ± 9% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts 6373 ± 27% +1031.4% 72110 ±107% interrupts.CPU11.RES:Rescheduling_interrupts 432202 ± 3% -83.2% 72442 ± 54% interrupts.CPU12.LOC:Local_timer_interrupts 155.00 ± 11% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts 155.00 ± 11% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts 14555 ± 92% +134.6% 34139 ± 84% interrupts.CPU12.RES:Rescheduling_interrupts 424325 -82.2% 75517 ± 79% interrupts.CPU13.LOC:Local_timer_interrupts 1967 ±119% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts 1967 ±119% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts 422820 -82.1% 75823 ± 81% interrupts.CPU14.LOC:Local_timer_interrupts 326.00 ± 41% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts 326.00 ± 41% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts 423656 -89.0% 46695 ± 34% interrupts.CPU15.LOC:Local_timer_interrupts 257.00 ± 41% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts 257.00 ± 41% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts 48763 ± 10% +24.5% 60720 ± 17% interrupts.CPU15.TLB:TLB_shootdowns 421342 -86.4% 57263 ± 29% interrupts.CPU16.LOC:Local_timer_interrupts 231.00 ± 38% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts 231.00 ± 38% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts 422879 -87.1% 54629 ± 37% interrupts.CPU17.LOC:Local_timer_interrupts 294.67 ± 68% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts 294.67 ± 68% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts 422954 -83.9% 68120 ± 61% interrupts.CPU18.LOC:Local_timer_interrupts 146.00 ± 45% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts 146.00 ± 45% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts 421457 -85.9% 59547 ± 33% interrupts.CPU19.LOC:Local_timer_interrupts 377.00 ± 61% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts 377.00 ± 61% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts 420805 -86.9% 55334 ± 13% interrupts.CPU2.LOC:Local_timer_interrupts 610.33 ± 18% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 610.33 ± 18% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 423266 -79.8% 85589 ± 53% interrupts.CPU20.LOC:Local_timer_interrupts 737.33 ±100% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts 737.33 ±100% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts 426117 -87.9% 51500 ± 23% interrupts.CPU21.LOC:Local_timer_interrupts 367.33 ± 67% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts 367.33 ± 67% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts 420346 -86.7% 55867 ± 35% interrupts.CPU22.LOC:Local_timer_interrupts 337.00 ± 76% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts 337.00 ± 76% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts 423461 -86.0% 59321 ± 41% interrupts.CPU23.LOC:Local_timer_interrupts 293.67 ± 44% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts 293.67 ± 44% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts 435976 ± 4% -72.0% 122171 ± 19% interrupts.CPU24.LOC:Local_timer_interrupts 1982 ± 84% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts 1982 ± 84% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts 203699 ± 68% -75.7% 49434 ± 26% interrupts.CPU24.TLB:TLB_shootdowns 0.33 ±141% +1.5e+05% 505.75 ±171% interrupts.CPU25.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25 471193 ± 7% -82.5% 82593 ± 8% interrupts.CPU25.LOC:Local_timer_interrupts 1705 ± 60% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts 1705 ± 60% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts 125024 ± 29% -65.3% 43415 ± 26% interrupts.CPU25.TLB:TLB_shootdowns 451259 ± 3% -81.2% 84692 ± 59% interrupts.CPU26.LOC:Local_timer_interrupts 463.33 ± 47% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts 463.33 ± 47% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts 501807 ± 2% -84.8% 76345 ± 45% interrupts.CPU27.LOC:Local_timer_interrupts 813.67 ± 78% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts 813.67 ± 78% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts 1427 ± 85% -100.0% 0.00 interrupts.CPU28.64:PCI-MSI.31981597-edge.i40e-eth0-TxRx-28 443659 ± 5% -80.0% 88624 ± 41% interrupts.CPU28.LOC:Local_timer_interrupts 2541 ± 62% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts 2541 ± 62% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts 5256 ± 9% +32.8% 6981 ± 28% interrupts.CPU28.RES:Rescheduling_interrupts 68542 ± 21% -29.0% 48680 ± 22% interrupts.CPU28.TLB:TLB_shootdowns 452585 ± 5% -85.7% 64860 ± 47% interrupts.CPU29.LOC:Local_timer_interrupts 496.33 ± 25% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts 496.33 ± 25% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts 116678 ± 5% -10.5% 104448 ± 3% interrupts.CPU3.CAL:Function_call_interrupts 421987 -85.9% 59360 ± 32% interrupts.CPU3.LOC:Local_timer_interrupts 679.33 ± 61% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 679.33 ± 61% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 442571 ± 3% -83.0% 75259 ± 65% interrupts.CPU30.LOC:Local_timer_interrupts 615.33 ± 32% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts 615.33 ± 32% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts 426536 -84.1% 67873 ± 41% interrupts.CPU31.LOC:Local_timer_interrupts 598.00 ± 22% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts 598.00 ± 22% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts 424280 -88.2% 49948 ± 36% interrupts.CPU32.LOC:Local_timer_interrupts 578.67 ± 36% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts 578.67 ± 36% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts 483.67 ± 72% -99.9% 0.50 ±173% interrupts.CPU33.69:PCI-MSI.31981602-edge.i40e-eth0-TxRx-33 446640 ± 7% -87.1% 57754 ± 30% interrupts.CPU33.LOC:Local_timer_interrupts 802.00 ± 38% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts 802.00 ± 38% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts 452552 ± 6% -87.8% 55067 ± 35% interrupts.CPU34.LOC:Local_timer_interrupts 799.33 ± 23% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts 799.33 ± 23% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts 96233 ± 32% -56.5% 41873 ± 15% interrupts.CPU34.TLB:TLB_shootdowns 429842 -85.6% 61746 ± 34% interrupts.CPU35.LOC:Local_timer_interrupts 505.67 ± 20% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts 505.67 ± 20% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts 451468 ± 8% -87.7% 55592 ± 35% interrupts.CPU36.LOC:Local_timer_interrupts 782.33 ± 34% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts 782.33 ± 34% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts 448017 ± 9% -82.7% 77491 ± 51% interrupts.CPU37.LOC:Local_timer_interrupts 975.00 ± 20% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts 975.00 ± 20% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts 78458 ± 29% -45.4% 42858 ± 21% interrupts.CPU37.TLB:TLB_shootdowns 428530 -87.5% 53432 ± 28% interrupts.CPU38.LOC:Local_timer_interrupts 1171 ± 28% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts 1171 ± 28% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts 106813 ± 6% +12.8% 120520 ± 13% interrupts.CPU39.CAL:Function_call_interrupts 430675 -88.3% 50524 ± 31% interrupts.CPU39.LOC:Local_timer_interrupts 1020 ± 17% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts 1020 ± 17% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts 428055 -85.1% 63861 ± 12% interrupts.CPU4.LOC:Local_timer_interrupts 383.67 ± 82% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts 383.67 ± 82% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts 5791 ± 12% +46.8% 8504 ± 7% interrupts.CPU4.RES:Rescheduling_interrupts 454935 ± 8% -88.7% 51246 ± 37% interrupts.CPU40.LOC:Local_timer_interrupts 1039 ± 57% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts 1039 ± 57% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts 433623 -90.0% 43537 ± 19% interrupts.CPU41.LOC:Local_timer_interrupts 742.00 ± 51% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts 742.00 ± 51% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts 449487 ± 8% -87.7% 55489 ± 26% interrupts.CPU42.LOC:Local_timer_interrupts 1010 ± 44% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts 1010 ± 44% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts 422697 -89.6% 44024 ± 24% interrupts.CPU43.LOC:Local_timer_interrupts 1343 ± 34% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts 1343 ± 34% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts 442675 ± 6% -90.0% 44091 ± 31% interrupts.CPU44.LOC:Local_timer_interrupts 1034 ± 28% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts 1034 ± 28% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts 5340 ± 5% +71.1% 9135 ± 36% interrupts.CPU44.RES:Rescheduling_interrupts 426296 ± 2% -85.8% 60415 ± 52% interrupts.CPU45.LOC:Local_timer_interrupts 1203 ± 77% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts 1203 ± 77% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts 76232 ± 47% -43.5% 43041 ± 11% interrupts.CPU45.TLB:TLB_shootdowns 116299 ± 5% -7.4% 107672 interrupts.CPU46.CAL:Function_call_interrupts 446780 ± 6% -90.3% 43466 ± 30% interrupts.CPU46.LOC:Local_timer_interrupts 991.33 ± 67% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts 991.33 ± 67% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts 438134 -91.0% 39253 ± 26% interrupts.CPU47.LOC:Local_timer_interrupts 1090 ± 73% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts 1090 ± 73% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts 62325 ± 10% -41.6% 36410 ± 9% interrupts.CPU47.TLB:TLB_shootdowns 427838 -85.9% 60314 ± 29% interrupts.CPU48.LOC:Local_timer_interrupts 1190 ± 22% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts 1190 ± 22% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts 418693 -85.9% 58942 ± 36% interrupts.CPU49.LOC:Local_timer_interrupts 645.33 ± 16% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts 645.33 ± 16% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts 425294 -80.5% 82728 ± 60% interrupts.CPU5.LOC:Local_timer_interrupts 426.33 ± 68% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts 426.33 ± 68% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts 419619 -85.0% 62779 ± 25% interrupts.CPU50.LOC:Local_timer_interrupts 448.33 ± 47% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts 448.33 ± 47% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts 415509 -83.6% 67948 ± 27% interrupts.CPU51.LOC:Local_timer_interrupts 612.67 ± 57% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts 612.67 ± 57% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts 420477 -86.2% 58098 ± 17% interrupts.CPU52.LOC:Local_timer_interrupts 537.00 ± 73% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts 537.00 ± 73% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts 417727 -84.0% 66794 ± 32% interrupts.CPU53.LOC:Local_timer_interrupts 536.00 ± 54% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts 536.00 ± 54% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts 423457 -83.8% 68550 ± 25% interrupts.CPU54.LOC:Local_timer_interrupts 454.33 ± 70% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts 454.33 ± 70% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts 428178 ± 2% -86.1% 59682 ± 21% interrupts.CPU55.LOC:Local_timer_interrupts 474.67 ± 61% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts 474.67 ± 61% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts 422267 -86.2% 58099 ± 24% interrupts.CPU56.LOC:Local_timer_interrupts 515.00 ± 53% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 515.00 ± 53% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 4472 ± 18% +1055.5% 51682 ±152% interrupts.CPU56.RES:Rescheduling_interrupts 423243 -84.6% 65308 ± 28% interrupts.CPU57.LOC:Local_timer_interrupts 468.00 ± 56% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts 468.00 ± 56% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts 420091 -79.6% 85741 ± 65% interrupts.CPU58.LOC:Local_timer_interrupts 821.00 ± 42% -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts 821.00 ± 42% -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts 424974 ± 3% -80.2% 84166 ± 62% interrupts.CPU59.LOC:Local_timer_interrupts 308.33 ± 13% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts 308.33 ± 13% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts 421775 -82.8% 72670 ± 56% interrupts.CPU6.LOC:Local_timer_interrupts 389.00 ± 70% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts 389.00 ± 70% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts 5638 ± 23% +799.2% 50697 ±100% interrupts.CPU6.RES:Rescheduling_interrupts 421575 -83.7% 68678 ± 31% interrupts.CPU60.LOC:Local_timer_interrupts 198.67 ± 32% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts 198.67 ± 32% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts 424053 -87.0% 55303 ± 23% interrupts.CPU61.LOC:Local_timer_interrupts 2025 ±121% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts 2025 ±121% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts 417676 -85.7% 59583 ± 29% interrupts.CPU62.LOC:Local_timer_interrupts 297.67 ± 11% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts 297.67 ± 11% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts 417855 -83.3% 69865 ± 30% interrupts.CPU63.LOC:Local_timer_interrupts 291.67 ± 27% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts 291.67 ± 27% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts 415612 -85.4% 60816 ± 22% interrupts.CPU64.LOC:Local_timer_interrupts 230.67 ± 16% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts 230.67 ± 16% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts 42651 ± 9% +81.3% 77313 ± 57% interrupts.CPU64.TLB:TLB_shootdowns 427273 -85.5% 61866 ± 24% interrupts.CPU65.LOC:Local_timer_interrupts 329.33 ± 62% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts 329.33 ± 62% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts 417024 -85.6% 59899 ± 22% interrupts.CPU66.LOC:Local_timer_interrupts 213.33 ± 25% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts 213.33 ± 25% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts 419856 -87.7% 51631 ± 23% interrupts.CPU67.LOC:Local_timer_interrupts 378.67 ± 60% -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts 378.67 ± 60% -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts 417008 -86.4% 56788 ± 26% interrupts.CPU68.LOC:Local_timer_interrupts 680.33 ± 60% -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts 680.33 ± 60% -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts 425694 ± 3% -86.8% 56110 ± 24% interrupts.CPU69.LOC:Local_timer_interrupts 471.67 ± 49% -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts 471.67 ± 49% -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts 178.67 ± 7% +16.0% 207.25 ± 3% interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc 416426 -86.1% 57942 ± 39% interrupts.CPU7.LOC:Local_timer_interrupts 289.33 ± 35% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts 289.33 ± 35% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts 424944 -86.6% 56734 ± 25% interrupts.CPU70.LOC:Local_timer_interrupts 353.00 ± 34% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts 353.00 ± 34% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts 3687 ± 23% +1638.2% 64086 ±157% interrupts.CPU70.RES:Rescheduling_interrupts 426727 -86.8% 56294 ± 18% interrupts.CPU71.LOC:Local_timer_interrupts 281.33 ± 12% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts 281.33 ± 12% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts 67729 ±126% -92.3% 5209 ± 13% interrupts.CPU71.RES:Rescheduling_interrupts 422305 -85.9% 59484 ± 39% interrupts.CPU72.LOC:Local_timer_interrupts 2638 ± 76% -100.0% 0.00 interrupts.CPU72.NMI:Non-maskable_interrupts 2638 ± 76% -100.0% 0.00 interrupts.CPU72.PMI:Performance_monitoring_interrupts 426176 -85.0% 63723 ± 60% interrupts.CPU73.LOC:Local_timer_interrupts 1282 ± 35% -100.0% 0.00 interrupts.CPU73.NMI:Non-maskable_interrupts 1282 ± 35% -100.0% 0.00 interrupts.CPU73.PMI:Performance_monitoring_interrupts 419630 -87.6% 51872 ± 32% interrupts.CPU74.LOC:Local_timer_interrupts 815.67 ± 73% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts 815.67 ± 73% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts 129292 ± 4% -11.3% 114645 ± 6% interrupts.CPU75.CAL:Function_call_interrupts 422390 -88.0% 50639 ± 37% interrupts.CPU75.LOC:Local_timer_interrupts 1492 ±103% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts 1492 ±103% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts 63378 ± 15% -37.2% 39781 ± 19% interrupts.CPU75.TLB:TLB_shootdowns 421966 -84.7% 64707 ± 43% interrupts.CPU76.LOC:Local_timer_interrupts 2020 ± 59% -100.0% 0.00 interrupts.CPU76.NMI:Non-maskable_interrupts 2020 ± 59% -100.0% 0.00 interrupts.CPU76.PMI:Performance_monitoring_interrupts 130362 ± 6% -11.6% 115230 ± 5% interrupts.CPU77.CAL:Function_call_interrupts 430179 ± 2% -82.2% 76501 ± 78% interrupts.CPU77.LOC:Local_timer_interrupts 673.33 ± 19% -100.0% 0.00 interrupts.CPU77.NMI:Non-maskable_interrupts 673.33 ± 19% -100.0% 0.00 interrupts.CPU77.PMI:Performance_monitoring_interrupts 63423 ± 18% -37.4% 39715 ± 18% interrupts.CPU77.TLB:TLB_shootdowns 427777 -84.5% 66430 ± 55% interrupts.CPU78.LOC:Local_timer_interrupts 489.67 ± 28% -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts 489.67 ± 28% -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts 4354 ± 24% +3448.6% 154517 ± 74% interrupts.CPU78.RES:Rescheduling_interrupts 424013 -84.9% 63864 ± 46% interrupts.CPU79.LOC:Local_timer_interrupts 760.67 ± 45% -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts 760.67 ± 45% -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts 4838 ± 10% +1806.0% 92212 ±150% interrupts.CPU79.RES:Rescheduling_interrupts 422159 -86.5% 56853 ± 22% interrupts.CPU8.LOC:Local_timer_interrupts 466.33 ± 77% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts 466.33 ± 77% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts 422530 -80.5% 82595 ± 64% interrupts.CPU80.LOC:Local_timer_interrupts 667.33 ± 41% -100.0% 0.00 interrupts.CPU80.NMI:Non-maskable_interrupts 667.33 ± 41% -100.0% 0.00 interrupts.CPU80.PMI:Performance_monitoring_interrupts 131616 ± 4% -9.2% 119523 ± 5% interrupts.CPU81.CAL:Function_call_interrupts 424480 -80.3% 83564 ± 68% interrupts.CPU81.LOC:Local_timer_interrupts 789.67 ± 42% -100.0% 0.00 interrupts.CPU81.NMI:Non-maskable_interrupts 789.67 ± 42% -100.0% 0.00 interrupts.CPU81.PMI:Performance_monitoring_interrupts 421319 -81.5% 77765 ± 77% interrupts.CPU82.LOC:Local_timer_interrupts 1294 ± 61% -100.0% 0.00 interrupts.CPU82.NMI:Non-maskable_interrupts 1294 ± 61% -100.0% 0.00 interrupts.CPU82.PMI:Performance_monitoring_interrupts 3929 ± 33% +3208.2% 129979 ± 83% interrupts.CPU82.RES:Rescheduling_interrupts 419478 -87.6% 52116 ± 27% interrupts.CPU83.LOC:Local_timer_interrupts 753.33 ± 37% -100.0% 0.00 interrupts.CPU83.NMI:Non-maskable_interrupts 753.33 ± 37% -100.0% 0.00 interrupts.CPU83.PMI:Performance_monitoring_interrupts 95030 ± 22% -57.1% 40721 ± 20% interrupts.CPU83.TLB:TLB_shootdowns 425396 ± 2% -86.8% 56089 ± 21% interrupts.CPU84.LOC:Local_timer_interrupts 1297 ± 87% -100.0% 0.00 interrupts.CPU84.NMI:Non-maskable_interrupts 1297 ± 87% -100.0% 0.00 interrupts.CPU84.PMI:Performance_monitoring_interrupts 421549 -85.4% 61481 ± 32% interrupts.CPU85.LOC:Local_timer_interrupts 1353 ± 80% -100.0% 0.00 interrupts.CPU85.NMI:Non-maskable_interrupts 1353 ± 80% -100.0% 0.00 interrupts.CPU85.PMI:Performance_monitoring_interrupts 71485 ± 33% -42.7% 40939 ± 20% interrupts.CPU85.TLB:TLB_shootdowns 417922 -80.6% 80914 ± 60% interrupts.CPU86.LOC:Local_timer_interrupts 1802 ± 54% -100.0% 0.00 interrupts.CPU86.NMI:Non-maskable_interrupts 1802 ± 54% -100.0% 0.00 interrupts.CPU86.PMI:Performance_monitoring_interrupts 474648 ± 8% -85.0% 71349 ± 40% interrupts.CPU87.LOC:Local_timer_interrupts 2061 ± 49% -100.0% 0.00 interrupts.CPU87.NMI:Non-maskable_interrupts 2061 ± 49% -100.0% 0.00 interrupts.CPU87.PMI:Performance_monitoring_interrupts 429105 ± 4% -80.0% 85962 ± 51% interrupts.CPU88.LOC:Local_timer_interrupts 1652 ± 46% -100.0% 0.00 interrupts.CPU88.NMI:Non-maskable_interrupts 1652 ± 46% -100.0% 0.00 interrupts.CPU88.PMI:Performance_monitoring_interrupts 453471 ± 8% -80.8% 86957 ± 53% interrupts.CPU89.LOC:Local_timer_interrupts 1740 ± 67% -100.0% 0.00 interrupts.CPU89.NMI:Non-maskable_interrupts 1740 ± 67% -100.0% 0.00 interrupts.CPU89.PMI:Performance_monitoring_interrupts 422214 -83.7% 68952 ± 20% interrupts.CPU9.LOC:Local_timer_interrupts 359.33 ± 54% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 359.33 ± 54% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 437404 ± 2% -81.9% 79299 ± 55% interrupts.CPU90.LOC:Local_timer_interrupts 1978 ± 56% -100.0% 0.00 interrupts.CPU90.NMI:Non-maskable_interrupts 1978 ± 56% -100.0% 0.00 interrupts.CPU90.PMI:Performance_monitoring_interrupts 455839 ± 7% -82.2% 81040 ± 41% interrupts.CPU91.LOC:Local_timer_interrupts 1383 ± 75% -100.0% 0.00 interrupts.CPU91.NMI:Non-maskable_interrupts 1383 ± 75% -100.0% 0.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts 467301 ± 5% -83.0% 79477 ± 50% interrupts.CPU92.LOC:Local_timer_interrupts 2345 ± 52% -100.0% 0.00 interrupts.CPU92.NMI:Non-maskable_interrupts 2345 ± 52% -100.0% 0.00 interrupts.CPU92.PMI:Performance_monitoring_interrupts 491381 ± 7% -88.3% 57449 ± 19% interrupts.CPU93.LOC:Local_timer_interrupts 1036 ± 51% -100.0% 0.00 interrupts.CPU93.NMI:Non-maskable_interrupts 1036 ± 51% -100.0% 0.00 interrupts.CPU93.PMI:Performance_monitoring_interrupts 461963 ± 8% -83.4% 76571 ± 32% interrupts.CPU94.LOC:Local_timer_interrupts 1500 ± 66% -100.0% 0.00 interrupts.CPU94.NMI:Non-maskable_interrupts 1500 ± 66% -100.0% 0.00 interrupts.CPU94.PMI:Performance_monitoring_interrupts 456548 ± 7% -83.8% 73929 ± 48% interrupts.CPU95.LOC:Local_timer_interrupts 1611 ± 62% -100.0% 0.00 interrupts.CPU95.NMI:Non-maskable_interrupts 1611 ± 62% -100.0% 0.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts 41453841 -84.8% 6296789 interrupts.LOC:Local_timer_interrupts 85079 ± 2% -100.0% 0.00 interrupts.NMI:Non-maskable_interrupts 85079 ± 2% -100.0% 0.00 interrupts.PMI:Performance_monitoring_interrupts 3246266 ± 9% +18.3% 3840488 ± 9% interrupts.RES:Rescheduling_interrupts Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

8 months, 3 weeks

1
0
0 / 0

[x86/head/64] e38f5f75b4: BUG:kernel_hang_in_boot_stage

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-7): commit: e38f5f75b4eb132530d90926d57eb4291ac28c0f ("x86/head/64: Setup TSS early for secondary CPUs") https://git.kernel.org/cgit/linux/kernel/git/joro/linux.git sev-es-client-v5.7-rc3 in testcase: boot on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +-------------------------------+------------+------------+ | | eee880e19b | e38f5f75b4 | +-------------------------------+------------+------------+ | boot_successes | 4 | 0 | | boot_failures | 0 | 11 | | BUG:kernel_hang_in_boot_stage | 0 | 11 | +-------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> needed_size: 0x0000000002000000 trampoline_32bit: 0x000000000009d000 Decompressing Linux... Parsing ELF... done. Booting the kernel. BUG: kernel hang in boot stage Linux version 5.7.0-rc3-00070-ge38f5f75b4eb1 #1 Command line: ip=::::vm-snb-ssd-35::dhcp root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-ssd-35/boot-1-quantal-x86_64-core-20190426.cgz-e38f5f75b4eb132530d90926d57eb4291ac28c0f-20200507-3373-105sgo3-2.yaml ARCH=x86_64 kconfig=x86_64-kexec branch=linux-devel/devel-hourly-2020050700 commit=e38f5f75b4eb132530d90926d57eb4291ac28c0f BOOT_IMAGE=/pkg/linux/x86_64-kexec/gcc-7/e38f5f75b4eb132530d90926d57eb4291ac28c0f/vmlinuz-5.7.0-rc3-00070-ge38f5f75b4eb1 max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb-ssd/quantal-x86_64-core-20190426.cgz/x86_64-kexec/gcc-7/e38f5f75b4eb132530d90926d57eb4291ac28c0f/3 LKP_SERVER=inn selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw rcuperf.shutdown=0 watchdog_thresh=60 Elapsed time: 420 To reproduce: # build kernel cd linux cp config-5.7.0-rc3-00070-ge38f5f75b4eb1 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, lkp

8 months, 3 weeks

1
0
0 / 0

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP May 2020