April 2021 - LKP - Ml01.01.Org

Re: [block] 658f2fb7d2: fxmark.hdd_f2fs_dbench_client_72_directio.works/sec -21.4% regression

by Chien Nguyenmau

1 year, 3 months

1
0
0 / 0

[block] 658f2fb7d2: fxmark.hdd_f2fs_dbench_client_72_directio.works/sec -21.4% regression

by kernel test robot

Greeting, FYI, we noticed a -21.4% regression of fxmark.hdd_f2fs_dbench_client_72_directio.works/sec due to commit: commit: 658f2fb7d2139d7a218083b7e8688dd288c6512d ("[PATCH] block: fix io hung by block throttle") url: https://github.com/0day-ci/linux/commits/Junxiao-Bi/block-fix-io-hung-by-... base: https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git for-next in testcase: fxmark on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory with following parameters: disk: 1HDD media: hdd test: dbench_client fstype: f2fs directio: directio cpufreq_governor: performance ucode: 0x11 If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/directio/1HDD/f2fs/x86_64-rhel-8.3/hdd/debian-10.4-x86_64-20200603.cgz/lkp-knm01/dbench_client/fxmark/0x11 commit: 901b8e44f6 ("Merge branch 'for-5.13/io_uring' into for-next") 658f2fb7d2 ("block: fix io hung by block throttle") 901b8e44f6b08436 658f2fb7d2139d7a218083b7e86 ---------------- --------------------------- %stddev %change %stddev \ | \ 66.02 ± 2% -14.2% 56.65 ± 3% fxmark.hdd_f2fs_dbench_client_36_directio.sys_sec 3.04 ± 2% -14.3% 2.61 ± 3% fxmark.hdd_f2fs_dbench_client_36_directio.sys_util 19.47 ± 3% -13.4% 16.85 ± 3% fxmark.hdd_f2fs_dbench_client_36_directio.user_sec 0.90 ± 3% -13.5% 0.78 ± 3% fxmark.hdd_f2fs_dbench_client_36_directio.user_util 128.18 ± 2% -18.2% 104.90 ± 5% fxmark.hdd_f2fs_dbench_client_36_directio.works/sec 8.65 -9.0% 7.87 ± 3% fxmark.hdd_f2fs_dbench_client_54_directio.softirq_sec 0.27 -9.1% 0.24 ± 3% fxmark.hdd_f2fs_dbench_client_54_directio.softirq_util 103.82 ± 3% -20.8% 82.22 ± 3% fxmark.hdd_f2fs_dbench_client_54_directio.sys_sec 3.19 ± 3% -20.9% 2.52 ± 3% fxmark.hdd_f2fs_dbench_client_54_directio.sys_util 29.38 ± 2% -18.4% 23.98 fxmark.hdd_f2fs_dbench_client_54_directio.user_sec 0.90 ± 2% -18.5% 0.74 fxmark.hdd_f2fs_dbench_client_54_directio.user_util 162.84 ± 4% -24.8% 122.39 ± 3% fxmark.hdd_f2fs_dbench_client_54_directio.works/sec 10.29 -8.7% 9.40 ± 3% fxmark.hdd_f2fs_dbench_client_72_directio.softirq_sec 0.24 -8.8% 0.22 ± 3% fxmark.hdd_f2fs_dbench_client_72_directio.softirq_util 132.66 ± 2% -19.3% 107.01 ± 2% fxmark.hdd_f2fs_dbench_client_72_directio.sys_sec 3.05 ± 2% -19.4% 2.46 ± 2% fxmark.hdd_f2fs_dbench_client_72_directio.sys_util 29.72 ± 3% -18.2% 24.31 ± 2% fxmark.hdd_f2fs_dbench_client_72_directio.user_sec 0.68 ± 3% -18.3% 0.56 ± 2% fxmark.hdd_f2fs_dbench_client_72_directio.user_util 183.55 ± 3% -21.4% 144.32 ± 3% fxmark.hdd_f2fs_dbench_client_72_directio.works/sec 26566804 -9.9% 23936362 fxmark.time.file_system_outputs 72184 -2.6% 70322 fxmark.time.involuntary_context_switches 88.33 -15.9% 74.29 fxmark.time.percent_of_cpu_this_job_got 357.13 -15.9% 300.26 fxmark.time.system_time 91.39 -14.5% 78.18 fxmark.time.user_time 930788 ± 3% -30.6% 645881 ± 2% fxmark.time.voluntary_context_switches 9.40 -2.7% 9.14 iostat.cpu.system 2.70 -2.0% 2.64 iostat.cpu.user 6281 ± 6% -21.4% 4939 ± 8% meminfo.Active(anon) 71273 ± 4% -20.3% 56818 ± 7% meminfo.Shmem 6267 ± 5% -21.1% 4946 ± 8% numa-meminfo.node0.Active(anon) 71236 ± 4% -20.2% 56833 ± 7% numa-meminfo.node0.Shmem 6729 -12.4% 5897 vmstat.io.bo 6385 ± 2% -17.6% 5262 vmstat.system.cs 6088 ± 2% -18.6% 4958 perf-stat.i.context-switches 176.01 -15.1% 149.46 perf-stat.i.cpu-migrations 6074 ± 2% -18.6% 4945 perf-stat.ps.context-switches 175.89 -15.1% 149.36 perf-stat.ps.cpu-migrations 1570 ± 6% -21.5% 1233 ± 8% numa-vmstat.node0.nr_active_anon 2302914 -11.0% 2050036 numa-vmstat.node0.nr_dirtied 17815 ± 4% -20.3% 14202 ± 7% numa-vmstat.node0.nr_shmem 540895 -14.6% 461814 numa-vmstat.node0.nr_written 1570 ± 6% -21.5% 1233 ± 8% numa-vmstat.node0.nr_zone_active_anon 2679 ± 4% -15.2% 2272 ± 4% slabinfo.dmaengine-unmap-16.active_objs 3266 ± 7% -13.0% 2841 ± 4% slabinfo.dmaengine-unmap-16.num_objs 5826 ± 4% -12.6% 5093 ± 3% slabinfo.ext4_io_end.active_objs 5994 ± 4% -12.3% 5257 ± 3% slabinfo.ext4_io_end.num_objs 18698 ± 2% -9.5% 16920 slabinfo.ext4_pending_reservation.active_objs 18783 ± 2% -9.5% 17007 slabinfo.ext4_pending_reservation.num_objs 32425 ± 2% -11.6% 28671 ± 2% slabinfo.f2fs_free_nid.active_objs 32622 ± 2% -11.6% 28825 ± 2% slabinfo.f2fs_free_nid.num_objs 1584 ± 5% -22.1% 1233 ± 8% proc-vmstat.nr_active_anon 20434 -2.0% 20027 proc-vmstat.nr_active_file 3316154 -9.9% 2986902 proc-vmstat.nr_dirtied 405121 -0.9% 401332 proc-vmstat.nr_file_pages 82265 -4.4% 78653 proc-vmstat.nr_inactive_anon 17866 ± 4% -20.2% 14252 ± 7% proc-vmstat.nr_shmem 850118 -12.4% 744638 proc-vmstat.nr_written 1584 ± 5% -22.1% 1233 ± 8% proc-vmstat.nr_zone_active_anon 20434 -2.0% 20027 proc-vmstat.nr_zone_active_file 82265 -4.4% 78653 proc-vmstat.nr_zone_inactive_anon 4971269 -6.1% 4667688 proc-vmstat.numa_hit 4971268 -6.1% 4667687 proc-vmstat.numa_local 293150 -4.6% 279668 proc-vmstat.pgactivate 5110253 -5.9% 4808331 proc-vmstat.pgalloc_normal 144088 -17.3% 119141 proc-vmstat.pgdeactivate 5087969 -5.8% 4791351 proc-vmstat.pgfree 3400472 -12.4% 2978555 proc-vmstat.pgpgout 126836 -16.8% 105510 proc-vmstat.pgrotated 616731 -5.1% 585318 interrupts.CAL:Function_call_interrupts 6785 ± 4% -11.8% 5983 ± 4% interrupts.CPU11.CAL:Function_call_interrupts 6543 ± 3% -12.3% 5735 ± 3% interrupts.CPU17.CAL:Function_call_interrupts 34.17 ± 19% +185.6% 97.57 ±108% interrupts.CPU18.TLB:TLB_shootdowns 402.33 ± 33% -42.1% 232.86 ± 21% interrupts.CPU2.NMI:Non-maskable_interrupts 402.33 ± 33% -42.1% 232.86 ± 21% interrupts.CPU2.PMI:Performance_monitoring_interrupts 5327 ± 6% -15.6% 4496 ± 6% interrupts.CPU20.CAL:Function_call_interrupts 5017 ± 4% -12.6% 4384 ± 6% interrupts.CPU21.CAL:Function_call_interrupts 5106 ± 4% -12.6% 4461 ± 6% interrupts.CPU22.CAL:Function_call_interrupts 5046 ± 3% -13.0% 4389 ± 5% interrupts.CPU25.CAL:Function_call_interrupts 5126 ± 2% -13.7% 4422 ± 5% interrupts.CPU26.CAL:Function_call_interrupts 5025 ± 5% -14.1% 4318 ± 3% interrupts.CPU27.CAL:Function_call_interrupts 5074 ± 5% -12.3% 4451 ± 4% interrupts.CPU28.CAL:Function_call_interrupts 4985 ± 3% -12.0% 4386 ± 3% interrupts.CPU29.CAL:Function_call_interrupts 5168 ± 7% -14.2% 4436 ± 6% interrupts.CPU31.CAL:Function_call_interrupts 5390 ± 10% -16.7% 4492 ± 8% interrupts.CPU33.CAL:Function_call_interrupts 4929 ± 2% -11.1% 4379 ± 5% interrupts.CPU34.CAL:Function_call_interrupts 4942 ± 3% -11.7% 4366 ± 5% interrupts.CPU35.CAL:Function_call_interrupts 3625 ± 3% -8.2% 3329 ± 5% interrupts.CPU39.CAL:Function_call_interrupts 3658 ± 4% -10.0% 3290 ± 3% interrupts.CPU40.CAL:Function_call_interrupts 3825 ± 6% -16.3% 3203 ± 6% interrupts.CPU42.CAL:Function_call_interrupts 3665 ± 4% -15.7% 3088 ± 10% interrupts.CPU43.CAL:Function_call_interrupts 3664 ± 6% -10.5% 3279 ± 6% interrupts.CPU44.CAL:Function_call_interrupts 3788 ± 7% -13.4% 3280 ± 4% interrupts.CPU46.CAL:Function_call_interrupts 3895 ± 15% -16.3% 3261 ± 4% interrupts.CPU47.CAL:Function_call_interrupts 3945 ± 7% -15.0% 3352 ± 5% interrupts.CPU48.CAL:Function_call_interrupts 3838 ± 10% -15.7% 3237 ± 4% interrupts.CPU49.CAL:Function_call_interrupts 6927 ± 5% -13.5% 5991 ± 8% interrupts.CPU5.CAL:Function_call_interrupts 3459 ± 6% -13.1% 3005 ± 6% interrupts.CPU53.CAL:Function_call_interrupts 2564 ± 3% -10.2% 2302 ± 7% interrupts.CPU54.CAL:Function_call_interrupts 2517 ± 2% -8.9% 2294 ± 6% interrupts.CPU56.CAL:Function_call_interrupts 2517 ± 3% -10.7% 2246 ± 7% interrupts.CPU70.CAL:Function_call_interrupts 20310 ± 4% -13.1% 17648 ± 8% interrupts.NMI:Non-maskable_interrupts 20310 ± 4% -13.1% 17648 ± 8% interrupts.PMI:Performance_monitoring_interrupts 0.06 ± 91% -86.0% 0.01 ±130% perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.__get_node_page.f2fs_update_inode_page 0.04 ± 6% +16.1% 0.05 ± 8% perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file 0.03 ± 9% +110.0% 0.07 ± 97% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages 0.67 ± 66% -66.9% 0.22 ± 43% perf-sched.sch_delay.max.ms.issue_flush_thread.kthread.ret_from_fork 0.19 ± 89% -93.9% 0.01 ±127% perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.__get_node_page.f2fs_update_inode_page 0.45 ± 43% -56.5% 0.20 ± 68% perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.f2fs_submit_merged_ipu_write.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback 8.98 ±198% -93.0% 0.63 ± 28% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page 0.08 ± 34% -41.0% 0.05 ± 38% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 45.26 ± 6% +28.5% 58.17 ± 7% perf-sched.total_wait_and_delay.average.ms 61502 ± 6% -27.7% 44475 ± 11% perf-sched.total_wait_and_delay.count.ms 45.23 ± 6% +28.5% 58.14 ± 7% perf-sched.total_wait_time.average.ms 6.59 ± 10% +39.7% 9.20 ± 10% perf-sched.wait_and_delay.avg.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.f2fs_wait_on_page_writeback 4.48 ± 9% -53.8% 2.07 ± 71% perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.f2fs_need_inode_block_update.f2fs_do_sync_file.do_fsync 9.17 ± 14% -31.8% 6.25 ± 11% perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages 7540 ± 11% -26.6% 5536 ± 17% perf-sched.wait_and_delay.count.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range 19838 ± 8% -35.6% 12768 ± 15% perf-sched.wait_and_delay.count.io_schedule.wait_on_page_bit.wait_on_page_writeback.f2fs_wait_on_page_writeback 911.33 ± 6% -29.8% 639.43 ± 10% perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file 2795 ± 6% -43.1% 1589 ± 20% perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback 491.67 ± 14% -46.0% 265.29 ± 49% perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_get_node_info.__write_node_page.f2fs_fsync_node_pages 1726 ± 28% -48.5% 889.29 ± 42% perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_do_write_data_page.f2fs_write_single_data_page 839.00 ± 31% -59.6% 338.57 ± 56% perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_get_node_info.truncate_node.f2fs_remove_inode_page 509.67 ± 12% -46.6% 272.00 ± 48% perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_need_dentry_mark.f2fs_fsync_node_pages.f2fs_do_sync_file 497.83 ± 12% -58.8% 205.00 ± 67% perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_need_inode_block_update.f2fs_do_sync_file.do_fsync 267.17 ± 23% -75.7% 64.86 ±128% perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.f2fs_merge_page_bio.f2fs_inplace_write_data.f2fs_do_write_data_page 1539 ± 5% -49.4% 779.14 ± 15% perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page 791.00 ± 13% -45.2% 433.57 ± 27% perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages 910.67 ± 20% -54.1% 417.86 ± 38% perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page 856.83 ± 29% -60.2% 341.14 ± 57% perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page 3465 ± 2% -21.4% 2725 ± 8% perf-sched.wait_and_delay.count.schedule_timeout.wait_for_completion.f2fs_issue_flush.f2fs_do_sync_file 142.28 ± 8% +1126.8% 1745 ± 97% perf-sched.wait_and_delay.max.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range 5186 ± 2% +22.8% 6370 ± 11% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork 6.56 ± 10% +39.7% 9.17 ± 10% perf-sched.wait_time.avg.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.f2fs_wait_on_page_writeback 15.00 ± 88% -75.6% 3.65 ±165% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.__get_node_page.f2fs_find_in_inline_dir 2.74 ± 17% +90.3% 5.22 ± 24% perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.f2fs_get_node_info.__write_node_page.f2fs_fsync_node_pages 4.46 ± 9% -43.4% 2.52 ± 35% perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.f2fs_need_inode_block_update.f2fs_do_sync_file.do_fsync 9.14 ± 14% -32.3% 6.18 ± 12% perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages 142.23 ± 8% +1127.2% 1745 ± 97% perf-sched.wait_time.max.ms.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range 33.23 ± 80% -62.6% 12.44 ±158% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mnt_want_write.do_unlinkat.do_syscall_64 52.12 ± 53% -73.9% 13.59 ±161% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.__get_node_page.f2fs_update_inode_page 5186 ± 2% +22.8% 6370 ± 11% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork 43.08 ± 8% -8.5 34.60 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 41.46 ± 9% -8.2 33.29 ± 11% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.91 ± 10% -1.2 3.67 ± 12% perf-profile.calltrace.cycles-pp.vfs_write.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.94 ± 10% -1.2 3.70 ± 12% perf-profile.calltrace.cycles-pp.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.77 ± 11% -1.2 3.57 ± 12% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.72 ± 11% -1.2 3.53 ± 12% perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_pwrite64.do_syscall_64 5.97 ± 9% -1.2 4.80 ± 11% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64 6.06 ± 9% -1.2 4.89 ± 10% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.52 ± 13% -0.9 2.60 ± 14% perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.06 ± 12% -0.8 2.30 ± 14% perf-profile.calltrace.cycles-pp.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_pwrite64 2.62 ± 15% -0.7 1.91 ± 16% perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.52 ± 15% -0.7 1.82 ± 16% perf-profile.calltrace.cycles-pp.f2fs_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.45 ± 5% -0.7 2.79 ± 11% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.16 ± 15% -0.6 1.52 ± 15% perf-profile.calltrace.cycles-pp.f2fs_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open 3.30 ± 6% -0.6 2.67 ± 11% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.43 ± 9% -0.6 1.84 ± 10% perf-profile.calltrace.cycles-pp.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.do_fsync.__x64_sys_fsync.do_syscall_64 2.37 ± 9% -0.6 1.80 ± 11% perf-profile.calltrace.cycles-pp.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.do_fsync.__x64_sys_fsync 2.19 ± 6% -0.4 1.75 ± 10% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64 1.39 ± 16% -0.4 0.96 ± 17% perf-profile.calltrace.cycles-pp.f2fs_do_add_link.f2fs_create.path_openat.do_filp_open.do_sys_openat2 1.34 ± 16% -0.4 0.91 ± 18% perf-profile.calltrace.cycles-pp.f2fs_add_dentry.f2fs_do_add_link.f2fs_create.path_openat.do_filp_open 1.33 ± 16% -0.4 0.90 ± 18% perf-profile.calltrace.cycles-pp.f2fs_add_inline_entry.f2fs_add_dentry.f2fs_do_add_link.f2fs_create.path_openat 1.71 ± 15% -0.4 1.29 ± 12% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 1.02 ± 14% -0.4 0.60 ± 43% perf-profile.calltrace.cycles-pp.f2fs_remove_inode_page.f2fs_evict_inode.evict.do_unlinkat.do_syscall_64 1.69 ± 15% -0.4 1.27 ± 12% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.59 ± 7% -0.4 1.18 ± 11% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat 1.47 ± 9% -0.4 1.06 ± 8% perf-profile.calltrace.cycles-pp.f2fs_preallocate_blocks.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_pwrite64 0.94 ± 14% -0.4 0.54 ± 44% perf-profile.calltrace.cycles-pp.truncate_node.f2fs_remove_inode_page.f2fs_evict_inode.evict.do_unlinkat 1.78 ± 10% -0.4 1.42 ± 16% perf-profile.calltrace.cycles-pp.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.74 ± 11% -0.3 1.39 ± 17% perf-profile.calltrace.cycles-pp.vfs_read.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.77 ± 15% -0.3 0.45 ± 65% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch 1.07 ± 7% -0.3 0.75 ± 9% perf-profile.calltrace.cycles-pp.f2fs_convert_inline_inode.f2fs_preallocate_blocks.f2fs_file_write_iter.new_sync_write.vfs_write 0.61 ± 9% -0.3 0.31 ± 86% perf-profile.calltrace.cycles-pp.user_path_at_empty.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.62 ± 12% -0.3 0.32 ± 87% perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.f2fs_evict_inode.evict.do_unlinkat.do_syscall_64 1.46 ± 10% -0.3 1.18 ± 20% perf-profile.calltrace.cycles-pp.f2fs_file_read_iter.new_sync_read.vfs_read.ksys_pread64.do_syscall_64 1.39 ± 10% -0.3 1.11 ± 19% perf-profile.calltrace.cycles-pp.filemap_read.f2fs_file_read_iter.new_sync_read.vfs_read.ksys_pread64 0.91 ± 8% -0.3 0.64 ± 13% perf-profile.calltrace.cycles-pp.f2fs_convert_inline_page.f2fs_convert_inline_inode.f2fs_preallocate_blocks.f2fs_file_write_iter.new_sync_write 1.40 ± 9% -0.3 1.13 ± 15% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe 1.00 ± 15% -0.2 0.77 ± 14% perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.file_write_and_wait_range.f2fs_do_sync_file.do_fsync.__x64_sys_fsync 0.95 ± 15% -0.2 0.72 ± 15% perf-profile.calltrace.cycles-pp.wait_on_page_writeback.__filemap_fdatawait_range.file_write_and_wait_range.f2fs_do_sync_file.do_fsync 0.94 ± 15% -0.2 0.71 ± 15% perf-profile.calltrace.cycles-pp.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range.file_write_and_wait_range.f2fs_do_sync_file 0.91 ± 15% -0.2 0.70 ± 16% perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range.file_write_and_wait_range 0.90 ± 15% -0.2 0.69 ± 15% perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit.wait_on_page_writeback.__filemap_fdatawait_range 1.16 ± 10% -0.2 0.96 ± 15% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe 1.12 ± 9% -0.2 0.93 ± 13% perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open 0.83 ± 9% -0.2 0.66 ± 10% perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_lookupat.filename_lookup.vfs_statx 0.74 ± 9% +0.1 0.88 ± 8% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start 0.82 ± 8% +0.1 0.96 ± 7% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu 1.16 ± 4% +0.2 1.38 ± 9% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt 1.36 ± 4% +0.2 1.61 ± 7% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.53 ± 4% +0.3 1.84 ± 11% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 1.05 ± 5% +0.3 1.36 ± 7% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 3.10 ± 2% +0.7 3.80 ± 9% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 3.54 ± 2% +0.7 4.26 ± 8% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 3.35 ± 2% +0.8 4.12 ± 10% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 4.14 ± 3% +0.8 4.97 ± 9% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 43.33 ± 7% +8.1 51.42 ± 9% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 42.38 ± 7% +8.2 50.54 ± 9% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 42.46 ± 7% +8.2 50.64 ± 9% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 42.46 ± 7% +8.2 50.64 ± 9% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 48.00 ± 6% -7.9 40.13 ± 9% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 46.03 ± 7% -7.6 38.47 ± 9% perf-profile.children.cycles-pp.do_syscall_64 8.17 ± 10% -1.9 6.28 ± 10% perf-profile.children.cycles-pp.__schedule 6.38 ± 9% -1.5 4.89 ± 11% perf-profile.children.cycles-pp.schedule 6.11 ± 10% -1.3 4.77 ± 12% perf-profile.children.cycles-pp.perf_tp_event 5.66 ± 9% -1.3 4.33 ± 10% perf-profile.children.cycles-pp.vfs_write 5.49 ± 9% -1.3 4.20 ± 10% perf-profile.children.cycles-pp.new_sync_write 5.86 ± 11% -1.3 4.58 ± 12% perf-profile.children.cycles-pp.perf_swevent_overflow 5.83 ± 11% -1.3 4.55 ± 12% perf-profile.children.cycles-pp.__perf_event_overflow 5.77 ± 11% -1.3 4.50 ± 12% perf-profile.children.cycles-pp.perf_event_output_forward 4.94 ± 10% -1.2 3.70 ± 12% perf-profile.children.cycles-pp.ksys_pwrite64 4.72 ± 11% -1.2 3.53 ± 12% perf-profile.children.cycles-pp.f2fs_file_write_iter 4.99 ± 11% -1.2 3.81 ± 13% perf-profile.children.cycles-pp.perf_prepare_sample 4.50 ± 11% -1.1 3.40 ± 13% perf-profile.children.cycles-pp.get_perf_callchain 4.52 ± 12% -1.1 3.43 ± 13% perf-profile.children.cycles-pp.perf_callchain 3.69 ± 12% -1.0 2.73 ± 12% perf-profile.children.cycles-pp.perf_callchain_kernel 3.52 ± 13% -0.9 2.60 ± 14% perf-profile.children.cycles-pp.do_unlinkat 3.54 ± 11% -0.8 2.72 ± 12% perf-profile.children.cycles-pp.__generic_file_write_iter 3.31 ± 13% -0.8 2.55 ± 12% perf-profile.children.cycles-pp.generic_perform_write 3.08 ± 10% -0.7 2.35 ± 10% perf-profile.children.cycles-pp.io_schedule 2.64 ± 15% -0.7 1.93 ± 16% perf-profile.children.cycles-pp.evict 2.53 ± 15% -0.7 1.83 ± 16% perf-profile.children.cycles-pp.f2fs_evict_inode 3.14 ± 9% -0.7 2.44 ± 11% perf-profile.children.cycles-pp.wait_on_page_writeback 2.93 ± 8% -0.7 2.24 ± 11% perf-profile.children.cycles-pp.f2fs_wait_on_page_writeback 3.11 ± 9% -0.7 2.41 ± 11% perf-profile.children.cycles-pp.wait_on_page_bit 1.45 ± 20% -0.7 0.79 ± 22% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 2.16 ± 15% -0.6 1.52 ± 15% perf-profile.children.cycles-pp.f2fs_create 3.93 ± 4% -0.6 3.33 ± 10% perf-profile.children.cycles-pp.__do_sys_newstat 3.79 ± 5% -0.6 3.20 ± 10% perf-profile.children.cycles-pp.vfs_statx 2.43 ± 9% -0.6 1.84 ± 10% perf-profile.children.cycles-pp.f2fs_wait_on_node_pages_writeback 3.30 ± 6% -0.6 2.71 ± 10% perf-profile.children.cycles-pp.link_path_walk 2.20 ± 13% -0.6 1.65 ± 13% perf-profile.children.cycles-pp.perf_trace_sched_switch 1.26 ± 11% -0.5 0.74 ± 21% perf-profile.children.cycles-pp.rwsem_down_read_slowpath 1.54 ± 15% -0.5 1.03 ± 17% perf-profile.children.cycles-pp.do_write_page 0.98 ± 22% -0.5 0.47 ± 18% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.96 ± 13% -0.5 1.47 ± 16% perf-profile.children.cycles-pp.pagecache_get_page 2.11 ± 11% -0.5 1.63 ± 9% perf-profile.children.cycles-pp.dequeue_task_fair 2.33 ± 11% -0.5 1.85 ± 15% perf-profile.children.cycles-pp.newidle_balance 1.45 ± 16% -0.5 0.99 ± 16% perf-profile.children.cycles-pp.f2fs_do_add_link 2.02 ± 11% -0.4 1.57 ± 10% perf-profile.children.cycles-pp.dequeue_entity 1.39 ± 16% -0.4 0.94 ± 17% perf-profile.children.cycles-pp.f2fs_add_dentry 1.38 ± 16% -0.4 0.94 ± 17% perf-profile.children.cycles-pp.f2fs_add_inline_entry 1.74 ± 15% -0.4 1.30 ± 12% perf-profile.children.cycles-pp.schedule_idle 3.00 ± 5% -0.4 2.58 ± 9% perf-profile.children.cycles-pp.filename_lookup 1.48 ± 9% -0.4 1.06 ± 8% perf-profile.children.cycles-pp.f2fs_preallocate_blocks 1.78 ± 10% -0.4 1.42 ± 16% perf-profile.children.cycles-pp.ksys_pread64 1.02 ± 14% -0.4 0.67 ± 18% perf-profile.children.cycles-pp.f2fs_remove_inode_page 0.95 ± 14% -0.3 0.60 ± 21% perf-profile.children.cycles-pp.truncate_node 1.10 ± 16% -0.3 0.75 ± 21% perf-profile.children.cycles-pp.f2fs_outplace_write_data 0.78 ± 19% -0.3 0.44 ± 20% perf-profile.children.cycles-pp.rwsem_wake 1.09 ± 7% -0.3 0.76 ± 9% perf-profile.children.cycles-pp.f2fs_convert_inline_inode 1.38 ± 11% -0.3 1.06 ± 11% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template 0.76 ± 20% -0.3 0.44 ± 20% perf-profile.children.cycles-pp.wake_up_q 1.50 ± 9% -0.3 1.20 ± 17% perf-profile.children.cycles-pp.filemap_read 0.81 ± 15% -0.3 0.52 ± 16% perf-profile.children.cycles-pp.f2fs_submit_page_write 0.63 ± 17% -0.3 0.34 ± 42% perf-profile.children.cycles-pp.f2fs_get_node_info 1.39 ± 12% -0.3 1.10 ± 15% perf-profile.children.cycles-pp.__unwind_start 0.52 ± 29% -0.3 0.24 ± 50% perf-profile.children.cycles-pp.osq_lock 1.46 ± 10% -0.3 1.18 ± 20% perf-profile.children.cycles-pp.f2fs_file_read_iter 1.19 ± 14% -0.3 0.91 ± 16% perf-profile.children.cycles-pp._raw_spin_lock 0.91 ± 8% -0.3 0.64 ± 13% perf-profile.children.cycles-pp.f2fs_convert_inline_page 0.95 ± 11% -0.2 0.70 ± 15% perf-profile.children.cycles-pp.__orc_find 1.18 ± 5% -0.2 0.93 ± 7% perf-profile.children.cycles-pp.__submit_merged_write_cond 1.00 ± 15% -0.2 0.77 ± 14% perf-profile.children.cycles-pp.__filemap_fdatawait_range 2.31 ± 7% -0.2 2.08 ± 10% perf-profile.children.cycles-pp.walk_component 0.84 ± 12% -0.2 0.63 ± 13% perf-profile.children.cycles-pp.inode_permission 0.78 ± 11% -0.2 0.57 ± 20% perf-profile.children.cycles-pp.add_to_page_cache_lru 0.65 ± 12% -0.2 0.46 ± 17% perf-profile.children.cycles-pp.__add_to_page_cache_locked 0.64 ± 11% -0.2 0.45 ± 12% perf-profile.children.cycles-pp.orc_find 0.36 ± 24% -0.2 0.19 ± 25% perf-profile.children.cycles-pp.f2fs_invalidate_blocks 0.53 ± 24% -0.2 0.37 ± 16% perf-profile.children.cycles-pp.generic_permission 0.58 ± 21% -0.2 0.41 ± 13% perf-profile.children.cycles-pp.finish_task_switch 0.76 ± 25% -0.2 0.61 ± 12% perf-profile.children.cycles-pp.kmem_cache_free 0.57 ± 10% -0.1 0.43 ± 14% perf-profile.children.cycles-pp.sched_ttwu_pending 0.52 ± 14% -0.1 0.37 ± 18% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 0.48 ± 24% -0.1 0.34 ± 14% perf-profile.children.cycles-pp.make_kuid 0.48 ± 14% -0.1 0.35 ± 17% perf-profile.children.cycles-pp.sysvec_call_function_single 0.60 ± 10% -0.1 0.48 ± 15% perf-profile.children.cycles-pp.__entry_text_start 0.32 ± 21% -0.1 0.20 ± 21% perf-profile.children.cycles-pp.filemap_get_pages 0.67 ± 14% -0.1 0.55 ± 18% perf-profile.children.cycles-pp.step_into 0.45 ± 24% -0.1 0.34 ± 13% perf-profile.children.cycles-pp.map_id_range_down 0.60 ± 10% -0.1 0.49 ± 10% perf-profile.children.cycles-pp.vfs_unlink 0.33 ± 13% -0.1 0.24 ± 16% perf-profile.children.cycles-pp.do_renameat2 0.24 ± 16% -0.1 0.17 ± 24% perf-profile.children.cycles-pp.aa_get_task_label 0.17 ± 26% -0.1 0.11 ± 10% perf-profile.children.cycles-pp.filename_parentat 0.20 ± 15% -0.1 0.14 ± 24% perf-profile.children.cycles-pp.__sysvec_call_function_single 0.09 ± 24% -0.1 0.04 ± 63% perf-profile.children.cycles-pp.xas_create 0.14 ± 10% -0.1 0.09 ± 27% perf-profile.children.cycles-pp.set_page_dirty 0.15 ± 18% -0.1 0.10 ± 12% perf-profile.children.cycles-pp.path_parentat 0.12 ± 35% -0.0 0.07 ± 25% perf-profile.children.cycles-pp.truncate_cleanup_page 0.17 ± 9% -0.0 0.12 ± 17% perf-profile.children.cycles-pp.__f2fs_find_entry 0.14 ± 11% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.mem_cgroup_uncharge_list 0.11 ± 19% -0.0 0.07 ± 49% perf-profile.children.cycles-pp.set_task_cpu 0.26 ± 7% -0.0 0.22 ± 17% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.19 ± 14% -0.0 0.14 ± 16% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.15 ± 11% -0.0 0.11 ± 15% perf-profile.children.cycles-pp.f2fs_find_in_inline_dir 0.08 ± 18% -0.0 0.04 ± 65% perf-profile.children.cycles-pp.perf_trace_buf_update 0.12 ± 16% -0.0 0.09 ± 26% perf-profile.children.cycles-pp.__test_set_page_writeback 0.14 ± 13% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.__mark_inode_dirty 0.15 ± 10% -0.0 0.12 ± 11% perf-profile.children.cycles-pp.__list_add_valid 0.15 ± 14% +0.0 0.20 ± 8% perf-profile.children.cycles-pp.nr_iowait_cpu 0.30 ± 9% +0.1 0.37 ± 11% perf-profile.children.cycles-pp.mmput 0.29 ± 8% +0.1 0.37 ± 10% perf-profile.children.cycles-pp.exit_mmap 0.16 ± 11% +0.1 0.23 ± 16% perf-profile.children.cycles-pp.rb_next 0.03 ±100% +0.1 0.11 ± 32% perf-profile.children.cycles-pp.sys_imageblit 0.03 ±100% +0.1 0.11 ± 32% perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit 0.18 ± 10% +0.1 0.27 ± 19% perf-profile.children.cycles-pp.trigger_load_balance 0.50 ± 5% +0.1 0.58 ± 11% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.05 ± 45% +0.1 0.14 ± 42% perf-profile.children.cycles-pp.vt_console_print 1.06 ± 7% +0.2 1.26 ± 12% perf-profile.children.cycles-pp.update_blocked_averages 1.43 ± 5% +0.3 1.69 ± 7% perf-profile.children.cycles-pp.rebalance_domains 1.79 ± 5% +0.4 2.20 ± 11% perf-profile.children.cycles-pp.clockevents_program_event 43.33 ± 7% +8.1 51.42 ± 9% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 43.33 ± 7% +8.1 51.42 ± 9% perf-profile.children.cycles-pp.cpu_startup_entry 43.32 ± 7% +8.1 51.42 ± 9% perf-profile.children.cycles-pp.do_idle 42.46 ± 7% +8.2 50.64 ± 9% perf-profile.children.cycles-pp.start_secondary 0.49 ± 28% -0.3 0.23 ± 48% perf-profile.self.cycles-pp.osq_lock 0.94 ± 11% -0.3 0.69 ± 16% perf-profile.self.cycles-pp.__orc_find 1.07 ± 7% -0.2 0.85 ± 13% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.60 ± 10% -0.1 0.46 ± 16% perf-profile.self.cycles-pp.__entry_text_start 0.29 ± 12% -0.1 0.23 ± 15% perf-profile.self.cycles-pp.__schedule 0.17 ± 15% -0.1 0.12 ± 14% perf-profile.self.cycles-pp.update_curr 0.14 ± 16% -0.0 0.10 ± 20% perf-profile.self.cycles-pp.path_init 0.21 ± 6% -0.0 0.17 ± 14% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.09 ± 18% -0.0 0.05 ± 42% perf-profile.self.cycles-pp.current_time 0.17 ± 8% -0.0 0.13 ± 17% perf-profile.self.cycles-pp.f2fs_fill_dentries 0.08 ± 11% -0.0 0.05 ± 44% perf-profile.self.cycles-pp.path_lookupat 0.10 ± 9% -0.0 0.08 ± 21% perf-profile.self.cycles-pp.__lookup_mnt 0.09 ± 20% +0.1 0.16 ± 24% perf-profile.self.cycles-pp.rcu_core 0.15 ± 14% +0.1 0.23 ± 19% perf-profile.self.cycles-pp.trigger_load_balance fxmark.hdd_f2fs_dbench_client_72_directio.works_sec 200 +---------------------------------------------------------------------+ | +. | 190 |.+.+ +. .+ +. .+.+.+.+.+.+.+. .+.+.+. .+.+. : +.| 180 |-+ + + +.+ + + + + + +. .+. : | | + +.+.+.+ + +.+ | 170 |-+ | | | 160 |-+ O | | O | 150 |-O O O O O O O O O O O O | 140 |-+ O O O O O O O O | | O O O O | 130 |-+ O O O O | | O | 120 +---------------------------------------------------------------------+ fxmark.hdd_f2fs_dbench_client_72_directio.sys_sec 145 +---------------------------------------------------------------------+ 140 |-+ + + + .+ | |.+. : + + : : + .+. .+.+. .+.+. +. .+ .+ : | 135 |-+ +. : + : : + + +.+ + +.+.+. + +.+ + .+ : | 130 |-+ + + + + + +.+.+ +.| 125 |-+ + | 120 |-+ | | | 115 |-O O O | 110 |-+ O O O O O O O O O O O O O O | 105 |-+ O O O O O O O O O O | 100 |-+ O O O | | | 95 |-+ O | 90 +---------------------------------------------------------------------+ fxmark.hdd_f2fs_dbench_client_72_directio.sys_util 3.4 +---------------------------------------------------------------------+ | + | 3.2 |.+ +. + : + .+ .+. .+. .+ | | +.+. + + : + + + + .+ +.+ +.+. .+.+. .+. .+ + | 3 |-+ + + +. + + +.+.+ + +. .+.+ +.| | + + | 2.8 |-+ | | | 2.6 |-O O O O | | O O O O O O O O O O O O O O O O | 2.4 |-+ O O O O O O O | | O O O | 2.2 |-+ O | | | 2 +---------------------------------------------------------------------+ fxmark.hdd_f2fs_dbench_client_54_directio.sys_sec 110 +---------------------------------------------------------------------+ |.+ .+.+ +.+ .+ .+.+ | 105 |-++ .+ +. +. .+ + + + .+.+ + +.+ : + + :| 100 |-+ + + + +.+ + +.+ + + + + + + : + +. :| | + + .+.+ + + + + | 95 |-+ + | | | 90 |-+ | | O | 85 |-+ O O O O O O O O O O O | 80 |-O O O O O O O | | O O O O O O O O O | 75 |-+ O O O | | | 70 +---------------------------------------------------------------------+ fxmark.hdd_f2fs_dbench_client_54_directio.sys_util 3.3 +---------------------------------------------------------------------+ 3.2 |-++ .+ .+.+ + .+ + .+ + +. .+ +.+.+. | | + + .+.+.+ .+.+.+ + +.+ + + + + + + + + :| 3.1 |-+ + : +.+ + + + + :| 3 |-+ : + + | 2.9 |-+ + | 2.8 |-+ | | | 2.7 |-+ O | 2.6 |-+ O O O O O O O O O | 2.5 |-O O O O O | 2.4 |-+ O O O O O O O O O O O O | | O O O | 2.3 |-+ O | 2.2 +---------------------------------------------------------------------+ fxmark.time.user_time 96 +----------------------------------------------------------------------+ 94 |-+ .+ | |.+. .+.+ + .+ +.+.+. .+.+. .+.+ .+.+ .+ | 92 |-+ + +.+.+. .+.+ + + +. + + .+.+.+ + .+.+.+ : | 90 |-+ + + + + : +| | + | 88 |-+ | 86 |-+ | 84 |-+ | | | 82 |-+ O | 80 |-O O O O O O O | | O O O O O O O O O | 78 |-+ O O O O O O O O O O O O O O | 76 +----------------------------------------------------------------------+ fxmark.time.system_time 370 +---------------------------------------------------------------------+ |.+.+. .+.+.+. .+. .+.+.+.+.+.+.+.+.+ +. .+. .+ | 360 |-+ + +.+. .+ + + + +.+ +. .+.+.+ : | 350 |-+ +.+ + + : +| | + | 340 |-+ | 330 |-+ | | | 320 |-+ | 310 |-O O | | O O O O O O O O O O | 300 |-+ O O O O O O O O O O O O O O O O O | 290 |-+ O | | O | 280 +---------------------------------------------------------------------+ fxmark.time.voluntary_context_switches 1.05e+06 +----------------------------------------------------------------+ 1e+06 |-+ +. | |.+.+.++.+.+.+.+. .+. + +.+.++.+.+.+.+ ++.+.+.+. .+ | 950000 |-+ +.++ + + + +. .++ : | 900000 |-+ + + : +| | + | 850000 |-+ | 800000 |-+ | 750000 |-+ | | O | 700000 |-O O O | 650000 |-+ O O O O O O O O OO O O O O O O O | | O O O O O O O O O | 600000 |-+ O | 550000 +----------------------------------------------------------------+ fxmark.time.file_system_outputs 2.75e+07 +----------------------------------------------------------------+ | + | 2.7e+07 |.+. .++.+.+. + + .+.+.+. +.+. .+.+ + .+.+ .+ | 2.65e+07 |-+ + +. .+. + + + + + + :.+ + .+.++ : :| | + + + + + : :| 2.6e+07 |-+ + | | | 2.55e+07 |-+ | | | 2.5e+07 |-+ | 2.45e+07 |-+ O O | | O O O O O O O | 2.4e+07 |-+ O O O O O OO O O O | | O O O O O O O O O O O | 2.35e+07 +----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

2
2
0 / 0

[genirq] cbe16f35be: will-it-scale.per_thread_ops -5.2% regression

by kernel test robot

Greeting, FYI, we noticed a -5.2% regression of will-it-scale.per_thread_ops due to commit: commit: cbe16f35bee6880becca6f20d2ebf6b457148552 ("genirq: Add IRQF_NO_AUTOEN for request_irq/nmi()") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale on test machine: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory with following parameters: nr_task: 100% mode: thread test: getppid1 cpufreq_governor: performance ucode: 0x5003006 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/thread/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/getppid1/will-it-scale/0x5003006 commit: v5.12-rc2 cbe16f35be ("genirq: Add IRQF_NO_AUTOEN for request_irq/nmi()") v5.12-rc2 cbe16f35bee6880becca6f20d2e ---------------- --------------------------- %stddev %change %stddev \ | \ 7.408e+08 -5.2% 7.021e+08 will-it-scale.88.threads 8417726 -5.2% 7978644 will-it-scale.per_thread_ops 7.408e+08 -5.2% 7.021e+08 will-it-scale.workload 3.851e+10 -5.2% 3.65e+10 perf-stat.i.branch-instructions 1.839e+08 -4.2% 1.763e+08 perf-stat.i.branch-misses 1.39 +5.3% 1.46 perf-stat.i.cpi 5.988e+10 -5.2% 5.674e+10 perf-stat.i.dTLB-loads 4.139e+10 -5.2% 3.922e+10 perf-stat.i.dTLB-stores 2.239e+08 ± 3% -14.6% 1.913e+08 ± 4% perf-stat.i.iTLB-load-misses 1.741e+11 -5.2% 1.65e+11 perf-stat.i.instructions 794.20 ± 2% +10.7% 879.14 ± 4% perf-stat.i.instructions-per-iTLB-miss 0.72 -5.0% 0.68 perf-stat.i.ipc 1588 -5.2% 1505 perf-stat.i.metric.M/sec 1.39 +5.4% 1.47 perf-stat.overall.cpi 0.00 +0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 778.31 ± 3% +11.1% 864.35 ± 4% perf-stat.overall.instructions-per-iTLB-miss 0.72 -5.1% 0.68 perf-stat.overall.ipc 3.838e+10 -5.2% 3.638e+10 perf-stat.ps.branch-instructions 1.833e+08 -4.2% 1.757e+08 perf-stat.ps.branch-misses 5.968e+10 -5.2% 5.655e+10 perf-stat.ps.dTLB-loads 4.125e+10 -5.2% 3.909e+10 perf-stat.ps.dTLB-stores 2.231e+08 ± 3% -14.6% 1.907e+08 ± 4% perf-stat.ps.iTLB-load-misses 1.735e+11 -5.2% 1.645e+11 perf-stat.ps.instructions 5.243e+13 -5.2% 4.971e+13 perf-stat.total.instructions 43.20 -2.3 40.87 perf-profile.calltrace.cycles-pp.__entry_text_start.getppid 9.09 -0.4 8.65 ± 2% perf-profile.calltrace.cycles-pp.testcase 8.01 -0.3 7.66 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.getppid 2.93 -0.2 2.73 perf-profile.calltrace.cycles-pp.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.getppid 3.02 -0.2 2.83 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.getppid 2.52 ± 3% +0.7 3.23 ± 9% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.getppid 18.24 +1.1 19.29 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.getppid 13.33 +1.1 14.46 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_getppid.do_syscall_64.entry_SYSCALL_64_after_hwframe.getppid 1.94 ± 2% +1.7 3.62 ± 3% perf-profile.calltrace.cycles-pp.rcu_nocb_flush_deferred_wakeup.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.getppid 7.47 ± 2% +1.8 9.25 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.getppid 11.51 ± 2% +2.3 13.81 ± 2% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.getppid 39.05 +3.1 42.17 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.getppid 27.84 -1.5 26.36 perf-profile.children.cycles-pp.__entry_text_start 23.40 -1.2 22.22 perf-profile.children.cycles-pp.syscall_return_via_sysret 2.97 -0.2 2.76 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 3.23 -0.2 3.03 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.41 ± 2% -0.0 0.37 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 2.56 ± 3% +0.7 3.25 ± 7% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 13.88 +1.2 15.05 ± 2% perf-profile.children.cycles-pp.__x64_sys_getppid 2.02 ± 2% +1.6 3.67 ± 3% perf-profile.children.cycles-pp.rcu_nocb_flush_deferred_wakeup 7.86 +1.7 9.60 perf-profile.children.cycles-pp.exit_to_user_mode_prepare 12.71 +2.3 15.04 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 39.65 +3.1 42.77 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 23.19 -1.2 22.01 perf-profile.self.cycles-pp.syscall_return_via_sysret 19.57 -1.1 18.50 perf-profile.self.cycles-pp.getppid 12.47 -0.6 11.83 perf-profile.self.cycles-pp.__entry_text_start 8.60 -0.2 8.36 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 3.22 -0.2 3.02 perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 2.52 -0.2 2.35 perf-profile.self.cycles-pp.syscall_enter_from_user_mode 3.58 +0.4 3.95 perf-profile.self.cycles-pp.__x64_sys_getppid 2.12 ± 3% +0.6 2.74 ± 8% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 1.72 ± 2% +1.7 3.41 ± 3% perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup will-it-scale.per_thread_ops 8.8e+06 +-----------------------------------------------------------------+ 8.7e+06 |-+ .+. | | .+ + | 8.6e+06 |.+.+.+.+ + .+. .+.+. .+.+. .+.+.+.+. | 8.5e+06 |-+ ++ + + + +. | | +.+.++.+.+.+.+.+.+.+.| 8.4e+06 |-+ | 8.3e+06 |-O O O O O O OO | 8.2e+06 |-+ O O | | | 8.1e+06 |-+ | 8e+06 |-+ O O O O O O O O O O O O O O | | O O O O | 7.9e+06 |-+ O O O | 7.8e+06 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

5
11
0 / 0

1e7f963d62: WARNING:at_drivers/scsi/scsi_error.c:#scsi_eh_restore_cmnd

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 1e7f963d622a659baf2a71c369b8b3fe0c5ea39a ("Use a member variable to track the SCSI command submitter") https://github.com/bvanassche/linux scsi-remove-done-callback in testcase: xfstests version: xfstests-x86_64-73c0871-1_20210401 with following parameters: disk: 4HDD fs: xfs test: xfs-group-36 ucode: 0x21 test-description: xfstests is a regression test suite for xfs and other files ystems. test-url: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git on test machine: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> [ 33.728095] WARNING: CPU: 1 PID: 320 at drivers/scsi/scsi_error.c:1046 scsi_eh_restore_cmnd (kbuild/src/consumer/drivers/scsi/scsi_error.c:1046 (discriminator 1)) [ 33.728246] Modules linked in: xfs dm_mod netconsole btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c sd_mod t10_pi sg intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal i915 intel_powerclamp coretemp intel_gtt crct10dif_pclmul drm_kms_helper ahci crc32_pclmul crc32c_intel syscopyarea ghash_clmulni_intel libahci rapl sysfillrect libata sysimgblt fb_sys_fops intel_cstate mei_me drm usb_storage intel_uncore mei ipmi_devintf ipmi_msghandler video ip_tables [ 33.728929] CPU: 1 PID: 320 Comm: usb-storage Tainted: G W 5.12.0-rc1-00472-g1e7f963d622a #1 [ 33.729083] Hardware name: Hewlett-Packard p6-1451cx/2ADA, BIOS 8.15 02/05/2013 [ 33.729200] RIP: 0010:scsi_eh_restore_cmnd (kbuild/src/consumer/drivers/scsi/scsi_error.c:1046 (discriminator 1)) [ 33.729284] Code: 06 89 87 70 01 00 00 8b 46 04 89 47 24 8b 46 10 89 87 08 01 00 00 0f b6 46 15 88 87 d0 00 00 00 8b 46 08 89 87 b8 00 00 00 c3 <0f> 0b eb 8a 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 All code ======== 0: 06 (bad) 1: 89 87 70 01 00 00 mov %eax,0x170(%rdi) 7: 8b 46 04 mov 0x4(%rsi),%eax a: 89 47 24 mov %eax,0x24(%rdi) d: 8b 46 10 mov 0x10(%rsi),%eax 10: 89 87 08 01 00 00 mov %eax,0x108(%rdi) 16: 0f b6 46 15 movzbl 0x15(%rsi),%eax 1a: 88 87 d0 00 00 00 mov %al,0xd0(%rdi) 20: 8b 46 08 mov 0x8(%rsi),%eax 23: 89 87 b8 00 00 00 mov %eax,0xb8(%rdi) 29: c3 retq 2a:* 0f 0b ud2 <-- trapping instruction 2c: eb 8a jmp 0xffffffffffffffb8 2e: 66 66 2e 0f 1f 84 00 data16 nopw %cs:0x0(%rax,%rax,1) 35: 00 00 00 00 39: 66 data16 3a: 66 data16 3b: 2e cs 3c: 0f .byte 0xf 3d: 1f (bad) 3e: 84 00 test %al,(%rax) Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: eb 8a jmp 0xffffffffffffff8e 4: 66 66 2e 0f 1f 84 00 data16 nopw %cs:0x0(%rax,%rax,1) b: 00 00 00 00 f: 66 data16 10: 66 data16 11: 2e cs 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 33.729588] RSP: 0018:ffffc9000065be00 EFLAGS: 00010246 [ 33.729675] RAX: 0000000000000000 RBX: ffff88821f139840 RCX: 0000000053425355 [ 33.729790] RDX: 0000000000000000 RSI: ffffc9000065be10 RDI: ffff888219ecd110 [ 33.729903] RBP: 0000000000000001 R08: 000000000000000d R09: ffff88821faabc58 [ 33.730017] R10: 000000000000000f R11: fffffffffff9878e R12: ffff888219ecd110 [ 33.730131] R13: 0000000000000000 R14: 0000000000000012 R15: ffffc90000567a20 [ 33.730246] FS: 0000000000000000(0000) GS:ffff88821fa80000(0000) knlGS:0000000000000000 [ 33.730377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 33.730482] CR2: 000055b65ec06dd0 CR3: 000000021e00a006 CR4: 00000000001706e0 [ 33.730600] Call Trace: [ 33.730646] usb_stor_invoke_transport (kbuild/src/consumer/arch/x86/include/asm/bitops.h:207 kbuild/src/consumer/include/asm-generic/bitops/instrumented-non-atomic.h:135 kbuild/src/consumer/drivers/usb/storage/transport.c:728) usb_storage [ 33.730757] usb_stor_control_thread (kbuild/src/consumer/drivers/usb/storage/usb.c:381) usb_storage [ 33.730857] ? fill_inquiry_response (kbuild/src/consumer/drivers/usb/storage/usb.c:299) usb_storage [ 33.730953] ? fill_inquiry_response (kbuild/src/consumer/drivers/usb/storage/usb.c:299) usb_storage [ 33.731050] kthread (kbuild/src/consumer/kernel/kthread.c:292) [ 33.731110] ? kthread_park (kbuild/src/consumer/kernel/kthread.c:245) [ 33.731175] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:300) [ 33.731241] ---[ end trace 924a042cfbdcc12c ]--- [ 33.859651] 262144 bytes (262 kB, 256 KiB) copied, 0.0245905 s, 10.7 MB/s [ 33.859656] [ 33.860165] 512+0 records in [ 33.860166] [ 33.860649] 512+0 records out [ 33.860651] [ 33.862084] 262144 bytes (262 kB, 256 KiB) copied, 0.120131 s, 2.2 MB/s [ 33.862086] [ 33.862596] 512+0 records in [ 33.862597] [ 33.863052] 512+0 records out [ 33.863053] [ 33.864562] 262144 bytes (262 kB, 256 KiB) copied, 0.0840291 s, 3.1 MB/s [ 33.864565] [ 36.323963] XFS (sdb4): Mounting V5 Filesystem [ 36.380824] XFS (sdb4): Ending clean mount [ 36.381681] XFS (sdb4): Quotacheck needed: Please wait. [ 36.394965] XFS (sdb4): Quotacheck: Done. [ 36.396103] xfs filesystem being mounted at /fs/scratch supports timestamps until 2038 (0x7fffffff) [ 89.840283] xfs/360 IPMI BMC is not supported on this machine, skip bmc-watchdog setup! [ 89.840288] [ 124.291626] XFS (sdb4): Unmounting Filesystem [ 124.606877] XFS (sdb4): Mounting V5 Filesystem [ 124.731287] XFS (sdb4): Ending clean mount [ 124.768740] xfs filesystem being mounted at /fs/scratch supports timestamps until 2038 (0x7fffffff) [ 124.852863] XFS (sdb4): Unmounting Filesystem [ 130.967825] XFS (sdb1): Unmounting Filesystem [ 131.069158] _check_dmesg: something found in dmesg (see /lkp/benchmarks/xfstests/results//xfs/360.dmesg) [ 131.069164] [ 131.070749] [ 131.070753] [ 131.090913] run fstests xfs/361 at 2021-04-28 04:34:28 [ 131.536011] XFS (sdb1): Mounting V5 Filesystem [ 131.638167] XFS (sdb1): Ending clean mount [ 131.701786] xfs filesystem being mounted at /fs/sdb1 supports timestamps until 2038 (0x7fffffff) [ 132.032132] ------------[ cut here ]------------ To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

1
0
0 / 0

[mm/writeback] e5dbd33218: will-it-scale.per_process_ops -3.8% regression

by kernel test robot

Greeting, FYI, we noticed a -3.8% regression of will-it-scale.per_process_ops due to commit: commit: e5dbd33218bd8d87ab69f730ab90aed5fab7eb26 ("mm/writeback: Add wait_on_page_writeback_killable") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale on test machine: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory with following parameters: nr_task: 16 mode: process test: mmap2 cpufreq_governor: performance ucode: 0x5003006 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -3.8% regression | | test machine | 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=50% | | | test=mmap2 | | | ucode=0x5003006 | +------------------+-----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -4.4% regression | | test machine | 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=50% | | | test=mmap1 | | | ucode=0x5003006 | +------------------+-----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -2.8% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=16 | | | test=mmap2 | | | ucode=0x2006a0a | +------------------+-----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -4.7% regression | | test machine | 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=100% | | | test=mmap1 | | | ucode=0x5003006 | +------------------+-----------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/process/16/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/mmap2/will-it-scale/0x5003006 commit: 39f985c8f6 ("fs/cachefiles: Remove wait_bit_key layout dependency") e5dbd33218 ("mm/writeback: Add wait_on_page_writeback_killable") 39f985c8f667c80a e5dbd33218bd8d87ab69f730ab9 ---------------- --------------------------- %stddev %change %stddev \ | \ 9359770 -3.8% 9001769 will-it-scale.16.processes 584985 -3.8% 562610 will-it-scale.per_process_ops 9359770 -3.8% 9001769 will-it-scale.workload 15996 -1.2% 15811 proc-vmstat.nr_kernel_stack 23577 ± 10% +18.5% 27937 ± 7% softirqs.CPU48.SCHED 5183 ± 41% +47.2% 7630 ± 7% interrupts.CPU1.NMI:Non-maskable_interrupts 5183 ± 41% +47.2% 7630 ± 7% interrupts.CPU1.PMI:Performance_monitoring_interrupts 54.33 ± 12% +18.4% 64.33 ± 7% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 153.34 ± 24% -45.9% 83.00 ± 25% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 153.33 ± 24% -45.9% 82.99 ± 25% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 2.424e+10 -3.8% 2.332e+10 perf-stat.i.branch-instructions 0.47 +3.7% 0.48 perf-stat.i.cpi 2.529e+10 -4.0% 2.428e+10 perf-stat.i.dTLB-loads 1.15e+10 -3.8% 1.106e+10 perf-stat.i.dTLB-stores 54249733 -4.8% 51627939 perf-stat.i.iTLB-load-misses 1.004e+11 -3.8% 9.661e+10 perf-stat.i.instructions 2.15 -3.6% 2.07 perf-stat.i.ipc 693.66 -3.9% 666.70 perf-stat.i.metric.M/sec 0.46 +3.7% 0.48 perf-stat.overall.cpi 2.15 -3.6% 2.08 perf-stat.overall.ipc 2.416e+10 -3.8% 2.324e+10 perf-stat.ps.branch-instructions 2.52e+10 -4.0% 2.419e+10 perf-stat.ps.dTLB-loads 1.146e+10 -3.8% 1.102e+10 perf-stat.ps.dTLB-stores 54065825 -4.8% 51454019 perf-stat.ps.iTLB-load-misses 1.001e+11 -3.8% 9.628e+10 perf-stat.ps.instructions 3.025e+13 -3.9% 2.908e+13 perf-stat.total.instructions 0.89 ± 14% -0.1 0.77 ± 11% perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.shmem_mmap.mmap_region.do_mmap 0.14 ± 13% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.common_mmap 0.61 ± 12% -0.1 0.52 ± 12% perf-profile.children.cycles-pp.common_file_perm 0.21 ± 8% -0.0 0.17 ± 11% perf-profile.children.cycles-pp.vma_set_page_prot 0.12 ± 8% -0.0 0.09 ± 12% perf-profile.children.cycles-pp.blocking_notifier_call_chain 0.12 ± 14% -0.0 0.09 ± 15% perf-profile.children.cycles-pp.get_mmap_base 0.09 ± 8% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.vm_pgprot_modify 0.13 ± 15% +0.1 0.19 ± 8% perf-profile.children.cycles-pp.cap_capable 0.03 ±102% +0.1 0.12 ± 12% perf-profile.children.cycles-pp.munmap@plt 0.14 ± 13% +0.1 0.24 ± 6% perf-profile.children.cycles-pp.testcase 0.33 ± 10% -0.1 0.23 ± 10% perf-profile.self.cycles-pp.cap_vm_enough_memory 0.13 ± 11% -0.1 0.03 ±100% perf-profile.self.cycles-pp.common_mmap 0.48 ± 12% -0.1 0.41 ± 12% perf-profile.self.cycles-pp.common_file_perm 0.49 ± 12% -0.1 0.43 ± 13% perf-profile.self.cycles-pp.vm_area_alloc 0.12 ± 8% -0.0 0.09 ± 12% perf-profile.self.cycles-pp.blocking_notifier_call_chain 0.12 ± 13% -0.0 0.09 ± 14% perf-profile.self.cycles-pp.get_mmap_base 0.11 ± 8% +0.0 0.16 ± 10% perf-profile.self.cycles-pp.__x64_sys_munmap 0.11 ± 14% +0.1 0.18 ± 8% perf-profile.self.cycles-pp.cap_capable 0.12 ± 11% +0.1 0.20 ± 6% perf-profile.self.cycles-pp.testcase 0.01 ±223% +0.1 0.11 ± 13% perf-profile.self.cycles-pp.munmap@plt will-it-scale.16.processes 9.4e+06 +----------------------------------------------------------------+ |............+............+............ ......| 9.35e+06 |-+ +............+...... | 9.3e+06 |-+ | | | 9.25e+06 |-+ | 9.2e+06 |-+ | | | 9.15e+06 |-+ | 9.1e+06 |-+ | | | 9.05e+06 |-+ | 9e+06 |-+ O O | | O O | 8.95e+06 +----------------------------------------------------------------+ will-it-scale.per_process_ops 590000 +------------------------------------------------------------------+ | | 585000 |............+.............+............ ......| | +.............+...... | | | 580000 |-+ | | | 575000 |-+ | | | 570000 |-+ | | | | | 565000 |-+ | | O O O O | 560000 +------------------------------------------------------------------+ will-it-scale.workload 9.4e+06 +----------------------------------------------------------------+ |............+............+............ ......| 9.35e+06 |-+ +............+...... | 9.3e+06 |-+ | | | 9.25e+06 |-+ | 9.2e+06 |-+ | | | 9.15e+06 |-+ | 9.1e+06 |-+ | | | 9.05e+06 |-+ | 9e+06 |-+ O O | | O O | 8.95e+06 +----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2sp9: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/process/50%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/mmap2/will-it-scale/0x5003006 commit: 39f985c8f6 ("fs/cachefiles: Remove wait_bit_key layout dependency") e5dbd33218 ("mm/writeback: Add wait_on_page_writeback_killable") 39f985c8f667c80a e5dbd33218bd8d87ab69f730ab9 ---------------- --------------------------- %stddev %change %stddev \ | \ 25487561 -3.8% 24516984 will-it-scale.44.processes 579262 -3.8% 557203 will-it-scale.per_process_ops 25487561 -3.8% 24516984 will-it-scale.workload 3365 ± 12% -18.1% 2758 ± 15% numa-meminfo.node1.PageTables 841.83 ± 12% -18.2% 688.67 ± 15% numa-vmstat.node1.nr_page_table_pages 1.68 ± 12% +24.0% 2.09 ± 8% perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork 511.00 ± 3% +18.7% 606.50 ± 9% interrupts.CPU13.CAL:Function_call_interrupts 985.50 ± 39% -42.1% 571.00 ± 13% interrupts.CPU51.CAL:Function_call_interrupts 0.14 ± 10% -0.1 0.05 ± 45% perf-profile.children.cycles-pp.common_mmap 0.14 ± 8% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.cap_capable 0.20 ± 9% +0.1 0.29 ± 11% perf-profile.children.cycles-pp.apparmor_mmap_file 0.02 ±141% +0.1 0.13 ± 17% perf-profile.children.cycles-pp.munmap@plt 0.15 ± 11% +0.1 0.28 ± 11% perf-profile.children.cycles-pp.testcase 0.14 ± 12% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.common_mmap 0.35 ± 8% -0.1 0.25 ± 8% perf-profile.self.cycles-pp.cap_vm_enough_memory 0.11 ± 9% +0.1 0.18 ± 9% perf-profile.self.cycles-pp.__x64_sys_munmap 0.12 ± 11% +0.1 0.19 ± 9% perf-profile.self.cycles-pp.cap_capable 0.19 ± 8% +0.1 0.28 ± 11% perf-profile.self.cycles-pp.apparmor_mmap_file 0.13 ± 10% +0.1 0.23 ± 11% perf-profile.self.cycles-pp.testcase 0.00 +0.1 0.12 ± 19% perf-profile.self.cycles-pp.munmap@plt 6.572e+10 -3.8% 6.322e+10 perf-stat.i.branch-instructions 1.979e+08 -3.2% 1.916e+08 perf-stat.i.branch-misses 0.45 +3.8% 0.47 perf-stat.i.cpi 6.853e+10 -3.8% 6.59e+10 perf-stat.i.dTLB-loads 3.112e+10 -3.8% 2.994e+10 perf-stat.i.dTLB-stores 1.438e+08 -4.0% 1.381e+08 perf-stat.i.iTLB-load-misses 1912507 +2.0% 1949928 perf-stat.i.iTLB-loads 2.721e+11 -3.8% 2.618e+11 perf-stat.i.instructions 2.21 -3.8% 2.13 perf-stat.i.ipc 1879 -3.8% 1807 perf-stat.i.metric.M/sec 0.45 +3.9% 0.47 perf-stat.overall.cpi 0.00 +0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 2.21 -3.8% 2.13 perf-stat.overall.ipc 6.549e+10 -3.8% 6.301e+10 perf-stat.ps.branch-instructions 1.972e+08 -3.2% 1.91e+08 perf-stat.ps.branch-misses 6.829e+10 -3.8% 6.568e+10 perf-stat.ps.dTLB-loads 3.101e+10 -3.8% 2.983e+10 perf-stat.ps.dTLB-stores 1.433e+08 -3.9% 1.376e+08 perf-stat.ps.iTLB-load-misses 1905905 +2.0% 1943294 perf-stat.ps.iTLB-loads 2.711e+11 -3.8% 2.609e+11 perf-stat.ps.instructions 8.194e+13 -3.8% 7.88e+13 perf-stat.total.instructions *************************************************************************************************** lkp-csl-2sp9: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/process/50%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/mmap1/will-it-scale/0x5003006 commit: 39f985c8f6 ("fs/cachefiles: Remove wait_bit_key layout dependency") e5dbd33218 ("mm/writeback: Add wait_on_page_writeback_killable") 39f985c8f667c80a e5dbd33218bd8d87ab69f730ab9 ---------------- --------------------------- %stddev %change %stddev \ | \ 29240907 -4.4% 27946132 will-it-scale.44.processes 664565 -4.4% 635139 will-it-scale.per_process_ops 29240907 -4.4% 27946132 will-it-scale.workload 10398 ± 30% -54.6% 4721 ± 13% proc-vmstat.numa_hint_faults -15.86 +42.0% -22.53 sched_debug.cpu.nr_uninterruptible.min 11801 ± 8% -15.8% 9934 ± 12% numa-vmstat.node0.nr_slab_reclaimable 24848 ± 6% -14.2% 21328 ± 9% numa-vmstat.node0.nr_slab_unreclaimable 4235 ± 8% -11.6% 3745 ± 6% slabinfo.skbuff_head_cache.active_objs 4256 ± 8% -11.6% 3761 ± 5% slabinfo.skbuff_head_cache.num_objs 3575 ± 44% +55.9% 5572 ± 41% interrupts.CPU62.NMI:Non-maskable_interrupts 3575 ± 44% +55.9% 5572 ± 41% interrupts.CPU62.PMI:Performance_monitoring_interrupts 534.33 ± 6% +34.5% 718.67 ± 27% interrupts.CPU74.CAL:Function_call_interrupts 47209 ± 8% -15.8% 39739 ± 12% numa-meminfo.node0.KReclaimable 1344385 ± 3% -15.6% 1134906 ± 8% numa-meminfo.node0.MemUsed 47209 ± 8% -15.8% 39739 ± 12% numa-meminfo.node0.SReclaimable 99393 ± 6% -14.2% 85314 ± 9% numa-meminfo.node0.SUnreclaim 1252104 ± 3% +17.0% 1464694 ± 7% numa-meminfo.node1.MemUsed 121777 ± 6% +17.4% 142988 ± 9% numa-meminfo.node1.Slab 0.15 ± 11% -0.1 0.07 ± 13% perf-profile.children.cycles-pp.__x64_sys_mmap 0.17 ± 9% -0.1 0.09 ± 11% perf-profile.children.cycles-pp.get_mmap_base 0.25 ± 6% -0.0 0.20 ± 9% perf-profile.children.cycles-pp.cap_mmap_file 0.18 ± 10% +0.0 0.21 ± 5% perf-profile.children.cycles-pp.tlb_flush_mmu 0.19 ± 10% +0.0 0.24 ± 10% perf-profile.children.cycles-pp.cap_capable 0.27 ± 8% +0.1 0.35 ± 12% perf-profile.children.cycles-pp.apparmor_mmap_file 0.17 ± 10% +0.1 0.26 ± 10% perf-profile.children.cycles-pp.testcase 0.44 ± 9% -0.2 0.27 ± 9% perf-profile.self.cycles-pp.cap_vm_enough_memory 0.13 ± 13% -0.1 0.05 ± 45% perf-profile.self.cycles-pp.__x64_sys_mmap 0.16 ± 12% -0.1 0.09 ± 11% perf-profile.self.cycles-pp.get_mmap_base 0.23 ± 7% -0.0 0.18 ± 10% perf-profile.self.cycles-pp.cap_mmap_file 0.12 ± 9% +0.0 0.15 ± 8% perf-profile.self.cycles-pp.tlb_flush_mmu 0.18 ± 12% +0.0 0.22 ± 11% perf-profile.self.cycles-pp.cap_capable 0.14 ± 11% +0.1 0.21 ± 10% perf-profile.self.cycles-pp.testcase 0.13 ± 7% +0.1 0.21 ± 12% perf-profile.self.cycles-pp.__x64_sys_munmap 0.25 ± 9% +0.1 0.33 ± 11% perf-profile.self.cycles-pp.apparmor_mmap_file 6.752e+10 -4.4% 6.455e+10 perf-stat.i.branch-instructions 1.885e+08 -3.4% 1.821e+08 perf-stat.i.branch-misses 0.44 +4.6% 0.46 perf-stat.i.cpi 6.851e+10 -4.2% 6.563e+10 perf-stat.i.dTLB-loads 3.064e+10 -4.4% 2.929e+10 perf-stat.i.dTLB-stores 1.251e+08 -2.1% 1.225e+08 perf-stat.i.iTLB-load-misses 1904859 +1.4% 1931131 perf-stat.i.iTLB-loads 2.786e+11 -4.4% 2.664e+11 perf-stat.i.instructions 2232 -2.4% 2178 perf-stat.i.instructions-per-iTLB-miss 2.26 -4.4% 2.16 perf-stat.i.ipc 1893 -4.3% 1812 perf-stat.i.metric.M/sec 0.28 +0.0 0.28 perf-stat.overall.branch-miss-rate% 0.44 +4.7% 0.46 perf-stat.overall.cpi 0.00 +0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 2227 -2.4% 2174 perf-stat.overall.instructions-per-iTLB-miss 2.27 -4.4% 2.17 perf-stat.overall.ipc 6.728e+10 -4.4% 6.433e+10 perf-stat.ps.branch-instructions 1.879e+08 -3.4% 1.815e+08 perf-stat.ps.branch-misses 6.827e+10 -4.2% 6.54e+10 perf-stat.ps.dTLB-loads 3.054e+10 -4.4% 2.92e+10 perf-stat.ps.dTLB-stores 1.246e+08 -2.1% 1.22e+08 perf-stat.ps.iTLB-load-misses 1898462 +1.4% 1924678 perf-stat.ps.iTLB-loads 2.776e+11 -4.4% 2.654e+11 perf-stat.ps.instructions 8.39e+13 -4.4% 8.022e+13 perf-stat.total.instructions *************************************************************************************************** lkp-skl-fpga01: 104 threads Skylake with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/process/16/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/mmap2/will-it-scale/0x2006a0a commit: 39f985c8f6 ("fs/cachefiles: Remove wait_bit_key layout dependency") e5dbd33218 ("mm/writeback: Add wait_on_page_writeback_killable") 39f985c8f667c80a e5dbd33218bd8d87ab69f730ab9 ---------------- --------------------------- %stddev %change %stddev \ | \ 6286058 -2.8% 6112306 will-it-scale.16.processes 392878 -2.8% 382018 will-it-scale.per_process_ops 6286058 -2.8% 6112306 will-it-scale.workload 11705 ± 7% +14.6% 13414 ± 7% softirqs.CPU23.RCU 0.75 ± 16% -0.2 0.58 ± 11% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.56 ± 12% -0.1 0.45 ± 12% perf-profile.children.cycles-pp.tick_sched_timer 0.51 ± 12% -0.1 0.41 ± 12% perf-profile.children.cycles-pp.tick_sched_handle 0.11 ± 10% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.get_mmap_base 0.16 ± 10% +0.0 0.20 ± 4% perf-profile.children.cycles-pp.testcase 0.12 ± 17% +0.1 0.17 ± 11% perf-profile.children.cycles-pp.cap_capable 0.00 +0.1 0.10 ± 13% perf-profile.children.cycles-pp.munmap@plt 0.43 ± 9% -0.1 0.29 ± 6% perf-profile.self.cycles-pp.cap_vm_enough_memory 0.45 ± 9% -0.1 0.36 ± 9% perf-profile.self.cycles-pp.common_file_perm 0.10 ± 9% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.get_mmap_base 0.00 +0.1 0.08 ± 14% perf-profile.self.cycles-pp.munmap@plt 0.19 ± 11% +0.1 0.27 ± 8% perf-profile.self.cycles-pp.__x64_sys_munmap 5721 ± 47% -62.2% 2162 ± 79% interrupts.41:PCI-MSI.67633156-edge.eth0-TxRx-3 108.83 ± 24% -42.7% 62.33 ± 30% interrupts.CPU101.NMI:Non-maskable_interrupts 108.83 ± 24% -42.7% 62.33 ± 30% interrupts.CPU101.PMI:Performance_monitoring_interrupts 29.67 ± 16% +146.1% 73.00 ± 77% interrupts.CPU11.RES:Rescheduling_interrupts 5721 ± 47% -62.2% 2162 ± 79% interrupts.CPU33.41:PCI-MSI.67633156-edge.eth0-TxRx-3 211.00 ± 99% -69.4% 64.67 ± 32% interrupts.CPU43.NMI:Non-maskable_interrupts 211.00 ± 99% -69.4% 64.67 ± 32% interrupts.CPU43.PMI:Performance_monitoring_interrupts 99.00 ± 21% -26.9% 72.33 ± 21% interrupts.CPU44.NMI:Non-maskable_interrupts 99.00 ± 21% -26.9% 72.33 ± 21% interrupts.CPU44.PMI:Performance_monitoring_interrupts 108.67 ± 33% -37.4% 68.00 ± 18% interrupts.CPU49.NMI:Non-maskable_interrupts 108.67 ± 33% -37.4% 68.00 ± 18% interrupts.CPU49.PMI:Performance_monitoring_interrupts 97.67 ± 20% -24.9% 73.33 ± 17% interrupts.CPU96.NMI:Non-maskable_interrupts 97.67 ± 20% -24.9% 73.33 ± 17% interrupts.CPU96.PMI:Performance_monitoring_interrupts 0.03 ± 47% -66.3% 0.01 ± 42% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 1678 ± 38% +59.5% 2677 ± 12% perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 1678 ± 38% +59.5% 2677 ± 12% perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read 4759 ± 44% +70.0% 8092 ± 10% perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 4759 ± 44% +70.0% 8092 ± 10% perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read 4762 ± 44% +70.0% 8094 ± 10% perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 4319 ± 49% +78.4% 7706 ± 11% perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 1678 ± 38% +59.5% 2677 ± 12% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 1678 ± 38% +59.5% 2677 ± 12% perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read 4759 ± 44% +70.0% 8092 ± 10% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 4759 ± 44% +70.0% 8092 ± 10% perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read 4762 ± 44% +70.0% 8094 ± 10% perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 4319 ± 49% +78.4% 7706 ± 11% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 1.647e+10 -2.6% 1.603e+10 perf-stat.i.branch-instructions 82309505 ± 3% -6.4% 77043640 perf-stat.i.branch-misses 5.99 ± 24% +7.5 13.49 ± 44% perf-stat.i.cache-miss-rate% 0.08 ± 4% -0.0 0.07 ± 2% perf-stat.i.dTLB-load-miss-rate% 13214811 ± 4% -6.1% 12407570 ± 2% perf-stat.i.dTLB-load-misses 1.719e+10 -2.6% 1.674e+10 perf-stat.i.dTLB-loads 7.906e+09 -2.6% 7.699e+09 perf-stat.i.dTLB-stores 87.09 ± 3% -5.7 81.35 ± 4% perf-stat.i.iTLB-load-miss-rate% 13127160 ± 2% -5.1% 12459801 perf-stat.i.iTLB-load-misses 1919142 ± 23% +49.4% 2867789 ± 24% perf-stat.i.iTLB-loads 6.825e+10 -2.6% 6.645e+10 perf-stat.i.instructions 399.89 -2.7% 389.17 perf-stat.i.metric.M/sec 6.22 ± 28% +7.3 13.52 ± 39% perf-stat.overall.cache-miss-rate% 0.08 ± 4% -0.0 0.07 ± 2% perf-stat.overall.dTLB-load-miss-rate% 87.27 ± 3% -5.8 81.44 ± 4% perf-stat.overall.iTLB-load-miss-rate% 1.641e+10 -2.6% 1.598e+10 perf-stat.ps.branch-instructions 82037823 ± 3% -6.4% 76784758 perf-stat.ps.branch-misses 13169722 ± 4% -6.1% 12365594 ± 2% perf-stat.ps.dTLB-load-misses 1.713e+10 -2.7% 1.668e+10 perf-stat.ps.dTLB-loads 7.879e+09 -2.6% 7.674e+09 perf-stat.ps.dTLB-stores 13082418 ± 2% -5.1% 12417659 perf-stat.ps.iTLB-load-misses 1912636 ± 23% +49.4% 2857819 ± 24% perf-stat.ps.iTLB-loads 6.802e+10 -2.6% 6.623e+10 perf-stat.ps.instructions 2.056e+13 -2.8% 1.999e+13 perf-stat.total.instructions *************************************************************************************************** lkp-csl-2sp9: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/mmap1/will-it-scale/0x5003006 commit: 39f985c8f6 ("fs/cachefiles: Remove wait_bit_key layout dependency") e5dbd33218 ("mm/writeback: Add wait_on_page_writeback_killable") 39f985c8f667c80a e5dbd33218bd8d87ab69f730ab9 ---------------- --------------------------- %stddev %change %stddev \ | \ 30112325 -4.7% 28693708 will-it-scale.88.processes 342185 -4.7% 326064 will-it-scale.per_process_ops 30112325 -4.7% 28693708 will-it-scale.workload 8800 ± 14% +130.8% 20313 ± 70% cpuidle.POLL.time 1884 ± 32% -45.5% 1026 ± 42% interrupts.CPU67.CAL:Function_call_interrupts 121.50 ± 12% -20.0% 97.17 ± 7% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap 78709 ± 60% -48.1% 40873 ±122% numa-meminfo.node0.Active 78545 ± 60% -48.2% 40686 ±122% numa-meminfo.node0.Active(anon) 19679 ± 60% -48.1% 10211 ±122% numa-vmstat.node0.nr_active_anon 19679 ± 60% -48.1% 10211 ±122% numa-vmstat.node0.nr_zone_active_anon 6.879e+10 -4.7% 6.556e+10 perf-stat.i.branch-instructions 0.28 -0.0 0.28 perf-stat.i.branch-miss-rate% 1.883e+08 -6.9% 1.753e+08 perf-stat.i.branch-misses 0.83 +5.7% 0.88 perf-stat.i.cpi 7.004e+10 -4.7% 6.678e+10 perf-stat.i.dTLB-loads 27999 ± 3% -10.2% 25154 ± 11% perf-stat.i.dTLB-store-misses 3.146e+10 -4.7% 2.999e+10 perf-stat.i.dTLB-stores 1.007e+08 -6.0% 94644724 perf-stat.i.iTLB-load-misses 2.842e+11 -4.7% 2.709e+11 perf-stat.i.instructions 2829 +1.4% 2870 perf-stat.i.instructions-per-iTLB-miss 1.21 -5.4% 1.14 perf-stat.i.ipc 1935 -4.7% 1844 perf-stat.i.metric.M/sec 0.27 -0.0 0.27 perf-stat.overall.branch-miss-rate% 0.83 +5.7% 0.88 perf-stat.overall.cpi 2822 +1.4% 2862 perf-stat.overall.instructions-per-iTLB-miss 1.21 -5.4% 1.14 perf-stat.overall.ipc 6.855e+10 -4.7% 6.534e+10 perf-stat.ps.branch-instructions 1.877e+08 -6.9% 1.748e+08 perf-stat.ps.branch-misses 6.98e+10 -4.7% 6.655e+10 perf-stat.ps.dTLB-loads 3.135e+10 -4.7% 2.989e+10 perf-stat.ps.dTLB-stores 1.003e+08 -6.0% 94321542 perf-stat.ps.iTLB-load-misses 2.832e+11 -4.7% 2.699e+11 perf-stat.ps.instructions 8.563e+13 -4.7% 8.161e+13 perf-stat.total.instructions 30.17 -1.3 28.83 perf-profile.calltrace.cycles-pp.__mmap 26.81 -1.2 25.57 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap 25.89 -1.1 24.77 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap 22.19 -1.1 21.08 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 25.44 -1.1 24.35 perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap 25.11 -1.1 24.02 perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap 17.98 -1.0 17.01 perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 5.83 -0.3 5.57 perf-profile.calltrace.cycles-pp.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 3.33 -0.2 3.08 perf-profile.calltrace.cycles-pp.perf_iterate_sb.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff 2.40 -0.1 2.26 perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.90 ± 4% -0.1 0.76 ± 4% perf-profile.calltrace.cycles-pp.perf_event_mmap_output.perf_iterate_sb.perf_event_mmap.mmap_region.do_mmap 3.35 -0.1 3.21 perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.unmap_region.__do_munmap 2.90 -0.1 2.76 perf-profile.calltrace.cycles-pp.rcu_all_qs.__cond_resched.unmap_page_range.unmap_vmas.unmap_region 0.87 ± 2% -0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.76 -0.1 1.66 perf-profile.calltrace.cycles-pp.__entry_text_start.__munmap 1.78 -0.1 1.68 perf-profile.calltrace.cycles-pp.__vma_link_rb.vma_link.mmap_region.do_mmap.vm_mmap_pgoff 1.74 -0.1 1.66 perf-profile.calltrace.cycles-pp.__entry_text_start.__mmap 0.76 -0.1 0.70 perf-profile.calltrace.cycles-pp.security_vm_enough_memory_mm.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 0.58 -0.0 0.53 ± 2% perf-profile.calltrace.cycles-pp.cap_vm_enough_memory.security_vm_enough_memory_mm.mmap_region.do_mmap.vm_mmap_pgoff 0.64 -0.0 0.59 perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 0.79 -0.0 0.75 perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.95 -0.0 0.91 perf-profile.calltrace.cycles-pp.__vma_rb_erase.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 0.70 -0.0 0.68 perf-profile.calltrace.cycles-pp.vmacache_find.find_vma.__do_munmap.__vm_munmap.__x64_sys_munmap 0.84 +0.1 0.90 perf-profile.calltrace.cycles-pp.security_mmap_file.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.27 +0.1 1.34 perf-profile.calltrace.cycles-pp.vm_unmapped_area.arch_get_unmapped_area_topdown.get_unmapped_area.do_mmap.vm_mmap_pgoff 5.84 +0.2 6.08 perf-profile.calltrace.cycles-pp.__cond_resched.unmap_page_range.unmap_vmas.unmap_region.__do_munmap 2.05 +0.4 2.43 perf-profile.calltrace.cycles-pp.find_vma.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 23.20 +0.6 23.76 perf-profile.calltrace.cycles-pp.___might_sleep.unmap_page_range.unmap_vmas.unmap_region.__do_munmap 36.46 +0.9 37.31 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap 9.76 +1.0 10.72 perf-profile.calltrace.cycles-pp.free_pgd_range.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 40.96 +1.0 41.96 perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap 70.65 +1.3 71.93 perf-profile.calltrace.cycles-pp.__munmap 67.34 +1.4 68.76 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap 66.47 +1.5 67.94 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap 66.18 +1.5 67.67 perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap 65.70 +1.6 67.29 perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap 53.60 +1.7 55.34 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64 63.66 +1.8 65.49 perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.19 -1.4 28.83 perf-profile.children.cycles-pp.__mmap 22.27 -1.1 21.16 perf-profile.children.cycles-pp.do_mmap 25.48 -1.1 24.39 perf-profile.children.cycles-pp.ksys_mmap_pgoff 25.17 -1.1 24.08 perf-profile.children.cycles-pp.vm_mmap_pgoff 18.11 -1.0 17.16 perf-profile.children.cycles-pp.mmap_region 6.00 -0.3 5.73 perf-profile.children.cycles-pp.perf_event_mmap 3.39 -0.3 3.14 perf-profile.children.cycles-pp.perf_iterate_sb 1.73 -0.2 1.55 perf-profile.children.cycles-pp.down_write_killable 0.90 -0.2 0.74 perf-profile.children.cycles-pp.__might_sleep 0.94 ± 4% -0.1 0.79 ± 4% perf-profile.children.cycles-pp.perf_event_mmap_output 2.48 -0.1 2.34 perf-profile.children.cycles-pp.vma_link 3.41 -0.1 3.27 perf-profile.children.cycles-pp.zap_pte_range 2.26 -0.1 2.13 perf-profile.children.cycles-pp.__entry_text_start 1.13 -0.1 1.01 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.58 ± 2% -0.1 0.47 ± 5% perf-profile.children.cycles-pp.tlb_gather_mmu 1.78 -0.1 1.68 perf-profile.children.cycles-pp.__vma_link_rb 2.16 ± 2% -0.1 2.07 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.90 -0.1 0.83 ± 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.51 -0.1 0.44 perf-profile.children.cycles-pp.downgrade_write 0.85 -0.1 0.78 ± 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.80 -0.1 0.74 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt 0.81 -0.1 0.74 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.78 -0.1 0.72 perf-profile.children.cycles-pp.security_vm_enough_memory_mm 0.43 ± 5% -0.1 0.38 ± 7% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.60 -0.1 0.55 perf-profile.children.cycles-pp.cap_vm_enough_memory 0.68 -0.1 0.63 perf-profile.children.cycles-pp.free_pgtables 0.50 -0.0 0.45 ± 2% perf-profile.children.cycles-pp.vma_set_page_prot 0.34 ± 4% -0.0 0.29 ± 3% perf-profile.children.cycles-pp.up_read 0.15 ± 4% -0.0 0.11 ± 30% perf-profile.children.cycles-pp.ktime_get 0.33 ± 2% -0.0 0.29 ± 5% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.58 -0.0 0.54 ± 6% perf-profile.children.cycles-pp.tick_sched_timer 0.96 -0.0 0.92 perf-profile.children.cycles-pp.__vma_rb_erase 0.61 -0.0 0.57 perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.28 ± 2% -0.0 0.24 ± 4% perf-profile.children.cycles-pp.userfaultfd_unmap_complete 0.33 ± 2% -0.0 0.30 perf-profile.children.cycles-pp.unlink_anon_vmas 0.48 -0.0 0.45 perf-profile.children.cycles-pp.sync_mm_rss 0.74 -0.0 0.71 perf-profile.children.cycles-pp.vmacache_find 0.37 -0.0 0.34 ± 2% perf-profile.children.cycles-pp.up_write 0.40 ± 2% -0.0 0.38 perf-profile.children.cycles-pp.__vm_enough_memory 0.46 -0.0 0.44 perf-profile.children.cycles-pp.security_mmap_addr 0.25 ± 3% -0.0 0.23 ± 2% perf-profile.children.cycles-pp.unmap_single_vma 0.18 ± 2% -0.0 0.16 ± 6% perf-profile.children.cycles-pp.kfree 0.34 ± 2% -0.0 0.32 perf-profile.children.cycles-pp.userfaultfd_unmap_prep 0.17 -0.0 0.15 ± 3% perf-profile.children.cycles-pp.blocking_notifier_call_chain 0.08 -0.0 0.06 ± 7% perf-profile.children.cycles-pp.should_failslab 0.11 ± 4% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.unlink_file_vma 0.19 -0.0 0.18 perf-profile.children.cycles-pp.rcu_read_unlock_strict 0.10 ± 23% +0.0 0.14 ± 7% perf-profile.children.cycles-pp.vm_area_free 0.06 +0.0 0.11 ± 3% perf-profile.children.cycles-pp.__x86_indirect_thunk_r9 2.22 +0.1 2.27 perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.26 ± 7% +0.1 0.31 ± 4% perf-profile.children.cycles-pp.strlen 1.28 +0.1 1.35 perf-profile.children.cycles-pp.vm_unmapped_area 0.27 ± 2% +0.1 0.36 ± 2% perf-profile.children.cycles-pp.vmacache_update 0.87 +0.1 0.98 perf-profile.children.cycles-pp.security_mmap_file 94.27 +0.2 94.46 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 21.07 +0.3 21.37 perf-profile.children.cycles-pp.___might_sleep 92.45 +0.4 92.80 perf-profile.children.cycles-pp.do_syscall_64 2.21 +0.4 2.59 perf-profile.children.cycles-pp.find_vma 9.79 +1.0 10.74 perf-profile.children.cycles-pp.free_pgd_range 41.00 +1.0 42.00 perf-profile.children.cycles-pp.unmap_vmas 39.84 +1.1 40.91 perf-profile.children.cycles-pp.unmap_page_range 70.67 +1.3 71.95 perf-profile.children.cycles-pp.__munmap 66.25 +1.5 67.75 perf-profile.children.cycles-pp.__x64_sys_munmap 65.75 +1.6 67.31 perf-profile.children.cycles-pp.__vm_munmap 53.70 +1.7 55.42 perf-profile.children.cycles-pp.unmap_region 63.84 +1.8 65.66 perf-profile.children.cycles-pp.__do_munmap 2.07 -0.2 1.89 perf-profile.self.cycles-pp.__do_munmap 0.99 -0.2 0.83 ± 2% perf-profile.self.cycles-pp.do_mmap 0.88 ± 4% -0.1 0.74 ± 4% perf-profile.self.cycles-pp.perf_event_mmap_output 0.81 -0.1 0.67 ± 2% perf-profile.self.cycles-pp.__might_sleep 3.10 -0.1 2.99 perf-profile.self.cycles-pp.__cond_resched 2.39 -0.1 2.29 perf-profile.self.cycles-pp.perf_iterate_sb 0.56 ± 2% -0.1 0.46 ± 5% perf-profile.self.cycles-pp.tlb_gather_mmu 1.75 -0.1 1.66 perf-profile.self.cycles-pp.__vma_link_rb 2.46 -0.1 2.37 perf-profile.self.cycles-pp.zap_pte_range 2.17 -0.1 2.09 perf-profile.self.cycles-pp.perf_event_mmap 2.15 ± 2% -0.1 2.06 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.54 -0.1 0.46 ± 4% perf-profile.self.cycles-pp.__vm_munmap 1.01 -0.1 0.94 perf-profile.self.cycles-pp.__entry_text_start 0.48 -0.1 0.42 perf-profile.self.cycles-pp.downgrade_write 0.42 ± 6% -0.1 0.36 ± 7% perf-profile.self.cycles-pp.lru_add_drain_cpu 0.25 ± 3% -0.1 0.19 ± 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.71 ± 2% -0.1 0.66 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.93 -0.0 0.88 perf-profile.self.cycles-pp.vm_area_alloc 0.36 ± 2% -0.0 0.31 ± 2% perf-profile.self.cycles-pp.cap_vm_enough_memory 0.68 -0.0 0.63 perf-profile.self.cycles-pp.down_write_killable 0.14 ± 4% -0.0 0.10 ± 30% perf-profile.self.cycles-pp.ktime_get 0.49 -0.0 0.45 perf-profile.self.cycles-pp.unmap_vmas 0.27 ± 3% -0.0 0.23 ± 5% perf-profile.self.cycles-pp.userfaultfd_unmap_complete 0.93 -0.0 0.90 perf-profile.self.cycles-pp.__vma_rb_erase 0.33 ± 2% -0.0 0.29 ± 3% perf-profile.self.cycles-pp.__x64_sys_munmap 0.46 ± 2% -0.0 0.43 perf-profile.self.cycles-pp.sync_mm_rss 0.43 -0.0 0.40 perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.31 -0.0 0.28 ± 2% perf-profile.self.cycles-pp.unlink_anon_vmas 0.39 ± 2% -0.0 0.36 ± 2% perf-profile.self.cycles-pp.unmap_region 0.64 -0.0 0.61 perf-profile.self.cycles-pp.vmacache_find 0.54 -0.0 0.52 ± 2% perf-profile.self.cycles-pp.get_unmapped_area 0.30 ± 2% -0.0 0.27 ± 5% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 0.35 -0.0 0.33 perf-profile.self.cycles-pp.up_write 0.16 ± 5% -0.0 0.14 ± 4% perf-profile.self.cycles-pp.__vm_enough_memory 0.29 ± 2% -0.0 0.27 ± 2% perf-profile.self.cycles-pp.up_read 0.17 ± 3% -0.0 0.15 ± 3% perf-profile.self.cycles-pp.blocking_notifier_call_chain 0.17 ± 4% -0.0 0.15 ± 2% perf-profile.self.cycles-pp.do_syscall_64 0.20 ± 3% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.07 ± 6% -0.0 0.06 perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.20 ± 3% +0.0 0.21 ± 2% perf-profile.self.cycles-pp.cap_mmap_file 0.82 +0.0 0.85 perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown 0.17 ± 2% +0.0 0.21 ± 4% perf-profile.self.cycles-pp.lru_add_drain 1.24 +0.0 1.28 perf-profile.self.cycles-pp.vm_unmapped_area 0.05 ± 75% +0.1 0.11 ± 9% perf-profile.self.cycles-pp.vm_area_free 0.00 +0.1 0.07 perf-profile.self.cycles-pp.arch_vma_name 0.24 ± 3% +0.1 0.32 ± 2% perf-profile.self.cycles-pp.vmacache_update 0.41 ± 2% +0.1 0.50 perf-profile.self.cycles-pp.security_mmap_file 2.32 +0.1 2.46 perf-profile.self.cycles-pp.rcu_all_qs 1.24 +0.3 1.57 perf-profile.self.cycles-pp.find_vma 9.70 +1.0 10.65 perf-profile.self.cycles-pp.free_pgd_range 14.97 +1.0 15.94 perf-profile.self.cycles-pp.unmap_page_range Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

4
3
0 / 0

[hrtimer] 1035fd3f86: BUG:kernel_NULL_pointer_dereference,address

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 1035fd3f86b85a49f456d43196ad5c90d5462cef ("hrtimer: Avoid more SMP function calls in clock_was_set()") https://git.kernel.org/cgit/linux/kernel/git/tglx/devel.git hrtimer in testcase: rcutorture version: with following parameters: runtime: 300s test: default torture_type: srcud test-description: rcutorture is rcutorture kernel module load/unload test. test-url: https://www.kernel.org/doc/Documentation/RCU/torture.txt on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +---------------------------------------------+------------+------------+ | | ff7c9a1a33 | 1035fd3f86 | +---------------------------------------------+------------+------------+ | boot_successes | 6 | 0 | | boot_failures | 0 | 7 | | BUG:kernel_NULL_pointer_dereference,address | 0 | 7 | | Oops:#[##] | 0 | 7 | | RIP:clock_was_set | 0 | 7 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 7 | +---------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> [ 4.233316] BUG: kernel NULL pointer dereference, address: 0000000000000018 [ 4.233605] #PF: supervisor read access in kernel mode [ 4.233605] #PF: error_code(0x0000) - not-present page [ 4.233605] PGD 0 P4D 0 [ 4.233605] Oops: 0000 [#1] SMP PTI [ 4.233605] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.12.0-00008-g1035fd3f86b8 #2 [ 4.233605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 4.233605] RIP: 0010:clock_was_set (kbuild/src/x86_64/kernel/time/hrtimer.c:922 kbuild/src/x86_64/kernel/time/hrtimer.c:972) [ 4.233605] Code: 34 24 8b 54 24 0c 89 f1 f7 d2 89 f0 48 83 c1 01 21 da 48 c1 e1 06 48 01 e9 74 5b 48 8b 75 20 48 8b 49 28 48 c1 e0 06 48 01 e8 <48> 8b 49 18 48 2b 48 78 48 39 f1 0f 8c a4 00 00 00 83 78 4c 03 7e All code ======== 0: 34 24 xor $0x24,%al 2: 8b 54 24 0c mov 0xc(%rsp),%edx 6: 89 f1 mov %esi,%ecx 8: f7 d2 not %edx a: 89 f0 mov %esi,%eax c: 48 83 c1 01 add $0x1,%rcx 10: 21 da and %ebx,%edx 12: 48 c1 e1 06 shl $0x6,%rcx 16: 48 01 e9 add %rbp,%rcx 19: 74 5b je 0x76 1b: 48 8b 75 20 mov 0x20(%rbp),%rsi 1f: 48 8b 49 28 mov 0x28(%rcx),%rcx 23: 48 c1 e0 06 shl $0x6,%rax 27: 48 01 e8 add %rbp,%rax 2a:* 48 8b 49 18 mov 0x18(%rcx),%rcx <-- trapping instruction 2e: 48 2b 48 78 sub 0x78(%rax),%rcx 32: 48 39 f1 cmp %rsi,%rcx 35: 0f 8c a4 00 00 00 jl 0xdf 3b: 83 78 4c 03 cmpl $0x3,0x4c(%rax) 3f: 7e .byte 0x7e Code starting with the faulting instruction =========================================== 0: 48 8b 49 18 mov 0x18(%rcx),%rcx 4: 48 2b 48 78 sub 0x78(%rax),%rcx 8: 48 39 f1 cmp %rsi,%rcx b: 0f 8c a4 00 00 00 jl 0xb5 11: 83 78 4c 03 cmpl $0x3,0x4c(%rax) 15: 7e .byte 0x7e [ 4.233605] RSP: 0000:ffffb29800013af8 EFLAGS: 00010082 [ 4.233605] RAX: ffff902e2fc1e140 RBX: 00000000000000aa RCX: 0000000000000000 [ 4.233605] RDX: 00000000000000a8 RSI: 0000001664b2f651 RDI: ffffffff8f801040 [ 4.233605] RBP: ffff902e2fc1e100 R08: 000000014ade12ab R09: 0000000000000000 [ 4.233605] R10: 0000000000000002 R11: 0000000000000005 R12: 000000000001e100 [ 4.233605] R13: 0000000000000001 R14: 0000000000000286 R15: 0000000000000000 [ 4.233605] FS: 0000000000000000(0000) GS:ffff902e2fd00000(0000) knlGS:0000000000000000 [ 4.233605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4.233605] CR2: 0000000000000018 CR3: 000000018f40a000 CR4: 00000000000406e0 [ 4.233605] Call Trace: [ 4.233605] do_settimeofday64 (kbuild/src/x86_64/kernel/time/timekeeping.c:1328 kbuild/src/x86_64/kernel/time/timekeeping.c:1292) [ 4.233605] __devm_rtc_register_device.cold+0x6f/0x115 [ 4.233605] cmos_do_probe (kbuild/src/x86_64/drivers/rtc/rtc-cmos.c:867) [ 4.233605] ? cmos_pnp_remove (kbuild/src/x86_64/drivers/rtc/rtc-cmos.c:589) [ 4.233605] ? cmos_nvram_read (kbuild/src/x86_64/drivers/rtc/rtc-cmos.c:610) [ 4.233605] ? cmos_do_probe (kbuild/src/x86_64/drivers/rtc/rtc-cmos.c:1302) [ 4.233605] pnp_device_probe (kbuild/src/x86_64/drivers/pnp/driver.c:111) [ 4.233605] really_probe (kbuild/src/x86_64/drivers/base/dd.c:559) [ 4.233605] driver_probe_device (kbuild/src/x86_64/drivers/base/dd.c:745) [ 4.233605] device_driver_attach (kbuild/src/x86_64/drivers/base/dd.c:1020) [ 4.233605] __driver_attach (kbuild/src/x86_64/drivers/base/dd.c:1099) [ 4.233605] ? device_driver_attach (kbuild/src/x86_64/drivers/base/dd.c:1052) [ 4.233605] bus_for_each_dev (kbuild/src/x86_64/drivers/base/bus.c:305) [ 4.233605] bus_add_driver (kbuild/src/x86_64/drivers/base/bus.c:623) [ 4.233605] driver_register (kbuild/src/x86_64/drivers/base/driver.c:171) [ 4.233605] ? rtc_dev_init (kbuild/src/x86_64/drivers/rtc/rtc-cmos.c:1461) [ 4.233605] cmos_init (kbuild/src/x86_64/drivers/rtc/rtc-cmos.c:1465) [ 4.233605] do_one_initcall (kbuild/src/x86_64/init/main.c:1226) [ 4.233605] kernel_init_freeable (kbuild/src/x86_64/init/main.c:1298 kbuild/src/x86_64/init/main.c:1315 kbuild/src/x86_64/init/main.c:1335 kbuild/src/x86_64/init/main.c:1537) [ 4.233605] ? rest_init (kbuild/src/x86_64/init/main.c:1421) [ 4.233605] kernel_init (kbuild/src/x86_64/init/main.c:1426) [ 4.233605] ret_from_fork (kbuild/src/x86_64/arch/x86/entry/entry_64.S:300) [ 4.233605] Modules linked in: [ 4.233605] CR2: 0000000000000018 [ 4.233605] ---[ end trace 62d34bc36d339c7a ]--- [ 4.233605] RIP: 0010:clock_was_set (kbuild/src/x86_64/kernel/time/hrtimer.c:922 kbuild/src/x86_64/kernel/time/hrtimer.c:972) [ 4.233605] Code: 34 24 8b 54 24 0c 89 f1 f7 d2 89 f0 48 83 c1 01 21 da 48 c1 e1 06 48 01 e9 74 5b 48 8b 75 20 48 8b 49 28 48 c1 e0 06 48 01 e8 <48> 8b 49 18 48 2b 48 78 48 39 f1 0f 8c a4 00 00 00 83 78 4c 03 7e All code ======== 0: 34 24 xor $0x24,%al 2: 8b 54 24 0c mov 0xc(%rsp),%edx 6: 89 f1 mov %esi,%ecx 8: f7 d2 not %edx a: 89 f0 mov %esi,%eax c: 48 83 c1 01 add $0x1,%rcx 10: 21 da and %ebx,%edx 12: 48 c1 e1 06 shl $0x6,%rcx 16: 48 01 e9 add %rbp,%rcx 19: 74 5b je 0x76 1b: 48 8b 75 20 mov 0x20(%rbp),%rsi 1f: 48 8b 49 28 mov 0x28(%rcx),%rcx 23: 48 c1 e0 06 shl $0x6,%rax 27: 48 01 e8 add %rbp,%rax 2a:* 48 8b 49 18 mov 0x18(%rcx),%rcx <-- trapping instruction 2e: 48 2b 48 78 sub 0x78(%rax),%rcx 32: 48 39 f1 cmp %rsi,%rcx 35: 0f 8c a4 00 00 00 jl 0xdf 3b: 83 78 4c 03 cmpl $0x3,0x4c(%rax) 3f: 7e .byte 0x7e Code starting with the faulting instruction =========================================== 0: 48 8b 49 18 mov 0x18(%rcx),%rcx 4: 48 2b 48 78 sub 0x78(%rax),%rcx 8: 48 39 f1 cmp %rsi,%rcx b: 0f 8c a4 00 00 00 jl 0xb5 11: 83 78 4c 03 cmpl $0x3,0x4c(%rax) 15: 7e .byte 0x7e To reproduce: # build kernel cd linux cp config-5.12.0-00008-g1035fd3f86b8 .config make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

1
0
0 / 0

[mm/page_alloc] c14b25d7bc: netperf.Throughput_Mbps -2.4% regression

by kernel test robot

Greeting, FYI, we noticed a -2.4% regression of netperf.Throughput_Mbps due to commit: commit: c14b25d7bc66d7b2a88902c19c3ce76b478b659e ("mm/page_alloc: Update PGFREE outside the zone lock in __free_pages_ok") https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git mm-percpu-local_lock-v5r2 in testcase: netperf on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory with following parameters: ip: ipv4 runtime: 300s nr_threads: 50% cluster: cs-localhost send_size: 10K test: SCTP_STREAM_MANY cpufreq_governor: performance ucode: 0x5003006 test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance. test-url: http://www.netperf.org/netperf/ If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode: cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/50%/debian-10.4-x86_64-20200603.cgz/300s/10K/lkp-csl-2ap3/SCTP_STREAM_MANY/netperf/0x5003006 commit: df7f47653d ("mm/page_alloc: Avoid conflating IRQs disabled with zone->lock") c14b25d7bc ("mm/page_alloc: Update PGFREE outside the zone lock in __free_pages_ok") df7f47653d805dd0 c14b25d7bc66d7b2a88902c19c3 ---------------- --------------------------- %stddev %change %stddev \ | \ 3446 -2.4% 3364 netperf.Throughput_Mbps 330894 -2.4% 322966 netperf.Throughput_total_Mbps 1.212e+09 -2.4% 1.183e+09 netperf.workload 0.05 ± 64% +65.4% 0.08 ± 7% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 1606542 ± 11% -3.9% 1543247 ± 11% sched_debug.cpu.nr_switches.avg 2278353 -2.5% 2221933 vmstat.system.cs 0.56 ± 6% -0.0 0.51 ± 4% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.08 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.irqtime_account_irq 0.17 ± 5% -0.0 0.14 ± 5% perf-profile.self.cycles-pp.sctp_outq_flush 0.00 +0.1 0.05 ± 6% perf-profile.self.cycles-pp.sctp_outq_sack 197881 +2.0% 201784 interrupts.CAL:Function_call_interrupts 5129 ± 21% -38.7% 3146 ± 41% interrupts.CPU134.RES:Rescheduling_interrupts 6741 ± 18% +23.5% 8328 ± 7% interrupts.CPU135.NMI:Non-maskable_interrupts 6741 ± 18% +23.5% 8328 ± 7% interrupts.CPU135.PMI:Performance_monitoring_interrupts 6239 ± 27% +34.0% 8360 ± 4% interrupts.CPU148.NMI:Non-maskable_interrupts 6239 ± 27% +34.0% 8360 ± 4% interrupts.CPU148.PMI:Performance_monitoring_interrupts 5515 ± 17% -31.9% 3753 ± 31% interrupts.CPU150.RES:Rescheduling_interrupts 1284 ± 14% -20.3% 1023 ± 2% interrupts.CPU78.CAL:Function_call_interrupts 1631705 -2.0% 1599723 proc-vmstat.nr_file_pages 1389640 -2.3% 1357659 proc-vmstat.nr_shmem 1.573e+09 -2.4% 1.535e+09 proc-vmstat.numa_hit 1.574e+09 -2.4% 1.537e+09 proc-vmstat.numa_local 19680 ± 33% +87.1% 36814 ± 24% proc-vmstat.numa_pages_migrated 9.007e+09 -2.4% 8.791e+09 proc-vmstat.pgalloc_normal 9.003e+09 -2.4% 8.787e+09 proc-vmstat.pgfree 19680 ± 33% +87.1% 36814 ± 24% proc-vmstat.pgmigrate_success 264898 ± 25% +129.1% 606819 ± 40% softirqs.CPU132.NET_RX 397413 ± 67% +141.5% 959557 ± 38% softirqs.CPU134.NET_RX 36247 ± 14% -34.4% 23781 ± 34% softirqs.CPU134.SCHED 1076321 ± 23% -55.7% 476706 ± 64% softirqs.CPU184.NET_RX 1521995 ± 5% -22.8% 1175718 ± 20% softirqs.CPU36.NET_RX 1418346 ± 16% -40.7% 841312 ± 43% softirqs.CPU38.NET_RX 15458 ± 35% +82.8% 28259 ± 30% softirqs.CPU38.SCHED 1354153 ± 17% -31.7% 924720 ± 39% softirqs.CPU66.NET_RX 706549 ± 34% +79.1% 1265156 ± 25% softirqs.CPU88.NET_RX 30720 ± 19% -43.0% 17517 ± 42% softirqs.CPU88.SCHED 1.933e+10 -1.4% 1.906e+10 perf-stat.i.branch-instructions 2.024e+08 -1.8% 1.987e+08 perf-stat.i.branch-misses 2.638e+09 -1.8% 2.591e+09 perf-stat.i.cache-references 2295338 -2.4% 2240128 perf-stat.i.context-switches 3.98 +1.5% 4.04 perf-stat.i.cpi 2.499e+10 -1.6% 2.459e+10 perf-stat.i.dTLB-loads 1.191e+10 -2.3% 1.164e+10 perf-stat.i.dTLB-stores 1.423e+08 -2.2% 1.392e+08 perf-stat.i.iTLB-load-misses 9.106e+10 -1.6% 8.963e+10 perf-stat.i.instructions 0.25 -1.7% 0.25 perf-stat.i.ipc 307.25 -1.7% 302.05 perf-stat.i.metric.M/sec 3.97 +1.4% 4.02 perf-stat.overall.cpi 0.25 -1.4% 0.25 perf-stat.overall.ipc 1.927e+10 -1.4% 1.9e+10 perf-stat.ps.branch-instructions 2.017e+08 -1.8% 1.98e+08 perf-stat.ps.branch-misses 2.629e+09 -1.8% 2.582e+09 perf-stat.ps.cache-references 2287484 -2.4% 2232226 perf-stat.ps.context-switches 2.491e+10 -1.6% 2.451e+10 perf-stat.ps.dTLB-loads 1.187e+10 -2.3% 1.16e+10 perf-stat.ps.dTLB-stores 1.418e+08 -2.2% 1.387e+08 perf-stat.ps.iTLB-load-misses 9.077e+10 -1.6% 8.933e+10 perf-stat.ps.instructions 2.744e+13 -1.5% 2.703e+13 perf-stat.total.instructions netperf.Throughput_Mbps 3480 +--------------------------------------------------------------------+ | .+.+.+ .+. .+. .+.+. | 3460 |.+ : +.+. .+.+.+ +.+ ++.+.+ +. | 3440 |-+ : : + +.+.+.+.+.+.+.+.+ | | : : | 3420 |-+ +. : | | + | 3400 |-+ | | | 3380 |-+ O | 3360 |-+ O O O O O O O O | | O O O O O O O O O O O O O O O O | 3340 |-O O O O O O | | O O | 3320 +--------------------------------------------------------------------+ netperf.Throughput_total_Mbps 334000 +------------------------------------------------------------------+ |.+.+.+.+ .+. .+. .+. .+.+.+ | 332000 |-+ : +.+.+ +.+ +.+ ++.+.+ + .+. | 330000 |-+ : : +.+.+.+ ++.+.+ | | : : | 328000 |-+ ++ | | | 326000 |-+ | | | 324000 |-+ O | 322000 |-+ O O O O O O O O O O O O O OO O O O | | O O O OO O O O O O O | 320000 |-+ O | | O O O | 318000 +------------------------------------------------------------------+ netperf.workload 1.22e+09 +----------------------------------------------------------------+ |.+.+.+ : +.+. .++.+.+.+.+.+.++.+.+ +.+. | 1.21e+09 |-+ : : + ++.+.+.+.+.+.++ | | : : | | +.+ | 1.2e+09 |-+ | | | 1.19e+09 |-+ | | O | 1.18e+09 |-+ O O O O O O O O OO O O O O O O O | | O O O O O O O O O O O O | | O | 1.17e+09 |-+ O O O | | | 1.16e+09 +----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

1
0
0 / 0

[drm/i915/dp] 4a8d79901d: WARNING:at_drivers/gpu/drm/i915/display/intel_display_power.c:#assert_can_disable_lcpll[i915]

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 4a8d79901d5bed0812d272c372aa40282937b50f ("drm/i915/dp: Enable Intel's HDR backlight interface (only SDR for now)") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: suspend-stress version: with following parameters: mode: freeze iterations: 10 on test machine: 4 threads Broadwell-Y with 8G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> kern :warn : [ 21.137894] ------------[ cut here ]------------ kern :warn : [ 21.137897] CPU PWM1 enabled kern :warn : [ 21.137930] WARNING: CPU: 3 PID: 575 at drivers/gpu/drm/i915/display/intel_display_power.c:4902 assert_can_disable_lcpll+0x335/0x3c0 [i915] kern :warn : [ 21.138084] Modules linked in: sd_mod t10_pi sg x86_pkg_temp_thermal intel_powerclamp i915 hid_rmi rmi_core hid_multitouch coretemp intel_ra pl_msr crct10dif_pclmul wmi_bmof crc32_pclmul intel_gtt crc32c_intel ghash_clmulni_intel rapl drm_kms_helper intel_cstate ahci libahci serio_raw mei_me proces sor_thermal_device intel_uncore processor_thermal_rfim syscopyarea sysfillrect intel_soc_dts_iosf sysimgblt intel_pch_thermal hid_sensor_custom fb_sys_fops li bata bcma processor_thermal_mbox joydev mei drm processor_thermal_rapl intel_rapl_common int340x_thermal_zone i2c_hid ideapad_laptop sparse_keymap wmi rfkill video int3400_thermal acpi_thermal_rel i2c_designware_platform dw_dmac i2c_designware_core acpi_pad ip_tables kern :warn : [ 21.138148] CPU: 3 PID: 575 Comm: kworker/u8:6 Tainted: G W I 5.11.0-rc2-00745-g4a8d79901d5b #1 kern :warn : [ 21.138153] Hardware name: LENOVO 80HE/VIUU4, BIOS A6CN38WW 09/30/2014 kern :warn : [ 21.138155] Workqueue: events_unbound async_run_entry_fn kern :warn : [ 21.138161] RIP: 0010:assert_can_disable_lcpll+0x335/0x3c0 [i915] kern :info : [ 21.138263] ahci 0000:00:1f.2: pci_pm_suspend_late+0x0/0x40 returned 0 after 0 usecs kern :warn : [ 21.138296] Code: c0 75 22 e8 8d 61 cb ff e9 f3 fd ff ff e8 fd bc 6a c1 0f 0b e9 0c fd ff ff e8 f1 bc 6a c1 0f 0b e9 05 fe ff ff e8 e5 bc 6a c1 <0f> 0b e9 cf fd ff ff e8 d9 bc 6a c1 0f 0b e9 a2 fd ff ff e8 cd bc kern :warn : [ 21.138300] RSP: 0018:ffffc900006f3d58 EFLAGS: 00010282 kern :warn : [ 21.138304] RAX: 0000000000000000 RBX: ffff8881451002e8 RCX: 0000000000000027 kern :warn : [ 21.138307] RDX: 0000000000000027 RSI: 0000000000000002 RDI: ffff888249397cf8 kern :warn : [ 21.138310] RBP: ffff888145100000 R08: ffff888249397cf0 R09: ffffc900006f3cf0 kern :warn : [ 21.138313] R10: 0000000000000001 R11: 3030302035313969 R12: ffff8881451007f0 kern :warn : [ 21.138316] R13: ffff888145106c60 R14: 0000000000000002 R15: 0000000000000000 kern :warn : [ 21.138319] FS: 0000000000000000(0000) GS:ffff888249380000(0000) knlGS:0000000000000000 kern :warn : [ 21.138322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kern :warn : [ 21.138325] CR2: 0000557c22ec68a0 CR3: 0000000004c0a003 CR4: 00000000003706e0 kern :warn : [ 21.138329] Call Trace: kern :warn : [ 21.138334] hsw_disable_lcpll+0x22/0x280 [i915] kern :warn : [ 21.138479] i915_drm_suspend_late+0x5e/0x100 [i915] kern :warn : [ 21.138578] ? pci_pm_poweroff_late+0x40/0x40 kern :warn : [ 21.138582] dpm_run_callback+0x4c/0x140 kern :warn : [ 21.138590] __device_suspend_late+0x98/0x1a0 kern :warn : [ 21.138596] async_suspend_late+0x1b/0xa0 kern :warn : [ 21.138602] async_run_entry_fn+0x39/0x160 kern :warn : [ 21.138608] process_one_work+0x1ed/0x3c0 kern :warn : [ 21.138612] worker_thread+0x50/0x3c0 kern :warn : [ 21.138617] ? process_one_work+0x3c0/0x3c0 kern :warn : [ 21.138620] kthread+0x116/0x160 kern :warn : [ 21.138626] ? kthread_park+0xa0/0xa0 kern :warn : [ 21.138631] ret_from_fork+0x22/0x30 kern :warn : [ 21.138640] ---[ end trace a93fe4d40a5a37bb ]--- To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Thanks, Oliver Sang

1 year, 3 months

4
9
0 / 0

[clocksource] 6c52b5f3cf: stress-ng.opcode.ops_per_sec -14.4% regression

by kernel test robot

Greeting, FYI, we noticed a -14.4% regression of stress-ng.opcode.ops_per_sec due to commit: commit: 6c52b5f3cfefd6e429efc4413fd25e3c394e959f ("clocksource: Reduce WATCHDOG_THRESHOLD") https://git.kernel.org/cgit/linux/kernel/git/paulmck/linux-rcu.git dev.2021.04.13a in testcase: stress-ng on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory with following parameters: nr_threads: 10% disk: 1HDD testtime: 60s fs: ext4 class: os test: opcode cpufreq_governor: performance ucode: 0x5003006 In addition to that, the commit also has significant impact on the following tests: +------------------+---------------------------------------------------------------------------+ | testcase: change | unixbench: boot-time.dhcp 2.0% regression | | test machine | 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | nr_task=1 | | | runtime=300s | | | test=execl | | | ucode=0x4003006 | +------------------+---------------------------------------------------------------------------+ | testcase: change | fsmark: fsmark.files_per_sec -37.8% regression | | test machine | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory | | test parameters | cpufreq_governor=performance | | | disk=1BRD_48G | | | filesize=4M | | | fs=btrfs | | | iterations=1x | | | nr_threads=1t | | | sync_method=NoSync | | | test_size=24G | | | ucode=0x5003006 | +------------------+---------------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode: os/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/opcode/stress-ng/60s/0x5003006 commit: d42c7a8d68 ("kcsan: Document "value changed" line") 6c52b5f3cf ("clocksource: Reduce WATCHDOG_THRESHOLD") d42c7a8d68038bc0 6c52b5f3cfefd6e429efc4413fd ---------------- --------------------------- %stddev %change %stddev \ | \ 570473 -14.4% 488107 stress-ng.opcode.ops 9507 -14.4% 8135 stress-ng.opcode.ops_per_sec 573594 -14.5% 490520 stress-ng.time.involuntary_context_switches 18566 ± 13% -33.6% 12331 ± 16% stress-ng.time.major_page_faults 149469 ± 44% -47.1% 79036 ± 5% stress-ng.time.maximum_resident_set_size 38932182 ± 2% -14.8% 33173409 stress-ng.time.minor_page_faults 800.67 +1.6% 813.67 stress-ng.time.percent_of_cpu_this_job_got 289.84 +12.8% 327.07 stress-ng.time.system_time 207.53 -14.1% 178.27 stress-ng.time.user_time 1045208 -11.5% 924508 stress-ng.time.voluntary_context_switches 15.47 +1.3% 15.67 boot-time.dhcp 2168 ± 2% +4.3% 2262 boot-time.idle 9.46 -11.9% 8.34 iostat.cpu.system 1.91 +18.3% 2.26 ± 5% iostat.cpu.user 17319 ± 5% +40.7% 24370 ± 2% numa-meminfo.node1.Active 16704 ± 5% +42.8% 23852 numa-meminfo.node1.Active(anon) 4179 ± 5% +42.8% 5968 numa-vmstat.node1.nr_active_anon 4179 ± 5% +42.8% 5968 numa-vmstat.node1.nr_zone_active_anon 19143 ± 5% +37.3% 26281 meminfo.Active 17924 ± 5% +39.8% 25063 meminfo.Active(anon) 40996 ± 2% +22.7% 50289 meminfo.Shmem 88.00 +1.1% 89.00 vmstat.cpu.id 68791 -13.1% 59772 vmstat.system.cs 2602342 ± 2% -21.6% 2038976 vmstat.system.in 4.034e+09 -26.6% 2.96e+09 cpuidle.C1.time 1.249e+08 ± 2% -30.6% 86713543 ± 2% cpuidle.C1.usage 10583871 ± 19% +106.1% 21814475 ± 3% cpuidle.POLL.time 486292 ± 18% +23.0% 598021 ± 2% cpuidle.POLL.usage 2.41 -2.4 0.00 mpstat.cpu.all.irq% 0.15 -0.1 0.01 ± 49% mpstat.cpu.all.soft% 6.03 +1.4 7.47 mpstat.cpu.all.sys% 1.78 +0.3 2.08 ± 5% mpstat.cpu.all.usr% 0.00 ± 42% +133.3% 0.01 ± 25% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown] 0.01 ± 14% +68.0% 0.01 ± 18% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.01 ± 14% +68.0% 0.01 ± 18% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 3.07 ± 5% +35.4% 4.16 ± 23% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 2.63 ± 14% +27.0% 3.34 ± 17% sched_debug.cpu.clock.stddev 581.67 ± 4% -98.6% 8.36 ± 16% sched_debug.cpu.clock_task.stddev 20072 ± 41% -60.5% 7934 ±114% sched_debug.cpu.nr_switches.avg 34831 ± 29% -52.3% 16613 ± 69% sched_debug.cpu.nr_switches.max 1.00 -100.0% 0.00 sched_debug.sched_clock_stable() 24399 ± 6% -13.3% 21148 ± 7% slabinfo.cred_jar.active_objs 24409 ± 6% -13.3% 21161 ± 7% slabinfo.cred_jar.num_objs 8294 ± 2% -11.1% 7375 slabinfo.pid.active_objs 8294 ± 2% -11.1% 7375 slabinfo.pid.num_objs 29772 ± 3% -10.7% 26590 ± 6% slabinfo.proc_inode_cache.active_objs 29862 ± 3% -10.7% 26670 ± 6% slabinfo.proc_inode_cache.num_objs 511.67 +128.8% 1170 turbostat.Avg_MHz 18.44 +23.7 42.13 turbostat.Busy% 1.249e+08 ± 2% -30.6% 86709326 ± 2% turbostat.C1 65.77 -17.4 48.42 turbostat.C1% 80.65 -28.5% 57.65 turbostat.CPU%c1 1.7e+08 -22.0% 1.325e+08 turbostat.IRQ 55.67 ± 2% +3.9% 57.83 turbostat.PkgTmp 186.07 +9.8% 204.40 turbostat.PkgWatt 4482 ± 5% +39.9% 6270 proc-vmstat.nr_active_anon 19665 -2.8% 19111 proc-vmstat.nr_kernel_stack 11209 +9.6% 12282 proc-vmstat.nr_mapped 10271 ± 2% +22.5% 12583 proc-vmstat.nr_shmem 24169 -3.0% 23450 proc-vmstat.nr_slab_reclaimable 52904 -1.4% 52156 proc-vmstat.nr_slab_unreclaimable 4482 ± 5% +39.9% 6270 proc-vmstat.nr_zone_active_anon 35096977 -13.7% 30280778 ± 2% proc-vmstat.numa_hit 35010436 -13.8% 30194248 ± 2% proc-vmstat.numa_local 10212 ± 2% +35.3% 13813 ± 4% proc-vmstat.pgactivate 37511123 -14.1% 32225979 ± 2% proc-vmstat.pgalloc_normal 39225190 ± 2% -14.7% 33455229 proc-vmstat.pgfault 37366865 -14.2% 32075663 ± 2% proc-vmstat.pgfree 9.58 ± 3% -7.8% 8.84 ± 2% perf-stat.i.MPKI 6.556e+09 -2.9% 6.366e+09 perf-stat.i.branch-instructions 0.86 ± 2% +0.1 0.96 perf-stat.i.branch-miss-rate% 52874393 +10.8% 58588777 perf-stat.i.branch-misses 46443118 ± 2% -8.2% 42638901 perf-stat.i.cache-misses 2.597e+08 -7.0% 2.415e+08 perf-stat.i.cache-references 70898 -13.3% 61457 perf-stat.i.context-switches 2.02 +113.8% 4.32 perf-stat.i.cpi 5.52e+10 +111.9% 1.17e+11 perf-stat.i.cpu-cycles 1881 ± 4% -28.7% 1341 ± 2% perf-stat.i.cpu-migrations 1214 ± 2% +131.8% 2815 perf-stat.i.cycles-between-cache-misses 47257151 ± 3% -11.2% 41949022 ± 7% perf-stat.i.dTLB-load-misses 6.448e+09 -7.1% 5.988e+09 perf-stat.i.dTLB-loads 0.15 ± 4% -0.0 0.14 ± 3% perf-stat.i.dTLB-store-miss-rate% 7070051 ± 4% -18.9% 5731351 ± 3% perf-stat.i.dTLB-store-misses 4.621e+09 -14.0% 3.974e+09 perf-stat.i.dTLB-stores 49.48 +5.4 54.88 perf-stat.i.iTLB-load-miss-rate% 29623646 -9.3% 26860981 perf-stat.i.iTLB-load-misses 30337180 ± 2% -27.7% 21933274 ± 2% perf-stat.i.iTLB-loads 989.66 +7.8% 1066 perf-stat.i.instructions-per-iTLB-miss 0.50 -53.4% 0.23 perf-stat.i.ipc 298.29 ± 13% -33.3% 198.98 ± 16% perf-stat.i.major-faults 0.57 +111.9% 1.22 perf-stat.i.metric.GHz 0.79 -12.6% 0.69 perf-stat.i.metric.K/sec 186.95 -7.4% 173.13 perf-stat.i.metric.M/sec 619972 -14.5% 529911 perf-stat.i.minor-faults 16950944 -12.8% 14775464 perf-stat.i.node-load-misses 1158174 -8.3% 1062165 perf-stat.i.node-loads 82.15 +1.0 83.16 perf-stat.i.node-store-miss-rate% 7266499 ± 3% -9.8% 6556262 ± 2% perf-stat.i.node-store-misses 1461472 ± 3% -14.1% 1254949 ± 3% perf-stat.i.node-stores 624047 -14.5% 533390 perf-stat.i.page-faults 9.52 -6.0% 8.95 perf-stat.overall.MPKI 0.81 +0.1 0.92 perf-stat.overall.branch-miss-rate% 2.02 +114.3% 4.33 perf-stat.overall.cpi 1190 ± 2% +130.8% 2747 perf-stat.overall.cycles-between-cache-misses 0.15 ± 4% -0.0 0.14 ± 3% perf-stat.overall.dTLB-store-miss-rate% 49.41 +5.6 55.05 perf-stat.overall.iTLB-load-miss-rate% 921.55 +9.1% 1004 perf-stat.overall.instructions-per-iTLB-miss 0.49 -53.3% 0.23 perf-stat.overall.ipc 6.456e+09 -2.9% 6.268e+09 perf-stat.ps.branch-instructions 52068658 +10.8% 57699339 perf-stat.ps.branch-misses 45689361 ± 2% -8.2% 41941220 perf-stat.ps.cache-misses 2.559e+08 -7.0% 2.379e+08 perf-stat.ps.cache-references 69875 -13.4% 60528 perf-stat.ps.context-switches 5.437e+10 +111.9% 1.152e+11 perf-stat.ps.cpu-cycles 1856 ± 4% -28.7% 1322 ± 2% perf-stat.ps.cpu-migrations 46573389 ± 3% -11.2% 41343641 ± 7% perf-stat.ps.dTLB-load-misses 6.354e+09 -7.1% 5.9e+09 perf-stat.ps.dTLB-loads 6967873 ± 4% -19.0% 5646326 ± 3% perf-stat.ps.dTLB-store-misses 4.553e+09 -14.0% 3.915e+09 perf-stat.ps.dTLB-stores 29182781 -9.3% 26458801 perf-stat.ps.iTLB-load-misses 29879883 ± 2% -27.7% 21606494 ± 2% perf-stat.ps.iTLB-loads 291.21 ± 13% -33.3% 194.27 ± 16% perf-stat.ps.major-faults 611056 ± 2% -14.6% 521852 perf-stat.ps.minor-faults 16664002 -12.8% 14524399 perf-stat.ps.node-load-misses 1161220 -8.4% 1063348 perf-stat.ps.node-loads 7141376 ± 3% -9.8% 6441085 ± 2% perf-stat.ps.node-store-misses 1446833 ± 3% -14.2% 1241728 ± 3% perf-stat.ps.node-stores 615068 ± 2% -14.6% 525276 perf-stat.ps.page-faults 1.71e+12 -2.0% 1.676e+12 perf-stat.total.instructions 14099 ± 12% -34.3% 9258 ± 4% softirqs.CPU0.RCU 12872 ± 6% +10.3% 14202 ± 3% softirqs.CPU0.SCHED 10233 ± 7% -23.9% 7788 ± 3% softirqs.CPU10.RCU 9723 ± 4% -20.3% 7752 ± 2% softirqs.CPU11.RCU 9728 ± 4% -17.5% 8022 ± 6% softirqs.CPU12.RCU 9552 ± 7% -19.0% 7739 ± 2% softirqs.CPU13.RCU 10179 ± 10% -22.5% 7891 ± 3% softirqs.CPU14.RCU 9655 ± 5% -16.5% 8064 ± 6% softirqs.CPU15.RCU 10140 ± 6% -24.1% 7694 softirqs.CPU16.RCU 10613 ± 6% -26.6% 7787 ± 2% softirqs.CPU17.RCU 10154 ± 4% -24.7% 7647 softirqs.CPU18.RCU 10153 ± 4% -25.0% 7614 ± 3% softirqs.CPU19.RCU 10271 ± 4% -23.3% 7875 ± 7% softirqs.CPU20.RCU 10452 ± 3% -24.6% 7884 ± 7% softirqs.CPU21.RCU 10249 ± 5% -25.0% 7682 ± 4% softirqs.CPU22.RCU 10037 ± 4% -24.3% 7595 ± 2% softirqs.CPU23.RCU 10086 ± 2% -22.8% 7791 ± 3% softirqs.CPU24.RCU 10789 ± 17% -25.4% 8052 ± 2% softirqs.CPU25.RCU 9938 ± 5% -21.0% 7847 ± 6% softirqs.CPU26.RCU 10044 ± 4% -24.3% 7604 ± 5% softirqs.CPU27.RCU 10007 ± 4% -24.6% 7544 ± 3% softirqs.CPU28.RCU 10093 ± 7% -24.6% 7613 ± 4% softirqs.CPU29.RCU 10018 ± 6% -24.5% 7560 ± 3% softirqs.CPU30.RCU 9857 ± 3% -22.5% 7642 ± 3% softirqs.CPU31.RCU 9618 ± 5% -20.6% 7633 ± 3% softirqs.CPU32.RCU 9621 ± 4% -21.7% 7528 ± 2% softirqs.CPU33.RCU 9840 ± 3% -22.3% 7641 ± 3% softirqs.CPU34.RCU 10038 ± 8% -25.6% 7465 ± 2% softirqs.CPU35.RCU 9852 ± 3% -22.1% 7678 ± 4% softirqs.CPU36.RCU 9936 ± 6% -22.7% 7676 ± 3% softirqs.CPU37.RCU 9721 ± 4% -22.7% 7516 ± 3% softirqs.CPU38.RCU 9628 ± 5% -21.4% 7572 ± 5% softirqs.CPU39.RCU 10003 ± 3% -22.2% 7787 ± 3% softirqs.CPU4.RCU 9694 ± 4% -22.1% 7550 softirqs.CPU40.RCU 9750 ± 4% -23.0% 7508 ± 2% softirqs.CPU41.RCU 9724 ± 5% -21.7% 7617 ± 5% softirqs.CPU42.RCU 9746 ± 3% -23.5% 7454 ± 2% softirqs.CPU43.RCU 9607 ± 5% -20.1% 7679 ± 5% softirqs.CPU44.RCU 9600 ± 5% -20.3% 7646 ± 7% softirqs.CPU45.RCU 10089 ± 8% -24.5% 7622 ± 5% softirqs.CPU46.RCU 9534 ± 5% -21.3% 7501 ± 2% softirqs.CPU47.RCU 10060 ± 4% -21.8% 7864 ± 5% softirqs.CPU48.RCU 9977 ± 5% -19.8% 8000 ± 4% softirqs.CPU49.RCU 10552 ± 7% -26.0% 7812 ± 3% softirqs.CPU50.RCU 10431 ± 5% -24.2% 7905 ± 4% softirqs.CPU51.RCU 10847 ± 16% -26.0% 8023 ± 5% softirqs.CPU52.RCU 10367 ± 6% -23.8% 7897 ± 4% softirqs.CPU53.RCU 10330 ± 3% -25.1% 7741 ± 3% softirqs.CPU54.RCU 10586 ± 6% -25.2% 7914 ± 2% softirqs.CPU55.RCU 10056 ± 7% -22.5% 7796 softirqs.CPU56.RCU 10704 ± 8% -25.8% 7942 ± 3% softirqs.CPU57.RCU 10371 ± 4% -24.7% 7812 ± 2% softirqs.CPU58.RCU 10294 ± 3% -24.1% 7812 ± 3% softirqs.CPU59.RCU 9981 ± 4% -20.8% 7909 ± 6% softirqs.CPU6.RCU 10241 ± 3% -22.3% 7954 ± 3% softirqs.CPU60.RCU 10376 ± 7% -23.6% 7929 ± 2% softirqs.CPU61.RCU 10290 ± 3% -22.7% 7958 ± 4% softirqs.CPU62.RCU 10234 ± 3% -23.2% 7856 ± 2% softirqs.CPU63.RCU 10616 ± 2% -23.3% 8143 ± 7% softirqs.CPU64.RCU 10725 ± 5% -26.9% 7835 softirqs.CPU65.RCU 10809 ± 3% -27.5% 7835 ± 3% softirqs.CPU66.RCU 10817 ± 5% -27.3% 7867 ± 3% softirqs.CPU67.RCU 10906 ± 3% -27.1% 7949 ± 4% softirqs.CPU68.RCU 11060 ± 3% -28.7% 7886 ± 5% softirqs.CPU69.RCU 9806 ± 4% -19.5% 7894 ± 3% softirqs.CPU7.RCU 10645 ± 4% -27.3% 7738 ± 2% softirqs.CPU70.RCU 10764 ± 8% -26.8% 7881 ± 2% softirqs.CPU71.RCU 10310 ± 5% -25.5% 7680 ± 3% softirqs.CPU72.RCU 10301 ± 3% -25.0% 7725 ± 3% softirqs.CPU73.RCU 10715 ± 2% -25.7% 7960 ± 4% softirqs.CPU74.RCU 10495 ± 4% -26.8% 7687 ± 2% softirqs.CPU75.RCU 10598 ± 8% -25.5% 7892 ± 4% softirqs.CPU76.RCU 10592 ± 9% -26.5% 7788 ± 3% softirqs.CPU77.RCU 10346 ± 4% -25.8% 7678 ± 5% softirqs.CPU78.RCU 10415 ± 5% -25.3% 7783 ± 3% softirqs.CPU79.RCU 9678 ± 5% -17.8% 7956 ± 4% softirqs.CPU8.RCU 10111 ± 5% -24.6% 7628 ± 3% softirqs.CPU80.RCU 10097 ± 3% -18.3% 8246 ± 4% softirqs.CPU81.RCU 10222 ± 4% -23.1% 7862 ± 3% softirqs.CPU82.RCU 10198 ± 5% -23.0% 7855 ± 2% softirqs.CPU83.RCU 10325 ± 4% -24.6% 7781 ± 4% softirqs.CPU84.RCU 10255 ± 3% -21.9% 8005 ± 3% softirqs.CPU85.RCU 10194 ± 4% -22.0% 7951 ± 2% softirqs.CPU86.RCU 10184 ± 5% -24.7% 7671 ± 2% softirqs.CPU87.RCU 10368 ± 3% -25.3% 7749 ± 2% softirqs.CPU88.RCU 10430 ± 5% -24.6% 7862 ± 4% softirqs.CPU89.RCU 9775 ± 5% -21.5% 7676 ± 2% softirqs.CPU9.RCU 9910 ± 4% -21.9% 7740 ± 6% softirqs.CPU90.RCU 10147 ± 4% -23.6% 7751 ± 3% softirqs.CPU91.RCU 10315 ± 7% -25.4% 7693 ± 3% softirqs.CPU92.RCU 10103 ± 5% -24.8% 7594 ± 2% softirqs.CPU93.RCU 10657 ± 9% -27.6% 7713 ± 3% softirqs.CPU94.RCU 10088 ± 4% -22.8% 7783 ± 5% softirqs.CPU95.RCU 980712 ± 4% -23.2% 753584 softirqs.RCU 0.00 +6.5e+103% 65.00 ±123% interrupts.192:PCI-MSI.31983644-edge.i40e-eth1-TxRx-27 1.584e+08 -24.6% 1.194e+08 interrupts.CAL:Function_call_interrupts 1553818 -21.9% 1212996 interrupts.CPU0.CAL:Function_call_interrupts 1916 ± 20% +99.9% 3829 ± 2% interrupts.CPU0.NMI:Non-maskable_interrupts 1916 ± 20% +99.9% 3829 ± 2% interrupts.CPU0.PMI:Performance_monitoring_interrupts 451.50 ± 10% +231.5% 1496 ± 9% interrupts.CPU0.RES:Rescheduling_interrupts 92.83 ± 27% +34.5% 124.83 ± 12% interrupts.CPU0.TLB:TLB_shootdowns 1555201 -22.0% 1213457 interrupts.CPU1.CAL:Function_call_interrupts 1693 ± 36% +134.4% 3969 ± 3% interrupts.CPU1.NMI:Non-maskable_interrupts 1693 ± 36% +134.4% 3969 ± 3% interrupts.CPU1.PMI:Performance_monitoring_interrupts 397.67 ± 6% +280.2% 1512 ± 9% interrupts.CPU1.RES:Rescheduling_interrupts 1560119 -22.1% 1215683 interrupts.CPU10.CAL:Function_call_interrupts 365.67 ± 10% +290.6% 1428 ± 11% interrupts.CPU10.RES:Rescheduling_interrupts 1560624 -22.1% 1215999 interrupts.CPU11.CAL:Function_call_interrupts 366.50 ± 14% +287.1% 1418 ± 11% interrupts.CPU11.RES:Rescheduling_interrupts 1561507 -22.1% 1215913 interrupts.CPU12.CAL:Function_call_interrupts 373.00 ± 8% +289.9% 1454 ± 13% interrupts.CPU12.RES:Rescheduling_interrupts 1562400 -22.1% 1216393 interrupts.CPU13.CAL:Function_call_interrupts 355.17 ± 11% +303.6% 1433 ± 14% interrupts.CPU13.RES:Rescheduling_interrupts 80.67 ± 23% +33.7% 107.83 ± 12% interrupts.CPU13.TLB:TLB_shootdowns 1562669 -22.2% 1216425 interrupts.CPU14.CAL:Function_call_interrupts 398.50 ± 13% +271.4% 1480 ± 11% interrupts.CPU14.RES:Rescheduling_interrupts 1564260 -22.2% 1217102 interrupts.CPU15.CAL:Function_call_interrupts 346.50 ± 9% +306.3% 1408 ± 8% interrupts.CPU15.RES:Rescheduling_interrupts 1564778 -22.2% 1217389 interrupts.CPU16.CAL:Function_call_interrupts 344.67 ± 14% +284.7% 1325 ± 11% interrupts.CPU16.RES:Rescheduling_interrupts 71.00 ± 10% +54.5% 109.67 ± 19% interrupts.CPU16.TLB:TLB_shootdowns 1565548 -22.2% 1217621 interrupts.CPU17.CAL:Function_call_interrupts 343.50 ± 14% +307.0% 1398 ± 11% interrupts.CPU17.RES:Rescheduling_interrupts 1566249 -22.3% 1217608 interrupts.CPU18.CAL:Function_call_interrupts 323.00 ± 8% +334.4% 1403 ± 10% interrupts.CPU18.RES:Rescheduling_interrupts 74.50 ± 12% +54.1% 114.83 ± 25% interrupts.CPU18.TLB:TLB_shootdowns 1566835 -22.2% 1218368 interrupts.CPU19.CAL:Function_call_interrupts 357.50 ± 13% +286.9% 1383 ± 15% interrupts.CPU19.RES:Rescheduling_interrupts 76.33 ± 15% +41.7% 108.17 ± 21% interrupts.CPU19.TLB:TLB_shootdowns 1555036 -21.9% 1213727 interrupts.CPU2.CAL:Function_call_interrupts 1916 ± 29% +103.5% 3900 ± 3% interrupts.CPU2.NMI:Non-maskable_interrupts 1916 ± 29% +103.5% 3900 ± 3% interrupts.CPU2.PMI:Performance_monitoring_interrupts 405.67 ± 3% +255.6% 1442 ± 13% interrupts.CPU2.RES:Rescheduling_interrupts 1566774 -22.2% 1218796 interrupts.CPU20.CAL:Function_call_interrupts 379.00 ± 9% +267.4% 1392 ± 11% interrupts.CPU20.RES:Rescheduling_interrupts 1567370 -22.3% 1218379 interrupts.CPU21.CAL:Function_call_interrupts 384.67 ± 7% +276.9% 1450 ± 9% interrupts.CPU21.RES:Rescheduling_interrupts 1568808 -22.3% 1219084 interrupts.CPU22.CAL:Function_call_interrupts 363.83 ± 8% +288.5% 1413 ± 9% interrupts.CPU22.RES:Rescheduling_interrupts 1570464 -22.3% 1219491 interrupts.CPU23.CAL:Function_call_interrupts 331.33 ± 16% +309.1% 1355 ± 12% interrupts.CPU23.RES:Rescheduling_interrupts 69.00 ± 22% +58.7% 109.50 ± 28% interrupts.CPU23.TLB:TLB_shootdowns 1570852 -22.4% 1218594 interrupts.CPU24.CAL:Function_call_interrupts 360.67 ± 10% +300.6% 1444 ± 15% interrupts.CPU24.RES:Rescheduling_interrupts 1572421 -22.5% 1218832 interrupts.CPU25.CAL:Function_call_interrupts 361.50 ± 12% +313.7% 1495 ± 18% interrupts.CPU25.RES:Rescheduling_interrupts 1573135 -22.5% 1219681 interrupts.CPU26.CAL:Function_call_interrupts 346.50 ± 8% +312.0% 1427 ± 16% interrupts.CPU26.RES:Rescheduling_interrupts 81.83 ± 22% +50.1% 122.83 ± 20% interrupts.CPU26.TLB:TLB_shootdowns 0.00 +6.4e+103% 64.33 ±124% interrupts.CPU27.192:PCI-MSI.31983644-edge.i40e-eth1-TxRx-27 1573607 -22.5% 1219860 interrupts.CPU27.CAL:Function_call_interrupts 364.33 ± 11% +299.3% 1454 ± 18% interrupts.CPU27.RES:Rescheduling_interrupts 1574486 -22.5% 1220253 interrupts.CPU28.CAL:Function_call_interrupts 366.17 ± 4% +272.5% 1363 ± 16% interrupts.CPU28.RES:Rescheduling_interrupts 73.67 ± 22% +57.2% 115.83 ± 15% interrupts.CPU28.TLB:TLB_shootdowns 1576181 -22.5% 1220758 interrupts.CPU29.CAL:Function_call_interrupts 338.00 ± 12% +308.9% 1382 ± 14% interrupts.CPU29.RES:Rescheduling_interrupts 72.17 ± 18% +71.4% 123.67 ± 21% interrupts.CPU29.TLB:TLB_shootdowns 1555885 -22.0% 1213686 interrupts.CPU3.CAL:Function_call_interrupts 408.33 ± 9% +252.6% 1439 ± 8% interrupts.CPU3.RES:Rescheduling_interrupts 1577545 -22.6% 1220782 interrupts.CPU30.CAL:Function_call_interrupts 340.50 ± 10% +306.1% 1382 ± 12% interrupts.CPU30.RES:Rescheduling_interrupts 1579433 -22.7% 1221357 interrupts.CPU31.CAL:Function_call_interrupts 340.83 ± 7% +295.1% 1346 ± 18% interrupts.CPU31.RES:Rescheduling_interrupts 1580661 -22.7% 1221936 interrupts.CPU32.CAL:Function_call_interrupts 342.17 ± 8% +281.4% 1305 ± 12% interrupts.CPU32.RES:Rescheduling_interrupts 60.33 ± 23% +76.5% 106.50 ± 22% interrupts.CPU32.TLB:TLB_shootdowns 1581769 -22.7% 1222330 interrupts.CPU33.CAL:Function_call_interrupts 2003 ± 32% +105.4% 4115 ± 4% interrupts.CPU33.NMI:Non-maskable_interrupts 2003 ± 32% +105.4% 4115 ± 4% interrupts.CPU33.PMI:Performance_monitoring_interrupts 342.00 ± 12% +301.3% 1372 ± 12% interrupts.CPU33.RES:Rescheduling_interrupts 1582830 -22.8% 1222702 interrupts.CPU34.CAL:Function_call_interrupts 357.00 ± 7% +288.3% 1386 ± 14% interrupts.CPU34.RES:Rescheduling_interrupts 1586009 -22.9% 1223200 interrupts.CPU35.CAL:Function_call_interrupts 318.67 ± 13% +291.8% 1248 ± 10% interrupts.CPU35.RES:Rescheduling_interrupts 1589398 -23.0% 1223144 interrupts.CPU36.CAL:Function_call_interrupts 2551 ± 10% +61.6% 4122 ± 5% interrupts.CPU36.NMI:Non-maskable_interrupts 2551 ± 10% +61.6% 4122 ± 5% interrupts.CPU36.PMI:Performance_monitoring_interrupts 327.17 ± 9% +332.6% 1415 ± 16% interrupts.CPU36.RES:Rescheduling_interrupts 79.33 ± 24% +62.6% 129.00 ± 19% interrupts.CPU36.TLB:TLB_shootdowns 1592078 -23.1% 1223994 interrupts.CPU37.CAL:Function_call_interrupts 2525 ± 5% +62.1% 4095 ± 5% interrupts.CPU37.NMI:Non-maskable_interrupts 2525 ± 5% +62.1% 4095 ± 5% interrupts.CPU37.PMI:Performance_monitoring_interrupts 327.67 ± 15% +300.2% 1311 ± 11% interrupts.CPU37.RES:Rescheduling_interrupts 1593159 -23.1% 1224772 interrupts.CPU38.CAL:Function_call_interrupts 2449 ± 5% +71.6% 4203 ± 3% interrupts.CPU38.NMI:Non-maskable_interrupts 2449 ± 5% +71.6% 4203 ± 3% interrupts.CPU38.PMI:Performance_monitoring_interrupts 329.17 ± 11% +284.5% 1265 ± 9% interrupts.CPU38.RES:Rescheduling_interrupts 77.33 ± 15% +53.0% 118.33 ± 17% interrupts.CPU38.TLB:TLB_shootdowns 1597729 -23.3% 1225532 interrupts.CPU39.CAL:Function_call_interrupts 2375 ± 7% +75.5% 4169 ± 4% interrupts.CPU39.NMI:Non-maskable_interrupts 2375 ± 7% +75.5% 4169 ± 4% interrupts.CPU39.PMI:Performance_monitoring_interrupts 321.33 ± 15% +294.8% 1268 ± 10% interrupts.CPU39.RES:Rescheduling_interrupts 68.33 ± 35% +68.3% 115.00 ± 26% interrupts.CPU39.TLB:TLB_shootdowns 1556572 -22.0% 1213891 interrupts.CPU4.CAL:Function_call_interrupts 370.17 ± 12% +288.8% 1439 ± 10% interrupts.CPU4.RES:Rescheduling_interrupts 76.17 ± 15% +49.9% 114.17 ± 12% interrupts.CPU4.TLB:TLB_shootdowns 1599710 -23.4% 1226042 interrupts.CPU40.CAL:Function_call_interrupts 2477 ± 12% +67.3% 4144 ± 5% interrupts.CPU40.NMI:Non-maskable_interrupts 2477 ± 12% +67.3% 4144 ± 5% interrupts.CPU40.PMI:Performance_monitoring_interrupts 324.83 ± 15% +304.5% 1314 ± 11% interrupts.CPU40.RES:Rescheduling_interrupts 1602951 -23.5% 1226709 interrupts.CPU41.CAL:Function_call_interrupts 2321 ± 8% +79.0% 4155 ± 3% interrupts.CPU41.NMI:Non-maskable_interrupts 2321 ± 8% +79.0% 4155 ± 3% interrupts.CPU41.PMI:Performance_monitoring_interrupts 320.67 ± 8% +285.3% 1235 ± 15% interrupts.CPU41.RES:Rescheduling_interrupts 73.00 ± 18% +55.9% 113.83 ± 19% interrupts.CPU41.TLB:TLB_shootdowns 1610138 -23.8% 1227114 interrupts.CPU42.CAL:Function_call_interrupts 2314 ± 5% +78.4% 4128 ± 5% interrupts.CPU42.NMI:Non-maskable_interrupts 2314 ± 5% +78.4% 4128 ± 5% interrupts.CPU42.PMI:Performance_monitoring_interrupts 310.33 ± 8% +312.2% 1279 ± 22% interrupts.CPU42.RES:Rescheduling_interrupts 66.17 ± 20% +80.4% 119.33 ± 30% interrupts.CPU42.TLB:TLB_shootdowns 1611973 -23.8% 1227608 interrupts.CPU43.CAL:Function_call_interrupts 2347 ± 9% +75.8% 4125 ± 5% interrupts.CPU43.NMI:Non-maskable_interrupts 2347 ± 9% +75.8% 4125 ± 5% interrupts.CPU43.PMI:Performance_monitoring_interrupts 303.50 ± 8% +310.8% 1246 ± 13% interrupts.CPU43.RES:Rescheduling_interrupts 64.50 ± 16% +117.1% 140.00 ± 24% interrupts.CPU43.TLB:TLB_shootdowns 1614258 -23.9% 1228447 interrupts.CPU44.CAL:Function_call_interrupts 2414 ± 8% +70.8% 4123 ± 5% interrupts.CPU44.NMI:Non-maskable_interrupts 2414 ± 8% +70.8% 4123 ± 5% interrupts.CPU44.PMI:Performance_monitoring_interrupts 305.33 ± 7% +316.4% 1271 ± 12% interrupts.CPU44.RES:Rescheduling_interrupts 1618727 -24.0% 1229501 interrupts.CPU45.CAL:Function_call_interrupts 2425 ± 8% +71.1% 4150 ± 6% interrupts.CPU45.NMI:Non-maskable_interrupts 2425 ± 8% +71.1% 4150 ± 6% interrupts.CPU45.PMI:Performance_monitoring_interrupts 292.83 ± 7% +320.0% 1229 ± 10% interrupts.CPU45.RES:Rescheduling_interrupts 1620732 -24.1% 1229767 interrupts.CPU46.CAL:Function_call_interrupts 2152 ± 20% +91.7% 4126 ± 5% interrupts.CPU46.NMI:Non-maskable_interrupts 2152 ± 20% +91.7% 4126 ± 5% interrupts.CPU46.PMI:Performance_monitoring_interrupts 307.33 ± 14% +313.6% 1271 ± 17% interrupts.CPU46.RES:Rescheduling_interrupts 63.17 ± 22% +91.6% 121.00 ± 22% interrupts.CPU46.TLB:TLB_shootdowns 1625711 -24.3% 1230630 interrupts.CPU47.CAL:Function_call_interrupts 1913 ± 29% +116.8% 4147 ± 3% interrupts.CPU47.NMI:Non-maskable_interrupts 1913 ± 29% +116.8% 4147 ± 3% interrupts.CPU47.PMI:Performance_monitoring_interrupts 310.33 ± 9% +331.0% 1337 ± 17% interrupts.CPU47.RES:Rescheduling_interrupts 63.67 ± 20% +103.9% 129.83 ± 28% interrupts.CPU47.TLB:TLB_shootdowns 1635071 -24.6% 1233173 interrupts.CPU48.CAL:Function_call_interrupts 1749 ± 33% +117.6% 3806 ± 2% interrupts.CPU48.NMI:Non-maskable_interrupts 1749 ± 33% +117.6% 3806 ± 2% interrupts.CPU48.PMI:Performance_monitoring_interrupts 440.50 ± 13% +285.3% 1697 ± 15% interrupts.CPU48.RES:Rescheduling_interrupts 1636847 -24.6% 1234586 interrupts.CPU49.CAL:Function_call_interrupts 1907 ± 32% +107.9% 3965 ± 3% interrupts.CPU49.NMI:Non-maskable_interrupts 1907 ± 32% +107.9% 3965 ± 3% interrupts.CPU49.PMI:Performance_monitoring_interrupts 528.83 ± 14% +261.5% 1911 ± 8% interrupts.CPU49.RES:Rescheduling_interrupts 63.67 ± 9% +74.3% 111.00 ± 19% interrupts.CPU49.TLB:TLB_shootdowns 1557508 -22.0% 1214275 interrupts.CPU5.CAL:Function_call_interrupts 360.00 ± 11% +304.7% 1456 ± 13% interrupts.CPU5.RES:Rescheduling_interrupts 1638985 -24.6% 1235069 interrupts.CPU50.CAL:Function_call_interrupts 1916 ± 28% +103.7% 3903 ± 4% interrupts.CPU50.NMI:Non-maskable_interrupts 1916 ± 28% +103.7% 3903 ± 4% interrupts.CPU50.PMI:Performance_monitoring_interrupts 543.83 ± 8% +252.8% 1918 ± 14% interrupts.CPU50.RES:Rescheduling_interrupts 81.67 ± 11% +34.9% 110.17 ± 13% interrupts.CPU50.TLB:TLB_shootdowns 1644500 -24.8% 1236087 interrupts.CPU51.CAL:Function_call_interrupts 1849 ± 35% +116.5% 4003 ± 2% interrupts.CPU51.NMI:Non-maskable_interrupts 1849 ± 35% +116.5% 4003 ± 2% interrupts.CPU51.PMI:Performance_monitoring_interrupts 521.17 ± 12% +261.6% 1884 ± 7% interrupts.CPU51.RES:Rescheduling_interrupts 73.17 ± 22% +47.4% 107.83 ± 28% interrupts.CPU51.TLB:TLB_shootdowns 1645993 -24.8% 1237456 interrupts.CPU52.CAL:Function_call_interrupts 529.00 ± 5% +265.5% 1933 ± 8% interrupts.CPU52.RES:Rescheduling_interrupts 73.00 ± 16% +50.0% 109.50 ± 16% interrupts.CPU52.TLB:TLB_shootdowns 1649801 -25.0% 1238136 interrupts.CPU53.CAL:Function_call_interrupts 543.17 ± 7% +249.9% 1900 ± 12% interrupts.CPU53.RES:Rescheduling_interrupts 73.50 ± 24% +54.4% 113.50 ± 18% interrupts.CPU53.TLB:TLB_shootdowns 1654935 -25.1% 1239258 interrupts.CPU54.CAL:Function_call_interrupts 560.50 ± 8% +239.8% 1904 ± 9% interrupts.CPU54.RES:Rescheduling_interrupts 72.33 ± 13% +53.7% 111.17 ± 15% interrupts.CPU54.TLB:TLB_shootdowns 1667829 -25.6% 1240627 interrupts.CPU55.CAL:Function_call_interrupts 553.50 ± 13% +251.8% 1947 ± 7% interrupts.CPU55.RES:Rescheduling_interrupts 1673782 -25.8% 1241296 interrupts.CPU56.CAL:Function_call_interrupts 2134 ± 26% +85.3% 3955 ± 3% interrupts.CPU56.NMI:Non-maskable_interrupts 2134 ± 26% +85.3% 3955 ± 3% interrupts.CPU56.PMI:Performance_monitoring_interrupts 558.50 ± 11% +232.8% 1858 ± 10% interrupts.CPU56.RES:Rescheduling_interrupts 1673858 -25.7% 1242929 interrupts.CPU57.CAL:Function_call_interrupts 2169 ± 23% +84.0% 3990 ± 4% interrupts.CPU57.NMI:Non-maskable_interrupts 2169 ± 23% +84.0% 3990 ± 4% interrupts.CPU57.PMI:Performance_monitoring_interrupts 558.17 ± 2% +259.2% 2004 ± 11% interrupts.CPU57.RES:Rescheduling_interrupts 1675001 -25.8% 1243500 interrupts.CPU58.CAL:Function_call_interrupts 2143 ± 25% +82.1% 3903 ± 3% interrupts.CPU58.NMI:Non-maskable_interrupts 2143 ± 25% +82.1% 3903 ± 3% interrupts.CPU58.PMI:Performance_monitoring_interrupts 586.00 ± 13% +241.5% 2001 ± 9% interrupts.CPU58.RES:Rescheduling_interrupts 76.67 ± 9% +33.5% 102.33 ± 11% interrupts.CPU58.TLB:TLB_shootdowns 1677140 -25.8% 1245091 interrupts.CPU59.CAL:Function_call_interrupts 603.83 ± 12% +219.1% 1927 ± 8% interrupts.CPU59.RES:Rescheduling_interrupts 1558142 -22.1% 1214507 interrupts.CPU6.CAL:Function_call_interrupts 345.83 ± 6% +304.8% 1399 ± 8% interrupts.CPU6.RES:Rescheduling_interrupts 78.17 ± 18% +47.8% 115.50 ± 15% interrupts.CPU6.TLB:TLB_shootdowns 1682308 -25.9% 1246000 interrupts.CPU60.CAL:Function_call_interrupts 2358 ± 8% +69.8% 4005 ± 4% interrupts.CPU60.NMI:Non-maskable_interrupts 2358 ± 8% +69.8% 4005 ± 4% interrupts.CPU60.PMI:Performance_monitoring_interrupts 556.00 ± 4% +254.7% 1972 ± 11% interrupts.CPU60.RES:Rescheduling_interrupts 1687753 -26.1% 1247152 interrupts.CPU61.CAL:Function_call_interrupts 571.67 ± 12% +238.6% 1935 ± 14% interrupts.CPU61.RES:Rescheduling_interrupts 1691195 -26.2% 1248322 interrupts.CPU62.CAL:Function_call_interrupts 554.00 ± 11% +256.3% 1973 ± 4% interrupts.CPU62.RES:Rescheduling_interrupts 1693421 -26.2% 1249765 interrupts.CPU63.CAL:Function_call_interrupts 578.00 ± 8% +252.9% 2040 ± 12% interrupts.CPU63.RES:Rescheduling_interrupts 1693802 -26.2% 1250259 interrupts.CPU64.CAL:Function_call_interrupts 576.50 ± 13% +237.9% 1947 ± 4% interrupts.CPU64.RES:Rescheduling_interrupts 70.83 ± 14% +48.2% 105.00 ± 18% interrupts.CPU64.TLB:TLB_shootdowns 1696044 -26.2% 1252071 interrupts.CPU65.CAL:Function_call_interrupts 2000 ± 20% +80.8% 3616 ± 17% interrupts.CPU65.NMI:Non-maskable_interrupts 2000 ± 20% +80.8% 3616 ± 17% interrupts.CPU65.PMI:Performance_monitoring_interrupts 592.67 ± 17% +238.7% 2007 ± 6% interrupts.CPU65.RES:Rescheduling_interrupts 1698659 -26.3% 1252352 interrupts.CPU66.CAL:Function_call_interrupts 544.00 ± 11% +261.3% 1965 ± 10% interrupts.CPU66.RES:Rescheduling_interrupts 1703027 -26.3% 1254746 interrupts.CPU67.CAL:Function_call_interrupts 558.33 ± 16% +264.0% 2032 ± 12% interrupts.CPU67.RES:Rescheduling_interrupts 1705223 -26.3% 1256151 interrupts.CPU68.CAL:Function_call_interrupts 546.50 ± 13% +271.8% 2031 ± 12% interrupts.CPU68.RES:Rescheduling_interrupts 1706279 -26.3% 1257075 interrupts.CPU69.CAL:Function_call_interrupts 2204 ± 26% +84.2% 4061 ± 2% interrupts.CPU69.NMI:Non-maskable_interrupts 2204 ± 26% +84.2% 4061 ± 2% interrupts.CPU69.PMI:Performance_monitoring_interrupts 594.17 ± 8% +233.8% 1983 ± 9% interrupts.CPU69.RES:Rescheduling_interrupts 1558831 -22.1% 1214955 interrupts.CPU7.CAL:Function_call_interrupts 375.17 ± 12% +281.9% 1432 ± 7% interrupts.CPU7.RES:Rescheduling_interrupts 80.33 ± 17% +32.0% 106.00 ± 17% interrupts.CPU7.TLB:TLB_shootdowns 1708194 -26.3% 1259777 interrupts.CPU70.CAL:Function_call_interrupts 2167 ± 19% +88.7% 4090 ± 2% interrupts.CPU70.NMI:Non-maskable_interrupts 2167 ± 19% +88.7% 4090 ± 2% interrupts.CPU70.PMI:Performance_monitoring_interrupts 609.17 ± 3% +235.2% 2042 ± 6% interrupts.CPU70.RES:Rescheduling_interrupts 1713148 -26.4% 1261613 interrupts.CPU71.CAL:Function_call_interrupts 2097 ± 27% +96.9% 4130 ± 3% interrupts.CPU71.NMI:Non-maskable_interrupts 2097 ± 27% +96.9% 4130 ± 3% interrupts.CPU71.PMI:Performance_monitoring_interrupts 578.50 ± 15% +254.4% 2050 ± 12% interrupts.CPU71.RES:Rescheduling_interrupts 70.17 ± 16% +54.6% 108.50 ± 21% interrupts.CPU71.TLB:TLB_shootdowns 1718192 -26.6% 1260936 interrupts.CPU72.CAL:Function_call_interrupts 1990 ± 26% +111.9% 4217 ± 6% interrupts.CPU72.NMI:Non-maskable_interrupts 1990 ± 26% +111.9% 4217 ± 6% interrupts.CPU72.PMI:Performance_monitoring_interrupts 519.33 ± 9% +267.9% 1910 ± 21% interrupts.CPU72.RES:Rescheduling_interrupts 1722067 -26.7% 1262027 interrupts.CPU73.CAL:Function_call_interrupts 2142 ± 23% +92.1% 4116 ± 3% interrupts.CPU73.NMI:Non-maskable_interrupts 2142 ± 23% +92.1% 4116 ± 3% interrupts.CPU73.PMI:Performance_monitoring_interrupts 531.17 ± 13% +271.4% 1972 ± 12% interrupts.CPU73.RES:Rescheduling_interrupts 1724351 -26.6% 1264962 interrupts.CPU74.CAL:Function_call_interrupts 2222 ± 23% +93.6% 4301 ± 6% interrupts.CPU74.NMI:Non-maskable_interrupts 2222 ± 23% +93.6% 4301 ± 6% interrupts.CPU74.PMI:Performance_monitoring_interrupts 547.17 ± 10% +287.1% 2118 ± 16% interrupts.CPU74.RES:Rescheduling_interrupts 73.50 ± 22% +56.7% 115.17 ± 18% interrupts.CPU74.TLB:TLB_shootdowns 1724876 -26.6% 1265941 interrupts.CPU75.CAL:Function_call_interrupts 2246 ± 24% +88.3% 4230 ± 6% interrupts.CPU75.NMI:Non-maskable_interrupts 2246 ± 24% +88.3% 4230 ± 6% interrupts.CPU75.PMI:Performance_monitoring_interrupts 530.83 ± 9% +276.8% 2000 ± 12% interrupts.CPU75.RES:Rescheduling_interrupts 1726311 -26.5% 1268269 interrupts.CPU76.CAL:Function_call_interrupts 2045 ± 25% +105.6% 4206 ± 5% interrupts.CPU76.NMI:Non-maskable_interrupts 2045 ± 25% +105.6% 4206 ± 5% interrupts.CPU76.PMI:Performance_monitoring_interrupts 547.17 ± 9% +288.8% 2127 ± 14% interrupts.CPU76.RES:Rescheduling_interrupts 77.17 ± 19% +51.4% 116.83 ± 12% interrupts.CPU76.TLB:TLB_shootdowns 1728950 -26.5% 1270182 interrupts.CPU77.CAL:Function_call_interrupts 2285 ± 17% +91.5% 4377 ± 4% interrupts.CPU77.NMI:Non-maskable_interrupts 2285 ± 17% +91.5% 4377 ± 4% interrupts.CPU77.PMI:Performance_monitoring_interrupts 564.67 ± 6% +269.6% 2087 ± 11% interrupts.CPU77.RES:Rescheduling_interrupts 74.17 ± 21% +65.4% 122.67 ± 23% interrupts.CPU77.TLB:TLB_shootdowns 1733767 -26.7% 1271681 interrupts.CPU78.CAL:Function_call_interrupts 1760 ± 33% +135.2% 4140 ± 4% interrupts.CPU78.NMI:Non-maskable_interrupts 1760 ± 33% +135.2% 4140 ± 4% interrupts.CPU78.PMI:Performance_monitoring_interrupts 536.67 ± 7% +272.3% 1998 ± 13% interrupts.CPU78.RES:Rescheduling_interrupts 1739078 -26.8% 1273254 interrupts.CPU79.CAL:Function_call_interrupts 2014 ± 24% +108.6% 4200 ± 3% interrupts.CPU79.NMI:Non-maskable_interrupts 2014 ± 24% +108.6% 4200 ± 3% interrupts.CPU79.PMI:Performance_monitoring_interrupts 550.83 ± 11% +256.2% 1961 ± 12% interrupts.CPU79.RES:Rescheduling_interrupts 80.17 ± 22% +62.4% 130.17 ± 20% interrupts.CPU79.TLB:TLB_shootdowns 1560071 -22.1% 1214673 interrupts.CPU8.CAL:Function_call_interrupts 376.67 ± 7% +307.9% 1536 ± 16% interrupts.CPU8.RES:Rescheduling_interrupts 1739255 -26.6% 1276667 interrupts.CPU80.CAL:Function_call_interrupts 1776 ± 38% +134.2% 4160 ± 3% interrupts.CPU80.NMI:Non-maskable_interrupts 1776 ± 38% +134.2% 4160 ± 3% interrupts.CPU80.PMI:Performance_monitoring_interrupts 593.00 ± 8% +247.9% 2063 ± 12% interrupts.CPU80.RES:Rescheduling_interrupts 65.17 ± 20% +72.6% 112.50 ± 24% interrupts.CPU80.TLB:TLB_shootdowns 1744994 -26.7% 1279747 interrupts.CPU81.CAL:Function_call_interrupts 2179 ± 18% +72.9% 3768 ± 17% interrupts.CPU81.NMI:Non-maskable_interrupts 2179 ± 18% +72.9% 3768 ± 17% interrupts.CPU81.PMI:Performance_monitoring_interrupts 556.33 ± 6% +283.2% 2131 ± 15% interrupts.CPU81.RES:Rescheduling_interrupts 84.00 ± 19% +42.3% 119.50 ± 30% interrupts.CPU81.TLB:TLB_shootdowns 1746355 -26.5% 1283194 interrupts.CPU82.CAL:Function_call_interrupts 580.67 ± 13% +275.6% 2181 ± 11% interrupts.CPU82.RES:Rescheduling_interrupts 82.33 ± 23% +50.2% 123.67 ± 11% interrupts.CPU82.TLB:TLB_shootdowns 1752956 -26.6% 1286255 interrupts.CPU83.CAL:Function_call_interrupts 581.17 ± 11% +267.1% 2133 ± 12% interrupts.CPU83.RES:Rescheduling_interrupts 1764860 -26.9% 1289469 interrupts.CPU84.CAL:Function_call_interrupts 586.50 ± 12% +244.7% 2021 ± 12% interrupts.CPU84.RES:Rescheduling_interrupts 1771742 -27.0% 1293323 interrupts.CPU85.CAL:Function_call_interrupts 588.50 ± 11% +256.0% 2095 ± 10% interrupts.CPU85.RES:Rescheduling_interrupts 1771025 -26.8% 1296747 interrupts.CPU86.CAL:Function_call_interrupts 2281 ± 18% +85.3% 4229 ± 3% interrupts.CPU86.NMI:Non-maskable_interrupts 2281 ± 18% +85.3% 4229 ± 3% interrupts.CPU86.PMI:Performance_monitoring_interrupts 580.50 ± 6% +270.5% 2150 ± 11% interrupts.CPU86.RES:Rescheduling_interrupts 1781041 -27.0% 1300147 interrupts.CPU87.CAL:Function_call_interrupts 580.17 ± 8% +261.4% 2096 ± 8% interrupts.CPU87.RES:Rescheduling_interrupts 1784978 -26.9% 1304062 interrupts.CPU88.CAL:Function_call_interrupts 2354 ± 26% +78.2% 4195 ± 5% interrupts.CPU88.NMI:Non-maskable_interrupts 2354 ± 26% +78.2% 4195 ± 5% interrupts.CPU88.PMI:Performance_monitoring_interrupts 594.33 ± 11% +265.4% 2171 ± 11% interrupts.CPU88.RES:Rescheduling_interrupts 1790968 -26.9% 1308585 interrupts.CPU89.CAL:Function_call_interrupts 629.67 ± 8% +241.2% 2148 ± 9% interrupts.CPU89.RES:Rescheduling_interrupts 1559678 -22.1% 1215175 interrupts.CPU9.CAL:Function_call_interrupts 371.00 ± 17% +297.6% 1475 ± 11% interrupts.CPU9.RES:Rescheduling_interrupts 1819853 -28.0% 1310330 interrupts.CPU90.CAL:Function_call_interrupts 545.67 ± 6% +283.6% 2093 ± 15% interrupts.CPU90.RES:Rescheduling_interrupts 67.17 ± 20% +66.3% 111.67 ± 29% interrupts.CPU90.TLB:TLB_shootdowns 1817004 -27.7% 1313826 interrupts.CPU91.CAL:Function_call_interrupts 1953 ± 25% +114.4% 4189 ± 5% interrupts.CPU91.NMI:Non-maskable_interrupts 1953 ± 25% +114.4% 4189 ± 5% interrupts.CPU91.PMI:Performance_monitoring_interrupts 580.67 ± 14% +263.2% 2108 ± 13% interrupts.CPU91.RES:Rescheduling_interrupts 65.33 ± 17% +109.4% 136.83 ± 22% interrupts.CPU91.TLB:TLB_shootdowns 1820669 -27.7% 1316902 interrupts.CPU92.CAL:Function_call_interrupts 2018 ± 23% +88.0% 3794 ± 18% interrupts.CPU92.NMI:Non-maskable_interrupts 2018 ± 23% +88.0% 3794 ± 18% interrupts.CPU92.PMI:Performance_monitoring_interrupts 565.67 ± 8% +269.1% 2087 ± 15% interrupts.CPU92.RES:Rescheduling_interrupts 1828689 -27.7% 1321787 interrupts.CPU93.CAL:Function_call_interrupts 2272 ± 21% +66.8% 3789 ± 16% interrupts.CPU93.NMI:Non-maskable_interrupts 2272 ± 21% +66.8% 3789 ± 16% interrupts.CPU93.PMI:Performance_monitoring_interrupts 584.17 ± 5% +269.9% 2160 ± 6% interrupts.CPU93.RES:Rescheduling_interrupts 70.33 ± 18% +51.4% 106.50 ± 24% interrupts.CPU93.TLB:TLB_shootdowns 1825869 -27.5% 1323766 interrupts.CPU94.CAL:Function_call_interrupts 594.50 ± 6% +245.0% 2051 ± 10% interrupts.CPU94.RES:Rescheduling_interrupts 1835732 -27.0% 1340021 interrupts.CPU95.CAL:Function_call_interrupts 2131 ± 18% +81.2% 3862 ± 20% interrupts.CPU95.NMI:Non-maskable_interrupts 2131 ± 18% +81.2% 3862 ± 20% interrupts.CPU95.PMI:Performance_monitoring_interrupts 530.83 ± 6% +304.6% 2147 ± 13% interrupts.CPU95.RES:Rescheduling_interrupts 80.33 ± 7% +79.3% 144.00 ± 23% interrupts.CPU95.TLB:TLB_shootdowns 205031 ± 6% +77.8% 364501 interrupts.NMI:Non-maskable_interrupts 205031 ± 6% +77.8% 364501 interrupts.PMI:Performance_monitoring_interrupts 43840 ± 2% +272.5% 163324 ± 2% interrupts.RES:Rescheduling_interrupts 7385 +49.5% 11042 ± 3% interrupts.TLB:TLB_shootdowns 52.22 -29.6 22.66 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 19.19 ± 3% -8.4 10.84 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 19.26 ± 3% -8.4 10.91 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 7.11 ± 5% -3.5 3.64 perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.11 ± 5% -3.5 3.64 perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.10 ± 5% -3.5 3.64 perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.56 ± 5% -3.2 3.34 perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 6.53 ± 5% -3.2 3.32 perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 5.78 ± 2% -3.1 2.67 perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.cpuidle_enter_state.cpuidle_enter 5.59 ± 2% -3.0 2.58 perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.cpuidle_enter_state 6.29 -2.9 3.41 ± 2% perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.29 -2.9 3.41 ± 2% perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.93 -2.7 3.21 ± 2% perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.27 -2.5 2.73 perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64 5.16 -2.5 2.67 ± 2% perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process.kernel_clone.__do_sys_clone 3.44 ± 2% -2.3 1.19 ± 2% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 3.88 ± 5% -1.8 2.04 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault 3.85 ± 5% -1.8 2.02 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 3.95 ± 5% -1.8 2.13 ± 6% perf-profile.calltrace.cycles-pp.asm_exc_page_fault 3.35 ± 5% -1.6 1.76 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 3.23 ± 5% -1.5 1.69 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 2.35 ± 5% -1.2 1.18 ± 4% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit 3.86 ± 4% -1.1 2.73 ± 3% perf-profile.calltrace.cycles-pp.do_seccomp.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.85 ± 4% -1.1 2.73 ± 3% perf-profile.calltrace.cycles-pp.seccomp_set_mode_filter.do_seccomp.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.25 ± 6% -1.1 1.16 ± 3% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 2.24 ± 6% -1.1 1.18 ± 3% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit 2.17 ± 5% -1.1 1.12 ± 3% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 2.11 ± 7% -1.0 1.10 ± 3% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit 1.99 ± 2% -1.0 0.99 ± 4% perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mmap.dup_mm.copy_process.kernel_clone 1.96 ± 7% -0.9 1.03 ± 4% perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap.mmput 1.86 ± 2% -0.9 0.96 perf-profile.calltrace.cycles-pp.__flush_tlb_all.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function 3.38 ± 4% -0.9 2.50 ± 3% perf-profile.calltrace.cycles-pp.bpf_prog_create_from_user.seccomp_set_mode_filter.do_seccomp.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.83 ± 2% -0.9 0.95 perf-profile.calltrace.cycles-pp.native_flush_tlb_global.__flush_tlb_all.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function 1.11 ± 2% -0.8 0.26 ±100% perf-profile.calltrace.cycles-pp.llist_reverse_order.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function 1.63 ± 5% -0.8 0.82 ± 3% perf-profile.calltrace.cycles-pp.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.do_exit 1.47 ± 2% -0.7 0.74 ± 3% perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm.copy_process 1.51 ± 5% -0.7 0.84 ± 6% perf-profile.calltrace.cycles-pp.copy_page_range.dup_mmap.dup_mm.copy_process.kernel_clone 2.07 -0.7 1.41 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork 2.78 ± 4% -0.6 2.15 ± 3% perf-profile.calltrace.cycles-pp.bpf_prepare_filter.bpf_prog_create_from_user.seccomp_set_mode_filter.do_seccomp.do_syscall_64 1.31 ± 8% -0.6 0.67 ± 3% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit 1.30 ± 8% -0.6 0.67 ± 3% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit 2.74 ± 4% -0.6 2.13 ± 3% perf-profile.calltrace.cycles-pp.bpf_prog_select_runtime.bpf_prepare_filter.bpf_prog_create_from_user.seccomp_set_mode_filter.do_seccomp 2.73 ± 4% -0.6 2.12 ± 3% perf-profile.calltrace.cycles-pp.bpf_int_jit_compile.bpf_prog_select_runtime.bpf_prepare_filter.bpf_prog_create_from_user.seccomp_set_mode_filter 1.27 ± 6% -0.6 0.67 ± 4% perf-profile.calltrace.cycles-pp.next_uptodate_page.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault 1.02 ± 9% -0.6 0.44 ± 44% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput 1.88 -0.6 1.31 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 1.85 -0.6 1.30 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 1.22 ± 7% -0.5 0.68 ± 8% perf-profile.calltrace.cycles-pp.copy_pte_range.copy_page_range.dup_mmap.dup_mm.copy_process 1.76 -0.5 1.25 ± 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 1.52 -0.4 1.12 ± 2% perf-profile.calltrace.cycles-pp.bpf_jit_free.process_one_work.worker_thread.kthread.ret_from_fork 1.51 -0.4 1.11 ± 2% perf-profile.calltrace.cycles-pp.__vunmap.bpf_jit_binary_free.bpf_jit_free.process_one_work.worker_thread 1.51 -0.4 1.12 ± 2% perf-profile.calltrace.cycles-pp.bpf_jit_binary_free.bpf_jit_free.process_one_work.worker_thread.kthread 1.42 -0.3 1.07 perf-profile.calltrace.cycles-pp._vm_unmap_aliases.__vunmap.bpf_jit_binary_free.bpf_jit_free.process_one_work 0.68 ± 5% -0.3 0.35 ± 70% perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu.cpa_flush.change_page_attr_set_clr.set_memory_ro 1.06 ± 3% -0.3 0.78 ± 3% perf-profile.calltrace.cycles-pp.change_page_attr_set_clr.set_memory_x.bpf_int_jit_compile.bpf_prog_select_runtime.bpf_prepare_filter 1.06 ± 3% -0.3 0.78 ± 3% perf-profile.calltrace.cycles-pp.set_memory_x.bpf_int_jit_compile.bpf_prog_select_runtime.bpf_prepare_filter.bpf_prog_create_from_user 1.47 ± 4% -0.2 1.23 ± 3% perf-profile.calltrace.cycles-pp.change_page_attr_set_clr.set_memory_ro.bpf_int_jit_compile.bpf_prog_select_runtime.bpf_prepare_filter 1.47 ± 4% -0.2 1.23 ± 3% perf-profile.calltrace.cycles-pp.set_memory_ro.bpf_int_jit_compile.bpf_prog_select_runtime.bpf_prepare_filter.bpf_prog_create_from_user 0.69 ± 5% -0.2 0.53 ± 4% perf-profile.calltrace.cycles-pp.cpa_flush.change_page_attr_set_clr.set_memory_ro.bpf_int_jit_compile.bpf_prog_select_runtime 0.69 ± 5% -0.2 0.53 ± 4% perf-profile.calltrace.cycles-pp.on_each_cpu.cpa_flush.change_page_attr_set_clr.set_memory_ro.bpf_int_jit_compile 0.67 ± 46% +0.3 0.94 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify 0.67 ± 46% +0.3 0.94 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify 0.67 ± 46% +0.3 0.94 perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify 0.62 ± 46% +0.3 0.92 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify 0.60 ± 46% +0.3 0.91 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel 0.00 +0.7 0.66 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +0.7 0.69 perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +0.7 0.69 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times 0.00 +0.8 0.76 perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle 0.00 +0.8 0.76 perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 0.00 +0.9 0.85 ± 2% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.00 +0.9 0.90 ± 2% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +0.9 0.90 perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +1.1 1.06 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +1.1 1.08 perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 0.00 +1.1 1.09 perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +1.2 1.17 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt 0.00 +1.2 1.18 perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +1.2 1.18 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.00 +1.2 1.20 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +1.2 1.20 perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt 0.00 +1.2 1.20 perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +1.2 1.22 ± 2% perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +1.3 1.29 perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.00 +1.3 1.33 perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.56 ± 4% +1.6 2.16 perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +1.6 1.61 perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.00 +2.4 2.41 perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +2.4 2.41 perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 1.06 ± 14% +3.7 4.75 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.04 ± 14% +3.7 4.74 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 2.02 ± 11% +6.7 8.68 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 1.87 ± 11% +6.7 8.59 perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 0.00 +9.5 9.51 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_nohz_irq_exit.sysvec_call_function.asm_sysvec_call_function 0.00 +9.6 9.60 perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.sysvec_call_function.asm_sysvec_call_function.cpuidle_enter_state 0.00 +9.6 9.62 perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.sysvec_call_function.asm_sysvec_call_function.cpuidle_enter_state.cpuidle_enter 0.00 +10.6 10.61 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state 0.00 +10.8 10.85 perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter 0.00 +10.9 10.93 perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle 0.00 +11.0 11.04 perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_call_function 0.00 +11.1 11.09 perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 72.76 +11.1 83.89 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 72.76 +11.1 83.89 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 72.73 +11.2 83.88 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 0.00 +11.2 11.22 perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_call_function.asm_sysvec_call_function 73.49 +11.3 84.83 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 0.62 ± 4% +11.9 12.51 perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_call_function.asm_sysvec_call_function 0.00 +12.0 12.05 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter 0.00 +12.2 12.16 perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu 0.00 +12.4 12.36 perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_call_function 68.20 +13.7 81.95 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 66.98 +14.1 81.05 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 0.82 ± 3% +23.0 23.86 perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_call_function.asm_sysvec_call_function.cpuidle_enter_state.cpuidle_enter 0.79 ± 3% +23.1 23.84 perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_call_function.asm_sysvec_call_function.cpuidle_enter_state 12.46 +25.8 38.27 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 8.02 +28.5 36.53 perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function.cpuidle_enter_state.cpuidle_enter.do_idle 52.41 -29.5 22.89 perf-profile.children.cycles-pp.intel_idle 19.27 ± 3% -8.4 10.89 perf-profile.children.cycles-pp.do_syscall_64 19.33 ± 3% -8.4 10.96 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 7.14 ± 5% -3.5 3.65 perf-profile.children.cycles-pp.do_group_exit 7.14 ± 5% -3.5 3.65 perf-profile.children.cycles-pp.do_exit 7.11 ± 5% -3.5 3.64 perf-profile.children.cycles-pp.__x64_sys_exit_group 6.45 ± 2% -3.5 3.00 perf-profile.children.cycles-pp.__sysvec_call_function 6.24 ± 2% -3.3 2.90 perf-profile.children.cycles-pp.flush_smp_call_function_queue 6.59 ± 5% -3.2 3.35 perf-profile.children.cycles-pp.mmput 6.57 ± 5% -3.2 3.34 perf-profile.children.cycles-pp.exit_mmap 6.29 -2.9 3.42 ± 2% perf-profile.children.cycles-pp.__do_sys_clone 6.29 -2.9 3.42 ± 2% perf-profile.children.cycles-pp.kernel_clone 5.94 -2.7 3.21 ± 2% perf-profile.children.cycles-pp.copy_process 5.28 -2.5 2.74 ± 2% perf-profile.children.cycles-pp.dup_mm 5.17 -2.5 2.68 ± 2% perf-profile.children.cycles-pp.dup_mmap 3.50 ± 2% -2.3 1.21 ± 2% perf-profile.children.cycles-pp.menu_select 4.19 ± 4% -1.9 2.25 ± 5% perf-profile.children.cycles-pp.asm_exc_page_fault 4.06 ± 4% -1.9 2.13 perf-profile.children.cycles-pp.exc_page_fault 4.04 ± 4% -1.9 2.12 perf-profile.children.cycles-pp.do_user_addr_fault 3.47 ± 4% -1.7 1.82 perf-profile.children.cycles-pp.handle_mm_fault 3.34 ± 4% -1.6 1.74 perf-profile.children.cycles-pp.__handle_mm_fault 2.40 ± 4% -1.2 1.20 ± 4% perf-profile.children.cycles-pp.free_pgtables 2.42 ± 6% -1.2 1.27 ± 3% perf-profile.children.cycles-pp.unmap_vmas 3.86 ± 4% -1.1 2.73 ± 3% perf-profile.children.cycles-pp.seccomp_set_mode_filter 3.86 ± 4% -1.1 2.73 ± 3% perf-profile.children.cycles-pp.do_seccomp 2.28 ± 5% -1.1 1.16 ± 3% perf-profile.children.cycles-pp.do_fault 2.28 ± 6% -1.1 1.19 ± 3% perf-profile.children.cycles-pp.unmap_page_range 2.21 ± 5% -1.1 1.13 ± 3% perf-profile.children.cycles-pp.filemap_map_pages 2.14 ± 7% -1.0 1.12 ± 3% perf-profile.children.cycles-pp.zap_pte_range 2.08 ± 2% -1.0 1.07 perf-profile.children.cycles-pp.__flush_tlb_all 1.99 ± 2% -1.0 0.99 ± 4% perf-profile.children.cycles-pp.anon_vma_fork 2.05 ± 2% -1.0 1.06 perf-profile.children.cycles-pp.native_flush_tlb_global 3.38 ± 4% -0.9 2.50 ± 3% perf-profile.children.cycles-pp.bpf_prog_create_from_user 1.65 ± 5% -0.8 0.83 ± 2% perf-profile.children.cycles-pp.unlink_anon_vmas 1.49 -0.8 0.69 ± 3% perf-profile.children.cycles-pp.native_irq_return_iret 1.47 ± 5% -0.7 0.73 ± 2% perf-profile.children.cycles-pp.kmem_cache_free 1.50 ± 2% -0.7 0.76 ± 3% perf-profile.children.cycles-pp.anon_vma_clone 1.52 ± 5% -0.7 0.84 ± 6% perf-profile.children.cycles-pp.copy_page_range 2.07 -0.7 1.41 ± 2% perf-profile.children.cycles-pp.ret_from_fork 1.26 -0.7 0.60 ± 3% perf-profile.children.cycles-pp.irq_exit_rcu 1.36 ± 8% -0.7 0.70 ± 4% perf-profile.children.cycles-pp.tlb_finish_mmu 1.35 ± 8% -0.7 0.69 ± 4% perf-profile.children.cycles-pp.tlb_flush_mmu 1.23 ± 2% -0.7 0.57 perf-profile.children.cycles-pp.llist_reverse_order 2.79 ± 4% -0.6 2.15 ± 3% perf-profile.children.cycles-pp.bpf_prepare_filter 1.30 ± 6% -0.6 0.68 ± 5% perf-profile.children.cycles-pp.next_uptodate_page 2.74 ± 4% -0.6 2.13 ± 3% perf-profile.children.cycles-pp.bpf_prog_select_runtime 2.73 ± 4% -0.6 2.12 ± 3% perf-profile.children.cycles-pp.bpf_int_jit_compile 2.37 ± 2% -0.6 1.80 ± 2% perf-profile.children.cycles-pp.on_each_cpu 1.88 -0.6 1.31 ± 2% perf-profile.children.cycles-pp.kthread 1.86 -0.6 1.30 perf-profile.children.cycles-pp.worker_thread 2.61 ± 2% -0.6 2.06 ± 2% perf-profile.children.cycles-pp._vm_unmap_aliases 2.34 ± 2% -0.5 1.79 ± 2% perf-profile.children.cycles-pp.smp_call_function_many_cond 1.22 ± 7% -0.5 0.69 ± 9% perf-profile.children.cycles-pp.copy_pte_range 0.93 ± 2% -0.5 0.39 perf-profile.children.cycles-pp.__cpa_flush_tlb 1.11 ± 2% -0.5 0.58 perf-profile.children.cycles-pp.kmem_cache_alloc 1.08 ± 8% -0.5 0.55 ± 4% perf-profile.children.cycles-pp.release_pages 2.53 ± 3% -0.5 2.00 ± 3% perf-profile.children.cycles-pp.change_page_attr_set_clr 1.76 -0.5 1.25 ± 2% perf-profile.children.cycles-pp.process_one_work 0.98 ± 11% -0.5 0.47 ± 8% perf-profile.children.cycles-pp.native_flush_tlb_one_user 1.03 ± 3% -0.5 0.53 ± 2% perf-profile.children.cycles-pp.llist_add_batch 0.95 ± 12% -0.5 0.46 ± 8% perf-profile.children.cycles-pp.flush_tlb_one_kernel 1.65 -0.5 1.18 perf-profile.children.cycles-pp.__vunmap 0.51 ± 3% -0.5 0.06 perf-profile.children.cycles-pp.irqtime_account_irq 1.07 ± 3% -0.4 0.63 ± 2% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.73 ± 18% -0.4 0.30 ± 14% perf-profile.children.cycles-pp.do_kernel_range_flush 0.71 ± 4% -0.4 0.28 ± 3% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 1.52 -0.4 1.12 ± 2% perf-profile.children.cycles-pp.bpf_jit_free 1.51 -0.4 1.12 ± 2% perf-profile.children.cycles-pp.bpf_jit_binary_free 0.74 ± 6% -0.4 0.39 ± 2% perf-profile.children.cycles-pp.page_remove_rmap 0.73 ± 2% -0.4 0.37 ± 4% perf-profile.children.cycles-pp.vm_area_dup 0.62 ± 4% -0.3 0.29 ± 3% perf-profile.children.cycles-pp.remove_vma 0.68 ± 3% -0.3 0.35 ± 5% perf-profile.children.cycles-pp.anon_vma_interval_tree_insert 1.12 ± 2% -0.3 0.81 ± 2% perf-profile.children.cycles-pp.__purge_vmap_area_lazy 0.76 ± 4% -0.3 0.45 ± 3% perf-profile.children.cycles-pp.get_page_from_freelist 0.47 ± 8% -0.3 0.17 ± 6% perf-profile.children.cycles-pp.tick_nohz_next_event 1.30 ± 3% -0.3 1.01 ± 3% perf-profile.children.cycles-pp.cpa_flush 0.57 ± 6% -0.3 0.28 ± 10% perf-profile.children.cycles-pp.unlink_file_vma 0.59 ± 2% -0.3 0.31 ± 2% perf-profile.children.cycles-pp.__do_sys_wait4 1.06 ± 3% -0.3 0.78 ± 3% perf-profile.children.cycles-pp.set_memory_x 0.59 ± 2% -0.3 0.31 ± 2% perf-profile.children.cycles-pp.kernel_wait4 0.57 ± 2% -0.3 0.29 ± 4% perf-profile.children.cycles-pp.do_wait 0.60 ± 3% -0.3 0.33 ± 3% perf-profile.children.cycles-pp.__schedule 0.51 ± 3% -0.3 0.24 ± 4% perf-profile.children.cycles-pp.down_write 0.50 ± 5% -0.3 0.24 ± 3% perf-profile.children.cycles-pp._raw_spin_lock 0.65 ± 2% -0.3 0.39 ± 3% perf-profile.children.cycles-pp.__softirqentry_text_start 0.58 ± 12% -0.2 0.34 ± 3% perf-profile.children.cycles-pp.bpf_prog_alloc 1.47 ± 4% -0.2 1.23 ± 3% perf-profile.children.cycles-pp.set_memory_ro 0.47 ± 6% -0.2 0.24 ± 3% perf-profile.children.cycles-pp.__x64_sys_munmap 0.47 ± 7% -0.2 0.24 ± 3% perf-profile.children.cycles-pp.__vm_munmap 0.47 ± 6% -0.2 0.24 ± 3% perf-profile.children.cycles-pp.__do_munmap 0.43 ± 6% -0.2 0.20 ± 6% perf-profile.children.cycles-pp.do_set_pte 0.52 ± 2% -0.2 0.30 ± 5% perf-profile.children.cycles-pp.pte_alloc_one 0.42 ± 6% -0.2 0.20 ± 7% perf-profile.children.cycles-pp.seccomp_attach_filter 0.42 ± 6% -0.2 0.20 ± 7% perf-profile.children.cycles-pp.seccomp_cache_prepare_bitmap 0.35 ± 8% -0.2 0.14 ± 10% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 0.49 ± 4% -0.2 0.28 ± 5% perf-profile.children.cycles-pp.__pte_alloc 0.53 ± 4% -0.2 0.34 ± 4% perf-profile.children.cycles-pp.prep_new_page 0.52 ± 5% -0.2 0.32 ± 4% perf-profile.children.cycles-pp.clear_page_erms 0.39 ± 3% -0.2 0.19 ± 8% perf-profile.children.cycles-pp.up_write 0.44 ± 7% -0.2 0.24 ± 7% perf-profile.children.cycles-pp.do_anonymous_page 0.38 ± 7% -0.2 0.19 ± 5% perf-profile.children.cycles-pp.__slab_free 0.37 ± 12% -0.2 0.18 ± 5% perf-profile.children.cycles-pp.do_sys_open 0.37 ± 12% -0.2 0.18 ± 5% perf-profile.children.cycles-pp.do_sys_openat2 0.36 ± 7% -0.2 0.18 ± 5% perf-profile.children.cycles-pp.refill_obj_stock 0.38 ± 3% -0.2 0.19 ± 6% perf-profile.children.cycles-pp.anon_vma_interval_tree_remove 0.40 ± 8% -0.2 0.22 ± 6% perf-profile.children.cycles-pp.rcu_core 0.36 ± 4% -0.2 0.18 ± 5% perf-profile.children.cycles-pp.__memcg_kmem_charge 0.33 ± 4% -0.2 0.15 ± 5% perf-profile.children.cycles-pp.page_counter_try_charge 0.66 ± 3% -0.2 0.48 perf-profile.children.cycles-pp.sched_clock_cpu 0.27 ± 7% -0.2 0.09 ± 4% perf-profile.children.cycles-pp.default_send_IPI_mask_sequence_phys 0.34 ± 3% -0.2 0.16 ± 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.34 ± 5% -0.2 0.17 ± 3% perf-profile.children.cycles-pp.cpumask_next 0.40 ± 7% -0.2 0.23 ± 3% perf-profile.children.cycles-pp.__vmalloc_node_range 0.31 ± 6% -0.2 0.14 ± 6% perf-profile.children.cycles-pp.page_add_file_rmap 0.41 ± 12% -0.2 0.24 ± 3% perf-profile.children.cycles-pp.pcpu_alloc 0.38 ± 8% -0.2 0.21 ± 3% perf-profile.children.cycles-pp.bpf_prog_alloc_no_stats 0.31 ± 6% -0.2 0.15 ± 10% perf-profile.children.cycles-pp.vma_interval_tree_remove 0.36 ± 11% -0.2 0.20 ± 7% perf-profile.children.cycles-pp.rcu_do_batch 0.27 ± 4% -0.2 0.12 ± 6% perf-profile.children.cycles-pp.rcu_idle_exit 0.23 ± 7% -0.2 0.08 ± 6% perf-profile.children.cycles-pp.__default_send_IPI_dest_field 0.32 ± 4% -0.2 0.17 ± 6% perf-profile.children.cycles-pp.schedule 0.34 ± 5% -0.1 0.19 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair 0.33 ± 4% -0.1 0.18 ± 4% perf-profile.children.cycles-pp.wp_page_copy 0.32 ± 3% -0.1 0.17 ± 4% perf-profile.children.cycles-pp.wait_task_zombie 0.30 ± 4% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.release_task 0.28 ± 7% -0.1 0.14 ± 4% perf-profile.children.cycles-pp.find_vma 0.30 ± 3% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.do_filp_open 0.30 ± 3% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.path_openat 0.19 ± 4% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.__cpa_addr 0.33 ± 4% -0.1 0.18 ± 4% perf-profile.children.cycles-pp.load_balance 0.35 ± 3% -0.1 0.21 ± 3% perf-profile.children.cycles-pp.wake_up_new_task 0.28 ± 11% -0.1 0.14 ± 6% perf-profile.children.cycles-pp.unmap_region 0.33 ± 5% -0.1 0.19 ± 2% perf-profile.children.cycles-pp.dup_task_struct 0.28 ± 6% -0.1 0.15 ± 4% perf-profile.children.cycles-pp.__memcg_kmem_charge_page 0.27 ± 6% -0.1 0.14 ± 4% perf-profile.children.cycles-pp.free_pages_and_swap_cache 0.27 ± 5% -0.1 0.14 ± 4% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.28 ± 3% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.newidle_balance 0.21 ± 7% -0.1 0.09 ± 12% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.23 ± 5% -0.1 0.11 ± 5% perf-profile.children.cycles-pp.sync_regs 0.29 ± 3% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.find_busiest_group 0.28 ± 3% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.update_sd_lb_stats 0.30 ± 4% -0.1 0.18 ± 4% perf-profile.children.cycles-pp.select_task_rq_fair 0.23 ± 10% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.__put_anon_vma 0.58 ± 2% -0.1 0.46 perf-profile.children.cycles-pp.sched_clock 0.23 ± 4% -0.1 0.12 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid 0.23 ± 4% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 0.19 ± 5% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.rcu_eqs_exit 0.25 ± 10% -0.1 0.14 ± 4% perf-profile.children.cycles-pp.__get_vm_area_node 0.23 ± 4% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.idle_cpu 0.21 ± 7% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.drain_obj_stock 0.54 ± 2% -0.1 0.44 perf-profile.children.cycles-pp.native_sched_clock 0.21 ± 4% -0.1 0.11 ± 6% perf-profile.children.cycles-pp._find_next_bit 0.26 ± 5% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.find_idlest_group 0.23 ± 12% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.alloc_vmap_area 0.24 ± 15% -0.1 0.14 ± 12% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.21 ± 3% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.dup_userfaultfd 0.20 ± 8% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.unlock_page 0.18 ± 15% -0.1 0.08 ± 6% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.19 ± 11% -0.1 0.10 ± 13% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.21 ± 6% -0.1 0.12 ± 8% perf-profile.children.cycles-pp.alloc_pages_vma 0.19 ± 6% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.___might_sleep 0.38 ± 5% -0.1 0.29 ± 5% perf-profile.children.cycles-pp.flush_tlb_kernel_range 0.20 ± 4% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.obj_cgroup_charge 0.20 ± 4% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.__x64_sys_mprotect 0.20 ± 4% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.do_mprotect_pkey 1.13 ± 3% -0.1 1.04 ± 4% perf-profile.children.cycles-pp.__mutex_lock 0.19 ± 5% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.rmqueue 0.17 ± 8% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.page_counter_uncharge 0.13 ± 8% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.tick_check_broadcast_expired 0.15 ± 4% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.call_cpuidle 0.17 ± 9% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.page_counter_cancel 0.13 ± 10% -0.1 0.04 ± 73% perf-profile.children.cycles-pp.__rb_erase_color 0.65 ± 4% -0.1 0.57 ± 4% perf-profile.children.cycles-pp.mutex_spin_on_owner 0.18 ± 2% -0.1 0.09 ± 6% perf-profile.children.cycles-pp.link_path_walk 0.18 ± 2% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.schedule_tail 0.16 ± 6% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.18 ± 4% -0.1 0.10 ± 6% perf-profile.children.cycles-pp.schedule_idle 0.24 ± 11% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.memset_erms 0.17 ± 2% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.mprotect_fixup 0.17 ± 8% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.finish_task_switch 0.16 ± 3% -0.1 0.08 ± 4% perf-profile.children.cycles-pp.walk_component 0.15 ± 4% -0.1 0.07 ± 9% perf-profile.children.cycles-pp.native_apic_mem_write 0.17 ± 7% -0.1 0.10 ± 6% perf-profile.children.cycles-pp.lock_page_memcg 0.17 ± 18% -0.1 0.10 ± 15% perf-profile.children.cycles-pp.arch_do_signal_or_restart 0.16 ± 10% -0.1 0.09 ± 12% perf-profile.children.cycles-pp.__get_free_pages 0.15 ± 9% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__pagevec_lru_add 0.14 ± 11% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.17 ± 11% -0.1 0.09 ± 10% perf-profile.children.cycles-pp.mem_cgroup_charge 0.12 ± 8% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.rcu_eqs_enter 0.14 ± 8% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.free_unref_page_list 0.13 ± 8% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.__memcg_kmem_uncharge 0.14 ± 7% -0.1 0.07 perf-profile.children.cycles-pp.__cond_resched 0.11 ± 4% -0.1 0.05 perf-profile.children.cycles-pp.update_rq_clock 0.15 ± 7% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.__slab_alloc 0.13 ± 8% -0.1 0.07 ± 11% perf-profile.children.cycles-pp.purge_fragmented_blocks_allcpus 0.14 ± 6% -0.1 0.08 ± 7% perf-profile.children.cycles-pp.___slab_alloc 0.09 ± 5% -0.1 0.03 ± 99% perf-profile.children.cycles-pp.__might_sleep 0.09 ± 15% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__pud_alloc 0.12 ± 8% -0.1 0.06 ± 7% perf-profile.children.cycles-pp.__rb_insert_augmented 0.11 ± 9% -0.1 0.06 ± 9% perf-profile.children.cycles-pp.lock_page_lruvec_irqsave 0.13 ± 14% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__vmalloc_node 0.11 ± 9% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.free_pcppages_bulk 0.08 ± 5% -0.1 0.03 ± 99% perf-profile.children.cycles-pp.perf_iterate_sb 0.13 ± 8% -0.1 0.07 ± 9% perf-profile.children.cycles-pp.proc_invalidate_siblings_dcache 0.11 ± 6% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.unmap_single_vma 0.13 ± 5% -0.1 0.07 ± 9% perf-profile.children.cycles-pp.bpf_jit_binary_alloc 0.10 ± 7% -0.1 0.04 ± 44% perf-profile.children.cycles-pp.fput_many 0.10 ± 4% -0.1 0.04 ± 44% perf-profile.children.cycles-pp.flush_tlb_mm_range 0.12 ± 5% -0.1 0.06 ± 7% perf-profile.children.cycles-pp.rebalance_domains 0.11 ± 3% -0.1 0.06 ± 8% perf-profile.children.cycles-pp.__split_vma 0.09 ± 6% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.mm_init 0.11 ± 9% -0.1 0.06 ± 8% perf-profile.children.cycles-pp.kfree 0.09 ± 10% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.do_task_dead 0.10 ± 7% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.vma_interval_tree_insert_after 0.09 ± 7% -0.0 0.04 ± 44% perf-profile.children.cycles-pp.lru_cache_add 0.11 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__pmd_alloc 0.11 ± 6% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.__mod_memcg_state 0.11 ± 6% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.__put_user_nocheck_4 0.12 ± 9% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.__setup_rt_frame 0.11 ± 6% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.copy_fpstate_to_sigframe 0.11 ± 8% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.acct_collect 0.11 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.nr_iowait_cpu 0.11 ± 4% -0.0 0.06 perf-profile.children.cycles-pp.__perf_sw_event 0.10 ± 7% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.try_to_wake_up 0.08 ± 19% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.put_cred_rcu 0.09 ± 4% -0.0 0.04 ± 44% perf-profile.children.cycles-pp.rcu_read_unlock_strict 0.09 ± 5% -0.0 0.04 ± 44% perf-profile.children.cycles-pp.sched_move_task 0.07 ± 8% -0.0 0.03 ± 99% perf-profile.children.cycles-pp.run_rebalance_domains 0.09 ± 5% -0.0 0.04 ± 44% perf-profile.children.cycles-pp.d_invalidate 0.10 ± 6% -0.0 0.05 ± 9% perf-profile.children.cycles-pp.module_alloc 0.08 ± 6% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.do_notify_parent 0.09 ± 5% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.update_load_avg 0.07 ± 10% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.09 ± 7% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.allocate_slab 0.11 ± 4% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.copy_page 0.06 ± 6% -0.0 0.03 ± 99% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.11 ± 4% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.update_ts_time_stats 0.10 ± 6% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.try_charge 0.00 +0.1 0.05 ± 7% perf-profile.children.cycles-pp.do_setitimer 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.__x64_sys_setitimer 0.00 +0.1 0.10 ± 3% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.00 +0.1 0.14 ± 5% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.73 ± 23% +0.2 0.94 perf-profile.children.cycles-pp.start_kernel 0.08 ± 7% +0.5 0.60 ± 3% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.46 ± 3% +0.6 1.09 ± 2% perf-profile.children.cycles-pp.tick_sched_handle 0.45 ± 4% +0.6 1.09 perf-profile.children.cycles-pp.update_process_times 0.29 ± 3% +0.7 1.02 perf-profile.children.cycles-pp.scheduler_tick 0.00 +0.8 0.77 perf-profile.children.cycles-pp.sched_clock_local 0.41 ± 34% +1.2 1.57 perf-profile.children.cycles-pp.clockevents_program_event 0.09 ± 10% +1.3 1.41 perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.55 ± 2% +1.4 1.92 perf-profile.children.cycles-pp.tick_sched_timer 0.70 ± 3% +1.9 2.58 perf-profile.children.cycles-pp.__hrtimer_run_queues 1.30 ± 13% +4.3 5.63 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 1.28 ± 13% +4.3 5.62 perf-profile.children.cycles-pp.hrtimer_interrupt 2.46 ± 10% +7.4 9.91 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 2.27 ± 9% +7.5 9.80 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.58 ± 7% +10.4 10.96 perf-profile.children.cycles-pp.tick_nohz_irq_exit 72.76 +11.1 83.89 perf-profile.children.cycles-pp.start_secondary 73.49 +11.3 84.83 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 73.49 +11.3 84.83 perf-profile.children.cycles-pp.cpu_startup_entry 73.49 +11.3 84.83 perf-profile.children.cycles-pp.do_idle 68.89 +14.0 82.87 perf-profile.children.cycles-pp.cpuidle_enter 68.86 +14.0 82.85 perf-profile.children.cycles-pp.cpuidle_enter_state 0.00 +24.0 23.97 perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event 0.00 +24.5 24.49 perf-profile.children.cycles-pp.sched_clock_tick 0.97 ± 2% +25.8 26.75 perf-profile.children.cycles-pp.irq_enter_rcu 0.95 ± 2% +25.8 26.74 perf-profile.children.cycles-pp.tick_irq_enter 11.42 +27.0 38.45 perf-profile.children.cycles-pp.asm_sysvec_call_function 8.87 ± 2% +28.5 37.41 perf-profile.children.cycles-pp.sysvec_call_function 1.72 ± 8% +50.6 52.31 perf-profile.children.cycles-pp.ktime_get 0.00 +53.0 53.01 perf-profile.children.cycles-pp.read_hpet 52.41 -29.5 22.89 perf-profile.self.cycles-pp.intel_idle 3.81 ± 2% -2.5 1.35 perf-profile.self.cycles-pp.cpuidle_enter_state 2.53 ± 2% -1.7 0.80 ± 2% perf-profile.self.cycles-pp.menu_select 2.05 ± 2% -1.0 1.05 perf-profile.self.cycles-pp.native_flush_tlb_global 1.49 -0.8 0.69 ± 2% perf-profile.self.cycles-pp.native_irq_return_iret 1.27 ± 2% -0.7 0.56 ± 2% perf-profile.self.cycles-pp.flush_smp_call_function_queue 1.22 ± 2% -0.7 0.57 perf-profile.self.cycles-pp.llist_reverse_order 1.37 ± 10% -0.6 0.72 ± 3% perf-profile.self.cycles-pp.ktime_get 1.26 ± 6% -0.6 0.64 ± 4% perf-profile.self.cycles-pp.next_uptodate_page 1.09 ± 8% -0.5 0.57 ± 4% perf-profile.self.cycles-pp.zap_pte_range 1.00 ± 3% -0.5 0.50 ± 2% perf-profile.self.cycles-pp.llist_add_batch 0.97 ± 11% -0.5 0.47 ± 8% perf-profile.self.cycles-pp.native_flush_tlb_one_user 0.71 ± 9% -0.4 0.36 ± 5% perf-profile.self.cycles-pp.release_pages 0.43 ± 22% -0.3 0.09 ± 5% perf-profile.self.cycles-pp.poll_idle 0.63 ± 6% -0.3 0.30 ± 3% perf-profile.self.cycles-pp.kmem_cache_free 0.65 ± 3% -0.3 0.33 ± 5% perf-profile.self.cycles-pp.anon_vma_interval_tree_insert 0.64 ± 6% -0.3 0.35 ± 10% perf-profile.self.cycles-pp.copy_pte_range 0.56 ± 7% -0.3 0.28 ± 2% perf-profile.self.cycles-pp.page_remove_rmap 0.36 ± 3% -0.2 0.13 ± 5% perf-profile.self.cycles-pp.__cpa_flush_tlb 0.44 ± 3% -0.2 0.21 ± 2% perf-profile.self.cycles-pp.kmem_cache_alloc 0.42 ± 6% -0.2 0.20 ± 7% perf-profile.self.cycles-pp.seccomp_cache_prepare_bitmap 0.41 ± 4% -0.2 0.20 ± 2% perf-profile.self.cycles-pp.down_write 0.43 ± 4% -0.2 0.21 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 0.37 ± 3% -0.2 0.18 ± 7% perf-profile.self.cycles-pp.up_write 0.49 ± 6% -0.2 0.30 ± 4% perf-profile.self.cycles-pp.clear_page_erms 0.36 ± 7% -0.2 0.18 ± 5% perf-profile.self.cycles-pp.__slab_free 0.36 ± 2% -0.2 0.18 ± 4% perf-profile.self.cycles-pp.anon_vma_interval_tree_remove 0.33 ± 2% -0.2 0.16 ± 6% perf-profile.self.cycles-pp.do_idle 0.29 ± 5% -0.2 0.13 ± 5% perf-profile.self.cycles-pp.page_counter_try_charge 0.30 ± 3% -0.2 0.15 ± 6% perf-profile.self.cycles-pp.anon_vma_clone 0.23 ± 7% -0.2 0.08 ± 6% perf-profile.self.cycles-pp.__default_send_IPI_dest_field 0.30 ± 6% -0.2 0.15 ± 11% perf-profile.self.cycles-pp.vma_interval_tree_remove 0.19 ± 4% -0.1 0.05 perf-profile.self.cycles-pp.__cpa_addr 0.29 ± 4% -0.1 0.15 ± 6% perf-profile.self.cycles-pp.vm_area_dup 0.26 ± 7% -0.1 0.11 ± 10% perf-profile.self.cycles-pp.page_add_file_rmap 0.19 ± 8% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.irqtime_account_irq 0.22 ± 6% -0.1 0.09 ± 7% perf-profile.self.cycles-pp.sysvec_call_function 0.26 ± 5% -0.1 0.13 ± 3% perf-profile.self.cycles-pp.free_pages_and_swap_cache 0.20 ± 7% -0.1 0.08 ± 14% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.23 ± 4% -0.1 0.11 ± 5% perf-profile.self.cycles-pp.sync_regs 0.23 ± 3% -0.1 0.11 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.16 ± 8% -0.1 0.05 ± 45% perf-profile.self.cycles-pp.do_kernel_range_flush 0.23 ± 8% -0.1 0.12 ± 3% perf-profile.self.cycles-pp.dup_mmap 0.21 ± 6% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.filemap_map_pages 0.23 ± 5% -0.1 0.12 ± 3% perf-profile.self.cycles-pp.__list_del_entry_valid 0.23 ± 4% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.idle_cpu 0.20 ± 4% -0.1 0.10 ± 3% perf-profile.self.cycles-pp.asm_sysvec_call_function 0.52 ± 3% -0.1 0.42 perf-profile.self.cycles-pp.native_sched_clock 0.18 ± 13% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.19 ± 3% -0.1 0.09 ± 5% perf-profile.self.cycles-pp._find_next_bit 0.19 ± 6% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.find_vma 0.20 ± 6% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.get_obj_cgroup_from_current 0.19 ± 3% -0.1 0.10 ± 3% perf-profile.self.cycles-pp.dup_userfaultfd 0.21 ± 4% -0.1 0.12 ± 5% perf-profile.self.cycles-pp.update_sd_lb_stats 0.20 ± 5% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.__handle_mm_fault 0.19 ± 9% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.unlock_page 0.18 ± 5% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.___might_sleep 0.12 ± 7% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.tick_check_broadcast_expired 0.62 ± 4% -0.1 0.54 ± 4% perf-profile.self.cycles-pp.mutex_spin_on_owner 0.15 ± 3% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.call_cpuidle 0.22 ± 7% -0.1 0.14 ± 4% perf-profile.self.cycles-pp.find_idlest_group 0.16 ± 10% -0.1 0.08 ± 8% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state 0.15 ± 9% -0.1 0.07 ± 8% perf-profile.self.cycles-pp.page_counter_cancel 0.22 ± 11% -0.1 0.15 ± 7% perf-profile.self.cycles-pp.memset_erms 0.15 ± 10% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.unlink_anon_vmas 0.17 ± 7% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.lock_page_memcg 0.14 ± 9% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.refill_obj_stock 0.10 ± 8% -0.1 0.03 ± 99% perf-profile.self.cycles-pp.__purge_vmap_area_lazy 0.14 ± 6% -0.1 0.07 ± 9% perf-profile.self.cycles-pp.native_apic_mem_write 0.13 ± 14% -0.1 0.06 perf-profile.self.cycles-pp.cpumask_next 0.09 ± 5% -0.1 0.03 ± 99% perf-profile.self.cycles-pp.unmap_page_range 0.10 ± 8% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.rcu_eqs_exit 0.09 ± 7% -0.1 0.03 ± 99% perf-profile.self.cycles-pp.kfree 0.09 ± 11% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.irq_exit_rcu 0.12 ± 8% -0.1 0.06 ± 13% perf-profile.self.cycles-pp.__rb_insert_augmented 0.11 ± 6% -0.1 0.05 ± 9% perf-profile.self.cycles-pp.unmap_single_vma 0.11 ± 5% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.nr_iowait_cpu 0.12 ± 6% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.alloc_vmap_area 0.11 ± 4% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.acct_collect 0.10 ± 7% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__mod_memcg_state 0.10 ± 9% -0.0 0.05 perf-profile.self.cycles-pp.vma_interval_tree_insert_after 0.09 ± 8% -0.0 0.04 ± 44% perf-profile.self.cycles-pp.purge_fragmented_blocks_allcpus 0.10 ± 4% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.copy_page 0.10 ± 4% -0.0 0.06 ± 7% perf-profile.self.cycles-pp._vm_unmap_aliases 0.07 ± 12% -0.0 0.05 ± 7% perf-profile.self.cycles-pp.sched_clock_cpu 0.93 ± 2% +0.1 1.07 ± 3% perf-profile.self.cycles-pp.smp_call_function_many_cond 0.00 +0.5 0.45 ± 2% perf-profile.self.cycles-pp.sched_clock_local 0.00 +53.0 52.99 perf-profile.self.cycles-pp.read_hpet stress-ng.time.user_time 215 +---------------------------------------------------------------------+ | + + .+| 210 |+. +. +.+++. +.+ + ++. + +. .++ +.+ +++.+ .+ +.++. + :| 205 |-++ + + + + + +.++ ++ + +. : :+ +. + + | | + ++ + + + | 200 |-+ | 195 |-+ | | | 190 |-+ | 185 |-+ O | | O O O | 180 |-+O OO OO OO O | 175 |O+ O O O O O | | O O OO | 170 +---------------------------------------------------------------------+ stress-ng.time.system_time 335 +---------------------------------------------------------------------+ 330 |-+ O O OO | |O O O OO OOO O O | 325 |-+OO OO O OO | 320 |-+ O | 315 |-+ | 310 |-+ | | | 305 |-+ | 300 |-+ | 295 |-+ + + .+ .+ | 290 |-.+ .+ + .+ +. .+ + +.++ .++. + ++ : +.+ + +. .+ | |+ + +.+ + +.+ +++ + + ++ +.+ ++.+ +++.++ +.+| 285 |-+ | 280 +---------------------------------------------------------------------+ stress-ng.time.percent_of_cpu_this_job_got 820 +---------------------------------------------------------------------+ | | | O | 815 |-+ O | | O O OO OOO O O OOO | |O O O O O | 810 |-+ O OO | | | 805 |-+ | | | | +. + .+ +. +.+ .+ | 800 |+.++.+ + + +. +.+++.++ +.+++.++.++.++ ++.++.+++.+ + ++.++.++.+| | + : : | | :: | 795 +---------------------------------------------------------------------+ stress-ng.time.minor_page_faults 4e+07 +-----------------------------------------------------------------+ |+ + +.+ + ++. :: :: + + : + .+ :+ ++. | 3.9e+07 |:+ +. + + + + +++.+ +.+ +.+ :.+++ + ++ ++.+ + + + +| 3.8e+07 |-+++ + + + :+ :+ :| | + + | 3.7e+07 |-+ | | | 3.6e+07 |-+ | | | 3.5e+07 |O+ OOO | 3.4e+07 |-+ O O | | OO O O O O O O | 3.3e+07 |-+ O OO O O OO | | | 3.2e+07 +-----------------------------------------------------------------+ stress-ng.time.involuntary_context_switches 590000 +------------------------------------------------------------------+ 580000 |-+ + +. .+| |+. :+.++. + .++ .++.++ .+++.+++.++. + +.+ +.++.+ .+++.+ + + | 570000 |-++ + + + + + + + :: ++ + | 560000 |-+ ++ + | 550000 |-+ | 540000 |-+ | | | 530000 |-+ | 520000 |-+ | 510000 |-+ O OO O | 500000 |O+O O OO O | | OO O O O O | 490000 |-+ O O OO O | 480000 +------------------------------------------------------------------+ stress-ng.opcode.ops 580000 +------------------------------------------------------------------+ 570000 |+. ++.++. + .++ .++.+ .+ +.+++.++. + +.+ +.++.+ ++.+ + + | | + + + + ++ + + + + :: ++.+ + | 560000 |-+ ++ + | 550000 |-+ | 540000 |-+ | 530000 |-+ | | | 520000 |-+ | 510000 |O+ O OO O O | 500000 |-+O O O O | 490000 |-+ OO O O O O | | O O OO O | 480000 |-+ O | 470000 +------------------------------------------------------------------+ stress-ng.opcode.ops_per_sec 9800 +--------------------------------------------------------------------+ 9600 |-+ +. + .+| |+. : +++. +. ++. +.++.+ +.++.+++.++. + .++ ++.++.+ +.++.+ +. + | 9400 |-++ + + + + + + + :+ + + | 9200 |-+ ++ + | | | 9000 |-+ | 8800 |-+ | 8600 |-+ | | O OO O | 8400 |O+O O O OO | 8200 |-+ O O OO O O | | O O O O O | 8000 |-+ O | 7800 +--------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2sp4: 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/1/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2sp4/execl/unixbench/0x4003006 commit: d42c7a8d68 ("kcsan: Document "value changed" line") 6c52b5f3cf ("clocksource: Reduce WATCHDOG_THRESHOLD") d42c7a8d68038bc0 6c52b5f3cfefd6e429efc4413fd ---------------- --------------------------- %stddev %change %stddev \ | \ 41269 ± 24% -24.8% 31015 ± 4% unixbench.time.involuntary_context_switches 87.00 +2.3% 89.00 unixbench.time.percent_of_cpu_this_job_got 9346 -6.5% 8736 ± 2% proc-vmstat.pgactivate 1.37 ± 16% -1.4 0.00 mpstat.cpu.all.irq% 0.10 ± 12% -0.1 0.00 ± 59% mpstat.cpu.all.soft% 1600 ± 4% -8.3% 1467 ± 6% slabinfo.pool_workqueue.active_objs 1609 ± 4% -8.8% 1467 ± 6% slabinfo.pool_workqueue.num_objs 96.50 +2.6% 99.00 vmstat.cpu.id 1918 ± 2% -19.7% 1539 vmstat.system.cs 27.42 +1.0% 27.69 boot-time.boot 14.59 +2.0% 14.89 boot-time.dhcp 2119 +7.3% 2273 boot-time.idle 55342935 ±146% -97.1% 1614545 ± 9% cpuidle.C1.time 369013 ±113% -87.0% 47814 ± 10% cpuidle.POLL.time 80151 ± 24% -94.7% 4258 ± 10% cpuidle.POLL.usage 72.00 ± 5% +139.4% 172.33 turbostat.Avg_MHz 3.76 ± 11% +4.7 8.50 turbostat.Busy% 113.08 +8.9% 123.12 turbostat.PkgWatt 15.27 ±100% +582.6% 104.25 ± 58% sched_debug.cfs_rq:/.MIN_vruntime.avg 144.93 ±100% +338.3% 635.26 ± 54% sched_debug.cfs_rq:/.MIN_vruntime.stddev 15.27 ±100% +582.6% 104.25 ± 58% sched_debug.cfs_rq:/.max_vruntime.avg 144.93 ±100% +338.3% 635.26 ± 54% sched_debug.cfs_rq:/.max_vruntime.stddev 584.77 ± 3% -98.3% 9.72 ± 18% sched_debug.cpu.clock_task.stddev 4639 ± 7% -11.6% 4102 sched_debug.cpu.nr_switches.avg 36958 ± 11% -35.6% 23798 ± 17% sched_debug.cpu.nr_switches.max 4896 ± 6% -34.3% 3216 ± 9% sched_debug.cpu.nr_switches.stddev 1.00 -100.0% 0.00 sched_debug.sched_clock_stable() 9.417e+08 ± 2% +7.3% 1.01e+09 ± 2% perf-stat.i.branch-instructions 5.82 ± 28% +3.6 9.39 ± 2% perf-stat.i.cache-miss-rate% 2168912 ± 14% +20.7% 2618168 ± 3% perf-stat.i.cache-misses 1868 ± 2% -19.3% 1508 perf-stat.i.context-switches 1.66 ± 15% +114.4% 3.57 ± 2% perf-stat.i.cpi 6.219e+09 ± 6% +155.1% 1.587e+10 perf-stat.i.cpu-cycles 2937 ± 9% +109.3% 6147 ± 2% perf-stat.i.cycles-between-cache-misses 0.12 ± 75% -0.1 0.03 ± 4% perf-stat.i.dTLB-load-miss-rate% 993756 ± 62% -63.4% 363963 ± 2% perf-stat.i.dTLB-load-misses 0.08 ± 30% -0.0 0.05 perf-stat.i.dTLB-store-miss-rate% 426111 ± 18% -22.7% 329275 ± 3% perf-stat.i.dTLB-store-misses 2650130 ± 3% +8.8% 2882167 perf-stat.i.iTLB-load-misses 4.549e+09 ± 2% +6.7% 4.853e+09 ± 2% perf-stat.i.instructions 0.72 ± 8% -58.5% 0.30 ± 2% perf-stat.i.ipc 0.06 ± 6% +155.1% 0.17 perf-stat.i.metric.GHz 0.57 ± 59% -63.9% 0.21 ± 6% perf-stat.i.metric.K/sec 83.19 +1.0 84.23 perf-stat.i.node-load-miss-rate% 542248 ± 9% +37.4% 745113 perf-stat.i.node-load-misses 112232 ± 16% +27.4% 143037 ± 3% perf-stat.i.node-loads 73.15 ± 29% +23.3 96.45 perf-stat.i.node-store-miss-rate% 142423 ± 19% +81.4% 258350 perf-stat.i.node-store-misses 5.57 ± 26% +3.5 9.07 ± 3% perf-stat.overall.cache-miss-rate% 1.37 ± 9% +138.8% 3.27 ± 2% perf-stat.overall.cpi 2902 ± 9% +109.0% 6065 ± 2% perf-stat.overall.cycles-between-cache-misses 0.08 ± 64% -0.1 0.03 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.07 ± 20% -0.0 0.05 perf-stat.overall.dTLB-store-miss-rate% 0.74 ± 8% -58.4% 0.31 ± 2% perf-stat.overall.ipc 72.83 ± 28% +23.1 95.94 perf-stat.overall.node-store-miss-rate% 1242941 +11.5% 1386048 perf-stat.overall.path-length 9.388e+08 ± 2% +7.3% 1.007e+09 ± 2% perf-stat.ps.branch-instructions 2162600 ± 14% +20.7% 2610503 ± 3% perf-stat.ps.cache-misses 1863 ± 2% -19.3% 1504 perf-stat.ps.context-switches 6.2e+09 ± 6% +155.2% 1.582e+10 perf-stat.ps.cpu-cycles 990761 ± 62% -63.4% 362845 ± 2% perf-stat.ps.dTLB-load-misses 424764 ± 18% -22.7% 328252 ± 3% perf-stat.ps.dTLB-store-misses 2641809 ± 3% +8.8% 2873343 perf-stat.ps.iTLB-load-misses 4.536e+09 ± 2% +6.7% 4.839e+09 ± 2% perf-stat.ps.instructions 540562 ± 9% +37.4% 742841 perf-stat.ps.node-load-misses 111898 ± 16% +27.5% 142630 ± 3% perf-stat.ps.node-loads 141992 ± 19% +81.4% 257582 perf-stat.ps.node-store-misses 1.5e+12 ± 2% +6.6% 1.6e+12 ± 2% perf-stat.total.instructions 0.01 ± 23% +5667.9% 0.51 ±109% perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.01 ± 33% +11121.1% 0.71 ± 81% perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read 0.00 ± 40% +114.3% 0.01 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.01 ± 47% +106.7% 0.02 ± 3% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.00 ± 49% +5141.7% 0.10 ±215% perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.01 ± 24% +95.8% 0.02 ± 15% perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf 0.00 ± 27% +268.8% 0.01 ± 6% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 0.01 ± 13% +26.1% 0.01 ± 3% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork 0.01 ± 27% +9866.7% 1.00 ±112% perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.01 ± 42% +19611.6% 1.41 ± 81% perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read 0.01 ± 43% +110.0% 0.01 ± 17% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown] 0.03 ± 72% +517.5% 0.16 ± 72% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.02 ± 29% +73.4% 0.03 ± 11% perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf 2.54 +79.1% 4.55 ± 2% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 129.77 ± 3% +17.4% 152.41 ± 4% perf-sched.total_wait_and_delay.average.ms 11677 ± 4% -12.9% 10170 ± 5% perf-sched.total_wait_and_delay.count.ms 129.77 ± 3% +17.4% 152.39 ± 4% perf-sched.total_wait_time.average.ms 188.49 ± 10% +27.2% 239.84 ± 9% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.11 ± 7% +13.5% 0.12 ± 5% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown] 71.26 ± 3% +26.4% 90.08 ± 2% perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 305.63 ± 6% +21.0% 369.75 ± 10% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 0.00 ± 27% +287.5% 0.01 ± 12% perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 209.83 ± 16% -27.3% 152.50 ± 16% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown] 105.33 ± 14% -32.6% 71.00 ± 14% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown] 2120 ± 3% -21.5% 1665 ± 2% perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read 144.17 ± 20% -33.1% 96.50 ± 21% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 89.00 ± 11% -41.0% 52.50 ± 45% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas 3080 ± 8% -16.7% 2566 ± 9% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork 4587 ± 50% -59.8% 1845 ±103% perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 188.49 ± 10% +27.2% 239.83 ± 9% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.11 ± 7% +13.5% 0.12 ± 5% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown] 71.26 ± 3% +26.3% 89.98 ± 2% perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.12 ± 31% -75.3% 0.03 ±142% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc.security_prepare_creds.prepare_creds 0.10 ± 7% +20.5% 0.12 ± 7% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_strings.isra.0 0.10 ± 31% +92.3% 0.20 ± 74% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_read.walk_component.path_lookupat 0.11 ± 8% +18.3% 0.13 ± 5% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.filemap_read.__kernel_read.exec_binprm 0.12 ± 37% -66.7% 0.04 ±104% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.getname_kernel.open_exec 0.10 ± 18% +34.6% 0.14 ± 12% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.exit_mmap 3.58 ± 34% -50.4% 1.78 ± 46% perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom 305.62 ± 6% +21.0% 369.74 ± 10% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 0.15 ± 37% -80.4% 0.03 ±142% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc.security_prepare_creds.prepare_creds 0.13 ± 36% +437.7% 0.72 ±155% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.walk_component.path_lookupat 0.16 ± 28% +42.2% 0.23 ± 5% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.setup_arg_pages.load_elf_binary 0.14 ± 37% -69.1% 0.04 ±103% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.getname_kernel.open_exec 4587 ± 50% -59.8% 1844 ±103% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 3.58 ± 34% -50.4% 1.78 ± 46% perf-sched.wait_time.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom 0.09 ± 3% +13.4% 0.11 ± 5% perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion_interruptible.usb_stor_control_thread.kthread 83958 ± 22% -39.2% 51071 ± 7% softirqs.CPU0.RCU 52781 ± 24% -42.2% 30516 ± 4% softirqs.CPU10.RCU 57678 ± 27% -45.6% 31359 ± 6% softirqs.CPU11.RCU 49173 ± 17% -38.0% 30501 ± 5% softirqs.CPU12.RCU 52583 ± 26% -38.5% 32315 ± 11% softirqs.CPU13.RCU 53806 ± 32% -42.7% 30806 ± 8% softirqs.CPU14.RCU 49603 ± 21% -36.6% 31471 ± 11% softirqs.CPU15.RCU 48176 ± 17% -37.4% 30155 ± 8% softirqs.CPU16.RCU 44644 ± 13% -30.2% 31144 ± 4% softirqs.CPU17.RCU 41356 ± 12% -26.0% 30596 ± 9% softirqs.CPU18.RCU 46651 ± 14% -34.9% 30377 ± 8% softirqs.CPU19.RCU 58926 ± 31% -41.1% 34685 ± 11% softirqs.CPU2.RCU 55066 ± 41% -46.6% 29403 ± 4% softirqs.CPU20.RCU 46142 ± 19% -34.0% 30462 ± 8% softirqs.CPU21.RCU 46125 ± 18% -36.6% 29254 ± 5% softirqs.CPU22.RCU 43186 ± 12% -27.6% 31271 ± 7% softirqs.CPU27.RCU 43298 ± 11% -30.5% 30080 ± 4% softirqs.CPU28.RCU 45410 ± 17% -33.3% 30308 ± 11% softirqs.CPU29.RCU 59941 ± 35% -47.8% 31271 ± 9% softirqs.CPU3.RCU 42403 ± 9% -32.1% 28793 ± 6% softirqs.CPU30.RCU 43296 ± 13% -30.7% 29985 ± 6% softirqs.CPU31.RCU 50475 ± 15% -39.1% 30719 ± 5% softirqs.CPU32.RCU 49326 ± 15% -35.2% 31981 ± 17% softirqs.CPU33.RCU 51237 ± 12% -39.6% 30970 ± 10% softirqs.CPU34.RCU 51711 ± 11% -45.2% 28360 ± 3% softirqs.CPU35.RCU 49924 ± 17% -43.2% 28373 ± 6% softirqs.CPU36.RCU 49802 ± 16% -40.2% 29775 ± 11% softirqs.CPU37.RCU 52954 ± 17% -46.1% 28545 ± 8% softirqs.CPU38.RCU 49661 ± 14% -41.5% 29067 ± 8% softirqs.CPU39.RCU 60707 ± 28% -43.7% 34155 ± 11% softirqs.CPU4.RCU 51390 ± 16% -45.8% 27866 ± 8% softirqs.CPU40.RCU 58695 ± 20% -51.3% 28563 ± 4% softirqs.CPU41.RCU 51871 ± 23% -38.8% 31769 ± 15% softirqs.CPU42.RCU 50183 ± 12% -42.7% 28735 ± 9% softirqs.CPU43.RCU 56295 ± 15% -46.9% 29880 ± 14% softirqs.CPU44.RCU 54323 ± 17% -45.3% 29698 ± 7% softirqs.CPU45.RCU 51361 ± 16% -46.3% 27564 ± 2% softirqs.CPU46.RCU 163543 ± 17% -43.8% 91839 ± 13% softirqs.CPU47.RCU 60813 ± 28% -45.2% 33329 ± 7% softirqs.CPU5.RCU 60658 ± 27% -44.3% 33808 ± 13% softirqs.CPU50.RCU 55217 ± 24% -39.3% 33499 ± 13% softirqs.CPU51.RCU 58356 ± 28% -43.8% 32790 ± 8% softirqs.CPU52.RCU 62221 ± 23% -45.9% 33655 ± 5% softirqs.CPU53.RCU 55815 ± 27% -44.1% 31202 ± 9% softirqs.CPU54.RCU 58726 ± 28% -43.6% 33117 ± 10% softirqs.CPU55.RCU 52856 ± 26% -42.7% 30294 ± 8% softirqs.CPU56.RCU 53839 ± 22% -44.0% 30174 ± 8% softirqs.CPU57.RCU 55030 ± 28% -45.0% 30274 ± 6% softirqs.CPU58.RCU 54756 ± 22% -42.8% 31304 ± 5% softirqs.CPU59.RCU 54918 ± 31% -47.1% 29032 ± 3% softirqs.CPU6.RCU 53535 ± 24% -41.9% 31090 ± 6% softirqs.CPU60.RCU 52174 ± 23% -41.2% 30704 ± 5% softirqs.CPU61.RCU 48552 ± 21% -36.8% 30663 ± 8% softirqs.CPU62.RCU 56162 ± 29% -43.7% 31643 ± 9% softirqs.CPU63.RCU 47055 ± 27% -39.2% 28624 ± 3% softirqs.CPU64.RCU 45531 ± 16% -36.9% 28741 ± 4% softirqs.CPU65.RCU 43623 ± 17% -29.5% 30756 ± 7% softirqs.CPU66.RCU 43174 ± 20% -30.6% 29945 ± 9% softirqs.CPU67.RCU 43982 ± 18% -31.1% 30302 ± 10% softirqs.CPU68.RCU 45056 ± 21% -37.4% 28219 softirqs.CPU69.RCU 56189 ± 26% -42.9% 32065 ± 9% softirqs.CPU7.RCU 47003 ± 20% -37.3% 29489 ± 4% softirqs.CPU70.RCU 51052 ± 20% -37.4% 31979 ± 10% softirqs.CPU74.RCU 42083 ± 10% -24.0% 31964 ± 20% softirqs.CPU75.RCU 48436 ± 20% -40.1% 29025 ± 7% softirqs.CPU76.RCU 47672 ± 22% -37.1% 29972 ± 8% softirqs.CPU77.RCU 45766 ± 12% -35.2% 29661 ± 8% softirqs.CPU79.RCU 55322 ± 30% -41.8% 32216 ± 11% softirqs.CPU8.RCU 54363 ± 11% -43.3% 30850 ± 6% softirqs.CPU80.RCU 53303 ± 15% -37.7% 33196 ± 13% softirqs.CPU81.RCU 51853 ± 13% -40.1% 31073 ± 5% softirqs.CPU82.RCU 51969 ± 13% -41.2% 30553 ± 7% softirqs.CPU83.RCU 50169 ± 16% -43.0% 28586 ± 4% softirqs.CPU84.RCU 51590 ± 19% -40.9% 30502 ± 7% softirqs.CPU85.RCU 52550 ± 14% -42.5% 30201 ± 8% softirqs.CPU86.RCU 51864 ± 16% -42.4% 29871 ± 6% softirqs.CPU87.RCU 51409 ± 13% -44.2% 28669 ± 9% softirqs.CPU88.RCU 53518 ± 15% -45.4% 29224 ± 7% softirqs.CPU89.RCU 53986 ± 26% -39.9% 32454 ± 12% softirqs.CPU9.RCU 48578 ± 14% -39.4% 29454 ± 4% softirqs.CPU90.RCU 51131 ± 16% -44.9% 28148 ± 4% softirqs.CPU91.RCU 51216 ± 13% -43.8% 28775 ± 7% softirqs.CPU92.RCU 53664 ± 23% -42.2% 31002 ± 12% softirqs.CPU93.RCU 53551 ± 14% -44.8% 29573 ± 6% softirqs.CPU94.RCU 74490 ± 20% -39.1% 45358 ± 11% softirqs.CPU95.RCU 5514177 ± 17% -37.5% 3445171 softirqs.RCU 93120 ± 31% +64.2% 152920 ± 10% interrupts.CAL:Function_call_interrupts 20.67 ± 14% +408.9% 105.17 ± 65% interrupts.CPU10.NMI:Non-maskable_interrupts 20.67 ± 14% +408.9% 105.17 ± 65% interrupts.CPU10.PMI:Performance_monitoring_interrupts 20.33 ± 11% +397.5% 101.17 ± 79% interrupts.CPU11.NMI:Non-maskable_interrupts 20.33 ± 11% +397.5% 101.17 ± 79% interrupts.CPU11.PMI:Performance_monitoring_interrupts 20.50 ± 33% +221.1% 65.83 ± 40% interrupts.CPU13.NMI:Non-maskable_interrupts 20.50 ± 33% +221.1% 65.83 ± 40% interrupts.CPU13.PMI:Performance_monitoring_interrupts 11.50 ± 43% +318.8% 48.17 ± 92% interrupts.CPU13.RES:Rescheduling_interrupts 16.00 ± 32% +1041.7% 182.67 ±144% interrupts.CPU14.NMI:Non-maskable_interrupts 16.00 ± 32% +1041.7% 182.67 ±144% interrupts.CPU14.PMI:Performance_monitoring_interrupts 18.67 ± 28% +361.6% 86.17 ± 36% interrupts.CPU15.NMI:Non-maskable_interrupts 18.67 ± 28% +361.6% 86.17 ± 36% interrupts.CPU15.PMI:Performance_monitoring_interrupts 18.50 ± 23% +581.1% 126.00 ± 77% interrupts.CPU16.NMI:Non-maskable_interrupts 18.50 ± 23% +581.1% 126.00 ± 77% interrupts.CPU16.PMI:Performance_monitoring_interrupts 20.33 ± 13% +287.7% 78.83 ± 24% interrupts.CPU17.NMI:Non-maskable_interrupts 20.33 ± 13% +287.7% 78.83 ± 24% interrupts.CPU17.PMI:Performance_monitoring_interrupts 20.33 ± 17% +304.1% 82.17 ± 22% interrupts.CPU18.NMI:Non-maskable_interrupts 20.33 ± 17% +304.1% 82.17 ± 22% interrupts.CPU18.PMI:Performance_monitoring_interrupts 19.33 ± 10% +309.5% 79.17 ± 36% interrupts.CPU19.NMI:Non-maskable_interrupts 19.33 ± 10% +309.5% 79.17 ± 36% interrupts.CPU19.PMI:Performance_monitoring_interrupts 20.00 ± 10% +261.7% 72.33 ± 31% interrupts.CPU20.NMI:Non-maskable_interrupts 20.00 ± 10% +261.7% 72.33 ± 31% interrupts.CPU20.PMI:Performance_monitoring_interrupts 20.00 ± 10% +292.5% 78.50 ± 24% interrupts.CPU21.NMI:Non-maskable_interrupts 20.00 ± 10% +292.5% 78.50 ± 24% interrupts.CPU21.PMI:Performance_monitoring_interrupts 20.33 ± 15% +223.8% 65.83 ± 39% interrupts.CPU22.NMI:Non-maskable_interrupts 20.33 ± 15% +223.8% 65.83 ± 39% interrupts.CPU22.PMI:Performance_monitoring_interrupts 41.67 ± 26% +102.8% 84.50 ± 26% interrupts.CPU23.TLB:TLB_shootdowns 17.50 ± 24% +473.3% 100.33 ± 65% interrupts.CPU25.NMI:Non-maskable_interrupts 17.50 ± 24% +473.3% 100.33 ± 65% interrupts.CPU25.PMI:Performance_monitoring_interrupts 551.83 ± 3% +108.4% 1150 ±110% interrupts.CPU26.CAL:Function_call_interrupts 25.50 ± 66% +359.5% 117.17 ± 59% interrupts.CPU26.NMI:Non-maskable_interrupts 25.50 ± 66% +359.5% 117.17 ± 59% interrupts.CPU26.PMI:Performance_monitoring_interrupts 22.50 ± 29% +343.7% 99.83 ± 84% interrupts.CPU28.NMI:Non-maskable_interrupts 22.50 ± 29% +343.7% 99.83 ± 84% interrupts.CPU28.PMI:Performance_monitoring_interrupts 18.83 ± 23% +448.7% 103.33 ± 64% interrupts.CPU31.NMI:Non-maskable_interrupts 18.83 ± 23% +448.7% 103.33 ± 64% interrupts.CPU31.PMI:Performance_monitoring_interrupts 22.33 ± 43% +264.9% 81.50 ± 42% interrupts.CPU32.NMI:Non-maskable_interrupts 22.33 ± 43% +264.9% 81.50 ± 42% interrupts.CPU32.PMI:Performance_monitoring_interrupts 18.00 ± 26% +402.8% 90.50 ± 51% interrupts.CPU34.NMI:Non-maskable_interrupts 18.00 ± 26% +402.8% 90.50 ± 51% interrupts.CPU34.PMI:Performance_monitoring_interrupts 6.00 ± 84% +522.2% 37.33 ±119% interrupts.CPU34.RES:Rescheduling_interrupts 23.83 ± 62% +325.9% 101.50 ± 54% interrupts.CPU36.NMI:Non-maskable_interrupts 23.83 ± 62% +325.9% 101.50 ± 54% interrupts.CPU36.PMI:Performance_monitoring_interrupts 25.17 ± 79% +437.7% 135.33 ± 75% interrupts.CPU37.NMI:Non-maskable_interrupts 25.17 ± 79% +437.7% 135.33 ± 75% interrupts.CPU37.PMI:Performance_monitoring_interrupts 18.17 ± 23% +337.6% 79.50 ± 41% interrupts.CPU38.NMI:Non-maskable_interrupts 18.17 ± 23% +337.6% 79.50 ± 41% interrupts.CPU38.PMI:Performance_monitoring_interrupts 13.83 ± 32% +636.1% 101.83 ± 78% interrupts.CPU39.NMI:Non-maskable_interrupts 13.83 ± 32% +636.1% 101.83 ± 78% interrupts.CPU39.PMI:Performance_monitoring_interrupts 20.83 ± 45% +346.4% 93.00 ± 57% interrupts.CPU45.NMI:Non-maskable_interrupts 20.83 ± 45% +346.4% 93.00 ± 57% interrupts.CPU45.PMI:Performance_monitoring_interrupts 677.83 ± 6% -12.9% 590.17 ± 3% interrupts.CPU47.CAL:Function_call_interrupts 20.67 ± 99% +317.7% 86.33 ± 33% interrupts.CPU48.TLB:TLB_shootdowns 21.17 ± 30% +353.5% 96.00 ± 18% interrupts.CPU5.NMI:Non-maskable_interrupts 21.17 ± 30% +353.5% 96.00 ± 18% interrupts.CPU5.PMI:Performance_monitoring_interrupts 9.50 ± 37% +649.1% 71.17 ± 84% interrupts.CPU52.RES:Rescheduling_interrupts 21.00 ± 26% +342.1% 92.83 ± 13% interrupts.CPU53.NMI:Non-maskable_interrupts 21.00 ± 26% +342.1% 92.83 ± 13% interrupts.CPU53.PMI:Performance_monitoring_interrupts 20.17 ± 18% +330.6% 86.83 ± 16% interrupts.CPU54.NMI:Non-maskable_interrupts 20.17 ± 18% +330.6% 86.83 ± 16% interrupts.CPU54.PMI:Performance_monitoring_interrupts 21.50 ± 22% +276.7% 81.00 ± 26% interrupts.CPU55.NMI:Non-maskable_interrupts 21.50 ± 22% +276.7% 81.00 ± 26% interrupts.CPU55.PMI:Performance_monitoring_interrupts 12.33 ± 75% +297.3% 49.00 ± 66% interrupts.CPU55.RES:Rescheduling_interrupts 18.50 ± 29% +418.9% 96.00 ± 23% interrupts.CPU56.NMI:Non-maskable_interrupts 18.50 ± 29% +418.9% 96.00 ± 23% interrupts.CPU56.PMI:Performance_monitoring_interrupts 19.67 ± 13% +444.1% 107.00 ± 51% interrupts.CPU57.NMI:Non-maskable_interrupts 19.67 ± 13% +444.1% 107.00 ± 51% interrupts.CPU57.PMI:Performance_monitoring_interrupts 21.17 ± 14% +336.2% 92.33 ± 25% interrupts.CPU58.NMI:Non-maskable_interrupts 21.17 ± 14% +336.2% 92.33 ± 25% interrupts.CPU58.PMI:Performance_monitoring_interrupts 21.17 ± 15% +475.6% 121.83 ± 73% interrupts.CPU59.NMI:Non-maskable_interrupts 21.17 ± 15% +475.6% 121.83 ± 73% interrupts.CPU59.PMI:Performance_monitoring_interrupts 18.17 ± 29% +379.8% 87.17 ± 16% interrupts.CPU6.NMI:Non-maskable_interrupts 18.17 ± 29% +379.8% 87.17 ± 16% interrupts.CPU6.PMI:Performance_monitoring_interrupts 6.33 ± 81% +473.7% 36.33 ± 87% interrupts.CPU6.RES:Rescheduling_interrupts 19.83 ± 15% +337.0% 86.67 ± 16% interrupts.CPU61.NMI:Non-maskable_interrupts 19.83 ± 15% +337.0% 86.67 ± 16% interrupts.CPU61.PMI:Performance_monitoring_interrupts 19.50 ± 14% +731.6% 162.17 ±109% interrupts.CPU62.NMI:Non-maskable_interrupts 19.50 ± 14% +731.6% 162.17 ±109% interrupts.CPU62.PMI:Performance_monitoring_interrupts 19.67 ± 33% +383.1% 95.00 ± 31% interrupts.CPU63.NMI:Non-maskable_interrupts 19.67 ± 33% +383.1% 95.00 ± 31% interrupts.CPU63.PMI:Performance_monitoring_interrupts 20.33 ± 11% +452.5% 112.33 ± 58% interrupts.CPU64.NMI:Non-maskable_interrupts 20.33 ± 11% +452.5% 112.33 ± 58% interrupts.CPU64.PMI:Performance_monitoring_interrupts 21.00 ± 16% +312.7% 86.67 ± 15% interrupts.CPU65.NMI:Non-maskable_interrupts 21.00 ± 16% +312.7% 86.67 ± 15% interrupts.CPU65.PMI:Performance_monitoring_interrupts 21.33 ± 20% +318.0% 89.17 ± 12% interrupts.CPU66.NMI:Non-maskable_interrupts 21.33 ± 20% +318.0% 89.17 ± 12% interrupts.CPU66.PMI:Performance_monitoring_interrupts 11.50 ± 41% +523.2% 71.67 ±143% interrupts.CPU66.RES:Rescheduling_interrupts 17.83 ± 13% +546.7% 115.33 ± 50% interrupts.CPU67.NMI:Non-maskable_interrupts 17.83 ± 13% +546.7% 115.33 ± 50% interrupts.CPU67.PMI:Performance_monitoring_interrupts 18.17 ± 14% +384.4% 88.00 ± 15% interrupts.CPU68.NMI:Non-maskable_interrupts 18.17 ± 14% +384.4% 88.00 ± 15% interrupts.CPU68.PMI:Performance_monitoring_interrupts 19.00 ± 16% +320.2% 79.83 ± 28% interrupts.CPU69.NMI:Non-maskable_interrupts 19.00 ± 16% +320.2% 79.83 ± 28% interrupts.CPU69.PMI:Performance_monitoring_interrupts 17.17 ± 37% +378.6% 82.17 ± 29% interrupts.CPU7.NMI:Non-maskable_interrupts 17.17 ± 37% +378.6% 82.17 ± 29% interrupts.CPU7.PMI:Performance_monitoring_interrupts 17.67 ± 16% +401.9% 88.67 ± 17% interrupts.CPU70.NMI:Non-maskable_interrupts 17.67 ± 16% +401.9% 88.67 ± 17% interrupts.CPU70.PMI:Performance_monitoring_interrupts 17.17 ± 21% +599.0% 120.00 ± 49% interrupts.CPU73.NMI:Non-maskable_interrupts 17.17 ± 21% +599.0% 120.00 ± 49% interrupts.CPU73.PMI:Performance_monitoring_interrupts 18.33 ± 17% +664.5% 140.17 ± 60% interrupts.CPU74.NMI:Non-maskable_interrupts 18.33 ± 17% +664.5% 140.17 ± 60% interrupts.CPU74.PMI:Performance_monitoring_interrupts 24.33 ± 75% +266.4% 89.17 ± 76% interrupts.CPU75.NMI:Non-maskable_interrupts 24.33 ± 75% +266.4% 89.17 ± 76% interrupts.CPU75.PMI:Performance_monitoring_interrupts 18.67 ± 24% +688.4% 147.17 ± 81% interrupts.CPU76.NMI:Non-maskable_interrupts 18.67 ± 24% +688.4% 147.17 ± 81% interrupts.CPU76.PMI:Performance_monitoring_interrupts 12.17 ± 44% +265.8% 44.50 ± 76% interrupts.CPU77.RES:Rescheduling_interrupts 18.83 ± 25% +410.6% 96.17 ± 36% interrupts.CPU79.NMI:Non-maskable_interrupts 18.83 ± 25% +410.6% 96.17 ± 36% interrupts.CPU79.PMI:Performance_monitoring_interrupts 19.50 ± 12% +582.9% 133.17 ± 89% interrupts.CPU8.NMI:Non-maskable_interrupts 19.50 ± 12% +582.9% 133.17 ± 89% interrupts.CPU8.PMI:Performance_monitoring_interrupts 9.67 ± 34% +286.2% 37.33 ± 80% interrupts.CPU80.RES:Rescheduling_interrupts 18.00 ± 26% +412.0% 92.17 ± 56% interrupts.CPU82.NMI:Non-maskable_interrupts 18.00 ± 26% +412.0% 92.17 ± 56% interrupts.CPU82.PMI:Performance_monitoring_interrupts 9.67 ± 50% +700.0% 77.33 ±125% interrupts.CPU82.RES:Rescheduling_interrupts 19.17 ± 36% +463.5% 108.00 ± 62% interrupts.CPU84.NMI:Non-maskable_interrupts 19.17 ± 36% +463.5% 108.00 ± 62% interrupts.CPU84.PMI:Performance_monitoring_interrupts 12.83 ± 42% +351.9% 58.00 ± 64% interrupts.CPU85.RES:Rescheduling_interrupts 17.83 ± 24% +379.4% 85.50 ± 48% interrupts.CPU86.NMI:Non-maskable_interrupts 17.83 ± 24% +379.4% 85.50 ± 48% interrupts.CPU86.PMI:Performance_monitoring_interrupts 17.00 ± 25% +597.1% 118.50 ± 96% interrupts.CPU87.NMI:Non-maskable_interrupts 17.00 ± 25% +597.1% 118.50 ± 96% interrupts.CPU87.PMI:Performance_monitoring_interrupts 18.33 ± 27% +469.1% 104.33 ± 61% interrupts.CPU9.NMI:Non-maskable_interrupts 18.33 ± 27% +469.1% 104.33 ± 61% interrupts.CPU9.PMI:Performance_monitoring_interrupts 8.00 ± 61% +472.9% 45.83 ± 66% interrupts.CPU9.RES:Rescheduling_interrupts 21.50 ± 43% +427.1% 113.33 ± 68% interrupts.CPU93.NMI:Non-maskable_interrupts 21.50 ± 43% +427.1% 113.33 ± 68% interrupts.CPU93.PMI:Performance_monitoring_interrupts 39.83 ± 13% +853.6% 379.83 ± 24% interrupts.CPU95.TLB:TLB_shootdowns 4076 ± 9% +145.6% 10013 ± 3% interrupts.NMI:Non-maskable_interrupts 4076 ± 9% +145.6% 10013 ± 3% interrupts.PMI:Performance_monitoring_interrupts 1164 ± 10% +192.5% 3404 ± 3% interrupts.RES:Rescheduling_interrupts 844.83 ± 36% +122.8% 1882 ± 17% interrupts.TLB:TLB_shootdowns 37.96 ± 13% -22.4 15.54 ± 12% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 14.65 ± 12% -9.1 5.59 ± 7% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 12.64 ± 10% -7.7 4.95 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 11.71 ± 12% -7.3 4.36 ± 11% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.27 ± 12% -6.8 4.46 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.12 ± 14% -6.6 3.49 ± 11% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 9.74 ± 15% -6.3 3.40 ± 12% perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve 9.39 ± 11% -6.0 3.43 ± 12% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 9.39 ± 11% -6.0 3.43 ± 12% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 9.39 ± 11% -6.0 3.43 ± 12% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 9.39 ± 11% -6.0 3.43 ± 12% perf-profile.calltrace.cycles-pp.execve 5.38 ± 18% -3.3 2.13 ± 12% perf-profile.calltrace.cycles-pp.asm_exc_page_fault 5.32 ± 16% -3.1 2.18 ± 5% perf-profile.calltrace.cycles-pp.main 5.29 ± 16% -3.1 2.18 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve.main 5.29 ± 16% -3.1 2.18 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve.main 5.29 ± 16% -3.1 2.18 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve.main 5.29 ± 16% -3.1 2.18 ± 4% perf-profile.calltrace.cycles-pp.execve.main 4.19 ± 17% -3.1 1.12 ± 17% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 4.65 ± 15% -3.0 1.65 ± 11% perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common 4.84 ± 17% -2.9 1.95 ± 15% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault 4.80 ± 17% -2.9 1.93 ± 15% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 4.36 ± 15% -2.8 1.51 ± 11% perf-profile.calltrace.cycles-pp.mmput.begin_new_exec.load_elf_binary.exec_binprm.bprm_execve 4.32 ± 15% -2.8 1.51 ± 11% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.begin_new_exec.load_elf_binary.exec_binprm 4.82 ± 12% -2.7 2.08 ± 6% perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.80 ± 13% -2.7 2.07 ± 7% perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.15 ± 33% -2.7 1.42 ± 12% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 4.07 ± 18% -2.5 1.53 ± 11% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 3.94 ± 13% -2.2 1.70 ± 4% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.91 ± 13% -2.2 1.68 ± 4% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64 2.63 ± 24% -1.7 0.90 ± 17% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 2.23 ± 17% -1.4 0.81 ± 11% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 2.15 ± 7% -1.4 0.74 ± 14% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.begin_new_exec.load_elf_binary 2.09 ± 6% -1.4 0.73 ± 14% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.begin_new_exec 2.40 ± 19% -1.3 1.05 ± 47% perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work 2.40 ± 19% -1.3 1.05 ± 47% perf-profile.calltrace.cycles-pp.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work 2.40 ± 19% -1.3 1.05 ± 47% perf-profile.calltrace.cycles-pp.printk.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work 2.40 ± 19% -1.3 1.05 ± 47% perf-profile.calltrace.cycles-pp.vprintk_emit.printk.irq_work_single.irq_work_run_list.irq_work_run 2.40 ± 19% -1.3 1.05 ± 47% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.irq_work_single.irq_work_run_list 2.03 ± 13% -1.3 0.69 ± 8% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.00 ± 11% -1.3 0.68 ± 7% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.31 ± 19% -1.3 1.00 ± 47% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.irq_work_single 2.02 ± 13% -1.3 0.75 ± 18% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 2.25 ± 19% -1.3 0.98 ± 47% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk 1.90 ± 7% -1.2 0.66 ± 16% perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap.mmput 1.87 ± 10% -1.2 0.64 ± 7% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.22 ± 13% -1.2 1.02 ± 15% perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open 2.13 ± 19% -1.2 0.94 ± 47% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock 2.13 ± 19% -1.2 0.94 ± 47% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit 1.68 ± 11% -1.2 0.49 ± 44% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 1.97 ± 10% -1.1 0.91 ± 16% perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_openat2 1.40 ± 15% -1.0 0.36 ± 71% perf-profile.calltrace.cycles-pp.ret_from_fork 1.40 ± 15% -1.0 0.36 ± 71% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 1.85 ± 9% -1.0 0.82 ± 23% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.84 ± 10% -1.0 0.81 ± 22% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.28 ± 29% -1.0 0.26 ±100% perf-profile.calltrace.cycles-pp.sched_exec.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 1.65 ± 15% -1.0 0.64 ± 11% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 1.34 ± 31% -1.0 0.35 ±107% perf-profile.calltrace.cycles-pp.load_elf_interp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common 1.71 ± 28% -1.0 0.73 ± 47% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write 1.80 ± 13% -1.0 0.84 ± 12% perf-profile.calltrace.cycles-pp.copy_strings.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.55 ± 10% -0.9 0.62 ± 50% perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.51 ± 12% -0.9 0.60 ± 50% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64 1.22 ± 18% -0.8 0.43 ± 75% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat 0.73 ± 21% +2.3 3.07 ± 7% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.00 +2.8 2.80 ± 6% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt 0.00 +2.8 2.84 ± 6% perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 2.52 ± 14% +3.5 6.06 ± 7% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 2.35 ± 11% +3.7 6.05 ± 7% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +4.5 4.51 ± 3% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.23 ± 11% +4.5 5.77 ± 6% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.00 +4.6 4.64 ± 3% perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +4.9 4.94 ± 7% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times 0.00 +5.3 5.30 ± 6% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle 0.00 +5.3 5.33 ± 6% perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 0.00 +6.2 6.22 ± 4% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state 0.00 +6.3 6.29 ± 4% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter 0.00 +6.4 6.35 ± 4% perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle 0.00 +6.5 6.48 ± 4% perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 0.59 ± 80% +6.8 7.38 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.49 ±103% +6.9 7.37 ± 4% perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 1.78 ± 19% +7.1 8.90 ± 2% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +7.1 7.13 ± 4% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.61 ± 51% +7.4 8.06 ± 3% perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.98 ± 49% +7.6 8.54 ± 2% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 2.93 ± 12% +7.9 10.80 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.00 +8.0 7.97 ± 3% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt 0.00 +8.0 8.02 ± 2% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +8.3 8.31 ± 3% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter 0.00 +8.4 8.39 ± 3% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu 0.00 +8.4 8.44 ± 2% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.09 ±223% +8.5 8.56 ± 2% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +8.5 8.47 ± 3% perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt 0.00 +8.6 8.61 ± 3% perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 4.41 ± 17% +9.8 14.19 ± 4% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.23 ± 22% +15.5 16.78 ± 3% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.20 ± 24% +15.6 16.77 ± 3% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 59.83 ± 5% +23.6 83.45 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 59.83 ± 5% +23.6 83.45 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 59.80 ± 5% +23.6 83.43 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 60.74 ± 5% +23.7 84.46 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 7.40 ± 11% +24.7 32.10 ± 2% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 7.27 ± 11% +24.8 32.06 ± 2% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 54.60 ± 6% +27.4 81.97 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 53.26 ± 7% +28.0 81.27 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 15.89 ± 8% +43.1 59.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 12.01 ± 6% +45.5 57.50 perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 38.06 ± 13% -22.4 15.66 ± 11% perf-profile.children.cycles-pp.intel_idle 27.72 ± 8% -17.0 10.74 ± 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 26.32 ± 10% -16.1 10.24 ± 6% perf-profile.children.cycles-pp.do_syscall_64 14.71 ± 12% -9.1 5.61 ± 7% perf-profile.children.cycles-pp.__x64_sys_execve 14.68 ± 12% -9.1 5.60 ± 7% perf-profile.children.cycles-pp.execve 14.67 ± 12% -9.1 5.60 ± 7% perf-profile.children.cycles-pp.do_execveat_common 11.72 ± 12% -7.3 4.37 ± 11% perf-profile.children.cycles-pp.bprm_execve 10.12 ± 14% -6.4 3.71 ± 10% perf-profile.children.cycles-pp.exec_binprm 10.05 ± 14% -6.4 3.67 ± 11% perf-profile.children.cycles-pp.load_elf_binary 8.40 ± 19% -5.1 3.32 ± 9% perf-profile.children.cycles-pp.asm_exc_page_fault 7.42 ± 19% -4.4 2.99 ± 11% perf-profile.children.cycles-pp.exc_page_fault 7.38 ± 19% -4.4 2.96 ± 11% perf-profile.children.cycles-pp.do_user_addr_fault 6.70 ± 20% -4.1 2.55 ± 6% perf-profile.children.cycles-pp.handle_mm_fault 6.34 ± 21% -4.0 2.39 ± 8% perf-profile.children.cycles-pp.__handle_mm_fault 5.50 ± 14% -3.3 2.21 ± 4% perf-profile.children.cycles-pp.main 4.25 ± 17% -3.1 1.14 ± 16% perf-profile.children.cycles-pp.menu_select 4.78 ± 15% -3.1 1.69 ± 12% perf-profile.children.cycles-pp.begin_new_exec 4.44 ± 16% -2.9 1.54 ± 11% perf-profile.children.cycles-pp.mmput 4.40 ± 16% -2.9 1.53 ± 11% perf-profile.children.cycles-pp.exit_mmap 4.86 ± 12% -2.8 2.09 ± 6% perf-profile.children.cycles-pp.do_sys_open 4.84 ± 13% -2.8 2.08 ± 6% perf-profile.children.cycles-pp.do_sys_openat2 4.38 ± 15% -2.7 1.63 ± 10% perf-profile.children.cycles-pp.do_fault 4.29 ± 14% -2.4 1.92 ± 5% perf-profile.children.cycles-pp.do_filp_open 4.26 ± 13% -2.4 1.91 ± 5% perf-profile.children.cycles-pp.path_openat 3.39 ± 8% -2.1 1.30 ± 11% perf-profile.children.cycles-pp.filemap_map_pages 3.46 ± 16% -1.9 1.52 ± 11% perf-profile.children.cycles-pp.walk_component 3.58 ± 15% -1.9 1.67 ± 13% perf-profile.children.cycles-pp.link_path_walk 2.95 ± 12% -1.7 1.21 ± 13% perf-profile.children.cycles-pp.vm_mmap_pgoff 2.63 ± 9% -1.7 0.98 ± 8% perf-profile.children.cycles-pp.__softirqentry_text_start 2.72 ± 12% -1.7 1.07 ± 9% perf-profile.children.cycles-pp.do_mmap 2.42 ± 7% -1.6 0.83 ± 13% perf-profile.children.cycles-pp.unmap_vmas 2.58 ± 16% -1.5 1.05 ± 10% perf-profile.children.cycles-pp.irq_exit_rcu 2.33 ± 6% -1.5 0.81 ± 13% perf-profile.children.cycles-pp.unmap_page_range 1.91 ± 19% -1.5 0.43 ± 18% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 2.43 ± 13% -1.5 0.96 ± 10% perf-profile.children.cycles-pp.mmap_region 2.12 ± 5% -1.4 0.73 ± 14% perf-profile.children.cycles-pp.zap_pte_range 2.18 ± 13% -1.4 0.80 ± 9% perf-profile.children.cycles-pp.kmem_cache_alloc 2.04 ± 12% -1.3 0.70 ± 7% perf-profile.children.cycles-pp.ksys_mmap_pgoff 2.04 ± 34% -1.2 0.79 ± 19% perf-profile.children.cycles-pp.filename_lookup 1.53 ± 27% -1.2 0.29 ± 17% perf-profile.children.cycles-pp.tick_nohz_next_event 1.78 ± 18% -1.1 0.68 ± 11% perf-profile.children.cycles-pp.__lookup_slow 1.50 ± 20% -1.1 0.41 ± 25% perf-profile.children.cycles-pp.kmem_cache_free 1.84 ± 21% -1.1 0.77 ± 19% perf-profile.children.cycles-pp.path_lookupat 1.56 ± 24% -1.0 0.51 ± 13% perf-profile.children.cycles-pp.__do_munmap 1.86 ± 9% -1.0 0.82 ± 23% perf-profile.children.cycles-pp.__do_sys_newstat 1.84 ± 9% -1.0 0.81 ± 22% perf-profile.children.cycles-pp.vfs_statx 1.42 ± 59% -1.0 0.41 ± 15% perf-profile.children.cycles-pp.tlb_finish_mmu 1.40 ± 60% -1.0 0.40 ± 16% perf-profile.children.cycles-pp.tlb_flush_mmu 1.81 ± 14% -0.9 0.86 ± 12% perf-profile.children.cycles-pp.copy_strings 1.45 ± 2% -0.9 0.52 ± 14% perf-profile.children.cycles-pp.next_uptodate_page 1.40 ± 15% -0.9 0.50 ± 14% perf-profile.children.cycles-pp.kthread 1.40 ± 15% -0.9 0.50 ± 15% perf-profile.children.cycles-pp.ret_from_fork 1.49 ± 14% -0.9 0.59 ± 15% perf-profile.children.cycles-pp.native_irq_return_iret 1.19 ± 70% -0.9 0.32 ± 17% perf-profile.children.cycles-pp.release_pages 1.93 ± 20% -0.9 1.08 ± 8% perf-profile.children.cycles-pp.io_serial_in 2.31 ± 19% -0.8 1.47 ± 6% perf-profile.children.cycles-pp.serial8250_console_write 1.22 ± 3% -0.8 0.39 ± 10% perf-profile.children.cycles-pp.rcu_core 1.26 ± 11% -0.8 0.43 ± 16% perf-profile.children.cycles-pp.create_elf_tables 1.28 ± 30% -0.8 0.49 ± 7% perf-profile.children.cycles-pp.sched_exec 1.23 ± 10% -0.8 0.44 ± 10% perf-profile.children.cycles-pp._raw_spin_lock 1.22 ± 25% -0.8 0.44 ± 17% perf-profile.children.cycles-pp.d_alloc_parallel 1.42 ± 29% -0.8 0.64 ± 26% perf-profile.children.cycles-pp.elf_map 1.34 ± 31% -0.8 0.57 ± 33% perf-profile.children.cycles-pp.load_elf_interp 1.15 ± 15% -0.7 0.42 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair 1.11 ± 13% -0.7 0.41 ± 20% perf-profile.children.cycles-pp.strnlen_user 1.10 ± 17% -0.7 0.41 ± 6% perf-profile.children.cycles-pp.find_idlest_group 1.27 ± 19% -0.7 0.58 ± 36% perf-profile.children.cycles-pp.dput 1.03 ± 35% -0.7 0.36 ± 15% perf-profile.children.cycles-pp.d_alloc 1.06 ± 47% -0.7 0.40 ± 11% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.97 ± 7% -0.7 0.32 ± 16% perf-profile.children.cycles-pp.rcu_do_batch 0.78 ± 94% -0.7 0.13 ± 25% perf-profile.children.cycles-pp.do_faccessat 0.99 ± 34% -0.6 0.35 ± 8% perf-profile.children.cycles-pp.__split_vma 0.95 ± 56% -0.6 0.34 ± 21% perf-profile.children.cycles-pp.__clear_user 0.99 ± 19% -0.6 0.38 ± 10% perf-profile.children.cycles-pp.perf_event_mmap 0.73 ± 17% -0.6 0.13 ± 31% perf-profile.children.cycles-pp.smpboot_thread_fn 0.98 ± 20% -0.6 0.38 ± 10% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.92 ± 18% -0.6 0.34 ± 14% perf-profile.children.cycles-pp.___might_sleep 0.85 ± 35% -0.6 0.28 ± 17% perf-profile.children.cycles-pp.__d_alloc 0.69 ± 24% -0.6 0.12 ± 17% perf-profile.children.cycles-pp.irqtime_account_irq 0.99 ± 32% -0.6 0.44 ± 14% perf-profile.children.cycles-pp.__x64_sys_mprotect 0.93 ± 19% -0.6 0.38 ± 19% perf-profile.children.cycles-pp.alloc_empty_file 0.99 ± 32% -0.6 0.44 ± 14% perf-profile.children.cycles-pp.do_mprotect_pkey 0.79 ± 41% -0.5 0.24 ± 26% perf-profile.children.cycles-pp.__vm_munmap 0.90 ± 21% -0.5 0.36 ± 17% perf-profile.children.cycles-pp.__alloc_file 0.79 ± 37% -0.5 0.26 ± 13% perf-profile.children.cycles-pp.__vma_adjust 0.65 ± 10% -0.5 0.12 ± 32% perf-profile.children.cycles-pp.run_ksoftirqd 0.91 ± 45% -0.5 0.38 ± 23% perf-profile.children.cycles-pp.do_set_pte 0.86 ± 16% -0.5 0.35 ± 24% perf-profile.children.cycles-pp.getname_flags 0.90 ± 33% -0.5 0.40 ± 13% perf-profile.children.cycles-pp.mprotect_fixup 1.02 ± 11% -0.5 0.54 ± 33% perf-profile.children.cycles-pp.step_into 0.70 ± 56% -0.5 0.23 ± 19% perf-profile.children.cycles-pp.wp_page_copy 0.73 ± 33% -0.5 0.26 ± 18% perf-profile.children.cycles-pp.task_work_run 0.83 ± 23% -0.5 0.37 ± 14% perf-profile.children.cycles-pp.rebalance_domains 0.71 ± 21% -0.4 0.28 ± 21% perf-profile.children.cycles-pp.unmap_region 0.71 ± 16% -0.4 0.28 ± 17% perf-profile.children.cycles-pp.do_anonymous_page 0.61 ± 13% -0.4 0.19 ± 30% perf-profile.children.cycles-pp.get_user_arg_ptr 0.67 ± 15% -0.4 0.25 ± 22% perf-profile.children.cycles-pp.strncpy_from_user 0.67 ± 8% -0.4 0.26 ± 14% perf-profile.children.cycles-pp.free_pgtables 0.51 ± 80% -0.4 0.11 ± 13% perf-profile.children.cycles-pp.flush_tlb_mm_range 0.53 ± 70% -0.4 0.16 ± 15% perf-profile.children.cycles-pp.__cond_resched 0.57 ± 58% -0.4 0.20 ± 30% perf-profile.children.cycles-pp.page_add_file_rmap 0.51 ± 53% -0.4 0.13 ± 37% perf-profile.children.cycles-pp.terminate_walk 0.58 ± 23% -0.4 0.23 ± 30% perf-profile.children.cycles-pp.page_remove_rmap 0.55 ± 9% -0.4 0.20 ± 17% perf-profile.children.cycles-pp.time 0.59 ± 17% -0.3 0.24 ± 15% perf-profile.children.cycles-pp.simple_lookup 0.48 ± 57% -0.3 0.13 ± 26% perf-profile.children.cycles-pp.__fput 0.55 ± 17% -0.3 0.21 ± 14% perf-profile.children.cycles-pp.d_add 0.70 ± 13% -0.3 0.37 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline 0.59 ± 23% -0.3 0.26 ± 21% perf-profile.children.cycles-pp.lookup_fast 0.44 ± 63% -0.3 0.11 ± 45% perf-profile.children.cycles-pp.vm_area_alloc 0.45 ± 78% -0.3 0.12 ± 17% perf-profile.children.cycles-pp.vma_interval_tree_insert 0.51 ± 25% -0.3 0.19 ± 29% perf-profile.children.cycles-pp.find_vma 0.53 ± 23% -0.3 0.21 ± 26% perf-profile.children.cycles-pp.sync_regs 0.40 ± 96% -0.3 0.09 ± 45% perf-profile.children.cycles-pp.__memcg_kmem_charge 0.49 ± 25% -0.3 0.17 ± 20% perf-profile.children.cycles-pp._IO_vfscanf 0.42 ± 90% -0.3 0.10 ± 10% perf-profile.children.cycles-pp.__do_fault 0.47 ± 25% -0.3 0.16 ± 23% perf-profile.children.cycles-pp.count 0.48 ± 22% -0.3 0.18 ± 34% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.40 ± 27% -0.3 0.10 ± 45% perf-profile.children.cycles-pp.rcu_idle_exit 0.41 ± 12% -0.3 0.11 ± 18% perf-profile.children.cycles-pp.__get_user_8 0.50 ± 27% -0.3 0.20 ± 12% perf-profile.children.cycles-pp.vma_link 0.45 ± 18% -0.3 0.15 ± 22% perf-profile.children.cycles-pp.unlock_page 0.51 ± 19% -0.3 0.23 ± 16% perf-profile.children.cycles-pp.load_balance 0.44 ± 23% -0.3 0.17 ± 36% perf-profile.children.cycles-pp.__might_fault 0.41 ± 93% -0.3 0.14 ± 33% perf-profile.children.cycles-pp.pte_alloc_one 0.46 ± 20% -0.3 0.19 ± 21% perf-profile.children.cycles-pp.perf_iterate_sb 0.47 ± 19% -0.3 0.20 ± 39% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.41 ± 31% -0.3 0.14 ± 48% perf-profile.children.cycles-pp.setup_arg_pages 0.53 ± 24% -0.3 0.26 ± 49% perf-profile.children.cycles-pp.__dentry_kill 0.33 ±104% -0.3 0.06 ± 48% perf-profile.children.cycles-pp.rcu_all_qs 0.37 ± 12% -0.3 0.10 ± 61% perf-profile.children.cycles-pp.__perf_sw_event 0.49 ± 29% -0.3 0.23 ± 11% perf-profile.children.cycles-pp.get_arg_page 0.51 ± 27% -0.3 0.25 ± 10% perf-profile.children.cycles-pp.get_page_from_freelist 0.48 ± 27% -0.3 0.22 ± 11% perf-profile.children.cycles-pp.__get_user_pages 0.42 ± 23% -0.3 0.15 ± 32% perf-profile.children.cycles-pp.__might_sleep 0.49 ± 28% -0.3 0.22 ± 9% perf-profile.children.cycles-pp.__get_user_pages_remote 0.32 ±107% -0.3 0.06 ± 45% perf-profile.children.cycles-pp.filemap_read 0.40 ± 30% -0.3 0.15 ± 17% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 0.39 ± 30% -0.2 0.14 ± 29% perf-profile.children.cycles-pp.alloc_pages_vma 0.30 ± 41% -0.2 0.06 ± 80% perf-profile.children.cycles-pp.update_ts_time_stats 0.30 ±114% -0.2 0.06 ± 50% perf-profile.children.cycles-pp.__kernel_read 0.31 ± 22% -0.2 0.07 ± 48% perf-profile.children.cycles-pp.rcu_eqs_exit 0.33 ± 75% -0.2 0.09 ± 31% perf-profile.children.cycles-pp.alloc_bprm 0.36 ± 22% -0.2 0.12 ± 26% perf-profile.children.cycles-pp.call_rcu 0.57 ± 21% -0.2 0.33 ± 11% perf-profile.children.cycles-pp.worker_thread 0.47 ± 28% -0.2 0.23 ± 18% perf-profile.children.cycles-pp._copy_from_user 0.39 ± 20% -0.2 0.15 ± 23% perf-profile.children.cycles-pp.try_to_unlazy 0.56 ± 19% -0.2 0.33 ± 10% perf-profile.children.cycles-pp.process_one_work 0.48 ± 22% -0.2 0.26 ± 44% perf-profile.children.cycles-pp.__check_object_size 0.28 ± 22% -0.2 0.07 ± 54% perf-profile.children.cycles-pp.nr_iowait_cpu 0.34 ± 27% -0.2 0.12 ± 25% perf-profile.children.cycles-pp.execl 0.54 ± 17% -0.2 0.33 ± 10% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 0.54 ± 18% -0.2 0.33 ± 10% perf-profile.children.cycles-pp.memcpy_toio 0.26 ± 26% -0.2 0.05 ± 78% perf-profile.children.cycles-pp.flush_tlb_func_common 0.31 ± 29% -0.2 0.10 ± 69% perf-profile.children.cycles-pp.shift_arg_pages 0.34 ± 24% -0.2 0.14 ± 37% perf-profile.children.cycles-pp.__x64_sys_munmap 0.32 ± 22% -0.2 0.12 ± 23% perf-profile.children.cycles-pp.__legitimize_path 0.33 ± 35% -0.2 0.12 ± 31% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.34 ± 21% -0.2 0.14 ± 32% perf-profile.children.cycles-pp.update_rq_clock 0.26 ± 17% -0.2 0.06 ± 80% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.32 ± 26% -0.2 0.12 ± 10% perf-profile.children.cycles-pp.copy_string_kernel 0.30 ± 26% -0.2 0.11 ± 32% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.32 ± 33% -0.2 0.13 ± 22% perf-profile.children.cycles-pp.user_path_at_empty 0.32 ± 22% -0.2 0.13 ± 26% perf-profile.children.cycles-pp.unlink_anon_vmas 0.26 ± 33% -0.2 0.08 ± 50% perf-profile.children.cycles-pp.__isoc99_sscanf 0.34 ± 14% -0.2 0.15 ± 22% perf-profile.children.cycles-pp.find_busiest_group 0.30 ± 32% -0.2 0.12 ± 15% perf-profile.children.cycles-pp.__anon_vma_prepare 0.24 ± 23% -0.2 0.06 ± 52% perf-profile.children.cycles-pp.__put_user_nocheck_8 0.34 ± 14% -0.2 0.17 ± 11% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.25 ± 26% -0.2 0.08 ± 24% perf-profile.children.cycles-pp.__cxa_atexit 0.22 ± 37% -0.2 0.05 ± 79% perf-profile.children.cycles-pp.map_vdso 0.25 ± 27% -0.2 0.08 ± 17% perf-profile.children.cycles-pp.get_unmapped_area 0.30 ± 27% -0.2 0.13 ± 18% perf-profile.children.cycles-pp.run_rebalance_domains 0.30 ± 28% -0.2 0.13 ± 21% perf-profile.children.cycles-pp.lru_add_drain 0.29 ± 26% -0.2 0.12 ± 15% perf-profile.children.cycles-pp.update_blocked_averages 0.28 ± 33% -0.2 0.12 ± 53% perf-profile.children.cycles-pp.memmove 0.23 ± 23% -0.2 0.06 ± 78% perf-profile.children.cycles-pp.___perf_sw_event 0.25 ± 14% -0.2 0.09 ± 33% perf-profile.children.cycles-pp.perf_event_mmap_output 0.21 ± 10% -0.2 0.05 ± 76% perf-profile.children.cycles-pp.memcpy_erms 0.28 ± 33% -0.2 0.12 ± 24% perf-profile.children.cycles-pp.__pagevec_lru_add 0.28 ± 25% -0.2 0.13 ± 17% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.29 ± 18% -0.2 0.14 ± 22% perf-profile.children.cycles-pp.update_sd_lb_stats 0.22 ± 46% -0.2 0.07 ± 47% perf-profile.children.cycles-pp.__isoc99_vsscanf 0.25 ± 51% -0.2 0.10 ± 57% perf-profile.children.cycles-pp.ksys_read 0.29 ± 28% -0.2 0.14 ± 21% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.22 ± 37% -0.2 0.07 ± 53% perf-profile.children.cycles-pp._start 0.20 ± 31% -0.1 0.05 ± 78% perf-profile.children.cycles-pp.note_gp_changes 0.24 ± 45% -0.1 0.09 ± 57% perf-profile.children.cycles-pp.vfs_read 0.18 ± 16% -0.1 0.04 ± 75% perf-profile.children.cycles-pp.do_brk_flags 0.19 ± 22% -0.1 0.06 ± 50% perf-profile.children.cycles-pp.vm_brk_flags 0.24 ± 15% -0.1 0.11 ± 22% perf-profile.children.cycles-pp.__d_lookup_done 0.16 ± 41% -0.1 0.03 ±100% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.19 ± 24% -0.1 0.06 ± 50% perf-profile.children.cycles-pp.page_counter_try_charge 0.19 ± 35% -0.1 0.06 ± 52% perf-profile.children.cycles-pp.fput_many 0.21 ± 50% -0.1 0.08 ± 26% perf-profile.children.cycles-pp.__rb_insert_augmented 0.18 ± 19% -0.1 0.06 ± 78% perf-profile.children.cycles-pp.__d_lookup_rcu 0.15 ± 42% -0.1 0.03 ±102% perf-profile.children.cycles-pp.up_read 0.21 ± 29% -0.1 0.09 ± 23% perf-profile.children.cycles-pp.unlink_file_vma 0.18 ± 26% -0.1 0.06 ± 47% perf-profile.children.cycles-pp.mark_page_accessed 0.19 ± 43% -0.1 0.07 ± 11% perf-profile.children.cycles-pp.__vma_link_rb 0.17 ± 22% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.__check_heap_object 0.18 ± 36% -0.1 0.06 ± 32% perf-profile.children.cycles-pp.vm_normal_page 0.26 ± 35% -0.1 0.14 ± 30% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.19 ± 31% -0.1 0.07 ± 52% perf-profile.children.cycles-pp.up_write 0.17 ± 67% -0.1 0.06 ± 73% perf-profile.children.cycles-pp.enqueue_hrtimer 0.24 ± 15% -0.1 0.12 ± 19% perf-profile.children.cycles-pp.arch_scale_freq_tick 0.14 ± 38% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__list_del_entry_valid 0.17 ± 53% -0.1 0.05 ± 53% perf-profile.children.cycles-pp.generic_permission 0.14 ± 36% -0.1 0.03 ±103% perf-profile.children.cycles-pp.xas_load 0.20 ± 41% -0.1 0.09 ± 33% perf-profile.children.cycles-pp.__get_free_pages 0.14 ± 23% -0.1 0.03 ±104% perf-profile.children.cycles-pp.lockref_get_not_dead 0.18 ± 24% -0.1 0.08 ± 25% perf-profile.children.cycles-pp.vm_area_dup 0.15 ± 41% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.refill_obj_stock 0.15 ± 29% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.__legitimize_mnt 0.14 ± 26% -0.1 0.05 ± 48% perf-profile.children.cycles-pp.call_cpuidle 0.16 ± 33% -0.1 0.07 ± 20% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.18 ± 14% -0.1 0.09 ± 50% perf-profile.children.cycles-pp.__pte_alloc 0.16 ± 26% -0.1 0.07 ± 30% perf-profile.children.cycles-pp.__mod_memcg_state 0.17 ± 23% -0.1 0.08 ± 54% perf-profile.children.cycles-pp.d_path 0.15 ± 18% -0.1 0.07 ± 54% perf-profile.children.cycles-pp.finish_fault 0.11 ± 27% -0.1 0.03 ±100% perf-profile.children.cycles-pp.down_read 0.11 ± 27% -0.1 0.03 ±101% perf-profile.children.cycles-pp.__slab_alloc 0.10 ± 27% -0.1 0.03 ±102% perf-profile.children.cycles-pp.___slab_alloc 0.10 ± 15% -0.1 0.03 ±103% perf-profile.children.cycles-pp.expand_downwards 0.12 ± 19% -0.1 0.05 ± 48% perf-profile.children.cycles-pp.io_serial_out 0.16 ± 9% -0.1 0.10 ± 28% perf-profile.children.cycles-pp.do_open 0.02 ±142% +0.2 0.18 ± 30% perf-profile.children.cycles-pp.update_load_avg 0.01 ±223% +0.2 0.18 ± 22% perf-profile.children.cycles-pp.update_curr 0.05 ± 74% +0.5 0.53 ± 20% perf-profile.children.cycles-pp.task_tick_fair 0.00 +0.7 0.73 ± 9% perf-profile.children.cycles-pp.sched_clock_local 0.74 ± 21% +2.4 3.14 ± 7% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 2.67 ± 12% +4.7 7.33 ± 8% perf-profile.children.cycles-pp.tick_sched_handle 2.50 ± 9% +4.8 7.29 ± 8% perf-profile.children.cycles-pp.update_process_times 1.32 ± 8% +5.5 6.79 ± 8% perf-profile.children.cycles-pp.scheduler_tick 0.70 ± 54% +6.8 7.46 ± 4% perf-profile.children.cycles-pp.tick_nohz_irq_exit 1.84 ± 19% +8.0 9.79 ± 3% perf-profile.children.cycles-pp.clockevents_program_event 0.40 ± 27% +8.3 8.72 ± 2% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 3.09 ± 10% +9.1 12.20 ± 5% perf-profile.children.cycles-pp.tick_sched_timer 4.60 ± 16% +11.1 15.70 ± 5% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.01 ±223% +15.3 15.26 ± 3% perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event 1.24 ± 22% +15.8 17.00 ± 2% perf-profile.children.cycles-pp.irq_enter_rcu 1.22 ± 24% +15.8 16.99 ± 2% perf-profile.children.cycles-pp.tick_irq_enter 0.00 +20.7 20.69 ± 3% perf-profile.children.cycles-pp.sched_clock_tick 59.83 ± 5% +23.6 83.45 perf-profile.children.cycles-pp.start_secondary 60.74 ± 5% +23.7 84.46 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 60.74 ± 5% +23.7 84.46 perf-profile.children.cycles-pp.cpu_startup_entry 60.74 ± 5% +23.7 84.46 perf-profile.children.cycles-pp.do_idle 7.66 ± 11% +27.0 34.62 ± 2% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 7.53 ± 11% +27.0 34.58 ± 2% perf-profile.children.cycles-pp.hrtimer_interrupt 55.45 ± 6% +27.5 82.93 perf-profile.children.cycles-pp.cpuidle_enter_state 55.45 ± 6% +27.5 82.95 perf-profile.children.cycles-pp.cpuidle_enter 14.73 ± 7% +46.3 61.01 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 12.64 ± 6% +47.6 60.23 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 2.77 ± 29% +50.5 53.26 ± 2% perf-profile.children.cycles-pp.ktime_get 0.00 +60.5 60.51 perf-profile.children.cycles-pp.read_hpet 38.06 ± 13% -22.4 15.66 ± 11% perf-profile.self.cycles-pp.intel_idle 3.26 ± 35% -2.2 1.02 ± 10% perf-profile.self.cycles-pp.cpuidle_enter_state 2.08 ± 27% -1.4 0.65 ± 25% perf-profile.self.cycles-pp.menu_select 1.45 ± 2% -1.0 0.44 ± 14% perf-profile.self.cycles-pp.next_uptodate_page 1.49 ± 14% -0.9 0.59 ± 15% perf-profile.self.cycles-pp.native_irq_return_iret 1.93 ± 20% -0.9 1.08 ± 8% perf-profile.self.cycles-pp.io_serial_in 1.18 ± 10% -0.8 0.40 ± 14% perf-profile.self.cycles-pp._raw_spin_lock 1.08 ± 14% -0.7 0.35 ± 17% perf-profile.self.cycles-pp.strnlen_user 1.03 ± 8% -0.7 0.31 ± 16% perf-profile.self.cycles-pp.zap_pte_range 0.85 ± 15% -0.7 0.19 ± 18% perf-profile.self.cycles-pp.kmem_cache_free 1.01 ± 15% -0.6 0.36 ± 7% perf-profile.self.cycles-pp.find_idlest_group 0.86 ± 18% -0.6 0.26 ± 26% perf-profile.self.cycles-pp.kmem_cache_alloc 0.88 ± 20% -0.6 0.29 ± 11% perf-profile.self.cycles-pp.___might_sleep 0.49 ± 35% -0.4 0.12 ± 17% perf-profile.self.cycles-pp.irqtime_account_irq 0.51 ± 22% -0.4 0.16 ± 26% perf-profile.self.cycles-pp.release_pages 0.70 ± 14% -0.3 0.37 ± 9% perf-profile.self.cycles-pp.lapic_next_deadline 0.52 ± 25% -0.3 0.21 ± 25% perf-profile.self.cycles-pp.sync_regs 0.42 ± 19% -0.3 0.12 ± 31% perf-profile.self.cycles-pp.unlock_page 0.39 ± 9% -0.3 0.09 ± 23% perf-profile.self.cycles-pp.__get_user_8 0.36 ± 32% -0.3 0.09 ± 19% perf-profile.self.cycles-pp.tick_nohz_next_event 0.46 ± 20% -0.3 0.19 ± 39% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.42 ± 26% -0.3 0.15 ± 11% perf-profile.self.cycles-pp.filemap_map_pages 0.32 ± 26% -0.3 0.06 ± 50% perf-profile.self.cycles-pp.update_process_times 0.39 ± 32% -0.2 0.14 ± 39% perf-profile.self.cycles-pp.page_remove_rmap 0.37 ± 16% -0.2 0.13 ± 34% perf-profile.self.cycles-pp.__might_sleep 0.39 ± 33% -0.2 0.16 ± 34% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.30 ± 24% -0.2 0.08 ± 22% perf-profile.self.cycles-pp.find_vma 0.54 ± 18% -0.2 0.32 ± 9% perf-profile.self.cycles-pp.memcpy_toio 0.28 ± 24% -0.2 0.06 ± 58% perf-profile.self.cycles-pp.nr_iowait_cpu 0.32 ± 45% -0.2 0.10 ± 15% perf-profile.self.cycles-pp.vma_interval_tree_insert 0.32 ± 33% -0.2 0.11 ± 35% perf-profile.self.cycles-pp.page_add_file_rmap 0.32 ± 27% -0.2 0.11 ± 23% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.26 ± 17% -0.2 0.06 ± 80% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.30 ± 31% -0.2 0.11 ± 16% perf-profile.self.cycles-pp.get_obj_cgroup_from_current 0.25 ± 18% -0.2 0.06 ± 57% perf-profile.self.cycles-pp.call_rcu 0.26 ± 28% -0.2 0.08 ± 50% perf-profile.self.cycles-pp.strncpy_from_user 0.21 ± 7% -0.2 0.03 ±100% perf-profile.self.cycles-pp.memcpy_erms 0.32 ± 37% -0.2 0.15 ± 25% perf-profile.self.cycles-pp.do_idle 0.21 ± 26% -0.2 0.05 ± 54% perf-profile.self.cycles-pp.__put_user_nocheck_8 0.20 ± 18% -0.2 0.04 ± 73% perf-profile.self.cycles-pp.perf_event_mmap 0.20 ± 31% -0.2 0.05 ± 79% perf-profile.self.cycles-pp.copy_strings 0.19 ± 27% -0.2 0.04 ± 72% perf-profile.self.cycles-pp.rcu_eqs_exit 0.26 ± 41% -0.1 0.11 ± 29% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.26 ± 15% -0.1 0.11 ± 8% perf-profile.self.cycles-pp.copy_user_generic_unrolled 0.19 ± 42% -0.1 0.05 ±101% perf-profile.self.cycles-pp.copy_page 0.22 ± 33% -0.1 0.08 ± 61% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.25 ± 27% -0.1 0.11 ± 27% perf-profile.self.cycles-pp.update_sd_lb_stats 0.18 ± 19% -0.1 0.05 ±102% perf-profile.self.cycles-pp.__d_lookup_rcu 0.19 ± 39% -0.1 0.05 ± 48% perf-profile.self.cycles-pp.__vma_link_rb 0.16 ± 31% -0.1 0.03 ± 99% perf-profile.self.cycles-pp._IO_vfscanf 0.17 ± 23% -0.1 0.03 ±108% perf-profile.self.cycles-pp.___perf_sw_event 0.20 ± 29% -0.1 0.07 ± 18% perf-profile.self.cycles-pp.__cond_resched 0.17 ± 22% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.__check_heap_object 0.18 ± 30% -0.1 0.05 ± 52% perf-profile.self.cycles-pp.page_counter_try_charge 0.16 ± 33% -0.1 0.03 ±105% perf-profile.self.cycles-pp.vm_normal_page 0.15 ± 29% -0.1 0.03 ±100% perf-profile.self.cycles-pp.__legitimize_mnt 0.15 ± 26% -0.1 0.03 ±100% perf-profile.self.cycles-pp.mark_page_accessed 0.26 ± 35% -0.1 0.14 ± 30% perf-profile.self.cycles-pp.__intel_pmu_enable_all 0.24 ± 15% -0.1 0.12 ± 19% perf-profile.self.cycles-pp.arch_scale_freq_tick 0.18 ± 44% -0.1 0.06 ± 60% perf-profile.self.cycles-pp.rcu_read_unlock_strict 0.17 ± 42% -0.1 0.06 ± 48% perf-profile.self.cycles-pp.__rb_insert_augmented 0.14 ± 23% -0.1 0.03 ±102% perf-profile.self.cycles-pp.lockref_get_not_dead 0.17 ± 32% -0.1 0.06 ± 54% perf-profile.self.cycles-pp.up_write 0.14 ± 41% -0.1 0.03 ±108% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.13 ± 39% -0.1 0.03 ± 99% perf-profile.self.cycles-pp.__list_del_entry_valid 0.16 ± 26% -0.1 0.06 ± 87% perf-profile.self.cycles-pp.vmacache_find 0.14 ± 27% -0.1 0.04 ± 75% perf-profile.self.cycles-pp.call_cpuidle 0.20 ± 30% -0.1 0.11 ± 24% perf-profile.self.cycles-pp.do_set_pte 0.13 ± 36% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.rcu_all_qs 0.15 ± 30% -0.1 0.06 ± 54% perf-profile.self.cycles-pp.__mod_memcg_state 0.13 ± 44% -0.1 0.04 ± 75% perf-profile.self.cycles-pp.update_rq_clock 0.15 ± 30% -0.1 0.06 ± 55% perf-profile.self.cycles-pp.__check_object_size 0.12 ± 38% -0.1 0.05 ± 83% perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt 0.11 ± 36% -0.1 0.03 ±103% perf-profile.self.cycles-pp.__hrtimer_get_next_event 0.14 ± 29% -0.1 0.07 ± 41% perf-profile.self.cycles-pp.handle_mm_fault 0.12 ± 19% -0.1 0.05 ± 48% perf-profile.self.cycles-pp.io_serial_out 0.00 +0.1 0.11 ± 22% perf-profile.self.cycles-pp.update_curr 0.00 +0.5 0.49 ± 11% perf-profile.self.cycles-pp.sched_clock_local 0.00 +60.5 60.51 perf-profile.self.cycles-pp.read_hpet *************************************************************************************************** lkp-csl-2ap2: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory ========================================================================================= compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode: gcc-9/performance/1BRD_48G/4M/btrfs/1x/x86_64-rhel-8.3/1t/debian-10.4-x86_64-20200603.cgz/NoSync/lkp-csl-2ap2/24G/fsmark/0x5003006 commit: d42c7a8d68 ("kcsan: Document "value changed" line") 6c52b5f3cf ("clocksource: Reduce WATCHDOG_THRESHOLD") d42c7a8d68038bc0 6c52b5f3cfefd6e429efc4413fd ---------------- --------------------------- %stddev %change %stddev \ | \ 264577 +1732.3% 4847771 fsmark.app_overhead 399.03 -37.8% 248.35 ± 2% fsmark.files_per_sec 21.70 +34.5% 29.19 ± 3% fsmark.time.elapsed_time 21.70 +34.5% 29.19 ± 3% fsmark.time.elapsed_time.max 79.67 +15.1% 91.67 fsmark.time.percent_of_cpu_this_job_got 17.21 +53.9% 26.49 ± 2% fsmark.time.system_time 169429 ± 2% -86.9% 22149 ± 8% fsmark.time.voluntary_context_switches 126301 ± 75% -98.4% 2017 ± 85% cpuidle.POLL.usage 5510 ± 6% +17.1% 6452 ± 3% boot-time.idle 1.72 +2.4% 1.77 boot-time.smp_boot 96.92 +2.0% 98.86 iostat.cpu.idle 2.86 ± 8% -65.9% 0.97 ± 5% iostat.cpu.system 48.83 ± 14% +161.8% 127.83 ± 34% turbostat.Avg_MHz 4.70 ± 6% +8.9 13.58 ± 14% turbostat.Busy% 62.27 ± 2% +16.4% 72.51 ± 2% uptime.boot 10421 ± 2% +24.8% 13004 ± 3% uptime.idle 0.05 ± 5% -0.0 0.02 ± 23% mpstat.cpu.all.iowait% 1.26 ± 17% -1.3 0.00 mpstat.cpu.all.irq% 0.09 ± 11% -0.1 0.00 ± 48% mpstat.cpu.all.soft% 7332 ± 62% +122.5% 16311 ± 39% numa-meminfo.node3.Active 4212 ± 20% +198.0% 12550 ± 24% numa-meminfo.node3.Active(anon) 7027 ± 48% +159.9% 18263 ± 33% numa-meminfo.node3.Shmem 1052 ± 20% +196.1% 3117 ± 24% numa-vmstat.node3.nr_active_anon 1756 ± 48% +159.5% 4557 ± 34% numa-vmstat.node3.nr_shmem 1052 ± 20% +196.1% 3117 ± 24% numa-vmstat.node3.nr_zone_active_anon 7855 ± 11% +39.8% 10981 ± 9% softirqs.CPU0.SCHED 5753 ± 13% +37.2% 7895 ± 27% softirqs.CPU1.SCHED 970575 ± 5% +20.3% 1167382 ± 3% softirqs.SCHED 96.67 +1.4% 98.00 vmstat.cpu.id 1008185 -23.6% 770151 ± 5% vmstat.io.bo 18059 ± 2% -74.6% 4595 vmstat.system.cs 0.01 ± 14% +51.7% 0.02 ± 11% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.01 ± 14% +51.7% 0.02 ± 11% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 3.00 ± 12% +54.8% 4.64 ± 16% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 2.53 ± 47% +83.5% 4.64 ± 16% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork 16335 ± 6% +44.7% 23644 ± 12% meminfo.Active 7042 ± 15% +103.1% 14304 ± 20% meminfo.Active(anon) 13816046 -10.7% 12342039 ± 3% meminfo.Inactive(file) 26979 ± 4% +31.6% 35497 ± 9% meminfo.Shmem 2269754 ± 3% -73.2% 607860 ± 4% meminfo.Writeback 1940 ± 26% -45.2% 1063 ± 41% sched_debug.cpu.avg_idle.min 32619 ± 3% +28.6% 41932 ± 2% sched_debug.cpu.clock_task.min 645.89 ± 20% -97.9% 13.74 ± 19% sched_debug.cpu.clock_task.stddev 165.42 ± 15% -24.1% 125.63 ± 13% sched_debug.cpu.curr->pid.avg 720.17 ± 7% -12.9% 627.50 ± 6% sched_debug.cpu.curr->pid.stddev 1.00 -100.0% 0.00 sched_debug.sched_clock_stable() 657.83 ± 3% -58.6% 272.50 ± 7% slabinfo.biovec-max.active_objs 684.00 ± 3% -54.9% 308.50 ± 8% slabinfo.biovec-max.num_objs 4867 -13.7% 4202 ± 5% slabinfo.btrfs_extent_map.active_objs 4982 -13.7% 4301 ± 4% slabinfo.btrfs_extent_map.num_objs 3531 -10.2% 3173 ± 2% slabinfo.btrfs_inode.active_objs 3546 -10.0% 3191 ± 2% slabinfo.btrfs_inode.num_objs 6938 ± 5% -11.2% 6158 ± 7% slabinfo.files_cache.active_objs 6938 ± 5% -11.2% 6158 ± 7% slabinfo.files_cache.num_objs 2146 -18.1% 1757 slabinfo.kmalloc-8k.active_objs 2164 -17.8% 1779 slabinfo.kmalloc-8k.num_objs 1753 ± 15% +104.0% 3575 ± 20% proc-vmstat.nr_active_anon 3742994 -10.4% 3351949 ± 2% proc-vmstat.nr_file_pages 42736503 +1.0% 43175533 proc-vmstat.nr_free_pages 3474846 -11.3% 3081667 ± 3% proc-vmstat.nr_inactive_file 6738 ± 4% +31.7% 8874 ± 9% proc-vmstat.nr_shmem 44643 -3.1% 43267 proc-vmstat.nr_slab_reclaimable 77603 -3.7% 74734 proc-vmstat.nr_slab_unreclaimable 570405 ± 3% -73.4% 151983 ± 4% proc-vmstat.nr_writeback 1753 ± 15% +104.0% 3575 ± 20% proc-vmstat.nr_zone_active_anon 3474846 -11.3% 3081667 ± 3% proc-vmstat.nr_zone_inactive_file 1652814 -22.3% 1284857 ± 4% proc-vmstat.nr_zone_write_pending 1359793 -61.0% 530557 ± 22% proc-vmstat.numa_foreign 11732111 +7.3% 12586943 proc-vmstat.numa_hit 11472411 +7.5% 12327232 proc-vmstat.numa_local 1359793 -61.0% 530557 ± 22% proc-vmstat.numa_miss 1619493 -51.2% 790267 ± 14% proc-vmstat.numa_other 3192 ± 5% +87.0% 5969 ± 13% proc-vmstat.pgactivate 126035 ± 2% +27.4% 160516 ± 4% proc-vmstat.pgfault 6459 ± 3% +27.7% 8247 ± 6% proc-vmstat.pgreuse 1.513e+09 -11.7% 1.337e+09 ± 3% perf-stat.i.branch-instructions 48492068 ± 3% -23.9% 36917714 ± 5% perf-stat.i.cache-misses 19647 ± 3% -77.1% 4504 perf-stat.i.context-switches 2.03 ± 8% +237.1% 6.83 ± 18% perf-stat.i.cpi 1.402e+10 ± 8% +200.4% 4.212e+10 ± 18% perf-stat.i.cpu-cycles 259.23 ± 2% -8.6% 237.01 perf-stat.i.cpu-migrations 824.65 ± 15% +329.0% 3537 ± 24% perf-stat.i.cycles-between-cache-misses 1.929e+09 ± 2% -14.1% 1.658e+09 ± 4% perf-stat.i.dTLB-loads 8.938e+08 -19.1% 7.228e+08 ± 3% perf-stat.i.dTLB-stores 7.72e+09 -13.0% 6.719e+09 ± 3% perf-stat.i.instructions 0.54 ± 8% -69.7% 0.16 ± 25% perf-stat.i.ipc 9.44 ± 6% -26.6% 6.93 ± 10% perf-stat.i.major-faults 0.07 ± 8% +200.1% 0.22 ± 18% perf-stat.i.metric.GHz 23.21 -14.5% 19.84 ± 2% perf-stat.i.metric.M/sec 4560 -2.9% 4426 perf-stat.i.minor-faults 11667068 ± 4% -16.9% 9696044 ± 16% perf-stat.i.node-loads 2789878 ± 8% -13.9% 2401011 ± 5% perf-stat.i.node-stores 4569 -3.0% 4433 perf-stat.i.page-faults 1.82 ± 8% +244.9% 6.26 ± 17% perf-stat.overall.cpi 289.37 ± 6% +301.9% 1162 ± 21% perf-stat.overall.cycles-between-cache-misses 0.55 ± 8% -69.9% 0.17 ± 25% perf-stat.overall.ipc 1.447e+09 -10.8% 1.291e+09 ± 3% perf-stat.ps.branch-instructions 46269702 ± 3% -23.4% 35462108 ± 6% perf-stat.ps.cache-misses 18723 ± 3% -76.8% 4334 perf-stat.ps.context-switches 183348 +1.3% 185661 perf-stat.ps.cpu-clock 1.34e+10 ± 8% +204.0% 4.075e+10 ± 18% perf-stat.ps.cpu-cycles 247.92 ± 2% -7.2% 230.01 perf-stat.ps.cpu-migrations 1.844e+09 ± 2% -13.2% 1.6e+09 ± 4% perf-stat.ps.dTLB-loads 8.542e+08 -18.3% 6.977e+08 ± 3% perf-stat.ps.dTLB-stores 7.381e+09 -12.1% 6.49e+09 ± 2% perf-stat.ps.instructions 9.01 ± 6% -26.3% 6.64 ± 10% perf-stat.ps.major-faults 11128894 ± 4% -16.4% 9301546 ± 16% perf-stat.ps.node-loads 183348 +1.3% 185661 perf-stat.ps.task-clock 1.666e+11 +20.4% 2.006e+11 ± 6% perf-stat.total.instructions 36.10 ± 5% -23.5 12.58 ± 30% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 19.48 ± 10% -15.3 4.21 ± 15% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 23.57 ± 6% -15.2 8.42 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 23.46 ± 6% -15.1 8.38 ± 11% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.86 ± 10% -14.7 4.12 ± 9% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.79 ± 10% -14.7 4.11 ± 9% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.66 ± 10% -14.6 4.08 ± 9% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 18.61 ± 10% -14.5 4.07 ± 9% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write 10.22 ± 5% -8.0 2.24 ± 16% perf-profile.calltrace.cycles-pp.ret_from_fork 10.22 ± 5% -8.0 2.24 ± 16% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 10.13 ± 5% -7.9 2.20 ± 15% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 10.11 ± 5% -7.9 2.19 ± 16% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 6.84 ± 10% -5.4 1.47 ± 7% perf-profile.calltrace.cycles-pp.btrfs_copy_from_user.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 6.62 ± 10% -5.2 1.43 ± 7% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.btrfs_copy_from_user.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 6.46 ± 10% -5.1 1.38 ± 7% perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.btrfs_copy_from_user.btrfs_buffered_write.btrfs_file_write_iter 6.40 ± 10% -5.0 1.37 ± 7% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.btrfs_copy_from_user.btrfs_buffered_write 6.03 ± 7% -4.7 1.32 ± 20% perf-profile.calltrace.cycles-pp.btrfs_work_helper.process_one_work.worker_thread.kthread.ret_from_fork 5.84 ± 7% -4.6 1.29 ± 21% perf-profile.calltrace.cycles-pp.brd_submit_bio.submit_bio_noacct.submit_bio.btrfs_map_bio.run_one_async_done 5.84 ± 7% -4.6 1.29 ± 21% perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.run_one_async_done.btrfs_work_helper.process_one_work 5.84 ± 7% -4.6 1.29 ± 21% perf-profile.calltrace.cycles-pp.submit_bio_noacct.submit_bio.btrfs_map_bio.run_one_async_done.btrfs_work_helper 5.84 ± 7% -4.5 1.29 ± 21% perf-profile.calltrace.cycles-pp.run_one_async_done.btrfs_work_helper.process_one_work.worker_thread.kthread 5.84 ± 7% -4.5 1.29 ± 21% perf-profile.calltrace.cycles-pp.btrfs_map_bio.run_one_async_done.btrfs_work_helper.process_one_work.worker_thread 5.81 ± 11% -4.5 1.27 ± 15% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 5.65 ± 11% -4.4 1.24 ± 15% perf-profile.calltrace.cycles-pp.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 5.64 ± 11% -4.4 1.24 ± 15% perf-profile.calltrace.cycles-pp.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 5.52 ± 11% -4.3 1.21 ± 15% perf-profile.calltrace.cycles-pp.need_preemptive_reclaim.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write 5.42 ± 11% -4.2 1.19 ± 15% perf-profile.calltrace.cycles-pp.__percpu_counter_sum.need_preemptive_reclaim.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata 5.20 ± 8% -4.1 1.15 ± 21% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_submit_bio.submit_bio_noacct.submit_bio.btrfs_map_bio 4.59 ± 19% -3.6 0.97 ± 26% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 3.20 ± 8% -2.6 0.62 ± 44% perf-profile.calltrace.cycles-pp.brd_insert_page.brd_do_bvec.brd_submit_bio.submit_bio_noacct.submit_bio 3.01 ± 10% -2.3 0.69 ± 8% perf-profile.calltrace.cycles-pp.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 2.79 ± 9% -2.1 0.64 ± 8% perf-profile.calltrace.cycles-pp.pagecache_get_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 2.23 ± 11% -1.6 0.65 ± 37% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +2.3 2.28 ± 9% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_real_ts64.__x64_sys_gettimeofday.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +2.6 2.55 ± 6% perf-profile.calltrace.cycles-pp.ktime_get_real_ts64.__x64_sys_gettimeofday.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +2.6 2.60 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_gettimeofday.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.82 ± 34% +4.0 4.84 ± 10% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 2.15 ± 18% +4.4 6.55 ± 8% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 2.04 ± 15% +4.5 6.50 ± 7% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +4.5 4.52 ± 7% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt 0.00 +4.6 4.56 ± 8% perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.99 ± 15% +5.2 6.22 ± 5% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.00 +5.4 5.40 ± 3% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.10 ±223% +5.5 5.55 ± 4% perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +5.6 5.60 ± 3% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times 0.00 +5.8 5.82 ± 3% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle 0.00 +5.8 5.84 ± 3% perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 0.00 +7.2 7.23 ± 16% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state 0.45 ± 72% +7.3 7.73 ± 14% perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 0.46 ± 72% +7.3 7.75 ± 14% perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +7.3 7.33 ± 15% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter 0.00 +7.4 7.37 ± 15% perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle 0.00 +7.5 7.49 ± 15% perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 0.00 +7.6 7.58 ± 14% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.96 ± 28% +7.6 9.59 ± 11% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.07 ± 56% +8.2 9.25 ± 11% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 1.00 ± 29% +8.3 9.26 ± 17% perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +9.0 8.98 ± 13% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter 0.00 +9.0 9.02 ± 11% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +9.1 9.10 ± 12% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu 0.41 ±106% +9.1 9.53 ± 9% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +9.1 9.14 ± 12% perf-profile.calltrace.cycles-pp.sched_clock_tick.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt 0.00 +9.2 9.17 ± 17% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt 0.00 +9.3 9.26 ± 12% perf-profile.calltrace.cycles-pp.sched_clock_idle_wakeup_event.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +9.4 9.38 ± 9% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 2.58 ± 14% +9.6 12.19 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 4.12 ± 18% +13.3 17.42 ± 7% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.36 ± 23% +17.3 18.64 ± 14% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.33 ± 23% +17.3 18.63 ± 14% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 60.63 ± 2% +26.7 87.34 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 60.63 ± 2% +26.7 87.34 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 60.59 ± 2% +26.7 87.33 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 60.85 ± 2% +27.0 87.88 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 7.59 ± 11% +29.3 36.93 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 7.43 ± 10% +29.4 36.85 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 54.95 ± 2% +31.0 85.98 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 53.41 ± 3% +32.0 85.40 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 17.19 ± 12% +48.3 65.47 ± 5% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 12.37 ± 10% +51.9 64.29 ± 6% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 36.20 ± 5% -23.6 12.64 ± 30% perf-profile.children.cycles-pp.intel_idle 24.67 ± 5% -15.7 8.99 ± 10% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 24.56 ± 5% -15.6 8.95 ± 10% perf-profile.children.cycles-pp.do_syscall_64 19.66 ± 10% -15.2 4.49 ± 12% perf-profile.children.cycles-pp.ksys_write 19.60 ± 10% -15.1 4.47 ± 12% perf-profile.children.cycles-pp.vfs_write 19.48 ± 10% -15.0 4.44 ± 12% perf-profile.children.cycles-pp.new_sync_write 18.67 ± 10% -14.6 4.08 ± 9% perf-profile.children.cycles-pp.btrfs_file_write_iter 18.62 ± 10% -14.6 4.07 ± 9% perf-profile.children.cycles-pp.btrfs_buffered_write 10.22 ± 5% -8.0 2.24 ± 16% perf-profile.children.cycles-pp.ret_from_fork 10.22 ± 5% -8.0 2.24 ± 16% perf-profile.children.cycles-pp.kthread 10.13 ± 5% -7.9 2.20 ± 15% perf-profile.children.cycles-pp.worker_thread 10.11 ± 5% -7.9 2.19 ± 16% perf-profile.children.cycles-pp.process_one_work 6.84 ± 10% -5.4 1.47 ± 7% perf-profile.children.cycles-pp.btrfs_copy_from_user 6.63 ± 10% -5.2 1.43 ± 7% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 6.49 ± 10% -5.1 1.39 ± 7% perf-profile.children.cycles-pp.copyin 6.44 ± 10% -5.1 1.38 ± 7% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 6.03 ± 7% -4.7 1.32 ± 20% perf-profile.children.cycles-pp.btrfs_work_helper 5.84 ± 7% -4.6 1.29 ± 21% perf-profile.children.cycles-pp.submit_bio 5.84 ± 7% -4.6 1.29 ± 21% perf-profile.children.cycles-pp.submit_bio_noacct 5.84 ± 7% -4.6 1.29 ± 21% perf-profile.children.cycles-pp.brd_submit_bio 5.84 ± 7% -4.5 1.29 ± 21% perf-profile.children.cycles-pp.run_one_async_done 5.84 ± 7% -4.5 1.29 ± 21% perf-profile.children.cycles-pp.btrfs_map_bio 5.81 ± 11% -4.5 1.27 ± 15% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 5.71 ± 11% -4.5 1.25 ± 15% perf-profile.children.cycles-pp.__reserve_bytes 5.67 ± 11% -4.4 1.24 ± 15% perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes 5.54 ± 11% -4.3 1.22 ± 15% perf-profile.children.cycles-pp.need_preemptive_reclaim 5.46 ± 11% -4.3 1.20 ± 15% perf-profile.children.cycles-pp.__percpu_counter_sum 5.21 ± 7% -4.1 1.15 ± 21% perf-profile.children.cycles-pp.brd_do_bvec 4.63 ± 19% -3.6 0.99 ± 26% perf-profile.children.cycles-pp.menu_select 3.09 ± 17% -2.7 0.42 ± 6% perf-profile.children.cycles-pp.__writeback_single_inode 3.09 ± 17% -2.7 0.42 ± 6% perf-profile.children.cycles-pp.do_writepages 3.09 ± 17% -2.7 0.42 ± 6% perf-profile.children.cycles-pp.extent_writepages 3.07 ± 17% -2.7 0.42 ± 6% perf-profile.children.cycles-pp.extent_write_cache_pages 3.22 ± 8% -2.5 0.69 ± 19% perf-profile.children.cycles-pp.brd_insert_page 3.02 ± 10% -2.3 0.70 ± 8% perf-profile.children.cycles-pp.prepare_pages 2.96 ± 7% -2.3 0.68 ± 13% perf-profile.children.cycles-pp._raw_spin_lock 2.92 ± 9% -2.3 0.66 ± 8% perf-profile.children.cycles-pp.pagecache_get_page 2.48 ± 17% -2.1 0.34 ± 5% perf-profile.children.cycles-pp.__extent_writepage 2.31 ± 11% -1.8 0.47 ± 22% perf-profile.children.cycles-pp.cpumask_next 2.36 ± 14% -1.7 0.69 ± 35% perf-profile.children.cycles-pp.irq_exit_rcu 1.94 ± 11% -1.5 0.40 ± 12% perf-profile.children.cycles-pp.btrfs_dirty_pages 1.78 ± 11% -1.4 0.35 ± 30% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 1.60 ± 17% -1.4 0.21 ± 7% perf-profile.children.cycles-pp.__extent_writepage_io 1.72 ± 9% -1.3 0.40 ± 8% perf-profile.children.cycles-pp.add_to_page_cache_lru 1.45 ± 16% -1.2 0.23 ± 28% perf-profile.children.cycles-pp.tick_nohz_next_event 1.74 ± 15% -1.2 0.56 ± 41% perf-profile.children.cycles-pp.__softirqentry_text_start 1.55 ± 10% -1.2 0.38 ± 14% perf-profile.children.cycles-pp.__alloc_pages_nodemask 1.37 ± 11% -1.1 0.30 ± 24% perf-profile.children.cycles-pp._find_next_bit 1.34 ± 8% -1.0 0.32 ± 15% perf-profile.children.cycles-pp.get_page_from_freelist 1.10 ± 10% -0.8 0.26 ± 8% perf-profile.children.cycles-pp.__add_to_page_cache_locked 0.96 ± 28% -0.8 0.14 ± 16% perf-profile.children.cycles-pp.irqtime_account_irq 1.11 ± 13% -0.8 0.35 ± 34% perf-profile.children.cycles-pp.rebalance_domains 0.88 ± 11% -0.7 0.18 ± 11% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.74 ± 17% -0.6 0.10 ± 12% perf-profile.children.cycles-pp.btrfs_set_range_writeback 0.85 ± 25% -0.6 0.24 ± 6% perf-profile.children.cycles-pp.native_irq_return_iret 0.69 ± 17% -0.6 0.10 ± 6% perf-profile.children.cycles-pp.writepage_delalloc 0.69 ± 7% -0.5 0.16 ± 14% perf-profile.children.cycles-pp.rmqueue 0.96 ± 16% -0.5 0.44 ± 35% perf-profile.children.cycles-pp.memcpy_toio 0.96 ± 16% -0.5 0.44 ± 34% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 0.62 ± 8% -0.5 0.14 ± 24% perf-profile.children.cycles-pp.end_bio_extent_writepage 0.62 ± 8% -0.5 0.14 ± 21% perf-profile.children.cycles-pp.btrfs_end_bio 0.54 ± 17% -0.5 0.07 ± 14% perf-profile.children.cycles-pp.__test_set_page_writeback 0.59 ± 9% -0.5 0.14 ± 12% perf-profile.children.cycles-pp.lru_cache_add 0.77 ± 19% -0.5 0.32 ± 10% perf-profile.children.cycles-pp.lapic_next_deadline 0.54 ± 17% -0.5 0.09 ± 26% perf-profile.children.cycles-pp.crc_128 0.80 ± 33% -0.4 0.36 ± 49% perf-profile.children.cycles-pp.write 0.79 ± 34% -0.4 0.35 ± 50% perf-profile.children.cycles-pp.devkmsg_write.cold 0.79 ± 34% -0.4 0.35 ± 50% perf-profile.children.cycles-pp.devkmsg_emit 0.49 ± 17% -0.4 0.07 ± 8% perf-profile.children.cycles-pp.__process_pages_contig 0.54 ± 34% -0.4 0.12 ± 10% perf-profile.children.cycles-pp._raw_spin_trylock 0.50 ± 13% -0.4 0.08 ± 17% perf-profile.children.cycles-pp.clear_extent_bit 0.49 ± 12% -0.4 0.08 ± 17% perf-profile.children.cycles-pp.__clear_extent_bit 0.53 ± 8% -0.4 0.12 ± 12% perf-profile.children.cycles-pp.__pagevec_lru_add 0.45 ± 22% -0.4 0.05 ± 48% perf-profile.children.cycles-pp.submit_extent_page 0.45 ± 14% -0.4 0.06 ± 57% perf-profile.children.cycles-pp.crc_42 0.44 ± 15% -0.4 0.06 ± 6% perf-profile.children.cycles-pp.find_lock_delalloc_range 0.45 ± 5% -0.4 0.09 ± 23% perf-profile.children.cycles-pp.rmqueue_bulk 0.43 ± 9% -0.3 0.10 ± 23% perf-profile.children.cycles-pp.end_page_writeback 0.38 ± 10% -0.3 0.06 ± 48% perf-profile.children.cycles-pp.alloc_extent_state 0.39 ± 10% -0.3 0.09 ± 22% perf-profile.children.cycles-pp.test_clear_page_writeback 0.41 ± 16% -0.3 0.11 ± 25% perf-profile.children.cycles-pp.prep_new_page 0.37 ± 24% -0.3 0.09 ± 63% perf-profile.children.cycles-pp.update_rq_clock 0.35 ± 9% -0.3 0.07 ± 16% perf-profile.children.cycles-pp.kmem_cache_alloc 0.33 ± 15% -0.3 0.06 ± 17% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.49 ± 18% -0.3 0.21 ± 45% perf-profile.children.cycles-pp.load_balance 0.33 ± 10% -0.3 0.06 ± 11% perf-profile.children.cycles-pp.xas_load 0.34 ± 14% -0.3 0.08 ± 17% perf-profile.children.cycles-pp.mem_cgroup_charge 0.28 ± 15% -0.2 0.04 ± 75% perf-profile.children.cycles-pp.find_next_bit 0.29 ± 11% -0.2 0.05 ± 7% perf-profile.children.cycles-pp.__mod_lruvec_state 0.29 ± 19% -0.2 0.06 ± 16% perf-profile.children.cycles-pp.account_page_dirtied 0.33 ± 18% -0.2 0.09 ± 27% perf-profile.children.cycles-pp.clear_page_erms 0.61 ± 20% -0.2 0.39 ± 8% perf-profile.children.cycles-pp.sched_clock_cpu 0.34 ± 29% -0.2 0.13 ± 43% perf-profile.children.cycles-pp.hrtimer_update_next_event 0.37 ± 13% -0.2 0.16 ± 44% perf-profile.children.cycles-pp.find_busiest_group 0.27 ± 7% -0.2 0.07 ± 16% perf-profile.children.cycles-pp.set_extent_bit 0.27 ± 7% -0.2 0.07 ± 17% perf-profile.children.cycles-pp.btrfs_set_extent_delalloc 0.31 ± 34% -0.2 0.11 ± 48% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.34 ± 14% -0.2 0.15 ± 46% perf-profile.children.cycles-pp.update_sd_lb_stats 0.52 ± 15% -0.2 0.33 ± 16% perf-profile.children.cycles-pp.sched_clock 0.29 ± 28% -0.2 0.11 ± 64% perf-profile.children.cycles-pp.rcu_idle_exit 0.29 ± 41% -0.2 0.10 ± 66% perf-profile.children.cycles-pp.update_blocked_averages 0.49 ± 16% -0.2 0.31 ± 16% perf-profile.children.cycles-pp.native_sched_clock 0.27 ± 18% -0.2 0.10 ± 18% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.20 ± 12% -0.2 0.04 ± 71% perf-profile.children.cycles-pp.iov_iter_fault_in_readable 0.30 ± 36% -0.2 0.14 ± 35% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.25 ± 30% -0.2 0.09 ± 35% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.21 ± 27% -0.1 0.07 ± 26% perf-profile.children.cycles-pp.do_sys_open 0.21 ± 27% -0.1 0.07 ± 26% perf-profile.children.cycles-pp.do_sys_openat2 0.21 ± 29% -0.1 0.07 ± 23% perf-profile.children.cycles-pp.do_filp_open 0.21 ± 29% -0.1 0.07 ± 23% perf-profile.children.cycles-pp.path_openat 0.24 ± 12% -0.1 0.12 ± 33% perf-profile.children.cycles-pp.arch_scale_freq_tick 0.20 ± 27% -0.1 0.08 ± 60% perf-profile.children.cycles-pp.rcu_eqs_exit 0.14 ± 35% -0.1 0.03 ±102% perf-profile.children.cycles-pp.idle_cpu 0.17 ± 27% -0.1 0.06 ± 59% perf-profile.children.cycles-pp.vt_console_print 0.17 ± 29% -0.1 0.06 ± 56% perf-profile.children.cycles-pp.fbcon_redraw 0.17 ± 29% -0.1 0.06 ± 59% perf-profile.children.cycles-pp.lf 0.17 ± 29% -0.1 0.06 ± 59% perf-profile.children.cycles-pp.con_scroll 0.17 ± 29% -0.1 0.06 ± 59% perf-profile.children.cycles-pp.fbcon_scroll 0.16 ± 28% -0.1 0.06 ± 59% perf-profile.children.cycles-pp.fbcon_putcs 0.16 ± 27% -0.1 0.06 ± 59% perf-profile.children.cycles-pp.bit_putcs 0.13 ± 16% -0.1 0.03 ±102% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.15 ± 27% -0.1 0.05 ± 59% perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit 0.12 ± 39% -0.1 0.03 ± 99% perf-profile.children.cycles-pp.btrfs_create 0.15 ± 26% -0.1 0.05 ± 59% perf-profile.children.cycles-pp.sys_imageblit 0.11 ± 18% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__schedule 0.16 ± 10% -0.1 0.08 ± 36% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.09 ± 19% -0.1 0.03 ±108% perf-profile.children.cycles-pp.cpumask_next_and 0.02 ±142% +0.1 0.08 ± 57% perf-profile.children.cycles-pp.seq_read_iter 0.21 ± 7% +0.3 0.54 ± 34% perf-profile.children.cycles-pp.start_kernel 0.00 +0.6 0.63 ± 6% perf-profile.children.cycles-pp.sched_clock_local 0.31 ± 17% +2.3 2.60 ± 6% perf-profile.children.cycles-pp.__x64_sys_gettimeofday 0.12 ± 20% +2.5 2.58 ± 6% perf-profile.children.cycles-pp.ktime_get_real_ts64 0.85 ± 33% +4.1 4.98 ± 9% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 2.25 ± 17% +4.6 6.87 ± 6% perf-profile.children.cycles-pp.tick_sched_handle 2.14 ± 14% +4.7 6.82 ± 6% perf-profile.children.cycles-pp.update_process_times 1.05 ± 15% +5.4 6.50 ± 3% perf-profile.children.cycles-pp.scheduler_tick 0.60 ± 24% +7.2 7.81 ± 14% perf-profile.children.cycles-pp.tick_nohz_irq_exit 2.02 ± 27% +8.0 9.99 ± 11% perf-profile.children.cycles-pp.clockevents_program_event 0.68 ± 31% +9.1 9.75 ± 9% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 2.69 ± 14% +10.0 12.69 ± 5% perf-profile.children.cycles-pp.tick_sched_timer 4.28 ± 18% +13.8 18.08 ± 6% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.00 +16.9 16.89 ± 13% perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event 1.37 ± 23% +17.4 18.78 ± 14% perf-profile.children.cycles-pp.irq_enter_rcu 1.35 ± 23% +17.4 18.78 ± 14% perf-profile.children.cycles-pp.tick_irq_enter 0.00 +22.7 22.69 ± 10% perf-profile.children.cycles-pp.sched_clock_tick 60.63 ± 2% +26.7 87.34 perf-profile.children.cycles-pp.start_secondary 60.85 ± 2% +27.0 87.88 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 60.85 ± 2% +27.0 87.88 perf-profile.children.cycles-pp.cpu_startup_entry 60.85 ± 2% +27.0 87.88 perf-profile.children.cycles-pp.do_idle 7.82 ± 11% +30.4 38.22 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 7.67 ± 10% +30.5 38.13 ± 2% perf-profile.children.cycles-pp.hrtimer_interrupt 55.14 ± 2% +31.4 86.51 perf-profile.children.cycles-pp.cpuidle_enter 55.10 ± 2% +31.4 86.49 perf-profile.children.cycles-pp.cpuidle_enter_state 15.30 ± 10% +51.0 66.31 ± 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 12.68 ± 10% +53.0 65.66 ± 6% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 3.22 ± 28% +56.7 59.87 ± 9% perf-profile.children.cycles-pp.ktime_get 0.00 +70.8 70.85 ± 9% perf-profile.children.cycles-pp.read_hpet 36.19 ± 5% -23.6 12.63 ± 30% perf-profile.self.cycles-pp.intel_idle 6.39 ± 10% -5.2 1.21 ± 11% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 4.33 ± 36% -3.4 0.90 ± 11% perf-profile.self.cycles-pp.cpuidle_enter_state 3.06 ± 11% -2.4 0.61 ± 19% perf-profile.self.cycles-pp.__percpu_counter_sum 2.90 ± 6% -2.3 0.64 ± 16% perf-profile.self.cycles-pp._raw_spin_lock 2.60 ± 27% -2.0 0.55 ± 24% perf-profile.self.cycles-pp.menu_select 2.72 ± 32% -1.7 0.97 ± 11% perf-profile.self.cycles-pp.ktime_get 1.87 ± 7% -1.4 0.42 ± 28% perf-profile.self.cycles-pp.brd_do_bvec 1.36 ± 11% -1.1 0.30 ± 24% perf-profile.self.cycles-pp._find_next_bit 0.85 ± 25% -0.6 0.24 ± 6% perf-profile.self.cycles-pp.native_irq_return_iret 0.74 ± 35% -0.6 0.14 ± 16% perf-profile.self.cycles-pp.irqtime_account_irq 0.73 ± 14% -0.6 0.15 ± 23% perf-profile.self.cycles-pp.cpumask_next 0.96 ± 16% -0.5 0.41 ± 35% perf-profile.self.cycles-pp.memcpy_toio 0.54 ± 17% -0.5 0.08 ± 32% perf-profile.self.cycles-pp.crc_128 0.77 ± 19% -0.5 0.32 ± 11% perf-profile.self.cycles-pp.lapic_next_deadline 0.57 ± 36% -0.4 0.14 ± 38% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.65 ± 10% -0.4 0.22 ± 10% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.53 ± 35% -0.4 0.12 ± 10% perf-profile.self.cycles-pp._raw_spin_trylock 0.44 ± 14% -0.4 0.06 ± 59% perf-profile.self.cycles-pp.crc_42 0.36 ± 15% -0.3 0.05 ± 93% perf-profile.self.cycles-pp.tick_nohz_next_event 0.38 ± 5% -0.3 0.08 ± 24% perf-profile.self.cycles-pp.rmqueue_bulk 0.28 ± 15% -0.2 0.04 ± 75% perf-profile.self.cycles-pp.find_next_bit 0.32 ± 18% -0.2 0.09 ± 25% perf-profile.self.cycles-pp.clear_page_erms 0.26 ± 9% -0.2 0.03 ± 70% perf-profile.self.cycles-pp.xas_load 0.25 ± 12% -0.2 0.04 ± 71% perf-profile.self.cycles-pp.__pagevec_lru_add 0.31 ± 22% -0.2 0.12 ± 46% perf-profile.self.cycles-pp.do_idle 0.47 ± 14% -0.2 0.30 ± 16% perf-profile.self.cycles-pp.native_sched_clock 0.30 ± 36% -0.2 0.14 ± 35% perf-profile.self.cycles-pp.__intel_pmu_enable_all 0.26 ± 34% -0.2 0.10 ± 49% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.18 ± 19% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt 0.24 ± 12% -0.1 0.12 ± 33% perf-profile.self.cycles-pp.arch_scale_freq_tick 0.22 ± 10% -0.1 0.10 ± 45% perf-profile.self.cycles-pp.update_sd_lb_stats 0.13 ± 36% -0.1 0.03 ±102% perf-profile.self.cycles-pp.idle_cpu 0.15 ± 26% -0.1 0.05 ± 59% perf-profile.self.cycles-pp.sys_imageblit 0.18 ± 27% -0.1 0.09 ± 50% perf-profile.self.cycles-pp.hrtimer_interrupt 0.16 ± 10% -0.1 0.08 ± 38% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.00 +0.4 0.40 ± 2% perf-profile.self.cycles-pp.sched_clock_local 0.00 +70.8 70.79 ± 9% perf-profile.self.cycles-pp.read_hpet 18.50 ± 33% +1172.1% 235.33 ±116% interrupts.34:PCI-MSI.524292-edge.eth0-TxRx-3 252658 ± 2% -19.6% 203234 ± 3% interrupts.CAL:Function_call_interrupts 953.00 ± 38% +165.4% 2529 ± 48% interrupts.CPU0.CAL:Function_call_interrupts 110.33 ± 17% +559.5% 727.67 ± 45% interrupts.CPU0.NMI:Non-maskable_interrupts 110.33 ± 17% +559.5% 727.67 ± 45% interrupts.CPU0.PMI:Performance_monitoring_interrupts 320.17 ±149% +406.0% 1620 ± 56% interrupts.CPU1.NMI:Non-maskable_interrupts 320.17 ±149% +406.0% 1620 ± 56% interrupts.CPU1.PMI:Performance_monitoring_interrupts 109.67 ± 35% +391.5% 539.00 ± 21% interrupts.CPU10.NMI:Non-maskable_interrupts 109.67 ± 35% +391.5% 539.00 ± 21% interrupts.CPU10.PMI:Performance_monitoring_interrupts 93.83 ± 27% +1274.6% 1289 ±155% interrupts.CPU101.NMI:Non-maskable_interrupts 93.83 ± 27% +1274.6% 1289 ±155% interrupts.CPU101.PMI:Performance_monitoring_interrupts 93.67 ± 27% +445.9% 511.33 ± 55% interrupts.CPU103.NMI:Non-maskable_interrupts 93.67 ± 27% +445.9% 511.33 ± 55% interrupts.CPU103.PMI:Performance_monitoring_interrupts 94.67 ± 27% +444.0% 515.00 ± 54% interrupts.CPU104.NMI:Non-maskable_interrupts 94.67 ± 27% +444.0% 515.00 ± 54% interrupts.CPU104.PMI:Performance_monitoring_interrupts 95.33 ± 26% +482.3% 555.17 ± 66% interrupts.CPU105.NMI:Non-maskable_interrupts 95.33 ± 26% +482.3% 555.17 ± 66% interrupts.CPU105.PMI:Performance_monitoring_interrupts 92.17 ± 27% +477.0% 531.83 ± 60% interrupts.CPU106.NMI:Non-maskable_interrupts 92.17 ± 27% +477.0% 531.83 ± 60% interrupts.CPU106.PMI:Performance_monitoring_interrupts 91.50 ± 30% +463.8% 515.83 ± 54% interrupts.CPU107.NMI:Non-maskable_interrupts 91.50 ± 30% +463.8% 515.83 ± 54% interrupts.CPU107.PMI:Performance_monitoring_interrupts 103.00 ± 28% +411.5% 526.83 ± 21% interrupts.CPU11.NMI:Non-maskable_interrupts 103.00 ± 28% +411.5% 526.83 ± 21% interrupts.CPU11.PMI:Performance_monitoring_interrupts 157.67 ± 61% +257.0% 562.83 ± 47% interrupts.CPU110.NMI:Non-maskable_interrupts 157.67 ± 61% +257.0% 562.83 ± 47% interrupts.CPU110.PMI:Performance_monitoring_interrupts 116.50 ± 21% +437.9% 626.67 ± 48% interrupts.CPU111.NMI:Non-maskable_interrupts 116.50 ± 21% +437.9% 626.67 ± 48% interrupts.CPU111.PMI:Performance_monitoring_interrupts 99.83 ± 39% +462.3% 561.33 ± 48% interrupts.CPU112.NMI:Non-maskable_interrupts 99.83 ± 39% +462.3% 561.33 ± 48% interrupts.CPU112.PMI:Performance_monitoring_interrupts 102.50 ± 40% +565.4% 682.00 ± 56% interrupts.CPU113.NMI:Non-maskable_interrupts 102.50 ± 40% +565.4% 682.00 ± 56% interrupts.CPU113.PMI:Performance_monitoring_interrupts 132.17 ± 66% +373.6% 626.00 ± 46% interrupts.CPU114.NMI:Non-maskable_interrupts 132.17 ± 66% +373.6% 626.00 ± 46% interrupts.CPU114.PMI:Performance_monitoring_interrupts 100.33 ± 38% +486.5% 588.50 ± 47% interrupts.CPU116.NMI:Non-maskable_interrupts 100.33 ± 38% +486.5% 588.50 ± 47% interrupts.CPU116.PMI:Performance_monitoring_interrupts 101.67 ± 37% +469.3% 578.83 ± 48% interrupts.CPU117.NMI:Non-maskable_interrupts 101.67 ± 37% +469.3% 578.83 ± 48% interrupts.CPU117.PMI:Performance_monitoring_interrupts 106.33 ± 38% +441.1% 575.33 ± 48% interrupts.CPU118.NMI:Non-maskable_interrupts 106.33 ± 38% +441.1% 575.33 ± 48% interrupts.CPU118.PMI:Performance_monitoring_interrupts 106.67 ± 43% +440.6% 576.67 ± 48% interrupts.CPU119.NMI:Non-maskable_interrupts 106.67 ± 43% +440.6% 576.67 ± 48% interrupts.CPU119.PMI:Performance_monitoring_interrupts 146.33 ± 86% +261.6% 529.17 ± 52% interrupts.CPU121.NMI:Non-maskable_interrupts 146.33 ± 86% +261.6% 529.17 ± 52% interrupts.CPU121.PMI:Performance_monitoring_interrupts 101.17 ± 32% +421.9% 528.00 ± 52% interrupts.CPU122.NMI:Non-maskable_interrupts 101.17 ± 32% +421.9% 528.00 ± 52% interrupts.CPU122.PMI:Performance_monitoring_interrupts 140.67 ± 51% +274.8% 527.17 ± 49% interrupts.CPU124.NMI:Non-maskable_interrupts 140.67 ± 51% +274.8% 527.17 ± 49% interrupts.CPU124.PMI:Performance_monitoring_interrupts 115.00 ± 32% +431.4% 611.17 ± 54% interrupts.CPU129.NMI:Non-maskable_interrupts 115.00 ± 32% +431.4% 611.17 ± 54% interrupts.CPU129.PMI:Performance_monitoring_interrupts 18.50 ± 33% +1172.1% 235.33 ±116% interrupts.CPU13.34:PCI-MSI.524292-edge.eth0-TxRx-3 113.67 ± 33% +438.3% 611.83 ± 54% interrupts.CPU130.NMI:Non-maskable_interrupts 113.67 ± 33% +438.3% 611.83 ± 54% interrupts.CPU130.PMI:Performance_monitoring_interrupts 118.17 ± 33% +1087.6% 1403 ±142% interrupts.CPU131.NMI:Non-maskable_interrupts 118.17 ± 33% +1087.6% 1403 ±142% interrupts.CPU131.PMI:Performance_monitoring_interrupts 115.17 ± 32% +446.3% 629.17 ± 57% interrupts.CPU133.NMI:Non-maskable_interrupts 115.17 ± 32% +446.3% 629.17 ± 57% interrupts.CPU133.PMI:Performance_monitoring_interrupts 213.33 ± 68% +184.6% 607.17 ± 55% interrupts.CPU134.NMI:Non-maskable_interrupts 213.33 ± 68% +184.6% 607.17 ± 55% interrupts.CPU134.PMI:Performance_monitoring_interrupts 114.00 ± 33% +438.6% 614.00 ± 55% interrupts.CPU135.NMI:Non-maskable_interrupts 114.00 ± 33% +438.6% 614.00 ± 55% interrupts.CPU135.PMI:Performance_monitoring_interrupts 115.83 ± 33% +349.2% 520.33 ± 51% interrupts.CPU136.NMI:Non-maskable_interrupts 115.83 ± 33% +349.2% 520.33 ± 51% interrupts.CPU136.PMI:Performance_monitoring_interrupts 136.50 ± 43% +280.3% 519.17 ± 51% interrupts.CPU137.NMI:Non-maskable_interrupts 136.50 ± 43% +280.3% 519.17 ± 51% interrupts.CPU137.PMI:Performance_monitoring_interrupts 152.83 ± 78% +208.6% 471.67 ± 27% interrupts.CPU14.NMI:Non-maskable_interrupts 152.83 ± 78% +208.6% 471.67 ± 27% interrupts.CPU14.PMI:Performance_monitoring_interrupts 123.50 ± 35% +515.4% 760.00 ±102% interrupts.CPU143.NMI:Non-maskable_interrupts 123.50 ± 35% +515.4% 760.00 ±102% interrupts.CPU143.PMI:Performance_monitoring_interrupts 137.00 ± 57% +190.0% 397.33 ± 26% interrupts.CPU145.NMI:Non-maskable_interrupts 137.00 ± 57% +190.0% 397.33 ± 26% interrupts.CPU145.PMI:Performance_monitoring_interrupts 124.33 ± 31% +678.0% 967.33 ±130% interrupts.CPU146.NMI:Non-maskable_interrupts 124.33 ± 31% +678.0% 967.33 ±130% interrupts.CPU146.PMI:Performance_monitoring_interrupts 108.00 ± 31% +273.5% 403.33 ± 25% interrupts.CPU147.NMI:Non-maskable_interrupts 108.00 ± 31% +273.5% 403.33 ± 25% interrupts.CPU147.PMI:Performance_monitoring_interrupts 110.33 ± 30% +327.2% 471.33 ± 36% interrupts.CPU148.NMI:Non-maskable_interrupts 110.33 ± 30% +327.2% 471.33 ± 36% interrupts.CPU148.PMI:Performance_monitoring_interrupts 108.83 ± 31% +296.6% 431.67 ± 27% interrupts.CPU149.NMI:Non-maskable_interrupts 108.83 ± 31% +296.6% 431.67 ± 27% interrupts.CPU149.PMI:Performance_monitoring_interrupts 111.50 ± 28% +383.3% 538.83 ± 42% interrupts.CPU15.NMI:Non-maskable_interrupts 111.50 ± 28% +383.3% 538.83 ± 42% interrupts.CPU15.PMI:Performance_monitoring_interrupts 110.00 ± 31% +302.6% 442.83 ± 11% interrupts.CPU150.NMI:Non-maskable_interrupts 110.00 ± 31% +302.6% 442.83 ± 11% interrupts.CPU150.PMI:Performance_monitoring_interrupts 112.67 ± 30% +282.0% 430.33 ± 14% interrupts.CPU151.NMI:Non-maskable_interrupts 112.67 ± 30% +282.0% 430.33 ± 14% interrupts.CPU151.PMI:Performance_monitoring_interrupts 118.33 ± 26% +264.6% 431.50 ± 14% interrupts.CPU152.NMI:Non-maskable_interrupts 118.33 ± 26% +264.6% 431.50 ± 14% interrupts.CPU152.PMI:Performance_monitoring_interrupts 109.67 ± 31% +290.3% 428.00 ± 15% interrupts.CPU153.NMI:Non-maskable_interrupts 109.67 ± 31% +290.3% 428.00 ± 15% interrupts.CPU153.PMI:Performance_monitoring_interrupts 110.33 ± 30% +289.3% 429.50 ± 14% interrupts.CPU154.NMI:Non-maskable_interrupts 110.33 ± 30% +289.3% 429.50 ± 14% interrupts.CPU154.PMI:Performance_monitoring_interrupts 104.33 ± 43% +312.8% 430.67 ± 14% interrupts.CPU155.NMI:Non-maskable_interrupts 104.33 ± 43% +312.8% 430.67 ± 14% interrupts.CPU155.PMI:Performance_monitoring_interrupts 105.33 ± 40% +418.8% 546.50 ± 58% interrupts.CPU156.NMI:Non-maskable_interrupts 105.33 ± 40% +418.8% 546.50 ± 58% interrupts.CPU156.PMI:Performance_monitoring_interrupts 108.83 ± 38% +348.2% 487.83 ± 37% interrupts.CPU157.NMI:Non-maskable_interrupts 108.83 ± 38% +348.2% 487.83 ± 37% interrupts.CPU157.PMI:Performance_monitoring_interrupts 148.50 ± 60% +189.5% 429.83 ± 14% interrupts.CPU158.NMI:Non-maskable_interrupts 148.50 ± 60% +189.5% 429.83 ± 14% interrupts.CPU158.PMI:Performance_monitoring_interrupts 116.17 ± 34% +271.2% 431.17 ± 14% interrupts.CPU159.NMI:Non-maskable_interrupts 116.17 ± 34% +271.2% 431.17 ± 14% interrupts.CPU159.PMI:Performance_monitoring_interrupts 104.00 ± 20% +354.3% 472.50 ± 27% interrupts.CPU16.NMI:Non-maskable_interrupts 104.00 ± 20% +354.3% 472.50 ± 27% interrupts.CPU16.PMI:Performance_monitoring_interrupts 115.17 ± 35% +275.3% 432.17 ± 13% interrupts.CPU160.NMI:Non-maskable_interrupts 115.17 ± 35% +275.3% 432.17 ± 13% interrupts.CPU160.PMI:Performance_monitoring_interrupts 121.67 ± 42% +235.1% 407.67 ± 25% interrupts.CPU161.NMI:Non-maskable_interrupts 121.67 ± 42% +235.1% 407.67 ± 25% interrupts.CPU161.PMI:Performance_monitoring_interrupts 113.50 ± 35% +260.4% 409.00 ± 32% interrupts.CPU162.NMI:Non-maskable_interrupts 113.50 ± 35% +260.4% 409.00 ± 32% interrupts.CPU162.PMI:Performance_monitoring_interrupts 115.67 ± 34% +239.2% 392.33 ± 25% interrupts.CPU163.NMI:Non-maskable_interrupts 115.67 ± 34% +239.2% 392.33 ± 25% interrupts.CPU163.PMI:Performance_monitoring_interrupts 94.50 ± 49% +361.2% 435.83 ± 41% interrupts.CPU164.NMI:Non-maskable_interrupts 94.50 ± 49% +361.2% 435.83 ± 41% interrupts.CPU164.PMI:Performance_monitoring_interrupts 104.50 ± 41% +295.1% 412.83 ± 24% interrupts.CPU165.NMI:Non-maskable_interrupts 104.50 ± 41% +295.1% 412.83 ± 24% interrupts.CPU165.PMI:Performance_monitoring_interrupts 105.67 ± 42% +275.6% 396.83 ± 25% interrupts.CPU166.NMI:Non-maskable_interrupts 105.67 ± 42% +275.6% 396.83 ± 25% interrupts.CPU166.PMI:Performance_monitoring_interrupts 110.33 ± 32% +257.4% 394.33 ± 26% interrupts.CPU167.NMI:Non-maskable_interrupts 110.33 ± 32% +257.4% 394.33 ± 26% interrupts.CPU167.PMI:Performance_monitoring_interrupts 121.67 ± 28% +376.0% 579.17 ± 47% interrupts.CPU169.NMI:Non-maskable_interrupts 121.67 ± 28% +376.0% 579.17 ± 47% interrupts.CPU169.PMI:Performance_monitoring_interrupts 97.00 ± 24% +414.1% 498.67 ± 31% interrupts.CPU17.NMI:Non-maskable_interrupts 97.00 ± 24% +414.1% 498.67 ± 31% interrupts.CPU17.PMI:Performance_monitoring_interrupts 132.33 ± 29% +1128.8% 1626 ±135% interrupts.CPU170.NMI:Non-maskable_interrupts 132.33 ± 29% +1128.8% 1626 ±135% interrupts.CPU170.PMI:Performance_monitoring_interrupts 127.00 ± 28% +357.5% 581.00 ± 47% interrupts.CPU171.NMI:Non-maskable_interrupts 127.00 ± 28% +357.5% 581.00 ± 47% interrupts.CPU171.PMI:Performance_monitoring_interrupts 126.83 ± 26% +390.3% 621.83 ± 46% interrupts.CPU172.NMI:Non-maskable_interrupts 126.83 ± 26% +390.3% 621.83 ± 46% interrupts.CPU172.PMI:Performance_monitoring_interrupts 147.33 ± 45% +376.5% 702.00 ± 56% interrupts.CPU174.NMI:Non-maskable_interrupts 147.33 ± 45% +376.5% 702.00 ± 56% interrupts.CPU174.PMI:Performance_monitoring_interrupts 129.67 ± 15% +421.3% 676.00 ± 50% interrupts.CPU176.NMI:Non-maskable_interrupts 129.67 ± 15% +421.3% 676.00 ± 50% interrupts.CPU176.PMI:Performance_monitoring_interrupts 147.33 ± 23% +362.7% 681.67 ± 52% interrupts.CPU177.NMI:Non-maskable_interrupts 147.33 ± 23% +362.7% 681.67 ± 52% interrupts.CPU177.PMI:Performance_monitoring_interrupts 132.83 ± 75% +368.0% 621.67 ± 44% interrupts.CPU18.NMI:Non-maskable_interrupts 132.83 ± 75% +368.0% 621.67 ± 44% interrupts.CPU18.PMI:Performance_monitoring_interrupts 127.83 ± 31% +430.0% 677.50 ± 51% interrupts.CPU180.NMI:Non-maskable_interrupts 127.83 ± 31% +430.0% 677.50 ± 51% interrupts.CPU180.PMI:Performance_monitoring_interrupts 127.50 ± 31% +605.1% 899.00 ± 85% interrupts.CPU181.NMI:Non-maskable_interrupts 127.50 ± 31% +605.1% 899.00 ± 85% interrupts.CPU181.PMI:Performance_monitoring_interrupts 180.00 ± 49% +715.6% 1468 ±131% interrupts.CPU182.NMI:Non-maskable_interrupts 180.00 ± 49% +715.6% 1468 ±131% interrupts.CPU182.PMI:Performance_monitoring_interrupts 123.17 ± 28% +451.6% 679.33 ± 62% interrupts.CPU184.NMI:Non-maskable_interrupts 123.17 ± 28% +451.6% 679.33 ± 62% interrupts.CPU184.PMI:Performance_monitoring_interrupts 145.17 ± 16% +306.0% 589.33 ± 57% interrupts.CPU185.NMI:Non-maskable_interrupts 145.17 ± 16% +306.0% 589.33 ± 57% interrupts.CPU185.PMI:Performance_monitoring_interrupts 127.17 ± 30% +355.3% 579.00 ± 56% interrupts.CPU187.NMI:Non-maskable_interrupts 127.17 ± 30% +355.3% 579.00 ± 56% interrupts.CPU187.PMI:Performance_monitoring_interrupts 124.33 ± 27% +383.4% 601.00 ± 57% interrupts.CPU188.NMI:Non-maskable_interrupts 124.33 ± 27% +383.4% 601.00 ± 57% interrupts.CPU188.PMI:Performance_monitoring_interrupts 106.83 ± 27% +493.1% 633.67 ± 38% interrupts.CPU20.NMI:Non-maskable_interrupts 106.83 ± 27% +493.1% 633.67 ± 38% interrupts.CPU20.PMI:Performance_monitoring_interrupts 106.50 ± 27% +478.2% 615.83 ± 40% interrupts.CPU21.NMI:Non-maskable_interrupts 106.50 ± 27% +478.2% 615.83 ± 40% interrupts.CPU21.PMI:Performance_monitoring_interrupts 121.33 ± 19% +403.8% 611.33 ± 39% interrupts.CPU22.NMI:Non-maskable_interrupts 121.33 ± 19% +403.8% 611.33 ± 39% interrupts.CPU22.PMI:Performance_monitoring_interrupts 121.33 ± 22% +403.3% 610.67 ± 39% interrupts.CPU23.NMI:Non-maskable_interrupts 121.33 ± 22% +403.3% 610.67 ± 39% interrupts.CPU23.PMI:Performance_monitoring_interrupts 149.83 ± 52% +307.7% 610.83 ± 55% interrupts.CPU25.NMI:Non-maskable_interrupts 149.83 ± 52% +307.7% 610.83 ± 55% interrupts.CPU25.PMI:Performance_monitoring_interrupts 115.17 ± 29% +409.3% 586.50 ± 65% interrupts.CPU26.NMI:Non-maskable_interrupts 115.17 ± 29% +409.3% 586.50 ± 65% interrupts.CPU26.PMI:Performance_monitoring_interrupts 114.00 ± 22% +353.7% 517.17 ± 55% interrupts.CPU3.NMI:Non-maskable_interrupts 114.00 ± 22% +353.7% 517.17 ± 55% interrupts.CPU3.PMI:Performance_monitoring_interrupts 115.17 ± 33% +402.6% 578.83 ± 63% interrupts.CPU33.NMI:Non-maskable_interrupts 115.17 ± 33% +402.6% 578.83 ± 63% interrupts.CPU33.PMI:Performance_monitoring_interrupts 112.50 ± 34% +411.4% 575.33 ± 63% interrupts.CPU34.NMI:Non-maskable_interrupts 112.50 ± 34% +411.4% 575.33 ± 63% interrupts.CPU34.PMI:Performance_monitoring_interrupts 115.83 ± 33% +1500.9% 1854 ±162% interrupts.CPU35.NMI:Non-maskable_interrupts 115.83 ± 33% +1500.9% 1854 ±162% interrupts.CPU35.PMI:Performance_monitoring_interrupts 113.83 ± 34% +458.1% 635.33 ± 59% interrupts.CPU37.NMI:Non-maskable_interrupts 113.83 ± 34% +458.1% 635.33 ± 59% interrupts.CPU37.PMI:Performance_monitoring_interrupts 100.50 ± 33% +384.9% 487.33 ± 60% interrupts.CPU39.NMI:Non-maskable_interrupts 100.50 ± 33% +384.9% 487.33 ± 60% interrupts.CPU39.PMI:Performance_monitoring_interrupts 172.67 ± 95% +306.5% 701.83 ± 83% interrupts.CPU4.NMI:Non-maskable_interrupts 172.67 ± 95% +306.5% 701.83 ± 83% interrupts.CPU4.PMI:Performance_monitoring_interrupts 106.83 ± 21% +353.4% 484.33 ± 60% interrupts.CPU40.NMI:Non-maskable_interrupts 106.83 ± 21% +353.4% 484.33 ± 60% interrupts.CPU40.PMI:Performance_monitoring_interrupts 137.17 ± 51% +253.6% 485.00 ± 59% interrupts.CPU41.NMI:Non-maskable_interrupts 137.17 ± 51% +253.6% 485.00 ± 59% interrupts.CPU41.PMI:Performance_monitoring_interrupts 118.17 ± 33% +479.1% 684.33 ± 88% interrupts.CPU47.NMI:Non-maskable_interrupts 118.17 ± 33% +479.1% 684.33 ± 88% interrupts.CPU47.PMI:Performance_monitoring_interrupts 93.33 ± 25% +1731.4% 1709 ±163% interrupts.CPU5.NMI:Non-maskable_interrupts 93.33 ± 25% +1731.4% 1709 ±163% interrupts.CPU5.PMI:Performance_monitoring_interrupts 130.83 ± 21% +624.3% 947.67 ±139% interrupts.CPU50.NMI:Non-maskable_interrupts 130.83 ± 21% +624.3% 947.67 ±139% interrupts.CPU50.PMI:Performance_monitoring_interrupts 121.17 ± 27% +205.6% 370.33 ± 32% interrupts.CPU51.NMI:Non-maskable_interrupts 121.17 ± 27% +205.6% 370.33 ± 32% interrupts.CPU51.PMI:Performance_monitoring_interrupts 120.67 ± 26% +302.2% 485.33 ± 42% interrupts.CPU52.NMI:Non-maskable_interrupts 120.67 ± 26% +302.2% 485.33 ± 42% interrupts.CPU52.PMI:Performance_monitoring_interrupts 120.83 ± 27% +247.6% 420.00 ± 26% interrupts.CPU53.NMI:Non-maskable_interrupts 120.83 ± 27% +247.6% 420.00 ± 26% interrupts.CPU53.PMI:Performance_monitoring_interrupts 94.67 ± 27% +298.9% 377.67 ± 29% interrupts.CPU54.NMI:Non-maskable_interrupts 94.67 ± 27% +298.9% 377.67 ± 29% interrupts.CPU54.PMI:Performance_monitoring_interrupts 97.00 ± 27% +272.0% 360.83 ± 36% interrupts.CPU55.NMI:Non-maskable_interrupts 97.00 ± 27% +272.0% 360.83 ± 36% interrupts.CPU55.PMI:Performance_monitoring_interrupts 119.00 ± 29% +256.9% 424.67 ± 15% interrupts.CPU56.NMI:Non-maskable_interrupts 119.00 ± 29% +256.9% 424.67 ± 15% interrupts.CPU56.PMI:Performance_monitoring_interrupts 117.50 ± 24% +260.9% 424.00 ± 15% interrupts.CPU57.NMI:Non-maskable_interrupts 117.50 ± 24% +260.9% 424.00 ± 15% interrupts.CPU57.PMI:Performance_monitoring_interrupts 111.17 ± 36% +281.7% 424.33 ± 15% interrupts.CPU58.NMI:Non-maskable_interrupts 111.17 ± 36% +281.7% 424.33 ± 15% interrupts.CPU58.PMI:Performance_monitoring_interrupts 120.50 ± 27% +231.8% 399.83 ± 26% interrupts.CPU59.NMI:Non-maskable_interrupts 120.50 ± 27% +231.8% 399.83 ± 26% interrupts.CPU59.PMI:Performance_monitoring_interrupts 120.00 ± 24% +410.4% 612.50 ± 77% interrupts.CPU60.NMI:Non-maskable_interrupts 120.00 ± 24% +410.4% 612.50 ± 77% interrupts.CPU60.PMI:Performance_monitoring_interrupts 123.33 ± 20% +302.6% 496.50 ± 44% interrupts.CPU61.NMI:Non-maskable_interrupts 123.33 ± 20% +302.6% 496.50 ± 44% interrupts.CPU61.PMI:Performance_monitoring_interrupts 141.67 ± 43% +202.5% 428.50 ± 14% interrupts.CPU62.NMI:Non-maskable_interrupts 141.67 ± 43% +202.5% 428.50 ± 14% interrupts.CPU62.PMI:Performance_monitoring_interrupts 108.00 ± 31% +297.1% 428.83 ± 14% interrupts.CPU63.NMI:Non-maskable_interrupts 108.00 ± 31% +297.1% 428.83 ± 14% interrupts.CPU63.PMI:Performance_monitoring_interrupts 89.83 ± 47% +376.3% 427.83 ± 15% interrupts.CPU64.NMI:Non-maskable_interrupts 89.83 ± 47% +376.3% 427.83 ± 15% interrupts.CPU64.PMI:Performance_monitoring_interrupts 103.50 ± 45% +347.7% 463.33 ± 14% interrupts.CPU65.NMI:Non-maskable_interrupts 103.50 ± 45% +347.7% 463.33 ± 14% interrupts.CPU65.PMI:Performance_monitoring_interrupts 110.17 ± 36% +310.9% 452.67 ± 21% interrupts.CPU66.NMI:Non-maskable_interrupts 110.17 ± 36% +310.9% 452.67 ± 21% interrupts.CPU66.PMI:Performance_monitoring_interrupts 102.17 ± 45% +321.2% 430.33 ± 13% interrupts.CPU67.NMI:Non-maskable_interrupts 102.17 ± 45% +321.2% 430.33 ± 13% interrupts.CPU67.PMI:Performance_monitoring_interrupts 109.33 ± 32% +332.6% 473.00 ± 33% interrupts.CPU68.NMI:Non-maskable_interrupts 109.33 ± 32% +332.6% 473.00 ± 33% interrupts.CPU68.PMI:Performance_monitoring_interrupts 102.50 ± 38% +325.4% 436.00 ± 28% interrupts.CPU69.NMI:Non-maskable_interrupts 102.50 ± 38% +325.4% 436.00 ± 28% interrupts.CPU69.PMI:Performance_monitoring_interrupts 93.17 ± 26% +504.7% 563.33 ± 47% interrupts.CPU7.NMI:Non-maskable_interrupts 93.17 ± 26% +504.7% 563.33 ± 47% interrupts.CPU7.PMI:Performance_monitoring_interrupts 108.67 ± 31% +260.1% 391.33 ± 26% interrupts.CPU70.NMI:Non-maskable_interrupts 108.67 ± 31% +260.1% 391.33 ± 26% interrupts.CPU70.PMI:Performance_monitoring_interrupts 109.50 ± 31% +258.0% 392.00 ± 26% interrupts.CPU71.NMI:Non-maskable_interrupts 109.50 ± 31% +258.0% 392.00 ± 26% interrupts.CPU71.PMI:Performance_monitoring_interrupts 115.33 ± 40% +374.6% 547.33 ± 55% interrupts.CPU73.NMI:Non-maskable_interrupts 115.33 ± 40% +374.6% 547.33 ± 55% interrupts.CPU73.PMI:Performance_monitoring_interrupts 123.67 ± 41% +1485.7% 1961 ±137% interrupts.CPU74.NMI:Non-maskable_interrupts 123.67 ± 41% +1485.7% 1961 ±137% interrupts.CPU74.PMI:Performance_monitoring_interrupts 128.67 ± 30% +402.5% 646.50 ± 60% interrupts.CPU75.NMI:Non-maskable_interrupts 128.67 ± 30% +402.5% 646.50 ± 60% interrupts.CPU75.PMI:Performance_monitoring_interrupts 138.00 ± 17% +426.1% 726.00 ± 71% interrupts.CPU76.NMI:Non-maskable_interrupts 138.00 ± 17% +426.1% 726.00 ± 71% interrupts.CPU76.PMI:Performance_monitoring_interrupts 157.67 ± 71% +346.3% 703.67 ± 61% interrupts.CPU78.NMI:Non-maskable_interrupts 157.67 ± 71% +346.3% 703.67 ± 61% interrupts.CPU78.PMI:Performance_monitoring_interrupts 93.83 ± 27% +559.0% 618.33 ± 38% interrupts.CPU8.NMI:Non-maskable_interrupts 93.83 ± 27% +559.0% 618.33 ± 38% interrupts.CPU8.PMI:Performance_monitoring_interrupts 119.67 ± 27% +463.2% 674.00 ± 50% interrupts.CPU80.NMI:Non-maskable_interrupts 119.67 ± 27% +463.2% 674.00 ± 50% interrupts.CPU80.PMI:Performance_monitoring_interrupts 139.17 ± 42% +392.8% 685.83 ± 52% interrupts.CPU81.NMI:Non-maskable_interrupts 139.17 ± 42% +392.8% 685.83 ± 52% interrupts.CPU81.PMI:Performance_monitoring_interrupts 110.33 ± 29% +482.6% 642.83 ± 58% interrupts.CPU84.NMI:Non-maskable_interrupts 110.33 ± 29% +482.6% 642.83 ± 58% interrupts.CPU84.PMI:Performance_monitoring_interrupts 121.33 ± 28% +540.9% 777.67 ± 80% interrupts.CPU85.NMI:Non-maskable_interrupts 121.33 ± 28% +540.9% 777.67 ± 80% interrupts.CPU85.PMI:Performance_monitoring_interrupts 145.83 ± 40% +1180.5% 1867 ±157% interrupts.CPU86.NMI:Non-maskable_interrupts 145.83 ± 40% +1180.5% 1867 ±157% interrupts.CPU86.PMI:Performance_monitoring_interrupts 121.17 ± 25% +417.9% 627.50 ± 68% interrupts.CPU88.NMI:Non-maskable_interrupts 121.17 ± 25% +417.9% 627.50 ± 68% interrupts.CPU88.PMI:Performance_monitoring_interrupts 167.67 ± 62% +229.6% 552.67 ± 56% interrupts.CPU89.NMI:Non-maskable_interrupts 167.67 ± 62% +229.6% 552.67 ± 56% interrupts.CPU89.PMI:Performance_monitoring_interrupts 109.50 ± 27% +391.8% 538.50 ± 21% interrupts.CPU9.NMI:Non-maskable_interrupts 109.50 ± 27% +391.8% 538.50 ± 21% interrupts.CPU9.PMI:Performance_monitoring_interrupts 107.83 ± 30% +418.7% 559.33 ± 53% interrupts.CPU91.NMI:Non-maskable_interrupts 107.83 ± 30% +418.7% 559.33 ± 53% interrupts.CPU91.PMI:Performance_monitoring_interrupts 97.00 ± 36% +503.8% 585.67 ± 57% interrupts.CPU92.NMI:Non-maskable_interrupts 97.00 ± 36% +503.8% 585.67 ± 57% interrupts.CPU92.PMI:Performance_monitoring_interrupts 152.00 ± 44% +377.4% 725.67 ± 53% interrupts.CPU95.NMI:Non-maskable_interrupts 152.00 ± 44% +377.4% 725.67 ± 53% interrupts.CPU95.PMI:Performance_monitoring_interrupts 99.33 ± 25% +625.5% 720.67 ± 45% interrupts.CPU96.NMI:Non-maskable_interrupts 99.33 ± 25% +625.5% 720.67 ± 45% interrupts.CPU96.PMI:Performance_monitoring_interrupts 251.67 ±126% +395.4% 1246 ± 45% interrupts.CPU97.NMI:Non-maskable_interrupts 251.67 ±126% +395.4% 1246 ± 45% interrupts.CPU97.PMI:Performance_monitoring_interrupts 201.83 ± 79% +293.3% 793.83 ± 90% interrupts.CPU98.NMI:Non-maskable_interrupts 201.83 ± 79% +293.3% 793.83 ± 90% interrupts.CPU98.PMI:Performance_monitoring_interrupts 111.00 ± 23% +362.2% 513.00 ± 55% interrupts.CPU99.NMI:Non-maskable_interrupts 111.00 ± 23% +362.2% 513.00 ± 55% interrupts.CPU99.PMI:Performance_monitoring_interrupts 46152 ± 10% +158.6% 119363 ± 19% interrupts.NMI:Non-maskable_interrupts 46152 ± 10% +158.6% 119363 ± 19% interrupts.PMI:Performance_monitoring_interrupts 398.67 ± 8% +240.2% 1356 ± 17% interrupts.RES:Rescheduling_interrupts Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

6
26
0 / 0

[KVM] 4fc096a99e: kernel-selftests.kvm.set_memory_region_test.fail

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 4fc096a99e01dd06dc55bef76ade7f8d76653245 ("KVM: Raise the maximum number of user memslots") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: kernel-selftests version: kernel-selftests-x86_64-cf9ae1bd-1_20210401 with following parameters: group: kvm ucode: 0x28 test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt on test machine: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 16G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> KERNEL SELFTESTS: linux_headers_dir is /usr/src/linux-headers-x86_64-rhel-8.3-kselftests-4fc096a99e01dd06dc55bef76ade7f8d76653245 2021-04-22 20:27:10 ln -sf /usr/bin/clang 2021-04-22 20:27:10 ln -sf /usr/bin/llc 2021-04-22 20:27:10 sed -i s/default_timeout=45/default_timeout=300/ kselftest/runner.sh LKP WARN miss config CONFIG_KVM_AMD= of kvm/config source /lkp/lkp/src/lib/tests/kernel-selftests-ext.sh 2021-04-22 20:27:10 cp kvm/settings /kselftests/kvm/settings 2021-04-22 20:27:10 /kselftests/run_kselftest.sh -c kvm TAP version 13 1..33 # selftests: kvm: cr4_cpuid_sync_test ok 1 selftests: kvm: cr4_cpuid_sync_test # selftests: kvm: evmcs_test ok 2 selftests: kvm: evmcs_test # selftests: kvm: get_cpuid_test ok 3 selftests: kvm: get_cpuid_test # selftests: kvm: hyperv_cpuid ok 4 selftests: kvm: hyperv_cpuid # selftests: kvm: kvm_pv_test # testing msr: MSR_KVM_SYSTEM_TIME (0x12) # testing msr: MSR_KVM_SYSTEM_TIME_NEW (0x4b564d01) # testing msr: MSR_KVM_WALL_CLOCK (0x11) # testing msr: MSR_KVM_WALL_CLOCK_NEW (0x4b564d00) # testing msr: MSR_KVM_ASYNC_PF_EN (0x4b564d02) # testing msr: MSR_KVM_STEAL_TIME (0x4b564d03) # testing msr: MSR_KVM_PV_EOI_EN (0x4b564d04) # testing msr: MSR_KVM_POLL_CONTROL (0x4b564d05) # testing msr: MSR_KVM_ASYNC_PF_INT (0x4b564d06) # testing msr: MSR_KVM_ASYNC_PF_ACK (0x4b564d07) # testing hcall: KVM_HC_KICK_CPU (5) # testing hcall: KVM_HC_SEND_IPI (10) # testing hcall: KVM_HC_SCHED_YIELD (11) ok 5 selftests: kvm: kvm_pv_test # selftests: kvm: mmio_warning_test # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 # ret1=0 exit_reason=17 suberror=1 # ret1=0 exit_reason=8 suberror=65530 ok 6 selftests: kvm: mmio_warning_test # selftests: kvm: platform_info_test ok 7 selftests: kvm: platform_info_test # selftests: kvm: set_sregs_test ok 8 selftests: kvm: set_sregs_test # selftests: kvm: smm_test ok 9 selftests: kvm: smm_test # selftests: kvm: state_test ok 10 selftests: kvm: state_test # selftests: kvm: vmx_preemption_timer_test # ==== Test Assertion Failure ==== # x86_64/vmx_preemption_timer_test.c:230: uc.args[4] < uc.args[5] # pid=1624 tid=1624 - Interrupted system call # 1 0x000000000040166f: main at vmx_preemption_timer_test.c:230 # 2 0x00007f346d85809a: ?? ??:0 # 3 0x00000000004016b9: _start at ??:? # Stage 2: L2 PT expiry TSC (3202417017) > L2 TSC deadline (3202312608) # Stage 2: L1 PT expiry TSC (3202584596) , L1 TSC deadline (3202189728) # Stage 2: L2 PT expiry TSC (3202417017) , L2 TSC deadline (3202312608) not ok 11 selftests: kvm: vmx_preemption_timer_test # exit=254 # selftests: kvm: svm_vmcall_test # nested SVM not enabled, skipping test ok 12 selftests: kvm: svm_vmcall_test # SKIP # selftests: kvm: sync_regs_test ok 13 selftests: kvm: sync_regs_test # selftests: kvm: userspace_msr_exit_test # To run the instruction emulated tests set the module parameter 'kvm.force_emulation_prefix=1' ok 14 selftests: kvm: userspace_msr_exit_test # selftests: kvm: vmx_apic_access_test ok 15 selftests: kvm: vmx_apic_access_test # selftests: kvm: vmx_close_while_nested_test ok 16 selftests: kvm: vmx_close_while_nested_test # selftests: kvm: vmx_dirty_log_test ok 17 selftests: kvm: vmx_dirty_log_test # selftests: kvm: vmx_set_nested_state_test ok 18 selftests: kvm: vmx_set_nested_state_test # selftests: kvm: vmx_tsc_adjust_test # IA32_TSC_ADJUST is -4294975532 (-1 * TSC_ADJUST_VALUE + -8236). # IA32_TSC_ADJUST is -4294975532 (-1 * TSC_ADJUST_VALUE + -8236). # IA32_TSC_ADJUST is -8589947763 (-2 * TSC_ADJUST_VALUE + -13171). # IA32_TSC_ADJUST is -8589947763 (-2 * TSC_ADJUST_VALUE + -13171). ok 19 selftests: kvm: vmx_tsc_adjust_test # selftests: kvm: xapic_ipi_test # Halter vCPU thread started # vCPU thread running vCPU 0 # Halter vCPU thread reported its APIC ID: 0 after 1 seconds. # IPI sender vCPU thread started. Letting vCPUs run for 3 seconds. # vCPU thread running vCPU 1 # Test successful after running for 3 seconds. # Sending vCPU sent 226153 IPIs to halting vCPU # Halting vCPU halted 226153 times, woke 226152 times, received 226152 IPIs. # Halter APIC ID=0 # Sender ICR value=0xa5 ICR2 value=0 # Halter TPR=0 PPR=0 LVR=0x50014 # Migrations attempted: 0 # Migrations completed: 0 ok 20 selftests: kvm: xapic_ipi_test # selftests: kvm: xss_msr_test # IA32_XSS is not supported by the vCPU, skipping test ok 21 selftests: kvm: xss_msr_test # SKIP # selftests: kvm: debug_regs ok 22 selftests: kvm: debug_regs # selftests: kvm: tsc_msrs_test ok 23 selftests: kvm: tsc_msrs_test # selftests: kvm: vmx_pmu_msrs_test ok 24 selftests: kvm: vmx_pmu_msrs_test # selftests: kvm: xen_shinfo_test ok 25 selftests: kvm: xen_shinfo_test # selftests: kvm: xen_vmcall_test ok 26 selftests: kvm: xen_vmcall_test # selftests: kvm: demand_paging_test # Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages # guest physical test memory offset: 0x7fbffff000 # Finished creating vCPUs and starting uffd threads # Started all vCPUs # All vCPU threads joined # Total guest execution time: 1.974251864s # Overall demand paging rate: 24402.471039 pgs/sec ok 27 selftests: kvm: demand_paging_test # selftests: kvm: dirty_log_test # Test iterations: 32, interval: 10 (ms) # Testing Log Mode 'dirty-log' # Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages # guest physical test memory offset: 0x7fbfffc000 # Dirtied 1024 pages # Total bits checked: dirty (68574), clear (8057983), track_next (4131) # Testing Log Mode 'clear-log' # Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages # guest physical test memory offset: 0x7fbfffc000 # Dirtied 1024 pages # Total bits checked: dirty (334670), clear (7791887), track_next (4882) # Testing Log Mode 'dirty-ring' # Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages # guest physical test memory offset: 0x7fbfffc000 # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 1 collected 1431 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 2 collected 34922 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 3 collected 1223 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 4 collected 1161 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 5 collected 1165 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 6 collected 1161 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 7 collected 1166 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 8 collected 1161 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 9 collected 1168 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 10 collected 1162 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 11 collected 1165 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 12 collected 1175 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 13 collected 1176 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 14 collected 1173 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 15 collected 1168 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 16 collected 1179 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 17 collected 1186 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 18 collected 1182 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 19 collected 1188 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 20 collected 1190 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 21 collected 1195 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 22 collected 1180 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 23 collected 1161 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 24 collected 1143 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 25 collected 1150 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 26 collected 1160 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 27 collected 1156 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 28 collected 1160 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 29 collected 1166 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 30 collected 678 pages # vcpu stops because vcpu is kicked out... # Notifying vcpu to continue # vcpu continues now. # Iteration 31 collected 849 pages # vcpu stops because dirty ring is full... # vcpu continues now. # Dirtied 32768 pages # Total bits checked: dirty (69500), clear (8057057), track_next (4738) ok 28 selftests: kvm: dirty_log_test # selftests: kvm: dirty_log_perf_test # Test iterations: 2 # Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages # guest physical test memory offset: 0x7fbffff000 # Populate memory time: 1.757522426s # Enabling dirty logging time: 0.000847277s # # Iteration 1 dirty memory time: 0.007516471s # Iteration 1 get dirty log time: 0.000023223s # Iteration 1 clear dirty log time: 0.005255955s # Iteration 2 dirty memory time: 0.337495562s # Iteration 2 get dirty log time: 0.000004629s # Iteration 2 clear dirty log time: 0.004977110s # Disabling dirty logging time: 0.007335534s # Get dirty log over 2 iterations took 0.000027852s. (Avg 0.000013926s/iteration) # Clear dirty log over 2 iterations took 0.010233065s. (Avg 0.005116532s/iteration) ok 29 selftests: kvm: dirty_log_perf_test # selftests: kvm: kvm_create_max_vcpus # KVM_CAP_MAX_VCPU_ID: 1023 # KVM_CAP_MAX_VCPUS: 288 # Testing creating 288 vCPUs, with IDs 0...287. # Testing creating 288 vCPUs, with IDs 735...1022. ok 30 selftests: kvm: kvm_create_max_vcpus # selftests: kvm: memslot_modification_stress_test # ==== Test Assertion Failure ==== # lib/kvm_util.c:693: (((guest_paddr >> vm->page_shift) + npages) - 1) <= vm->max_gfn # pid=2134 tid=2134 - Success # 1 0x0000000000405291: vm_userspace_mem_region_add at kvm_util.c:693 # 2 0x000000000040281d: add_remove_memslot at memslot_modification_stress_test.c:80 # 3 (inlined by) run_test at memslot_modification_stress_test.c:122 # 4 0x0000000000408033: for_each_guest_mode at guest_modes.c:37 # 5 0x0000000000402494: main at memslot_modification_stress_test.c:209 # 6 0x00007f6dbe3f709a: ?? ??:0 # 7 0x0000000000402539: _start at ??:? # Physical range beyond maximum supported physical address, # guest_paddr: 0x8000001000 npages: 0x1 # vm->max_gfn: 0x7ffffff vm->page_size: 0x1000 # Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages # guest physical test memory offset: 0x7fbffff000 # Finished creating vCPUs # Started all vCPUs not ok 31 selftests: kvm: memslot_modification_stress_test # exit=254 # selftests: kvm: set_memory_region_test # Testing KVM_RUN with zero added memory regions # Allowed number of memory slots: 32764 # Adding slots 0..32763, each memory region with 2048K size # not ok 32 selftests: kvm: set_memory_region_test # TIMEOUT 120 seconds # selftests: kvm: steal_time ok 33 selftests: kvm: steal_time To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

3
3
0 / 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP April 2021