Greeting, FYI, we noticed a -43.8% regression of will-it-scale.per_thread_ops due to commit: commit: 63a17eea7d5516f9e927fdc52b45f1d040fdcf35 ("posix-cpu-timers: Recalc next expiration when timer_settime() ends up not queueing") https://git.kernel.org/cgit/linux/kernel/git/frederic/linux-dynticks.git timers/core in testcase: will-it-scale on test machine: 104 threads 2 sockets Skylake with 192G memory with following parameters: nr_task: 100% mode: thread test: lseek1 cpufreq_governor: performance ucode: 0x2006a0a test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com&gt; Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/thread/100%/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/lseek1/will-it-scale/0x2006a0a commit: 053fe3ff19 ("posix-cpu-timers: Consolidate timer base accessor") 63a17eea7d ("posix-cpu-timers: Recalc next expiration when timer_settime() ends up not queueing") 053fe3ff1960ebdb 63a17eea7d5516f9e927fdc52b4 ---------------- --------------------------- %stddev %change %stddev \ | \ 48848859 -43.8% 27453975 ± 2% will-it-scale.104.threads 469700 -43.8% 263980 ± 2% will-it-scale.per_thread_ops 48848859 -43.8% 27453975 ± 2% will-it-scale.workload 2709 +3.3% 2799 turbostat.Bzy_MHz 208822 ± 28% +158.9% 540658 ± 72% turbostat.C1E 0.20 ± 72% +0.6 0.77 ± 70% turbostat.C1E% 198.22 -127.8 70.45 ± 44% turbostat.PKG_% 385.18 -4.5% 367.72 turbostat.PkgWatt 103.61 +1.4% 105.05 turbostat.RAMWatt 57572 +7.9% 62144 proc-vmstat.nr_active_anon 87313 -2.6% 85003 proc-vmstat.nr_inactive_anon 65155 +3.6% 67472 proc-vmstat.nr_shmem 57572 +7.9% 62144 proc-vmstat.nr_zone_active_anon 87313 -2.6% 85003 proc-vmstat.nr_zone_inactive_anon 13556 ± 38% -55.9% 5973 ± 18% proc-vmstat.numa_hint_faults_local 927915 -1.2% 916444 proc-vmstat.numa_hit 833923 -1.4% 822260 proc-vmstat.numa_local 92055 -67.4% 30029 ± 13% proc-vmstat.pgactivate 937373 -1.3% 925443 proc-vmstat.pgalloc_normal 1000036 -2.4% 975714 proc-vmstat.pgfault 879135 -1.7% 863774 proc-vmstat.pgfree 30594 ± 7% +41.5% 43285 ± 7% softirqs.CPU26.RCU 30411 ± 4% +43.1% 43509 ± 6% softirqs.CPU27.RCU 27396 ± 17% +58.7% 43473 ± 6% softirqs.CPU28.RCU 30052 ± 4% +43.9% 43231 ± 7% softirqs.CPU29.RCU 36180 ± 7% +24.2% 44925 ± 4% softirqs.CPU45.RCU 36447 ± 6% +23.2% 44913 ± 4% softirqs.CPU46.RCU 35505 ± 5% +26.9% 45042 ± 5% softirqs.CPU47.RCU 34935 ± 5% +28.7% 44951 ± 5% softirqs.CPU48.RCU 34821 ± 4% +30.0% 45255 ± 3% softirqs.CPU51.RCU 34430 ± 26% +26.8% 43646 ± 3% softirqs.CPU6.RCU 40937 ± 2% +11.8% 45772 ± 2% softirqs.CPU78.RCU 41112 ± 2% +11.4% 45815 softirqs.CPU79.RCU 38741 ± 15% +18.7% 45972 softirqs.CPU80.RCU 40536 ± 2% +12.3% 45502 ± 2% softirqs.CPU81.RCU 40977 ± 4% +14.2% 46782 ± 3% softirqs.CPU82.RCU 41552 ± 3% +10.3% 45834 softirqs.CPU85.RCU 40318 ± 2% +11.9% 45114 ± 3% softirqs.CPU87.RCU 39977 +14.1% 45607 ± 2% softirqs.CPU88.RCU 40633 +11.6% 45335 ± 4% softirqs.CPU89.RCU 364.50 ± 9% +40.4% 511.86 ± 20% interrupts.CPU29.RES:Rescheduling_interrupts 359.17 ± 4% +82.0% 653.86 ± 56% interrupts.CPU34.RES:Rescheduling_interrupts 396.33 ± 8% -58.2% 165.71 ± 56% interrupts.CPU40.RES:Rescheduling_interrupts 463.50 ± 28% -69.7% 140.43 ± 73% interrupts.CPU42.RES:Rescheduling_interrupts 404.17 ± 14% -65.0% 141.57 ± 62% interrupts.CPU43.RES:Rescheduling_interrupts 390.67 ± 23% -62.0% 148.57 ± 56% interrupts.CPU44.RES:Rescheduling_interrupts 528.33 ± 51% -70.6% 155.14 ± 18% interrupts.CPU46.RES:Rescheduling_interrupts 356.33 ± 11% -59.1% 145.57 ± 27% interrupts.CPU47.RES:Rescheduling_interrupts 329.00 ± 4% -38.1% 203.71 ± 46% interrupts.CPU48.RES:Rescheduling_interrupts 692.17 ± 70% -72.4% 191.00 ± 51% interrupts.CPU49.RES:Rescheduling_interrupts 477.67 ± 6% -54.4% 217.71 ± 32% interrupts.CPU54.RES:Rescheduling_interrupts 4846 ± 35% +36.4% 6610 ± 26% interrupts.CPU55.NMI:Non-maskable_interrupts 4846 ± 35% +36.4% 6610 ± 26% interrupts.CPU55.PMI:Performance_monitoring_interrupts 494.83 ± 17% -47.3% 260.57 ± 28% interrupts.CPU55.RES:Rescheduling_interrupts 484.00 ± 17% -58.0% 203.14 ± 42% interrupts.CPU62.RES:Rescheduling_interrupts 524.50 ± 23% -57.1% 225.14 ± 55% interrupts.CPU64.RES:Rescheduling_interrupts 495.50 ± 26% -64.3% 176.86 ± 22% interrupts.CPU65.RES:Rescheduling_interrupts 4839 ± 35% +25.1% 6054 ± 31% interrupts.CPU74.NMI:Non-maskable_interrupts 4839 ± 35% +25.1% 6054 ± 31% interrupts.CPU74.PMI:Performance_monitoring_interrupts 128.67 ± 10% +103.1% 261.29 ± 33% interrupts.CPU83.RES:Rescheduling_interrupts 48635 ± 6% -19.4% 39210 ± 5% interrupts.RES:Rescheduling_interrupts 0.03 ± 5% +2492.5% 0.67 ± 4% perf-stat.i.MPKI 4.791e+10 -43.1% 2.728e+10 ± 2% perf-stat.i.branch-instructions 0.22 +0.0 0.24 perf-stat.i.branch-miss-rate% 1.041e+08 -39.1% 63457370 ± 2% perf-stat.i.branch-misses 10.43 ± 5% -3.7 6.71 ± 12% perf-stat.i.cache-miss-rate% 551690 +958.4% 5839328 ± 15% perf-stat.i.cache-misses 5068760 ± 3% +1605.3% 86435112 ± 2% perf-stat.i.cache-references 1.21 +81.4% 2.20 ± 2% perf-stat.i.cpi 2.751e+11 +3.5% 2.846e+11 perf-stat.i.cpu-cycles 138.04 -5.3% 130.67 perf-stat.i.cpu-migrations 651980 ± 3% -91.9% 52850 ± 12% perf-stat.i.cycles-between-cache-misses 0.07 -0.0 0.07 perf-stat.i.dTLB-load-miss-rate% 48759822 -43.9% 27367271 ± 2% perf-stat.i.dTLB-load-misses 6.958e+10 -43.1% 3.962e+10 ± 2% perf-stat.i.dTLB-loads 0.00 +0.0 0.00 ± 2% perf-stat.i.dTLB-store-miss-rate% 65887 -3.0% 63941 perf-stat.i.dTLB-store-misses 4.232e+10 -43.1% 2.41e+10 ± 2% perf-stat.i.dTLB-stores 48798060 -43.6% 27508900 ± 2% perf-stat.i.iTLB-load-misses 2.264e+11 -43.0% 1.29e+11 ± 2% perf-stat.i.instructions 0.82 -44.7% 0.46 ± 2% perf-stat.i.ipc 2.65 +3.5% 2.74 perf-stat.i.metric.GHz 72.75 ± 11% +1072.1% 852.68 ± 2% perf-stat.i.metric.K/sec 1536 -43.1% 874.89 ± 2% perf-stat.i.metric.M/sec 3089 -2.6% 3009 perf-stat.i.minor-faults 89.95 +2.1 92.07 perf-stat.i.node-load-miss-rate% 113978 +849.5% 1082204 ± 2% perf-stat.i.node-load-misses 40862 ± 8% +137.7% 97111 ± 3% perf-stat.i.node-loads 72.10 +18.0 90.13 perf-stat.i.node-store-miss-rate% 19439 ± 4% +290.3% 75873 ± 8% perf-stat.i.node-store-misses 3090 -2.6% 3010 perf-stat.i.page-faults 0.02 ± 3% +2857.3% 0.67 ± 4% perf-stat.overall.MPKI 0.22 +0.0 0.23 perf-stat.overall.branch-miss-rate% 10.96 ± 4% -4.2 6.74 ± 13% perf-stat.overall.cache-miss-rate% 1.22 +81.7% 2.21 ± 2% perf-stat.overall.cpi 489587 -89.8% 49736 ± 14% perf-stat.overall.cycles-between-cache-misses 0.07 -0.0 0.07 perf-stat.overall.dTLB-load-miss-rate% 0.00 +0.0 0.00 ± 2% perf-stat.overall.dTLB-store-miss-rate% 0.82 -44.9% 0.45 ± 2% perf-stat.overall.ipc 70.36 +21.3 91.63 perf-stat.overall.node-load-miss-rate% 61.00 ± 2% +24.9 85.93 perf-stat.overall.node-store-miss-rate% 1396343 +1.7% 1420390 perf-stat.overall.path-length 4.775e+10 -43.0% 2.72e+10 ± 2% perf-stat.ps.branch-instructions 1.038e+08 -39.0% 63303178 ± 2% perf-stat.ps.branch-misses 560203 +940.0% 5826326 ± 15% perf-stat.ps.cache-misses 5117932 ± 3% +1583.9% 86179654 ± 2% perf-stat.ps.cache-references 2.742e+11 +3.5% 2.837e+11 perf-stat.ps.cpu-cycles 137.44 -5.2% 130.26 perf-stat.ps.cpu-migrations 48602770 -43.9% 27281277 ± 2% perf-stat.ps.dTLB-load-misses 6.934e+10 -43.1% 3.949e+10 ± 2% perf-stat.ps.dTLB-loads 65959 -3.2% 63847 perf-stat.ps.dTLB-store-misses 4.218e+10 -43.1% 2.402e+10 ± 2% perf-stat.ps.dTLB-stores 48632430 -43.6% 27420183 ± 2% perf-stat.ps.iTLB-load-misses 2.256e+11 -43.0% 1.286e+11 ± 2% perf-stat.ps.instructions 3096 -2.6% 3016 perf-stat.ps.minor-faults 113687 +849.6% 1079609 ± 2% perf-stat.ps.node-load-misses 47974 ± 7% +105.6% 98645 ± 2% perf-stat.ps.node-loads 19347 ± 4% +291.8% 75805 ± 8% perf-stat.ps.node-store-misses 3097 -2.6% 3017 perf-stat.ps.page-faults 6.821e+13 -42.8% 3.899e+13 ± 2% perf-stat.total.instructions 0.03 ± 2% +47.2% 0.05 ± 4% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.03 ± 7% +7826.5% 2.42 ± 59% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 0.01 ± 5% +128.1% 0.02 ± 78% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.03 ± 7% +39.3% 0.05 ± 9% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.vprintk_emit.devkmsg_emit.constprop 0.03 ± 7% +29.7% 0.04 ± 3% perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork 0.03 ± 2% +40.6% 0.04 ± 4% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.03 ± 9% +43.7% 0.04 ± 15% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.02 ± 9% +42.5% 0.03 ± 5% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 0.05 ± 8% +1799.3% 0.96 ± 58% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork 0.05 ± 8% +36.5% 0.06 ± 13% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 2.91 ±139% +357.9% 13.34 ± 97% perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.07 ± 73% +70433.0% 49.84 ± 60% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 0.05 ± 12% +3147.7% 1.62 ±234% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 22.18 ± 6% -10.9% 19.75 ± 10% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.stop_one_cpu.sched_exec 0.03 ± 7% +39.3% 0.05 ± 9% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.vprintk_emit.devkmsg_emit.constprop 15.66 ± 28% +3873.6% 622.35 ± 55% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 177.89 ± 3% -31.4% 122.05 ± 9% perf-sched.total_wait_and_delay.average.ms 12230 ± 2% +38.7% 16966 ± 6% perf-sched.total_wait_and_delay.count.ms 177.68 ± 3% -31.4% 121.83 ± 9% perf-sched.total_wait_time.average.ms 568.61 ± 4% +96.7% 1118 ± 16% perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 644.19 ± 19% +88.8% 1216 ± 19% perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read 0.86 ± 12% +257.3% 3.07 ± 37% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 0.32 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 276.67 ± 18% -74.4% 70.74 ± 34% perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.78 ± 17% +126.0% 1.75 ± 9% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.__mutex_lock.__fdget_pos.ksys_lseek 0.94 ± 18% +174.6% 2.57 ± 18% perf-sched.wait_and_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.__fget_files 0.89 ± 36% +233.9% 2.98 ± 16% perf-sched.wait_and_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__fget_files 0.88 ± 21% +210.6% 2.73 ± 12% perf-sched.wait_and_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__mutex_unlock_slowpath 0.92 ± 17% +268.7% 3.38 ± 55% perf-sched.wait_and_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.syscall_enter_from_user_mode.do_syscall_64 0.84 ± 24% +139.4% 2.01 ± 19% perf-sched.wait_and_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.__fget_files.__fget_light 0.79 ± 29% +139.6% 1.88 ± 21% perf-sched.wait_and_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__fget_files.__fget_light 0.75 ± 15% +136.4% 1.78 ± 18% perf-sched.wait_and_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__mutex_unlock_slowpath.ksys_lseek 119.95 ±100% +213.4% 375.89 ± 20% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 392.66 ± 10% +53.4% 602.35 ± 3% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 7.71 ± 9% -24.5% 5.82 ± 9% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.42 ± 39% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 766.62 ± 3% -12.3% 672.70 ± 2% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 942.67 ± 3% +29.8% 1223 ± 3% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 110.00 ± 7% +96.1% 215.71 ± 19% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 77.50 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 490.50 ± 27% +355.3% 2233 ± 30% perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read 707.33 ± 2% +42.0% 1004 ± 6% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk.__mutex_lock.__fdget_pos.ksys_lseek 286.67 ± 9% +15.4% 330.71 ± 2% perf-sched.wait_and_delay.count.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.__fget_files 153.67 ± 9% +31.2% 201.57 ± 8% perf-sched.wait_and_delay.count.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_is_held_type.___might_sleep 98.17 ± 7% +29.1% 126.71 ± 8% perf-sched.wait_and_delay.count.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__fget_files 78.00 ± 6% +57.7% 123.00 ± 5% perf-sched.wait_and_delay.count.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__mutex_unlock_slowpath 547.83 ± 2% +40.4% 769.00 ± 5% perf-sched.wait_and_delay.count.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.syscall_enter_from_user_mode.do_syscall_64 198.83 ± 7% +44.6% 287.57 ± 5% perf-sched.wait_and_delay.count.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.__fget_files.__fget_light 195.50 ± 5% +59.9% 312.57 ± 7% perf-sched.wait_and_delay.count.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__fget_files.__fget_light 189.00 ± 8% +82.2% 344.43 ± 5% perf-sched.wait_and_delay.count.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__mutex_unlock_slowpath.ksys_lseek 62.50 ± 11% -50.4% 31.00 ± 5% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 1275 ± 9% +32.3% 1688 ± 8% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 65.00 -100.0% 0.00 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 2809 ± 5% +74.5% 4901 ± 9% perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 1000 +10.0% 1099 ± 5% perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 2834 ± 6% +73.1% 4905 ± 9% perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read 23.63 ± 47% +2466.0% 606.27 ±210% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 1.83 ± 23% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 2821 ± 5% +73.6% 4898 ± 9% perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 22.18 ± 6% -10.9% 19.75 ± 10% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.stop_one_cpu.sched_exec 13.01 +218.0% 41.37 ±137% perf-sched.wait_and_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__fget_files 1251 ±100% +284.7% 4813 ± 5% perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 5092 ± 2% -70.5% 1501 ± 81% perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 442.99 ± 15% -37.5% 276.98 ± 33% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 6.35 ± 61% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 568.10 ± 4% +96.4% 1115 ± 16% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 640.01 ± 19% +89.4% 1212 ± 19% perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read 0.86 ± 12% +257.3% 3.07 ± 37% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 276.27 ± 18% -74.5% 70.51 ± 34% perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.78 ± 17% +126.0% 1.75 ± 9% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.__mutex_lock.__fdget_pos.ksys_lseek 0.00 ±223% +1.6e+06% 73.72 ±236% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 8.53 ± 4% -8.2% 7.83 ± 4% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.affine_move_task.__set_cpus_allowed_ptr 0.91 ± 55% +3376.7% 31.58 ±226% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.___might_sleep.__mutex_lock 0.62 ± 17% +357.8% 2.84 ± 42% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.__fget_files.__fget_light 0.50 ±110% +645.5% 3.76 ± 15% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.debug_lockdep_rcu_enabled.rcu_read_lock_held_common 0.35 ± 84% +767.2% 3.04 ± 52% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.64 ± 24% +336.3% 2.79 ± 56% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.ksys_lseek.do_syscall_64 0.94 ± 18% +174.6% 2.57 ± 18% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.__fget_files 1.23 ± 44% +255.5% 4.39 ± 26% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_is_held_type.rcu_read_lock_held 2.16 ±223% +505.8% 13.11 ± 6% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_is_held_type.xas_descend 0.89 ± 36% +233.9% 2.98 ± 16% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__fget_files 0.88 ± 21% +210.6% 2.73 ± 12% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__mutex_unlock_slowpath 0.65 ±117% +266.3% 2.38 ±100% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.preempt_count_sub.__mutex_lock 0.66 ± 40% +216.3% 2.10 ± 32% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.shmem_file_llseek.ksys_lseek 0.92 ± 17% +268.7% 3.38 ± 55% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.syscall_enter_from_user_mode.do_syscall_64 0.25 ±114% -92.6% 0.02 ±120% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_call_function_single.__rcu_read_lock.__fget_files 0.12 ± 93% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_call_function_single.mutex_unlock.ksys_lseek 0.28 ± 70% -83.6% 0.05 ± 78% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__fdget_pos.ksys_lseek 0.84 ± 24% +139.4% 2.01 ± 19% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.__fget_files.__fget_light 0.79 ± 29% +139.6% 1.88 ± 21% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__fget_files.__fget_light 0.75 ± 15% +136.4% 1.78 ± 18% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__mutex_unlock_slowpath.ksys_lseek 0.55 ± 37% +215.8% 1.74 ± 36% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.__fget_files.__fget_light 0.63 ± 48% +356.0% 2.87 ± 33% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_lockdep_current_cpu_online.rcu_read_lock_held_common.rcu_read_lock_held 192.29 ± 26% +95.3% 375.54 ± 20% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 392.63 ± 10% +53.4% 602.33 ± 3% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 7.68 ± 9% -24.7% 5.78 ± 9% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.42 ± 39% +304.8% 1.72 ± 6% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 766.57 ± 3% -12.4% 671.74 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 2809 ± 5% +74.3% 4896 ± 9% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 2821 ± 5% +73.6% 4899 ± 9% perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read 23.63 ± 47% +2466.0% 606.27 ±210% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 2821 ± 5% +73.6% 4898 ± 9% perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.00 ±223% +3.1e+06% 146.37 ±238% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 51.68 ±174% +1374.8% 762.09 ± 43% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.__wake_up_common_lock.pipe_write 9.16 ± 30% +243.5% 31.48 ±124% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.__fget_files.__fget_light 3.11 ±143% +506.4% 18.85 ± 18% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.debug_lockdep_rcu_enabled.rcu_read_lock_held_common 1.61 ±144% +488.9% 9.48 ± 51% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.17 ±223% +1217.1% 28.52 ±119% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.fs_reclaim_acquire 2.17 ±223% +605.6% 15.28 ± 21% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.pagecache_get_page 11.80 ± 33% +454.3% 65.42 ±183% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_is_held_type.rcu_read_lock_held 2.16 ±223% +639.2% 16.00 ± 17% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_is_held_type.xas_descend 13.01 +218.0% 41.37 ±137% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.__fget_files 0.35 ±110% -94.7% 0.02 ±120% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_call_function_single.__rcu_read_lock.__fget_files 0.20 ±172% +578.3% 1.37 ± 50% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_call_function_single.debug_lockdep_rcu_enabled.rcu_read_lock_held_common 0.21 ±145% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_call_function_single.mutex_unlock.ksys_lseek 0.76 ± 78% -91.7% 0.06 ± 99% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__fdget_pos.ksys_lseek 1.22 ± 22% +52.9% 1.87 ± 21% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__fget_files.__fget_light 8.58 ± 48% +51.5% 13.00 perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.__fget_files.__fget_light 6.06 ± 35% +131.6% 14.03 ± 13% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_lockdep_current_cpu_online.rcu_read_lock_held_common.rcu_read_lock_held 2470 ± 7% +94.9% 4813 ± 5% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 5092 ± 2% -70.5% 1501 ± 81% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 442.96 ± 15% -37.5% 276.93 ± 33% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 5.96 -3.2 2.77 ± 2% perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_lseek64 5.39 -2.2 3.17 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_lseek64 19.24 -2.1 17.13 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 1.99 -1.1 0.92 ± 2% perf-profile.calltrace.cycles-pp.check_preemption_disabled.lock_is_held_type.___might_sleep.__mutex_lock.__fdget_pos 99.03 -0.9 98.12 perf-profile.calltrace.cycles-pp.__libc_lseek64 2.29 -0.8 1.46 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.__libc_lseek64 1.86 ± 3% -0.8 1.05 ± 2% perf-profile.calltrace.cycles-pp.mark_lock.__lock_acquire.lock_acquire.__mutex_lock.__fdget_pos 1.22 -0.6 0.62 ± 3% perf-profile.calltrace.cycles-pp.shmem_file_llseek.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 3.73 -0.5 3.21 perf-profile.calltrace.cycles-pp.trace_hardirqs_on_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 5.08 -0.5 4.59 perf-profile.calltrace.cycles-pp.___might_sleep.__mutex_lock.__fdget_pos.ksys_lseek.do_syscall_64 3.89 -0.3 3.55 perf-profile.calltrace.cycles-pp.trace_hardirqs_off_finish.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 5.43 -0.3 5.10 perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire.__fget_files.__fget_light.__fdget_pos 0.85 ± 3% -0.3 0.59 ± 4% perf-profile.calltrace.cycles-pp.preempt_count_sub.__mutex_lock.__fdget_pos.ksys_lseek.do_syscall_64 1.41 -0.3 1.14 perf-profile.calltrace.cycles-pp.rcu_read_lock_held_common.rcu_read_lock_sched_held.trace_hardirqs_on_prepare.syscall_exit_to_user_mode.do_syscall_64 1.62 -0.2 1.46 perf-profile.calltrace.cycles-pp.rcu_read_lock_held_common.rcu_read_lock_sched_held.trace_hardirqs_off_finish.syscall_enter_from_user_mode.do_syscall_64 2.84 -0.1 2.73 perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.trace_hardirqs_on_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.24 -0.1 4.14 perf-profile.calltrace.cycles-pp.lock_is_held_type.___might_sleep.__mutex_lock.__fdget_pos.ksys_lseek 2.04 +0.1 2.11 perf-profile.calltrace.cycles-pp.trace_hardirqs_on.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 1.66 +0.1 1.74 perf-profile.calltrace.cycles-pp.lock_is_held_type.rcu_read_lock_sched_held.lock_release.__fget_files.__fget_light 1.55 +0.1 1.66 perf-profile.calltrace.cycles-pp.lock_is_held_type.rcu_read_lock_sched_held.lock_release.__mutex_unlock_slowpath.ksys_lseek 2.98 +0.1 3.09 perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.trace_hardirqs_off_finish.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.42 ± 2% +0.2 1.61 perf-profile.calltrace.cycles-pp.lock_is_held_type.rcu_read_lock_sched_held.lock_acquire.__mutex_lock.__fdget_pos 1.37 ± 2% +0.2 1.59 perf-profile.calltrace.cycles-pp.lock_is_held_type.rcu_read_lock_sched_held.lock_acquire.__fget_files.__fget_light 1.63 +0.3 1.89 perf-profile.calltrace.cycles-pp.lock_is_held_type.rcu_read_lock_held.__fget_files.__fget_light.__fdget_pos 1.50 +0.3 1.77 perf-profile.calltrace.cycles-pp.rcu_read_lock_held_common.rcu_read_lock_held.__fget_files.__fget_light.__fdget_pos 1.11 +0.3 1.41 perf-profile.calltrace.cycles-pp.lock_is_held_type.rcu_read_lock_sched_held.trace_hardirqs_on_prepare.syscall_exit_to_user_mode.do_syscall_64 1.43 +0.3 1.74 perf-profile.calltrace.cycles-pp.lockdep_hardirqs_on_prepare.trace_hardirqs_on.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.52 +0.3 1.83 perf-profile.calltrace.cycles-pp.lockdep_hardirqs_on_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 0.82 ± 3% +0.4 1.20 perf-profile.calltrace.cycles-pp.lockdep_hardirqs_on.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 3.42 +0.4 3.82 perf-profile.calltrace.cycles-pp.rcu_read_lock_held.__fget_files.__fget_light.__fdget_pos.ksys_lseek 0.99 +0.4 1.42 perf-profile.calltrace.cycles-pp.lock_is_held_type.rcu_read_lock_sched_held.trace_hardirqs_off_finish.syscall_enter_from_user_mode.do_syscall_64 0.94 ± 21% +0.4 1.39 ± 11% perf-profile.calltrace.cycles-pp.validate_chain.__lock_acquire.lock_acquire.__mutex_lock.__fdget_pos 23.74 +0.5 24.19 perf-profile.calltrace.cycles-pp.__fget_files.__fget_light.__fdget_pos.ksys_lseek.do_syscall_64 0.00 +0.6 0.61 ± 14% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 1.79 ± 3% +0.7 2.45 perf-profile.calltrace.cycles-pp.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_acquire.__fget_files.__fget_light 0.76 +0.7 1.42 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 6.49 ± 3% +0.7 7.17 ± 2% perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire.__mutex_lock.__fdget_pos.ksys_lseek 0.00 +0.7 0.68 ± 15% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 0.00 +0.7 0.69 ± 2% perf-profile.calltrace.cycles-pp.validate_chain.__lock_acquire.lock_acquire.__fget_files.__fget_light 3.52 ± 2% +0.7 4.24 perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_acquire.__fget_files.__fget_light.__fdget_pos 0.00 +0.7 0.73 ± 15% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write 0.00 +0.7 0.73 ± 15% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 0.00 +0.7 0.74 ± 15% perf-profile.calltrace.cycles-pp.__libc_write 0.00 +0.8 0.78 ± 2% perf-profile.calltrace.cycles-pp.debug_lockdep_rcu_enabled.rcu_read_lock_held_common.rcu_read_lock_sched_held.trace_hardirqs_off_finish.syscall_enter_from_user_mode 1.43 +0.8 2.26 perf-profile.calltrace.cycles-pp.trace_hardirqs_off.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 1.60 +1.0 2.60 perf-profile.calltrace.cycles-pp.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_acquire.__mutex_lock.__fdget_pos 3.55 ± 2% +1.0 4.56 perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_release.__fget_files.__fget_light.__fdget_pos 3.37 +1.0 4.40 perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_acquire.__mutex_lock.__fdget_pos.ksys_lseek 0.98 +1.0 2.02 perf-profile.calltrace.cycles-pp.lockdep_hardirqs_off.trace_hardirqs_off.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.1 1.06 ± 3% perf-profile.calltrace.cycles-pp.debug_lockdep_rcu_enabled.rcu_read_lock_held_common.rcu_read_lock_held.__fget_files.__fget_light 1.53 ± 4% +1.1 2.63 perf-profile.calltrace.cycles-pp.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_release.__fget_files.__fget_light 6.23 ± 2% +1.2 7.43 perf-profile.calltrace.cycles-pp.lock_release.__fget_files.__fget_light.__fdget_pos.ksys_lseek 7.23 +1.4 8.62 perf-profile.calltrace.cycles-pp.__mutex_unlock_slowpath.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 3.67 +1.6 5.30 perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_release.__mutex_unlock_slowpath.ksys_lseek.do_syscall_64 11.98 +1.7 13.66 perf-profile.calltrace.cycles-pp.lock_acquire.__mutex_lock.__fdget_pos.ksys_lseek.do_syscall_64 1.75 +1.7 3.47 perf-profile.calltrace.cycles-pp.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_release.__mutex_unlock_slowpath.ksys_lseek 0.00 +1.7 1.72 ± 2% perf-profile.calltrace.cycles-pp.debug_lockdep_rcu_enabled.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_acquire.__fget_files 6.23 +1.8 8.05 perf-profile.calltrace.cycles-pp.lock_release.__mutex_unlock_slowpath.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.9 1.93 ± 2% perf-profile.calltrace.cycles-pp.debug_lockdep_rcu_enabled.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_acquire.__mutex_lock 0.00 +2.0 1.95 ± 2% perf-profile.calltrace.cycles-pp.debug_lockdep_rcu_enabled.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_release.__fget_files 0.00 +2.8 2.82 perf-profile.calltrace.cycles-pp.debug_lockdep_rcu_enabled.rcu_read_lock_held_common.rcu_read_lock_sched_held.lock_release.__mutex_unlock_slowpath 0.66 +3.9 4.55 ± 2% perf-profile.calltrace.cycles-pp.lockdep_hardirqs_on.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 8.63 +4.3 12.97 perf-profile.calltrace.cycles-pp.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 1.22 +4.4 5.61 ± 2% perf-profile.calltrace.cycles-pp.lockdep_hardirqs_off.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 85.52 +5.2 90.70 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_lseek64 84.90 +5.5 90.43 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_lseek64 13.05 -6.0 7.09 ± 2% perf-profile.children.cycles-pp.check_preemption_disabled 5.95 -2.6 3.38 ± 2% perf-profile.children.cycles-pp.__entry_text_start 6.16 -2.6 3.59 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 5.00 -2.5 2.51 ± 2% perf-profile.children.cycles-pp.rcu_is_watching 5.22 -2.4 2.82 ± 2% perf-profile.children.cycles-pp.rcu_lockdep_current_cpu_online 19.40 -2.2 17.24 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 2.77 ± 2% -1.2 1.54 ± 2% perf-profile.children.cycles-pp.mark_lock 1.87 -0.8 1.04 ± 2% perf-profile.children.cycles-pp.__this_cpu_preempt_check 1.49 -0.6 0.86 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 1.27 -0.6 0.64 ± 3% perf-profile.children.cycles-pp.shmem_file_llseek 99.27 -0.6 98.66 perf-profile.children.cycles-pp.__libc_lseek64 3.76 -0.5 3.24 perf-profile.children.cycles-pp.trace_hardirqs_on_prepare 5.15 -0.5 4.68 perf-profile.children.cycles-pp.___might_sleep 0.83 ± 4% -0.4 0.41 ± 4% perf-profile.children.cycles-pp.__x64_sys_lseek 3.95 -0.4 3.60 perf-profile.children.cycles-pp.trace_hardirqs_off_finish 0.71 -0.3 0.36 ± 3% perf-profile.children.cycles-pp.find_held_lock 1.00 ± 3% -0.3 0.68 ± 4% perf-profile.children.cycles-pp.preempt_count_sub 0.75 -0.3 0.45 ± 2% perf-profile.children.cycles-pp.match_held_lock 0.63 ± 3% -0.3 0.34 ± 2% perf-profile.children.cycles-pp.debug_smp_processor_id 1.94 -0.3 1.65 ± 3% perf-profile.children.cycles-pp.hrtimer_interrupt 1.98 -0.3 1.69 ± 3% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 2.21 -0.3 1.94 ± 3% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 2.11 -0.3 1.84 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 1.59 -0.2 1.36 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues 1.28 ± 2% -0.2 1.07 ± 3% perf-profile.children.cycles-pp.tick_sched_timer 1.23 ± 2% -0.2 1.02 ± 3% perf-profile.children.cycles-pp.tick_sched_handle 1.23 -0.2 1.02 ± 3% perf-profile.children.cycles-pp.update_process_times 0.45 ± 2% -0.2 0.25 ± 3% perf-profile.children.cycles-pp.tracer_hardirqs_off 0.45 -0.2 0.26 ± 4% perf-profile.children.cycles-pp.preempt_count_add 0.40 ± 3% -0.2 0.22 ± 4% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.44 ± 2% -0.2 0.26 ± 3% perf-profile.children.cycles-pp.fput_many 0.47 -0.2 0.30 ± 3% perf-profile.children.cycles-pp.tracer_hardirqs_on 0.35 ± 3% -0.2 0.18 ± 4% perf-profile.children.cycles-pp.testcase 0.38 -0.2 0.22 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.30 -0.2 0.14 ± 3% perf-profile.children.cycles-pp.generic_file_llseek_size 0.31 ± 5% -0.2 0.16 ± 7% perf-profile.children.cycles-pp.__rcu_read_lock 0.96 -0.2 0.81 ± 4% perf-profile.children.cycles-pp.scheduler_tick 0.86 -0.1 0.72 ± 4% perf-profile.children.cycles-pp.task_tick_fair 0.76 -0.1 0.64 ± 5% perf-profile.children.cycles-pp.update_curr 0.33 ± 3% -0.1 0.20 ± 3% perf-profile.children.cycles-pp.mark_held_locks 0.24 ± 3% -0.1 0.13 ± 2% perf-profile.children.cycles-pp.__might_sleep 0.62 ± 2% -0.1 0.50 ± 6% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime 0.62 ± 2% -0.1 0.51 ± 6% perf-profile.children.cycles-pp.perf_tp_event 0.61 ± 2% -0.1 0.50 ± 6% perf-profile.children.cycles-pp.perf_event_output_forward 0.61 ± 2% -0.1 0.50 ± 6% perf-profile.children.cycles-pp.__perf_event_overflow 0.61 ± 2% -0.1 0.50 ± 6% perf-profile.children.cycles-pp.perf_swevent_overflow 0.52 ± 2% -0.1 0.42 ± 6% perf-profile.children.cycles-pp.perf_prepare_sample 0.25 ± 3% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.trace_preempt_on 0.16 ± 3% -0.1 0.07 ± 4% perf-profile.children.cycles-pp.mutex_unlock 0.44 ± 2% -0.1 0.35 ± 6% perf-profile.children.cycles-pp.perf_callchain 0.43 ± 3% -0.1 0.35 ± 6% perf-profile.children.cycles-pp.get_perf_callchain 0.14 ± 3% -0.1 0.06 ± 7% perf-profile.children.cycles-pp.__f_unlock_pos 0.13 ± 4% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.lockdep_sys_exit 0.16 ± 2% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.in_lock_functions 0.08 -0.0 0.04 ± 63% perf-profile.children.cycles-pp.fput 0.28 ± 3% -0.0 0.23 ± 6% perf-profile.children.cycles-pp.perf_callchain_kernel 0.09 ± 4% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.trace_preempt_off 0.13 ± 7% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.15 ± 2% -0.0 0.11 ± 9% perf-profile.children.cycles-pp.perf_callchain_user 0.08 ± 5% -0.0 0.05 perf-profile.children.cycles-pp.mutex_lock_nested 0.08 ± 6% -0.0 0.04 ± 40% perf-profile.children.cycles-pp.tracer_preempt_on 0.14 ± 3% -0.0 0.11 ± 9% perf-profile.children.cycles-pp.__get_user_nocheck_8 0.15 ± 3% -0.0 0.12 ± 7% perf-profile.children.cycles-pp.unwind_next_frame 0.09 -0.0 0.07 ± 5% perf-profile.children.cycles-pp.native_irq_return_iret 0.08 ± 6% -0.0 0.06 perf-profile.children.cycles-pp.__perf_event_header__init_id 0.08 ± 5% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.account_system_index_time 0.04 ± 72% +0.0 0.08 ± 12% perf-profile.children.cycles-pp.shmem_add_to_page_cache 0.05 ± 46% +0.0 0.10 ± 11% perf-profile.children.cycles-pp.iov_iter_fault_in_readable 0.07 ± 12% +0.0 0.12 ± 12% perf-profile.children.cycles-pp.__handle_mm_fault 0.05 ± 45% +0.0 0.09 ± 16% perf-profile.children.cycles-pp.do_fault 0.19 ± 7% +0.1 0.25 ± 8% perf-profile.children.cycles-pp.asm_exc_page_fault 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.kmem_cache_alloc 0.00 +0.1 0.06 ± 8% perf-profile.children.cycles-pp.lock_page_memcg 0.16 ± 9% +0.1 0.22 ± 8% perf-profile.children.cycles-pp.exc_page_fault 0.09 ± 9% +0.1 0.15 ± 12% perf-profile.children.cycles-pp.handle_mm_fault 2.13 +0.1 2.19 perf-profile.children.cycles-pp.trace_hardirqs_on 0.11 ± 10% +0.1 0.17 ± 12% perf-profile.children.cycles-pp.do_user_addr_fault 0.00 +0.1 0.12 ± 21% perf-profile.children.cycles-pp.pagecache_get_page 0.10 ± 23% +0.2 0.27 ± 12% perf-profile.children.cycles-pp.shmem_getpage_gfp 0.10 ± 23% +0.2 0.28 ± 12% perf-profile.children.cycles-pp.shmem_write_begin 0.18 ± 23% +0.2 0.42 ± 12% perf-profile.children.cycles-pp.generic_perform_write 0.18 ± 23% +0.3 0.46 ± 12% perf-profile.children.cycles-pp.__generic_file_write_iter 0.18 ± 23% +0.3 0.51 ± 13% perf-profile.children.cycles-pp.generic_file_write_iter 0.19 ± 23% +0.3 0.52 ± 13% perf-profile.children.cycles-pp.new_sync_write 3.55 +0.4 3.97 perf-profile.children.cycles-pp.rcu_read_lock_held 23.87 +0.4 24.29 perf-profile.children.cycles-pp.__fget_files 0.19 ± 23% +0.4 0.62 ± 14% perf-profile.children.cycles-pp.vfs_write 0.19 ± 23% +0.5 0.69 ± 14% perf-profile.children.cycles-pp.ksys_write 3.26 +0.5 3.78 perf-profile.children.cycles-pp.lockdep_hardirqs_on_prepare 0.18 ± 22% +0.6 0.74 ± 15% perf-profile.children.cycles-pp.__libc_write 0.77 +0.7 1.43 perf-profile.children.cycles-pp.exit_to_user_mode_prepare 1.48 ± 14% +0.7 2.16 ± 7% perf-profile.children.cycles-pp.validate_chain 1.47 +0.8 2.31 perf-profile.children.cycles-pp.trace_hardirqs_off 15.08 +1.3 16.40 perf-profile.children.cycles-pp.lock_is_held_type 7.28 +1.4 8.66 perf-profile.children.cycles-pp.__mutex_unlock_slowpath 24.17 +1.9 26.08 perf-profile.children.cycles-pp.lock_acquire 13.01 +3.0 16.05 perf-profile.children.cycles-pp.lock_release 12.13 +4.2 16.30 perf-profile.children.cycles-pp.rcu_read_lock_held_common 1.51 +4.3 5.80 ± 2% perf-profile.children.cycles-pp.lockdep_hardirqs_on 8.70 +4.3 13.03 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 20.90 +4.4 25.28 perf-profile.children.cycles-pp.rcu_read_lock_sched_held 2.32 +5.4 7.73 perf-profile.children.cycles-pp.lockdep_hardirqs_off 85.91 +5.8 91.75 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 85.33 +6.2 91.51 perf-profile.children.cycles-pp.do_syscall_64 1.94 +9.0 10.98 ± 2% perf-profile.children.cycles-pp.debug_lockdep_rcu_enabled 12.24 -5.6 6.65 ± 2% perf-profile.self.cycles-pp.check_preemption_disabled 11.38 -3.2 8.18 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 6.15 -2.6 3.59 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 5.12 -2.2 2.94 ± 2% perf-profile.self.cycles-pp.__entry_text_start 3.77 -1.7 2.04 ± 2% perf-profile.self.cycles-pp.rcu_lockdep_current_cpu_online 3.10 -1.6 1.53 ± 2% perf-profile.self.cycles-pp.rcu_is_watching 2.58 ± 2% -1.1 1.44 ± 2% perf-profile.self.cycles-pp.mark_lock 1.78 -0.8 1.00 ± 2% perf-profile.self.cycles-pp.rcu_read_lock_sched_held 1.10 -0.6 0.55 ± 2% perf-profile.self.cycles-pp.shmem_file_llseek 1.22 ± 4% -0.5 0.69 ± 10% perf-profile.self.cycles-pp.__fget_files 1.11 -0.5 0.59 ± 3% perf-profile.self.cycles-pp.rcu_read_lock_held_common 1.10 ± 2% -0.5 0.61 ± 4% perf-profile.self.cycles-pp.__libc_lseek64 0.66 ± 3% -0.4 0.25 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 1.22 -0.4 0.81 perf-profile.self.cycles-pp.__mutex_lock 0.92 -0.4 0.51 ± 3% perf-profile.self.cycles-pp.__mutex_unlock_slowpath 0.81 -0.4 0.42 ± 2% perf-profile.self.cycles-pp.___might_sleep 0.92 ± 2% -0.4 0.53 ± 2% perf-profile.self.cycles-pp.__this_cpu_preempt_check 0.67 ± 5% -0.3 0.33 ± 4% perf-profile.self.cycles-pp.__x64_sys_lseek 0.58 ± 2% -0.3 0.25 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.66 -0.3 0.33 ± 4% perf-profile.self.cycles-pp.trace_hardirqs_off_finish 0.70 ± 2% -0.3 0.38 ± 2% perf-profile.self.cycles-pp.ksys_lseek 0.64 -0.3 0.33 ± 3% perf-profile.self.cycles-pp.trace_hardirqs_on_prepare 0.66 ± 2% -0.3 0.39 ± 2% perf-profile.self.cycles-pp.match_held_lock 0.46 -0.2 0.24 ± 3% perf-profile.self.cycles-pp.find_held_lock 0.63 ± 4% -0.2 0.43 ± 4% perf-profile.self.cycles-pp.preempt_count_sub 0.43 -0.2 0.24 ± 3% perf-profile.self.cycles-pp.tracer_hardirqs_off 0.36 ± 18% -0.2 0.17 ± 10% perf-profile.self.cycles-pp.__fget_light 0.36 -0.2 0.20 ± 4% perf-profile.self.cycles-pp.debug_smp_processor_id 0.38 -0.2 0.22 ± 3% perf-profile.self.cycles-pp.preempt_count_add 0.41 -0.2 0.25 ± 2% perf-profile.self.cycles-pp.fput_many 0.38 ± 2% -0.2 0.21 ± 2% perf-profile.self.cycles-pp.__fdget_pos 0.36 -0.2 0.21 ± 3% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 0.29 -0.2 0.14 ± 4% perf-profile.self.cycles-pp.generic_file_llseek_size 0.33 ± 3% -0.2 0.18 ± 5% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.32 ± 2% -0.2 0.17 ± 3% perf-profile.self.cycles-pp.do_syscall_64 0.41 ± 2% -0.1 0.27 ± 2% perf-profile.self.cycles-pp.tracer_hardirqs_on 0.28 ± 4% -0.1 0.14 ± 8% perf-profile.self.cycles-pp.__rcu_read_lock 0.28 ± 4% -0.1 0.16 ± 4% perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.29 ± 2% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.trace_hardirqs_on 0.22 ± 3% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.__might_sleep 0.22 -0.1 0.13 ± 3% perf-profile.self.cycles-pp.trace_hardirqs_off 0.23 ± 2% -0.1 0.13 ± 3% perf-profile.self.cycles-pp.rcu_read_lock_held 0.17 ± 6% -0.1 0.08 ± 17% perf-profile.self.cycles-pp.testcase 0.23 ± 3% -0.1 0.14 ± 4% perf-profile.self.cycles-pp.mark_held_locks 0.20 ± 3% -0.1 0.13 ± 5% perf-profile.self.cycles-pp.trace_preempt_on 0.13 ± 2% -0.1 0.06 ± 5% perf-profile.self.cycles-pp.mutex_unlock 0.13 ± 5% -0.1 0.06 ± 11% perf-profile.self.cycles-pp.lockdep_sys_exit 0.11 ± 3% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.in_lock_functions 0.09 ± 7% -0.0 0.05 perf-profile.self.cycles-pp.trace_preempt_off 0.09 -0.0 0.07 ± 5% perf-profile.self.cycles-pp.native_irq_return_iret 3.11 +0.3 3.42 perf-profile.self.cycles-pp.lock_acquire 0.69 +0.7 1.39 perf-profile.self.cycles-pp.exit_to_user_mode_prepare 1.44 ± 14% +0.7 2.14 ± 7% perf-profile.self.cycles-pp.validate_chain 8.47 +0.8 9.24 perf-profile.self.cycles-pp.__lock_acquire 1.54 +1.2 2.73 perf-profile.self.cycles-pp.lockdep_hardirqs_on_prepare 3.01 +1.4 4.40 perf-profile.self.cycles-pp.lock_release 9.36 +3.8 13.19 perf-profile.self.cycles-pp.lock_is_held_type 1.00 +4.5 5.54 ± 2% perf-profile.self.cycles-pp.lockdep_hardirqs_on 1.10 +6.1 7.19 ± 2% perf-profile.self.cycles-pp.lockdep_hardirqs_off 1.74 +9.0 10.71 ± 2% perf-profile.self.cycles-pp.debug_lockdep_rcu_enabled 6 +-----------------------------------------------------------------------+ | | 5 |-+ | | | | | 4 |-+ | | O O | 3 |-+ O O OO O | | O O O O O OO O OO O O | 2 |-+ | | | | .++. .+. .+. .+. +. .+. .+. | 1 |.+ +.+.+.++ + ++.+.+.+.++.+.+ + +.+ ++ +.+.++.+.+.+.++.+.| | | 0 +-----------------------------------------------------------------------+ 6 +-----------------------------------------------------------------------+ | | 5 |-+ | | | | | 4 |-+ | | O O | 3 |-+ O O OO O | | O O O O O OO O OO O O | 2 |-+ | | | | .++. .+. .+. .+. +. .+. .+. | 1 |.+ +.+.+.++ + ++.+.+.+.++.+.+ + +.+ ++ +.+.++.+.+.+.++.+.| | | 0 +-----------------------------------------------------------------------+ 2.6 +---------------------------------------------------------------------+ 2.4 |-+ O | | O O | 2.2 |-+ | 2 |-+ OO O O O O O | 1.8 |-O O O O O | 1.6 |-+ O O | | O O O O | 1.4 |-+ | 1.2 |-+ | 1 |-+ .+ | 0.8 |-+ + + .+ + | |+ + :.+.+. +.+.+.+ .+.+. +.+. + +.+.+. .+. .+. +.+.+. +.+ +| 0.6 |-+ + + + + + ++.+ ++ +.+ + | 0.4 +---------------------------------------------------------------------+ 2.6 +---------------------------------------------------------------------+ 2.4 |-+ O | | O O | 2.2 |-+ | 2 |-+ OO O O O O O | 1.8 |-O O O O O | 1.6 |-+ O O | | O O O O | 1.4 |-+ | 1.2 |-+ | 1 |-+ .+ | 0.8 |-+ + + .+ + | |+ + :.+.+. +.+.+.+ .+.+. +.+. + +.+.+. .+. .+. +.+.+. +.+ +| 0.6 |-+ + + + + + ++.+ ++ +.+ + | 0.4 +---------------------------------------------------------------------+ 8 +-----------------------------------------------------------------------+ | | 7 |-+ | 6 |-+ | | | 5 |-+ | | O | 4 |-+ O | | O | 3 |-+ O O OO O O O O | 2 |-O O O O O O OO | | | 1 |-+ +. .+. +. .+.+.+ +. .+.+. +.+.+. .++.+.+. .+ .+.| |.+.+ +.+ + +.+.+.++.+ +.+.+.+.+ + + + + + | 0 +-----------------------------------------------------------------------+ 8 +-----------------------------------------------------------------------+ | | 7 |-+ | 6 |-+ | | | 5 |-+ | | O | 4 |-+ O | | O | 3 |-+ O O OO O O O O | 2 |-O O O O O O OO | | | 1 |-+ +. .+. +. .+.+.+ +. .+.+. +.+.+. .++.+.+. .+ .+.| |.+.+ +.+ + +.+.+.++.+ +.+.+.+.+ + + + + + | 0 +-----------------------------------------------------------------------+ 25 +----------------------------------------------------------------------+ | | | O | 20 |-+ | | O O O | | | 15 |-+O O O O | | O O O O O O O O O O O O | 10 |-+ | | | | | 5 |-+ | | | | .+ | 0 +----------------------------------------------------------------------+ 14 +----------------------------------------------------------------------+ | O O O O O O O | 12 |-+O O O | | O O O O O | 10 |-+ | | O O | 8 |-+ | | | 6 |-+ | | | 4 |-+ | | | 2 |-+ + | | .. | 0 +----------------------------------------------------------------------+ 60 +----------------------------------------------------------------------+ | | 50 |-+ | | | | | 40 |-+ | | | 30 |-+ | | | 20 |-+ | | O O O | | O O O O O O O + O O O O O O O O O | 10 |-+ + | | + | 0 +----------------------------------------------------------------------+ 20 +----------------------------------------------------------------------+ 18 |-+ | | | 16 |-+ | 14 |-+ O | | O O O O + O O O O O O O O | 12 |-+O O O O O : O | 10 |-+ : | 8 |-+ : | | : | 6 |-+ : | 4 |-+ : | | : | 2 |-+ : | 0 +----------------------------------------------------------------------+ 1200 +--------------------------------------------------------------------+ | | 1000 |-+O | | | | | 800 |-+ | | | 600 |-+ | | | 400 |-+ | | | | | 200 |-+ | | | 0 +--------------------------------------------------------------------+ 350 +---------------------------------------------------------------------+ | | 300 |-+ | | | 250 |-+ | | | 200 |-+ | | | 150 |-+ | | | 100 |-+ | | | 50 |-+ | | O O O O O O O O O O | 0 +---------------------------------------------------------------------+ 20 +----------------------------------------------------------------------+ 18 |-+ O O O O O | | O O | 16 |-+ | 14 |-+ O | | O O O O O O O O O O O O O | 12 |-+ | 10 |-+ | 8 |-+ | | | 6 |-+ | 4 |-+ | | | 2 |-+ | 0 +----------------------------------------------------------------------+ 18 +----------------------------------------------------------------------+ | O | 16 |-+ | 14 |-+ O O | | O O O O O O O O O O O O O O O | 12 |-+ O | 10 |-+ | | | 8 |-+ | 6 |-+ | | | 4 |-+ | 2 |-+ | | | 0 +----------------------------------------------------------------------+ will-it-scale.104.threads 5e+07 +-----------------------------------------------------------------+ | +.++.+.++.+.++.+.++ + +.++.+.++.+.++.| 4.5e+07 |-+ | | | | | 4e+07 |-+ | | | 3.5e+07 |-+ | | | 3e+07 |-+ | | O O O OO O O O O O O O | | O O O OO O O O | 2.5e+07 |-+ O | | | 2e+07 +-----------------------------------------------------------------+ will-it-scale.per_thread_ops 500000 +------------------------------------------------------------------+ |.++.+.++.+.++.+.++.+.+ .++.+.++.+.++.+.++.+.++.+.++.| 450000 |-+ +.+.++.+.+.++.+ | | | | | 400000 |-+ | | | 350000 |-+ | | | 300000 |-+ | | | | OO O OO O O OO O OO OO O O OO | 250000 |-+ O O O | | | 200000 +------------------------------------------------------------------+ will-it-scale.workload 5e+07 +-----------------------------------------------------------------+ | +.++.+.++.+.++.+.++ + +.++.+.++.+.++.| 4.5e+07 |-+ | | | | | 4e+07 |-+ | | | 3.5e+07 |-+ | | | 3e+07 |-+ | | O O O OO O O O O O O O | | O O O OO O O O | 2.5e+07 |-+ O | | | 2e+07 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang