Greeting, FYI, we noticed a -2.4% regression of will-it-scale.per_process_ops due to commit: commit: d884021f52609407c7943705b3e54b1642fa10cb ("[PATCH 1/2] mm/swap: Add pagevec locking") url: https://github.com/0day-ci/linux/commits/Sebastian-Andrzej-Siewior/mm-swa... in testcase: will-it-scale on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory with following parameters: nr_task: 50% mode: process test: brk1 cpufreq_governor: performance test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | fio-basic: | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory | | test parameters | bs=4k | | | cpufreq_governor=performance | | | disk=1HDD | | | fs2=nfsv4 | | | fs=ext4 | | | ioengine=sync | | | nr_task=100% | | | runtime=300s | | | rw=write | | | test_size=128G | | | ucode=0x42d | +------------------+-----------------------------------------------------------------------+ | testcase: change | fio-basic: fio.latency_100us% 1.6% improvement | | test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 128G memory | | test parameters | bs=4k | | | cpufreq_governor=performance | | | disk=1SSD | | | fs=ext4 | | | ioengine=sync | | | nr_task=64 | | | runtime=300s | | | rw=randwrite | | | test_size=512g | | | ucode=0x3d | +------------------+-----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.2/process/50%/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep3d/brk1/will-it-scale commit: f3c0b8ce48 (" dmaengine fix for dmaengine-fix-4.19-rc4") d884021f52 ("mm/swap: Add pagevec locking") f3c0b8ce48408b74 d884021f52609407c7943705b3 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x :4 75% 3:4 dmesg.WARNING:at_ip_perf_event_mmap_output/0x %stddev %change %stddev \ | \ 743536 -2.4% 725610 will-it-scale.per_process_ops 32715630 -2.4% 31926873 will-it-scale.workload 0.00 ± 71% +0.0 0.01 ± 34% mpstat.cpu.soft% 2457 ± 5% +12.2% 2757 ± 5% slabinfo.kmalloc-512.active_objs 67.33 ± 9% -17.3% 55.67 ± 4% sched_debug.cpu.cpu_load[0].max 10817 ± 11% -16.7% 9008 ± 5% sched_debug.cpu.load.stddev 1.26 -0.0 1.23 perf-stat.branch-miss-rate% 4.635e+10 -3.0% 4.496e+10 perf-stat.branch-misses 3.046e+12 +1.7% 3.097e+12 perf-stat.dTLB-stores 11.55 ± 2% +11.5 23.10 ± 6% perf-stat.iTLB-load-miss-rate% 1.57e+09 +125.1% 3.534e+09 ± 7% perf-stat.iTLB-load-misses 11183 -55.4% 4991 ± 7% perf-stat.instructions-per-iTLB-miss 55.41 ± 19% +21.4 76.78 ± 15% perf-stat.node-store-miss-rate% 3387268 ± 21% -36.2% 2161442 ± 7% perf-stat.node-stores 536500 +2.4% 549279 perf-stat.path-length 31.36 -1.1 30.24 perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.85 -0.6 8.21 perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.do_munmap.__x64_sys_brk.do_syscall_64 7.70 -0.4 7.33 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.do_munmap.__x64_sys_brk 11.19 -0.2 10.96 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 5.01 -0.2 4.79 perf-profile.calltrace.cycles-pp.security_vm_enough_memory_mm.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.41 -0.2 3.21 perf-profile.calltrace.cycles-pp.down_write_killable.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.23 -0.2 5.05 perf-profile.calltrace.cycles-pp.find_vma.do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.62 -0.2 4.44 perf-profile.calltrace.cycles-pp.get_unmapped_area.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.67 -0.2 1.48 perf-profile.calltrace.cycles-pp.find_vma.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.59 -0.2 3.42 perf-profile.calltrace.cycles-pp.vma_link.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.23 -0.1 3.08 perf-profile.calltrace.cycles-pp.selinux_vm_enough_memory.security_vm_enough_memory_mm.do_brk_flags.__x64_sys_brk.do_syscall_64 2.05 -0.1 1.91 perf-profile.calltrace.cycles-pp.vmacache_find.find_vma.do_munmap.__x64_sys_brk.do_syscall_64 3.94 -0.1 3.79 perf-profile.calltrace.cycles-pp.vm_area_alloc.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.96 -0.1 7.84 perf-profile.calltrace.cycles-pp.perf_event_mmap.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.42 -0.1 2.30 perf-profile.calltrace.cycles-pp.cred_has_capability.selinux_vm_enough_memory.security_vm_enough_memory_mm.do_brk_flags.__x64_sys_brk 2.50 -0.1 2.38 perf-profile.calltrace.cycles-pp.remove_vma.do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.33 -0.1 2.23 ± 2% perf-profile.calltrace.cycles-pp.kmem_cache_alloc.vm_area_alloc.do_brk_flags.__x64_sys_brk.do_syscall_64 1.16 -0.1 1.07 ± 2% perf-profile.calltrace.cycles-pp.vma_compute_subtree_gap.do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.25 -0.1 2.15 perf-profile.calltrace.cycles-pp.__vma_link_rb.vma_link.do_brk_flags.__x64_sys_brk.do_syscall_64 1.49 -0.1 1.40 perf-profile.calltrace.cycles-pp.vma_compute_subtree_gap.__vma_link_rb.vma_link.do_brk_flags.__x64_sys_brk 1.06 -0.1 0.97 perf-profile.calltrace.cycles-pp.__vm_enough_memory.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.63 -0.1 1.55 perf-profile.calltrace.cycles-pp.avc_has_perm_noaudit.cred_has_capability.selinux_vm_enough_memory.security_vm_enough_memory_mm.do_brk_flags 1.15 -0.1 1.07 ± 2% perf-profile.calltrace.cycles-pp.up_write.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.91 -0.1 0.84 ± 2% perf-profile.calltrace.cycles-pp.tlb_gather_mmu.unmap_region.do_munmap.__x64_sys_brk.do_syscall_64 0.88 ± 3% -0.1 0.81 perf-profile.calltrace.cycles-pp.___might_sleep.down_write_killable.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 ± 2% -0.1 0.68 perf-profile.calltrace.cycles-pp.free_pgd_range.unmap_region.do_munmap.__x64_sys_brk.do_syscall_64 1.16 -0.1 1.10 perf-profile.calltrace.cycles-pp.vma_merge.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.27 -0.1 1.22 ± 2% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.do_munmap.__x64_sys_brk.do_syscall_64 1.13 -0.1 1.08 perf-profile.calltrace.cycles-pp.__vma_rb_erase.do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.37 ± 2% -0.0 1.32 perf-profile.calltrace.cycles-pp.cap_vm_enough_memory.security_vm_enough_memory_mm.do_brk_flags.__x64_sys_brk.do_syscall_64 1.58 -0.0 1.54 perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.get_unmapped_area.do_brk_flags.__x64_sys_brk.do_syscall_64 0.94 -0.0 0.90 perf-profile.calltrace.cycles-pp.kmem_cache_free.remove_vma.do_munmap.__x64_sys_brk.do_syscall_64 1.14 -0.0 1.10 perf-profile.calltrace.cycles-pp.perf_event_mmap_output.perf_iterate_sb.perf_event_mmap.do_brk_flags.__x64_sys_brk 0.73 -0.0 0.69 ± 2% perf-profile.calltrace.cycles-pp.cap_mmap_addr.security_mmap_addr.get_unmapped_area.do_brk_flags.__x64_sys_brk 0.96 -0.0 0.93 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_stage2 74.01 +0.1 74.13 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 72.69 +0.2 72.89 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 69.82 +0.3 70.16 perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.7 0.73 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.lru_add_drain_cpu.unmap_region.do_munmap.__x64_sys_brk 0.00 +0.8 0.83 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.lru_add_drain_cpu.unmap_region.do_munmap.__x64_sys_brk 0.00 +2.2 2.23 perf-profile.calltrace.cycles-pp._raw_spin_lock.lru_add_drain_cpu.unmap_region.do_munmap.__x64_sys_brk 27.94 +2.4 30.30 perf-profile.calltrace.cycles-pp.do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 14.50 +2.9 17.43 perf-profile.calltrace.cycles-pp.unmap_region.do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +4.4 4.43 perf-profile.calltrace.cycles-pp.lru_add_drain_cpu.unmap_region.do_munmap.__x64_sys_brk.do_syscall_64 31.57 -1.1 30.43 perf-profile.children.cycles-pp.do_brk_flags 0.82 ± 3% -0.8 0.07 perf-profile.children.cycles-pp.lru_add_drain 8.91 -0.7 8.26 perf-profile.children.cycles-pp.unmap_vmas 7.98 -0.4 7.59 perf-profile.children.cycles-pp.unmap_page_range 7.18 -0.4 6.81 perf-profile.children.cycles-pp.find_vma 12.90 -0.2 12.65 perf-profile.children.cycles-pp.syscall_return_via_sysret 3.27 -0.2 3.03 perf-profile.children.cycles-pp.vma_compute_subtree_gap 0.60 ± 3% -0.2 0.37 ± 3% perf-profile.children.cycles-pp.unmap_single_vma 3.62 ± 2% -0.2 3.40 perf-profile.children.cycles-pp.down_write_killable 5.06 -0.2 4.84 perf-profile.children.cycles-pp.security_vm_enough_memory_mm 1.71 -0.2 1.52 perf-profile.children.cycles-pp._cond_resched 4.74 -0.2 4.55 perf-profile.children.cycles-pp.get_unmapped_area 2.41 -0.2 2.25 ± 2% perf-profile.children.cycles-pp.vmacache_find 3.74 -0.2 3.58 perf-profile.children.cycles-pp.vma_link 3.27 -0.1 3.12 perf-profile.children.cycles-pp.selinux_vm_enough_memory 3.95 -0.1 3.80 ± 2% perf-profile.children.cycles-pp.vm_area_alloc 8.28 -0.1 8.14 perf-profile.children.cycles-pp.perf_event_mmap 2.94 -0.1 2.81 perf-profile.children.cycles-pp.___might_sleep 2.47 -0.1 2.35 perf-profile.children.cycles-pp.__vma_link_rb 2.43 -0.1 2.31 perf-profile.children.cycles-pp.cred_has_capability 0.34 ± 2% -0.1 0.22 ± 5% perf-profile.children.cycles-pp.arch_tlb_gather_mmu 2.64 -0.1 2.53 perf-profile.children.cycles-pp.remove_vma 0.62 ± 3% -0.1 0.53 ± 4% perf-profile.children.cycles-pp.vmacache_update 1.14 -0.1 1.04 perf-profile.children.cycles-pp.__vm_enough_memory 0.87 ± 2% -0.1 0.79 ± 3% perf-profile.children.cycles-pp.rcu_all_qs 2.46 -0.1 2.37 ± 2% perf-profile.children.cycles-pp.kmem_cache_alloc 1.22 -0.1 1.14 ± 2% perf-profile.children.cycles-pp.up_write 0.92 -0.1 0.84 ± 2% perf-profile.children.cycles-pp.tlb_gather_mmu 1.63 -0.1 1.55 perf-profile.children.cycles-pp.avc_has_perm_noaudit 1.26 ± 2% -0.1 1.19 perf-profile.children.cycles-pp.__vma_rb_erase 0.78 ± 2% -0.1 0.72 perf-profile.children.cycles-pp.free_pgd_range 1.19 -0.1 1.12 perf-profile.children.cycles-pp.vma_merge 1.41 ± 2% -0.1 1.35 perf-profile.children.cycles-pp.cap_vm_enough_memory 0.82 -0.1 0.77 perf-profile.children.cycles-pp.percpu_counter_add_batch 1.30 -0.0 1.25 ± 2% perf-profile.children.cycles-pp.tlb_finish_mmu 1.64 -0.0 1.59 perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.95 -0.0 0.90 perf-profile.children.cycles-pp.kmem_cache_free 0.44 -0.0 0.40 perf-profile.children.cycles-pp.strlen 1.22 -0.0 1.18 perf-profile.children.cycles-pp.perf_event_mmap_output 0.53 ± 3% -0.0 0.49 ± 3% perf-profile.children.cycles-pp.cap_capable 0.96 ± 2% -0.0 0.93 perf-profile.children.cycles-pp.entry_SYSCALL_64_stage2 0.37 ± 2% -0.0 0.35 ± 4% perf-profile.children.cycles-pp.tlb_flush_mmu_free 74.20 +0.1 74.31 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 72.84 +0.2 73.04 perf-profile.children.cycles-pp.do_syscall_64 70.18 +0.3 70.48 perf-profile.children.cycles-pp.__x64_sys_brk 0.00 +0.8 0.84 ± 3% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.00 +1.0 0.97 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.72 +2.3 3.02 perf-profile.children.cycles-pp._raw_spin_lock 28.33 +2.3 30.68 perf-profile.children.cycles-pp.do_munmap 14.65 +2.9 17.56 perf-profile.children.cycles-pp.unmap_region 0.66 ± 4% +3.9 4.55 perf-profile.children.cycles-pp.lru_add_drain_cpu 3.19 -0.4 2.78 ± 2% perf-profile.self.cycles-pp.__x64_sys_brk 5.09 -0.4 4.71 perf-profile.self.cycles-pp.unmap_page_range 12.87 -0.2 12.62 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.55 ± 4% -0.2 0.34 ± 4% perf-profile.self.cycles-pp.unmap_single_vma 2.81 -0.2 2.64 perf-profile.self.cycles-pp.vma_compute_subtree_gap 2.28 -0.2 2.12 perf-profile.self.cycles-pp.vmacache_find 2.84 -0.1 2.70 perf-profile.self.cycles-pp.___might_sleep 0.80 -0.1 0.69 ± 2% perf-profile.self.cycles-pp._cond_resched 3.24 -0.1 3.12 perf-profile.self.cycles-pp.do_brk_flags 0.33 ± 3% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.arch_tlb_gather_mmu 4.25 -0.1 4.14 perf-profile.self.cycles-pp.find_vma 2.52 -0.1 2.41 perf-profile.self.cycles-pp.do_munmap 2.43 ± 2% -0.1 2.33 perf-profile.self.cycles-pp.do_syscall_64 1.42 -0.1 1.32 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.56 ± 3% -0.1 0.47 ± 5% perf-profile.self.cycles-pp.vmacache_update 1.62 -0.1 1.54 perf-profile.self.cycles-pp.avc_has_perm_noaudit 1.14 -0.1 1.07 perf-profile.self.cycles-pp.up_write 1.14 -0.1 1.07 perf-profile.self.cycles-pp.vma_merge 0.77 ± 2% -0.1 0.71 ± 2% perf-profile.self.cycles-pp.free_pgd_range 0.78 -0.1 0.73 ± 2% perf-profile.self.cycles-pp.security_mmap_addr 0.77 -0.1 0.72 ± 3% perf-profile.self.cycles-pp.get_unmapped_area 0.77 -0.1 0.72 ± 4% perf-profile.self.cycles-pp.cred_has_capability 0.39 ± 2% -0.0 0.34 ± 3% perf-profile.self.cycles-pp.unmap_vmas 1.44 ± 2% -0.0 1.39 perf-profile.self.cycles-pp.vm_area_alloc 0.89 -0.0 0.84 perf-profile.self.cycles-pp.__vma_rb_erase 0.78 -0.0 0.73 perf-profile.self.cycles-pp.__vm_enough_memory 1.12 ± 2% -0.0 1.07 perf-profile.self.cycles-pp.__might_sleep 0.78 -0.0 0.73 perf-profile.self.cycles-pp.percpu_counter_add_batch 1.16 -0.0 1.12 perf-profile.self.cycles-pp.perf_event_mmap_output 0.94 -0.0 0.90 perf-profile.self.cycles-pp.kmem_cache_free 0.66 -0.0 0.62 perf-profile.self.cycles-pp.rcu_all_qs 0.49 ± 3% -0.0 0.46 ± 2% perf-profile.self.cycles-pp.cap_capable 0.40 ± 2% -0.0 0.36 ± 2% perf-profile.self.cycles-pp.strlen 0.65 -0.0 0.63 ± 2% perf-profile.self.cycles-pp.cap_mmap_addr 0.63 ± 4% +0.1 0.75 perf-profile.self.cycles-pp.lru_add_drain_cpu 0.00 +0.8 0.81 ± 2% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.00 +0.9 0.94 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.68 ± 2% +2.2 2.92 perf-profile.self.cycles-pp._raw_spin_lock will-it-scale.per_process_ops 760000 +-+----------------------------------------------------------------+ | .+. + +. : | 755000 +-+ +. .+.+.+ : | 750000 +-+ : +. : | |.+. : +. .+.. .+.+. | 745000 +-+ +..+.+ + + +..+. .+.| | +.+.+..+ | 740000 +-+ | | | 735000 +-+ | 730000 +-+ O | | O O O | 725000 O-O O O O O O O O O O O O | | O O O O O O O O | 720000 +-+----------------------------------------------------------------+ will-it-scale.workload 3.36e+07 +-+--------------------------------------------------------------+ 3.34e+07 +-+ .+. .+. .+.+ | | +.+..+.+ + +. : | 3.32e+07 +-+ +. + : | 3.3e+07 +-+ + + : | |.+.+..+.+ +.+.+.+..+.+. | 3.28e+07 +-+ +.+.+.+..+.+.+.| 3.26e+07 +-+ | 3.24e+07 +-+ | | | 3.22e+07 +-+ O O | 3.2e+07 +-O O O O O O O | O O O O O O O O O O O O O O | 3.18e+07 +-+ O O | 3.16e+07 +-+--------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** ivb44: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory ========================================================================================= bs/compiler/cpufreq_governor/disk/fs2/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode: 4k/gcc-7/performance/1HDD/nfsv4/ext4/sync/x86_64-rhel-7.2/100%/debian-x86_64-2018-04-03.cgz/300s/write/ivb44/128G/fio-basic/0x42d commit: f3c0b8ce48 (" dmaengine fix for dmaengine-fix-4.19-rc4") d884021f52 ("mm/swap: Add pagevec locking") f3c0b8ce48408b74 d884021f52609407c7943705b3 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | :2 50% 1:4 kmsg.usb#-#:device_descriptor_read,error :2 150% 3:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :2 150% 3:4 dmesg.WARNING:stack_recursion 0:2 15% 0:4 perf-profile.children.cycles-pp.io_schedule_timeout 0:2 17% 0:4 perf-profile.children.cycles-pp.schedule_timeout %stddev %change %stddev \ | \ 4.1e+09 ± 76% -50.6% 2.025e+09 ±134% cpuidle.C3.time 7382534 ± 46% -61.3% 2854923 ± 99% cpuidle.C3.usage 29610 -1.1% 29281 proc-vmstat.nr_slab_unreclaimable 797236 +6.3% 847312 ± 4% proc-vmstat.pgfree 7382736 ± 46% -61.3% 2854836 ± 99% turbostat.C3 28.31 ± 76% -14.3 13.97 ±134% turbostat.C3% 2520 ± 2% -20.2% 2010 ± 14% slabinfo.eventpoll_pwq.active_objs 2520 ± 2% -20.2% 2010 ± 14% slabinfo.eventpoll_pwq.num_objs 350.00 ± 9% -27.1% 255.00 ± 17% slabinfo.secpath_cache.active_objs 350.00 ± 9% -27.1% 255.00 ± 17% slabinfo.secpath_cache.num_objs 1972 ± 2% -27.8% 1425 ± 24% slabinfo.task_group.active_objs 1972 ± 2% -27.8% 1425 ± 24% slabinfo.task_group.num_objs 86415 +65.8% 143246 ± 2% numa-meminfo.node0.Active 80095 ± 7% +69.3% 135603 ± 3% numa-meminfo.node0.Active(anon) 78653 ± 9% +69.4% 133251 ± 3% numa-meminfo.node0.AnonPages 8516 ± 75% +1396.8% 127471 ± 65% numa-meminfo.node0.Inactive(anon) 9990 ± 53% +1200.9% 129966 ± 63% numa-meminfo.node0.Shmem 198827 -27.4% 144403 ± 2% numa-meminfo.node1.Active 190365 ± 3% -27.9% 137259 ± 2% numa-meminfo.node1.Active(anon) 187188 ± 4% -28.1% 134639 ± 2% numa-meminfo.node1.AnonPages 231370 ± 2% -51.6% 112031 ± 74% numa-meminfo.node1.Inactive(anon) 240802 -48.7% 123588 ± 66% numa-meminfo.node1.Mapped 234636 ± 2% -51.3% 114274 ± 72% numa-meminfo.node1.Shmem 165.66 ± 44% -55.4% 73.84 ± 33% sched_debug.cfs_rq:/.load_avg.avg 304.80 ± 34% -37.0% 191.89 ± 6% sched_debug.cfs_rq:/.removed.load_avg.max 14073 ± 33% -36.8% 8887 ± 6% sched_debug.cfs_rq:/.removed.runnable_sum.max 125.80 ± 39% -60.3% 49.99 ± 26% sched_debug.cfs_rq:/.removed.util_avg.max 21.17 ± 48% -50.6% 10.45 ± 57% sched_debug.cfs_rq:/.removed.util_avg.stddev 506292 ± 2% -33.1% 338712 ± 18% sched_debug.cpu.avg_idle.min 103884 ± 5% +368.2% 486437 ± 71% sched_debug.cpu.avg_idle.stddev 7.96 +14.2% 9.10 ± 5% sched_debug.cpu.cpu_load[3].avg 246.40 ± 19% +37.1% 337.89 ± 9% sched_debug.cpu.cpu_load[3].max 37.79 ± 12% +29.3% 48.86 ± 9% sched_debug.cpu.cpu_load[3].stddev 160.10 ± 4% +43.3% 229.43 ± 13% sched_debug.cpu.cpu_load[4].max 24.69 +34.6% 33.25 ± 13% sched_debug.cpu.cpu_load[4].stddev 0.00 ± 6% +32.1% 0.00 ± 11% sched_debug.cpu.next_balance.stddev 20024 ± 7% +69.3% 33898 ± 3% numa-vmstat.node0.nr_active_anon 19661 ± 9% +69.4% 33311 ± 3% numa-vmstat.node0.nr_anon_pages 2128 ± 75% +1395.3% 31827 ± 65% numa-vmstat.node0.nr_inactive_anon 2499 ± 52% +1198.6% 32451 ± 63% numa-vmstat.node0.nr_shmem 20024 ± 7% +69.3% 33898 ± 3% numa-vmstat.node0.nr_zone_active_anon 2128 ± 75% +1395.3% 31827 ± 65% numa-vmstat.node0.nr_zone_inactive_anon 16608 ± 4% -26.0% 12293 ± 28% numa-vmstat.node1 47590 ± 3% -27.9% 34315 ± 2% numa-vmstat.node1.nr_active_anon 46795 ± 4% -28.1% 33660 ± 2% numa-vmstat.node1.nr_anon_pages 57747 ± 2% -51.5% 27994 ± 74% numa-vmstat.node1.nr_inactive_anon 60219 -48.6% 30929 ± 66% numa-vmstat.node1.nr_mapped 58564 ± 2% -51.2% 28556 ± 72% numa-vmstat.node1.nr_shmem 47590 ± 3% -27.9% 34315 ± 2% numa-vmstat.node1.nr_zone_active_anon 57747 ± 2% -51.5% 27994 ± 74% numa-vmstat.node1.nr_zone_inactive_anon 75.04 ± 6% -3.9 71.14 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 87.77 -2.6 85.21 perf-profile.calltrace.cycles-pp.secondary_startup_64 1.31 ± 3% -0.2 1.11 ± 12% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write 1.24 ± 4% -0.2 1.08 ± 13% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter 1.24 ± 4% -0.1 1.09 ± 12% perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter 1.25 +0.1 1.33 ± 3% perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 78.72 -3.3 75.42 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 87.77 -2.6 85.21 perf-profile.children.cycles-pp.secondary_startup_64 87.77 -2.6 85.21 perf-profile.children.cycles-pp.cpu_startup_entry 87.82 -2.5 85.29 perf-profile.children.cycles-pp.do_idle 1.37 ± 49% -0.7 0.62 ± 19% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 1.31 ± 3% -0.2 1.11 ± 12% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.30 ± 9% -0.2 0.14 ± 30% perf-profile.children.cycles-pp.__hrtimer_get_next_event 1.25 ± 4% -0.1 1.10 ± 12% perf-profile.children.cycles-pp.copyin 1.25 ± 3% -0.1 1.11 ± 14% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.34 ± 18% -0.1 0.25 ± 28% perf-profile.children.cycles-pp.timerqueue_del 0.18 ± 13% -0.1 0.10 ± 58% perf-profile.children.cycles-pp.nr_iowait_cpu 0.24 ± 19% -0.1 0.15 ± 28% perf-profile.children.cycles-pp.interrupt_entry 0.26 ± 17% -0.1 0.19 ± 16% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.10 ± 40% -0.1 0.03 ±105% perf-profile.children.cycles-pp.__list_del_entry_valid 0.10 ± 40% -0.1 0.03 ±105% perf-profile.children.cycles-pp.pick_next_task_fair 0.07 ± 20% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.mem_cgroup_try_charge 0.10 ± 14% -0.0 0.07 ± 26% perf-profile.children.cycles-pp.can_stop_idle_tick 0.20 ± 2% +0.0 0.24 ± 3% perf-profile.children.cycles-pp.radix_tree_next_chunk 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.alloc_pages_current 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.sys_imageblit 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.mga_imageblit 0.10 ± 30% +0.1 0.17 ± 4% perf-profile.children.cycles-pp.rcu_needs_cpu 0.06 ± 16% +0.1 0.13 ± 42% perf-profile.children.cycles-pp.__wake_up_common_lock 0.08 ± 5% +0.1 0.17 ± 28% perf-profile.children.cycles-pp.__x64_sys_execve 0.08 ± 5% +0.1 0.17 ± 25% perf-profile.children.cycles-pp.execve 0.08 ± 6% +0.1 0.16 ± 26% perf-profile.children.cycles-pp.__do_execve_file 0.00 +0.1 0.09 ± 40% perf-profile.children.cycles-pp.proc_reg_read 0.00 +0.1 0.11 ± 37% perf-profile.children.cycles-pp.seq_read 0.00 +0.1 0.13 ± 42% perf-profile.children.cycles-pp.__mutex_lock 0.30 ± 6% +0.2 0.46 ± 28% perf-profile.children.cycles-pp._raw_spin_lock 0.26 +0.2 0.45 ± 23% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.21 ± 23% -0.1 0.08 ± 15% perf-profile.self.cycles-pp.__hrtimer_get_next_event 0.18 ± 13% -0.1 0.09 ± 59% perf-profile.self.cycles-pp.nr_iowait_cpu 0.24 ± 19% -0.1 0.14 ± 30% perf-profile.self.cycles-pp.interrupt_entry 0.12 ± 28% -0.1 0.04 ±110% perf-profile.self.cycles-pp.timerqueue_del 0.24 ± 16% -0.1 0.16 ± 16% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.22 ± 4% -0.1 0.15 ± 18% perf-profile.self.cycles-pp.get_next_timer_interrupt 0.22 ± 4% -0.0 0.18 ± 12% perf-profile.self.cycles-pp.timerqueue_add 0.10 ± 14% -0.0 0.07 ± 26% perf-profile.self.cycles-pp.can_stop_idle_tick 0.20 ± 2% +0.0 0.24 ± 3% perf-profile.self.cycles-pp.radix_tree_next_chunk 0.10 ± 14% +0.1 0.17 ± 15% perf-profile.self.cycles-pp.do_syscall_64 0.09 +0.1 0.15 ± 24% perf-profile.self.cycles-pp.__update_load_avg_se 0.00 +0.1 0.07 ± 23% perf-profile.self.cycles-pp.sys_imageblit 0.00 +0.1 0.07 ± 31% perf-profile.self.cycles-pp.ext4_block_write_begin 0.10 ± 26% +0.1 0.17 ± 4% perf-profile.self.cycles-pp.rcu_needs_cpu 0.00 +0.1 0.07 ± 38% perf-profile.self.cycles-pp.hrtimer_get_next_event 0.00 +0.1 0.08 ± 19% perf-profile.self.cycles-pp.__mutex_lock 0.23 ± 2% +0.2 0.39 ± 23% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.29 +0.2 0.44 ± 28% perf-profile.self.cycles-pp._raw_spin_lock 0.50 ± 2% +0.2 0.68 ± 16% perf-profile.self.cycles-pp.__next_timer_interrupt *************************************************************************************************** lkp-hsw-ep2: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 128G memory ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode: 4k/gcc-7/performance/1SSD/ext4/sync/x86_64-rhel-7.2/64/debian-x86_64-2018-04-03.cgz/300s/randwrite/lkp-hsw-ep2/512g/fio-basic/0x3d commit: f3c0b8ce48 (" dmaengine fix for dmaengine-fix-4.19-rc4") d884021f52 ("mm/swap: Add pagevec locking") f3c0b8ce48408b74 d884021f52609407c7943705b3 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | :2 50% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x :2 50% 1:4 dmesg.WARNING:at_ip_native_sched_clock/0x 0:2 22% 1:4 perf-profile.children.cycles-pp.schedule_timeout 1:2 60% 2:4 perf-profile.children.cycles-pp.io_schedule_timeout %stddev %change %stddev \ | \ 5.06 +1.6 6.65 ± 11% fio.latency_100us% 0.86 ± 3% +0.1 0.99 ± 3% fio.latency_250us% 256.63 +7.7% 276.50 fio.time.system_time 45.00 +7.8% 48.50 ± 3% fio.write_clat_90%_us 58.00 +7.8% 62.50 ± 3% fio.write_clat_95%_us 14118389 ± 51% +79.1% 25292065 ± 9% cpuidle.C6.usage 2571 -1.1% 2543 proc-vmstat.nr_page_table_pages 19228 ± 79% -80.4% 3774 ± 3% proc-vmstat.numa_pte_updates 14115889 ± 51% +79.2% 25289182 ± 9% turbostat.C6 0.21 ± 17% -64.6% 0.07 ± 40% turbostat.Pkg%pc3 7321922 ± 2% -8.3% 6717403 ± 6% numa-numastat.node0.local_node 436778 ± 29% -67.9% 140332 ±173% numa-numastat.node0.numa_foreign 7336061 ± 2% -8.4% 6721069 ± 6% numa-numastat.node0.numa_hit 436778 ± 29% -67.9% 140332 ±173% numa-numastat.node1.numa_miss 865.00 ± 13% -20.7% 686.00 ± 26% slabinfo.nfs_commit_data.active_objs 865.00 ± 13% -20.7% 686.00 ± 26% slabinfo.nfs_commit_data.num_objs 2627 ± 24% -22.1% 2045 ± 34% slabinfo.scsi_sense_cache.active_objs 2628 ± 24% -21.4% 2065 ± 32% slabinfo.scsi_sense_cache.num_objs 51178 ± 4% -7.7% 47251 ± 4% slabinfo.vm_area_struct.active_objs 1287 ± 4% -7.8% 1186 ± 4% slabinfo.vm_area_struct.active_slabs 51507 ± 4% -7.8% 47465 ± 4% slabinfo.vm_area_struct.num_objs 1287 ± 4% -7.8% 1186 ± 4% slabinfo.vm_area_struct.num_slabs 2194 ± 3% -119.1% -419.80 sched_debug.cfs_rq:/.spread0.avg 5.71 ± 3% -14.8% 4.87 ± 12% sched_debug.cfs_rq:/.util_est_enqueued.avg 324306 ± 10% -9.0% 295152 ± 4% sched_debug.cpu.nr_load_updates.max 5621 ± 68% -69.7% 1702 ± 12% sched_debug.cpu.nr_load_updates.stddev 11.00 ± 13% -18.6% 8.95 ± 20% sched_debug.cpu.nr_uninterruptible.max 1027 ± 7% +45.0% 1490 ± 19% sched_debug.cpu.ttwu_count.min 546.55 -30.2% 381.53 ± 28% sched_debug.cpu.ttwu_local.min 0.00 ± 49% +122.3% 0.00 ± 30% sched_debug.rt_rq:/.rt_time.avg 0.02 ± 49% +122.3% 0.03 ± 30% sched_debug.rt_rq:/.rt_time.max 0.00 ± 49% +122.3% 0.00 ± 30% sched_debug.rt_rq:/.rt_time.stddev 289208 ± 15% -26.5% 212596 ± 23% numa-meminfo.node0.Active 190569 -33.4% 126938 ± 23% numa-meminfo.node0.Active(anon) 157915 ± 2% -36.2% 100710 ± 26% numa-meminfo.node0.AnonHugePages 190832 -35.5% 123030 ± 22% numa-meminfo.node0.AnonPages 7548 -16.5% 6301 ± 4% numa-meminfo.node0.KernelStack 8824 ± 2% -64.8% 3105 ± 78% numa-meminfo.node0.PageTables 69548 -11.1% 61798 ± 8% numa-meminfo.node0.SUnreclaim 87222 +74.1% 151868 ± 18% numa-meminfo.node1.Active(anon) 62739 ± 2% +93.2% 121230 ± 22% numa-meminfo.node1.AnonHugePages 79918 ± 2% +85.9% 148605 ± 18% numa-meminfo.node1.AnonPages 15315 +531.7% 96745 ± 49% numa-meminfo.node1.Inactive(anon) 4801 ± 4% +23.8% 5941 ± 5% numa-meminfo.node1.KernelStack 16370 +510.7% 99979 ± 48% numa-meminfo.node1.Mapped 1468 ± 12% +381.8% 7073 ± 34% numa-meminfo.node1.PageTables 47642 -33.4% 31737 ± 23% numa-vmstat.node0.nr_active_anon 47710 -35.5% 30760 ± 22% numa-vmstat.node0.nr_anon_pages 7549 -16.5% 6301 ± 4% numa-vmstat.node0.nr_kernel_stack 2206 ± 2% -64.8% 776.75 ± 78% numa-vmstat.node0.nr_page_table_pages 17386 -11.1% 15449 ± 8% numa-vmstat.node0.nr_slab_unreclaimable 3677874 -11.3% 3261486 ± 3% numa-vmstat.node0.nr_written 47642 -33.4% 31737 ± 23% numa-vmstat.node0.nr_zone_active_anon 432470 ± 29% -67.9% 138936 ±173% numa-vmstat.node0.numa_foreign 6791323 -12.1% 5968091 ± 4% numa-vmstat.node0.numa_hit 6776182 -12.0% 5962755 ± 4% numa-vmstat.node0.numa_local 21807 +74.1% 37971 ± 18% numa-vmstat.node1.nr_active_anon 19981 ± 2% +86.0% 37155 ± 18% numa-vmstat.node1.nr_anon_pages 3829 +532.5% 24221 ± 49% numa-vmstat.node1.nr_inactive_anon 4804 ± 4% +23.6% 5939 ± 5% numa-vmstat.node1.nr_kernel_stack 4094 +513.7% 25127 ± 48% numa-vmstat.node1.nr_mapped 367.50 ± 12% +381.1% 1768 ± 34% numa-vmstat.node1.nr_page_table_pages 3117498 ± 3% +14.3% 3562760 ± 4% numa-vmstat.node1.nr_written 21807 +74.1% 37971 ± 18% numa-vmstat.node1.nr_zone_active_anon 3829 +532.5% 24221 ± 49% numa-vmstat.node1.nr_zone_inactive_anon 5695266 +15.9% 6602176 ± 6% numa-vmstat.node1.numa_hit 5557113 +16.1% 6453474 ± 6% numa-vmstat.node1.numa_local 432474 ± 29% -67.9% 138938 ±173% numa-vmstat.node1.numa_miss 19.04 -4.1 14.96 ± 16% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 16.23 ± 4% -3.6 12.66 ± 18% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 6.30 ± 11% -2.2 4.15 ± 34% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 4.96 ± 14% -1.7 3.30 ± 39% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 3.58 ± 3% -1.0 2.55 ± 19% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 5.03 ± 5% -0.8 4.21 ± 12% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 3.06 ± 2% -0.8 2.26 ± 18% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 2.86 ± 3% -0.8 2.09 ± 18% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.50 ± 23% -0.7 0.76 ± 80% perf-profile.calltrace.cycles-pp.rcu_process_callbacks.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 2.65 ± 22% -0.7 1.92 ± 38% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 1.66 ± 34% -0.7 0.96 ± 75% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 3.17 -0.6 2.54 ± 15% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 2.56 -0.6 2.00 ± 16% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 1.58 -0.4 1.15 ± 24% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.77 ± 9% -0.4 0.33 ±104% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt 0.78 ± 3% -0.3 0.46 ± 58% perf-profile.calltrace.cycles-pp.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 18.20 ± 4% -3.9 14.30 ± 16% perf-profile.children.cycles-pp.apic_timer_interrupt 16.71 ± 6% -3.7 13.05 ± 18% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 8.34 ± 14% -1.6 6.73 ± 21% perf-profile.children.cycles-pp.irq_exit 8.96 ± 4% -1.5 7.42 ± 14% perf-profile.children.cycles-pp.hrtimer_interrupt 3.20 ± 19% -1.2 2.01 ± 36% perf-profile.children.cycles-pp.ktime_get 3.79 -1.1 2.68 ± 19% perf-profile.children.cycles-pp.tick_sched_timer 3.26 ± 2% -0.9 2.38 ± 18% perf-profile.children.cycles-pp.tick_sched_handle 3.05 ± 2% -0.8 2.21 ± 18% perf-profile.children.cycles-pp.update_process_times 5.28 -0.8 4.47 ± 11% perf-profile.children.cycles-pp.__hrtimer_run_queues 5.86 ± 2% -0.8 5.08 ± 13% perf-profile.children.cycles-pp.menu_select 2.71 ± 23% -0.7 1.97 ± 37% perf-profile.children.cycles-pp.clockevents_program_event 3.24 -0.7 2.57 ± 15% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 1.54 ± 24% -0.6 0.90 ± 55% perf-profile.children.cycles-pp.rcu_process_callbacks 2.62 -0.6 2.03 ± 16% perf-profile.children.cycles-pp.tick_nohz_next_event 0.86 ± 11% -0.5 0.40 ± 31% perf-profile.children.cycles-pp.tick_nohz_irq_exit 0.53 ± 34% -0.4 0.09 ± 39% perf-profile.children.cycles-pp.timekeeping_max_deferment 0.71 ± 14% -0.3 0.42 ± 51% perf-profile.children.cycles-pp.note_gp_changes 0.80 -0.2 0.58 ± 19% perf-profile.children.cycles-pp.run_timer_softirq 0.23 ± 53% -0.2 0.06 ± 64% perf-profile.children.cycles-pp.page_fault 0.34 ± 49% -0.2 0.16 ±111% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.23 ± 52% -0.2 0.05 ± 62% perf-profile.children.cycles-pp.do_page_fault 0.22 ± 55% -0.2 0.05 ± 62% perf-profile.children.cycles-pp.__do_page_fault 0.57 ± 12% -0.1 0.43 ± 31% perf-profile.children.cycles-pp.schedule 0.67 ± 5% -0.1 0.53 ± 23% perf-profile.children.cycles-pp.rcu_check_callbacks 0.68 ± 4% -0.1 0.54 ± 7% perf-profile.children.cycles-pp._raw_spin_lock 0.52 ± 6% -0.1 0.42 ± 15% perf-profile.children.cycles-pp.unwind_next_frame 0.19 ± 21% -0.1 0.10 ± 24% perf-profile.children.cycles-pp.raise_softirq 0.16 ± 9% -0.1 0.08 ± 15% perf-profile.children.cycles-pp.cpu_load_update 0.56 -0.1 0.49 ± 10% perf-profile.children.cycles-pp.__entry_SYSCALL_64_trampoline 0.17 ± 23% -0.1 0.10 ± 29% perf-profile.children.cycles-pp.irq_entries_start 0.11 ± 4% -0.1 0.05 ± 67% perf-profile.children.cycles-pp.orc_find 0.23 ± 6% -0.1 0.18 ± 8% perf-profile.children.cycles-pp.rcu_eqs_enter 0.17 ± 3% -0.0 0.12 ± 25% perf-profile.children.cycles-pp.calc_global_load_tick 0.09 ± 17% -0.0 0.04 ±103% perf-profile.children.cycles-pp.blk_queue_start_tag 0.15 ± 6% -0.0 0.12 ± 23% perf-profile.children.cycles-pp.rcu_nmi_exit 0.09 -0.0 0.06 ± 59% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.09 ± 22% -0.0 0.06 ± 22% perf-profile.children.cycles-pp.update_rt_rq_load_avg 0.10 ± 5% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.08 ± 12% -0.0 0.06 ± 17% perf-profile.children.cycles-pp.task_tick_idle 0.10 ± 5% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.get_iowait_load 0.10 ± 5% +0.0 0.14 ± 15% perf-profile.children.cycles-pp.nr_iowait_cpu 0.00 +0.1 0.06 ± 20% perf-profile.children.cycles-pp.blk_account_io_done 0.09 ± 17% +0.1 0.16 ± 21% perf-profile.children.cycles-pp.blk_finish_request 0.10 ± 26% +0.1 0.22 ± 45% perf-profile.children.cycles-pp.kmem_cache_free 0.30 ± 6% +0.2 0.48 ± 39% perf-profile.children.cycles-pp.ext4_end_bio 0.00 +0.2 0.18 ± 27% perf-profile.children.cycles-pp.pipe_read 0.75 ± 6% +0.2 0.93 ± 8% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.07 ± 7% +0.2 0.29 ± 26% perf-profile.children.cycles-pp.__vfs_read 0.07 ± 20% +0.2 0.32 ± 24% perf-profile.children.cycles-pp.vfs_read 0.07 ± 20% +0.3 0.34 ± 24% perf-profile.children.cycles-pp.ksys_read 0.03 ±100% +0.3 0.34 ± 23% perf-profile.children.cycles-pp.read 2.52 ± 25% -1.2 1.35 ± 56% perf-profile.self.cycles-pp.ktime_get 0.53 ± 35% -0.4 0.08 ± 42% perf-profile.self.cycles-pp.timekeeping_max_deferment 2.04 ± 9% -0.3 1.72 ± 6% perf-profile.self.cycles-pp.cpuidle_enter_state 0.42 ± 12% -0.2 0.23 ± 46% perf-profile.self.cycles-pp.__softirqentry_text_start 0.44 ± 21% -0.2 0.25 ± 50% perf-profile.self.cycles-pp.note_gp_changes 0.34 ± 49% -0.2 0.16 ±111% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.41 -0.2 0.24 ± 22% perf-profile.self.cycles-pp.run_timer_softirq 0.29 -0.2 0.13 ± 34% perf-profile.self.cycles-pp.tick_sched_timer 0.67 ± 3% -0.1 0.53 ± 8% perf-profile.self.cycles-pp._raw_spin_lock 0.38 ± 6% -0.1 0.25 ± 35% perf-profile.self.cycles-pp.update_blocked_averages 0.18 ± 18% -0.1 0.09 ± 27% perf-profile.self.cycles-pp.raise_softirq 0.21 ± 2% -0.1 0.12 ± 21% perf-profile.self.cycles-pp.load_balance 0.16 ± 9% -0.1 0.08 ± 15% perf-profile.self.cycles-pp.cpu_load_update 0.28 ± 14% -0.1 0.21 ± 24% perf-profile.self.cycles-pp.rcu_process_callbacks 0.26 ± 21% -0.1 0.18 ± 10% perf-profile.self.cycles-pp.idle_cpu 0.54 -0.1 0.46 ± 12% perf-profile.self.cycles-pp.__entry_SYSCALL_64_trampoline 0.18 ± 8% -0.1 0.12 ± 20% perf-profile.self.cycles-pp.scheduler_tick 0.23 ± 6% -0.1 0.17 ± 4% perf-profile.self.cycles-pp.rcu_eqs_enter 0.11 -0.1 0.05 ± 70% perf-profile.self.cycles-pp.orc_find 0.09 ± 17% -0.1 0.03 ±102% perf-profile.self.cycles-pp.update_rt_rq_load_avg 0.09 ± 22% -0.1 0.04 ±102% perf-profile.self.cycles-pp.tick_nohz_get_sleep_length 0.15 ± 17% -0.1 0.09 ± 24% perf-profile.self.cycles-pp.rcu_needs_cpu 0.35 ± 9% -0.1 0.30 ± 9% perf-profile.self.cycles-pp.tick_nohz_next_event 0.15 ± 6% -0.0 0.12 ± 23% perf-profile.self.cycles-pp.rcu_nmi_exit 0.07 ± 20% -0.0 0.04 ± 63% perf-profile.self.cycles-pp.ext4_bio_write_page 0.10 ± 5% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.get_iowait_load 0.09 ± 11% +0.0 0.14 ± 15% perf-profile.self.cycles-pp.nr_iowait_cpu 0.15 ± 33% +0.1 0.24 ± 17% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen