1:32 a.m.
Greeting,
FYI, we noticed a -25.0% regression of phoronix-test-suite.aom-av1.0.frames_per_second due
to commit:
commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa: replace
runnable_load_avg by load_avg")
https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git sched-lbnuma-rewrite-v2r6
in testcase: phoronix-test-suite
on test machine: 16 threads Intel(R) Xeon(R) CPU X5570 @ 2.93GHz with 48G memory
with following parameters:
test: aom-av1-1.2.0
cpufreq_governor: performance
ucode: 0x1d
test-description: The Phoronix Test Suite is the most comprehensive testing and
benchmarking platform available that provides an extensible framework for which new tests
can be easily added.
test-url: http://www.phoronix-test-suite.com/
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
commit:
91d7637912 ("sched/fair: reorder enqueue/dequeue_task_fair path")
f6183ef98b ("sched/numa: replace runnable_load_avg by load_avg")
91d76379120e4640 f6183ef98bba39a5de563d4ab4b
---------------- ---------------------------
%stddev %change %stddev
\ | \
0.04 -25.0% 0.03
phoronix-test-suite.aom-av1.0.frames_per_second
1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time
1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time.max
1935 -19.6% 1556 meminfo.max_used_kB
1.518e+10 ± 29% +39.9% 2.124e+10 cpuidle.C1E.time
33131701 ± 34% +35.9% 45011103 ± 2% cpuidle.C1E.usage
672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.active_objs
672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.num_objs
67.50 -2.2% 66.00 vmstat.cpu.id
31.00 +6.5% 33.00 vmstat.cpu.us
467.75 +5.0% 491.00 proc-vmstat.nr_mlock
484.75 +4.8% 508.00 proc-vmstat.nr_unevictable
484.75 +4.8% 508.00 proc-vmstat.nr_zone_unevictable
3492646 ± 7% +67.4% 5846384 ± 3% proc-vmstat.numa_hint_faults
2478506 ± 6% +127.8% 5646434 ± 4% proc-vmstat.numa_hint_faults_local
3650509 +13.5% 4141674 proc-vmstat.numa_hit
76734 +98.1% 151980 ± 2% proc-vmstat.numa_huge_pte_updates
3647139 +13.5% 4138249 proc-vmstat.numa_local
3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.numa_pages_migrated
42828088 +95.3% 83628988 ± 2% proc-vmstat.numa_pte_updates
7766914 ± 3% -25.6% 5776215 proc-vmstat.pgalloc_normal
7771023 ± 3% +43.5% 11147904 ± 2% proc-vmstat.pgfault
7716195 ± 3% -25.8% 5726088 proc-vmstat.pgfree
1152 ± 19% -88.9% 128.00 ±173% proc-vmstat.pgmigrate_fail
3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.pgmigrate_success
0.13 ±173% +0.5 0.63 ± 7%
perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
0.14 ±173% +0.5 0.67 ± 8%
perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
4.40 ± 47% +3.5 7.87 ± 16%
perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
4.40 ± 47% +3.5 7.87 ± 16%
perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
0.05 ± 67% +0.1 0.12 ± 17% perf-profile.children.cycles-pp.ktime_get
0.01 ±173% +0.1 0.10 ± 27%
perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.00 +0.1 0.10 ± 36% perf-profile.children.cycles-pp.rcu_core
0.32 ± 86% +0.5 0.85 ± 54%
perf-profile.children.cycles-pp.__libc_start_main
0.60 ± 46% +0.6 1.19 ± 41%
perf-profile.children.cycles-pp.do_syscall_64
0.60 ± 46% +0.6 1.19 ± 41%
perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
0.34 ± 82% +0.6 0.94 ± 51% perf-profile.children.cycles-pp.vfs_read
0.34 ± 82% +0.6 0.94 ± 51% perf-profile.children.cycles-pp.ksys_read
0.29 ± 96% +0.6 0.90 ± 53%
perf-profile.children.cycles-pp.perf_event_read
0.29 ± 96% +0.6 0.90 ± 53%
perf-profile.children.cycles-pp.smp_call_function_single
0.30 ± 96% +0.6 0.91 ± 53% perf-profile.children.cycles-pp.__libc_read
0.29 ± 96% +0.6 0.91 ± 52% perf-profile.children.cycles-pp.perf_read
4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.children.cycles-pp.start_kernel
0.01 ±173% +0.1 0.07 ± 19%
perf-profile.self.cycles-pp.cpuidle_enter_state
0.29 ± 98% +0.6 0.90 ± 53%
perf-profile.self.cycles-pp.smp_call_function_single
161465 ± 7% +66.5% 268796 ± 16% softirqs.CPU0.SCHED
154149 ± 4% +57.3% 242412 ± 5% softirqs.CPU1.SCHED
217783 ± 10% +17.4% 255655 ± 3% softirqs.CPU10.RCU
685538 ± 6% +16.3% 797133 ± 3% softirqs.CPU10.TIMER
647482 +29.6% 839256 ± 14% softirqs.CPU11.TIMER
215207 ± 9% +19.8% 257893 ± 5% softirqs.CPU12.RCU
676241 ± 4% +20.5% 814890 softirqs.CPU12.TIMER
218513 ± 8% +17.8% 257505 ± 4% softirqs.CPU14.RCU
665815 ± 5% +20.2% 800244 ± 3% softirqs.CPU14.TIMER
217764 ± 10% +16.0% 252579 ± 2% softirqs.CPU15.RCU
671996 ± 2% +18.0% 792743 ± 10% softirqs.CPU15.TIMER
216810 ± 10% +16.2% 251864 ± 3% softirqs.CPU2.RCU
168050 ± 8% +53.2% 257377 ± 4% softirqs.CPU2.SCHED
687608 ± 7% +13.9% 783359 ± 5% softirqs.CPU2.TIMER
214659 ± 10% +17.4% 252030 ± 2% softirqs.CPU3.RCU
201381 ± 8% +26.8% 255401 ± 10% softirqs.CPU3.SCHED
218567 ± 9% +15.1% 251588 ± 4% softirqs.CPU4.RCU
162228 ± 14% +39.7% 226690 ± 21% softirqs.CPU4.SCHED
686728 ± 7% +15.4% 792699 ± 4% softirqs.CPU4.TIMER
598763 ± 4% +33.1% 796983 ± 15% softirqs.CPU5.TIMER
219495 ± 10% +15.8% 254260 ± 4% softirqs.CPU6.RCU
685621 ± 6% +15.4% 790913 ± 4% softirqs.CPU6.TIMER
213675 ± 10% +18.1% 252314 ± 2% softirqs.CPU7.RCU
607580 ± 4% +27.7% 776035 ± 7% softirqs.CPU7.TIMER
217328 ± 10% +17.9% 256261 ± 4% softirqs.CPU8.RCU
672277 ± 8% +18.4% 795798 ± 3% softirqs.CPU8.TIMER
662696 ± 15% +22.2% 809936 ± 6% softirqs.CPU9.TIMER
3471815 ± 10% +16.7% 4051741 ± 2% softirqs.RCU
2788089 +21.0% 3372673 ± 2% softirqs.SCHED
10668573 ± 2% +17.5% 12539415 ± 5% softirqs.TIMER
250541 +31.8% 330137 ± 3% sched_debug.cfs_rq:/.exec_clock.avg
446877 ± 7% +68.2% 751493 ± 18% sched_debug.cfs_rq:/.exec_clock.max
112331 ± 17% +122.7% 250151 ± 35% sched_debug.cfs_rq:/.exec_clock.stddev
1517645 +35.0% 2049553 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg
2633991 ± 6% +65.2% 4352247 ± 18% sched_debug.cfs_rq:/.min_vruntime.max
684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.min_vruntime.stddev
0.46 ± 2% +8.1% 0.50 ± 4% sched_debug.cfs_rq:/.nr_running.avg
-292907 -321.0% 647293 ± 49% sched_debug.cfs_rq:/.spread0.avg
823435 ± 35% +258.3% 2949987 ± 32% sched_debug.cfs_rq:/.spread0.max
684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.spread0.stddev
389.08 +10.3% 429.08 ± 3% sched_debug.cfs_rq:/.util_avg.avg
282.99 ± 2% +13.1% 320.08 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg
383.98 ± 2% +8.6% 416.84 ± 3%
sched_debug.cfs_rq:/.util_est_enqueued.stddev
842328 +25.8% 1059503 sched_debug.cpu.clock.avg
842329 +25.8% 1059503 sched_debug.cpu.clock.max
842327 +25.8% 1059502 sched_debug.cpu.clock.min
1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock.stddev
842328 +25.8% 1059503 sched_debug.cpu.clock_task.avg
842329 +25.8% 1059503 sched_debug.cpu.clock_task.max
842327 +25.8% 1059502 sched_debug.cpu.clock_task.min
1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock_task.stddev
6159 ± 3% +29.3% 7966 ± 6% sched_debug.cpu.curr->pid.avg
21434 +25.0% 26795 sched_debug.cpu.curr->pid.max
7691 ± 2% +26.8% 9755 sched_debug.cpu.curr->pid.stddev
57507 ± 2% +27.2% 73171 sched_debug.cpu.nr_switches.avg
98634 ± 8% +51.9% 149817 ± 22% sched_debug.cpu.nr_switches.max
21105 ± 18% +85.3% 39113 ± 41% sched_debug.cpu.nr_switches.stddev
52692 ± 2% +29.9% 68458 sched_debug.cpu.sched_count.avg
94966 ± 8% +50.3% 142769 ± 25% sched_debug.cpu.sched_count.max
25787 ± 2% +27.6% 32917 sched_debug.cpu.sched_goidle.avg
47100 ± 8% +48.8% 70064 ± 26% sched_debug.cpu.sched_goidle.max
25109 ± 2% +28.2% 32200 sched_debug.cpu.ttwu_count.avg
47736 ± 4% +55.9% 74430 ± 9% sched_debug.cpu.ttwu_count.max
10778 ± 13% +80.8% 19490 ± 29% sched_debug.cpu.ttwu_count.stddev
11888 ± 4% +30.1% 15462 sched_debug.cpu.ttwu_local.avg
842327 +25.8% 1059502 sched_debug.cpu_clk
839741 +25.9% 1056916 sched_debug.ktime
843284 +25.8% 1060461 sched_debug.sched_clk
1307 +27.3% 1663 ± 10% interrupts.35:PCI-MSI.524289-edge.eth0-rx-0
884.50 ± 2% +45.9% 1290 ± 26% interrupts.42:PCI-MSI.524296-edge.eth0-tx-3
1810554 ± 9% -24.9% 1359040 interrupts.CAL:Function_call_interrupts
128994 ± 13% -54.5% 58688 ± 11%
interrupts.CPU0.CAL:Function_call_interrupts
3136833 ± 9% +30.6% 4097106 interrupts.CPU0.LOC:Local_timer_interrupts
147624 ± 12% -63.5% 53903 ± 7% interrupts.CPU0.TLB:TLB_shootdowns
131316 ± 12% -58.5% 54545 ± 3%
interrupts.CPU1.CAL:Function_call_interrupts
3196967 ± 6% +28.8% 4117877 interrupts.CPU1.LOC:Local_timer_interrupts
1316 ± 44% +134.5% 3086 ± 54% interrupts.CPU1.RES:Rescheduling_interrupts
150414 ± 17% -65.7% 51585 ± 14% interrupts.CPU1.TLB:TLB_shootdowns
3214233 ± 5% +28.6% 4132526 interrupts.CPU10.LOC:Local_timer_interrupts
9879 ± 80% -85.4% 1445 ± 45%
interrupts.CPU10.RES:Rescheduling_interrupts
3177700 ± 6% +29.8% 4126192 interrupts.CPU11.LOC:Local_timer_interrupts
559.50 ± 46% +253.3% 1976 ± 51%
interrupts.CPU11.RES:Rescheduling_interrupts
1307 +27.3% 1663 ± 10%
interrupts.CPU12.35:PCI-MSI.524289-edge.eth0-rx-0
3234073 ± 4% +27.7% 4130544 interrupts.CPU12.LOC:Local_timer_interrupts
9543 ± 47% -86.8% 1263 ± 55%
interrupts.CPU12.RES:Rescheduling_interrupts
3178037 ± 7% +29.8% 4124112 interrupts.CPU13.LOC:Local_timer_interrupts
3234950 ± 4% +27.7% 4130372 interrupts.CPU14.LOC:Local_timer_interrupts
7845 ± 61% -86.3% 1074 ± 77%
interrupts.CPU14.RES:Rescheduling_interrupts
3177518 ± 7% +29.8% 4123846 interrupts.CPU15.LOC:Local_timer_interrupts
115381 ± 14% -46.8% 61375 ± 38%
interrupts.CPU2.CAL:Function_call_interrupts
3211923 ± 4% +28.3% 4122113 interrupts.CPU2.LOC:Local_timer_interrupts
128739 ± 17% -57.7% 54470 ± 48% interrupts.CPU2.TLB:TLB_shootdowns
884.50 ± 2% +45.9% 1290 ± 26%
interrupts.CPU3.42:PCI-MSI.524296-edge.eth0-tx-3
3189990 ± 6% +29.1% 4119429 interrupts.CPU3.LOC:Local_timer_interrupts
1680 ± 89% +222.4% 5418 ± 50% interrupts.CPU3.RES:Rescheduling_interrupts
3207702 ± 5% +28.7% 4126732 interrupts.CPU4.LOC:Local_timer_interrupts
132646 ± 21% -46.4% 71034 ± 60% interrupts.CPU4.TLB:TLB_shootdowns
3185259 ± 7% +29.4% 4121092 interrupts.CPU5.LOC:Local_timer_interrupts
1237 ± 56% +203.8% 3757 ± 37% interrupts.CPU5.RES:Rescheduling_interrupts
3200005 ± 5% +29.0% 4128870 interrupts.CPU6.LOC:Local_timer_interrupts
3187267 ± 7% +29.4% 4125551 interrupts.CPU7.LOC:Local_timer_interrupts
802.00 ± 53% +280.5% 3051 ± 26% interrupts.CPU7.RES:Rescheduling_interrupts
3266308 ± 2% +26.5% 4132734 interrupts.CPU8.LOC:Local_timer_interrupts
23706 ± 90% -82.0% 4262 ± 66% interrupts.CPU8.RES:Rescheduling_interrupts
3163000 ± 8% +30.5% 4128371 interrupts.CPU9.LOC:Local_timer_interrupts
51161770 ± 6% +29.0% 65987472 interrupts.LOC:Local_timer_interrupts
1944653 ± 10% -31.9% 1325214 interrupts.TLB:TLB_shootdowns
4.79 +19.6% 5.73 perf-stat.i.MPKI
1.596e+09 -19.9% 1.278e+09 perf-stat.i.branch-instructions
1.66 +0.1 1.76 perf-stat.i.branch-miss-rate%
26901389 -14.4% 23036625 perf-stat.i.branch-misses
1.39e+08 -2.8% 1.351e+08 perf-stat.i.cache-references
0.61 ± 6% +17.9% 0.72 perf-stat.i.cpi
16.50 ± 5% +145.5% 40.52 ± 2% perf-stat.i.cpu-migrations
5919 -3.6% 5706 ± 2% perf-stat.i.cycles-between-cache-misses
0.05 ± 2% +0.0 0.07 perf-stat.i.dTLB-load-miss-rate%
3285384 ± 3% +15.2% 3784496 perf-stat.i.dTLB-load-misses
7.201e+09 -19.5% 5.793e+09 perf-stat.i.dTLB-loads
0.20 ± 2% +0.0 0.22 ± 4% perf-stat.i.dTLB-store-miss-rate%
3641778 ± 2% -7.6% 3366197 ± 5% perf-stat.i.dTLB-store-misses
1.925e+09 -19.5% 1.549e+09 perf-stat.i.dTLB-stores
0.01 ± 5% +0.0 0.02 ± 2% perf-stat.i.iTLB-load-miss-rate%
2872660 ± 2% +25.9% 3617061 ± 2% perf-stat.i.iTLB-load-misses
2.816e+10 -19.7% 2.261e+10 perf-stat.i.iTLB-loads
2.819e+10 -19.8% 2.261e+10 perf-stat.i.instructions
10654 ± 3% -27.1% 7764 ± 2% perf-stat.i.instructions-per-iTLB-miss
1.70 ± 4% -14.9% 1.45 perf-stat.i.ipc
0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.i.major-faults
4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.minor-faults
4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.page-faults
4.93 +21.2% 5.97 perf-stat.overall.MPKI
1.69 +0.1 1.80 perf-stat.overall.branch-miss-rate%
2.48 ± 4% +0.2 2.66 ± 2% perf-stat.overall.cache-miss-rate%
0.58 ± 2% +25.2% 0.73 perf-stat.overall.cpi
0.05 ± 3% +0.0 0.07 perf-stat.overall.dTLB-load-miss-rate%
0.19 ± 2% +0.0 0.22 ± 5% perf-stat.overall.dTLB-store-miss-rate%
0.01 +0.0 0.02 ± 2% perf-stat.overall.iTLB-load-miss-rate%
9815 ± 2% -36.3% 6253 ± 2%
perf-stat.overall.instructions-per-iTLB-miss
1.72 ± 2% -20.2% 1.37 perf-stat.overall.ipc
1.595e+09 -19.9% 1.278e+09 perf-stat.ps.branch-instructions
26891393 -14.3% 23033727 perf-stat.ps.branch-misses
1.389e+08 -2.8% 1.35e+08 perf-stat.ps.cache-references
16.50 ± 5% +145.4% 40.50 ± 2% perf-stat.ps.cpu-migrations
3283187 ± 3% +15.2% 3782278 perf-stat.ps.dTLB-load-misses
7.196e+09 -19.5% 5.79e+09 perf-stat.ps.dTLB-loads
3639422 ± 2% -7.6% 3364334 ± 5% perf-stat.ps.dTLB-store-misses
1.924e+09 -19.5% 1.548e+09 perf-stat.ps.dTLB-stores
2870820 ± 2% +25.9% 3614797 ± 2% perf-stat.ps.iTLB-load-misses
2.814e+10 -19.7% 2.26e+10 perf-stat.ps.iTLB-loads
2.817e+10 -19.8% 2.259e+10 perf-stat.ps.instructions
0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.ps.major-faults
4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.minor-faults
4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.page-faults
phoronix-test-suite.aom-av1.0.frames_per_second
0.04 +-------------------------------------------------------------------+
| |
0.035 |-+ |
0.03 |-O O O O O O O O O O O O O O O O O O O O O O O O O O O |
| |
0.025 |-+ |
| |
0.02 |-+ |
| |
0.015 |-+ |
0.01 |-+ |
| |
0.005 |-+ |
| |
0 +-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
8:03 a.m.
Hi Rong,
On Wed, 26 Feb 2020 at 02:33, kernel test robot <rong.a.chen(a)intel.com> wrote:
Greeting,
FYI, we noticed a -25.0% regression of phoronix-test-suite.aom-av1.0.frames_per_second
due to commit:
commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa: replace
runnable_load_avg by load_avg")
https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git sched-lbnuma-rewrite-v2r6
The following patches in the branch recover the perf regression. Do
you see regression on the whole branch ?
>
> in testcase: phoronix-test-suite
> on test machine: 16 threads Intel(R) Xeon(R) CPU X5570 @ 2.93GHz with 48G memory
> with following parameters:
>
> test: aom-av1-1.2.0
> cpufreq_governor: performance
> ucode: 0x1d
>
> test-description: The Phoronix Test Suite is the most comprehensive testing and
benchmarking platform available that provides an extensible framework for which new tests
can be easily added.
> test-url: http://www.phoronix-test-suite.com/
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <rong.a.chen(a)intel.com>
>
>
> Details are as below:
>
-------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
>
=========================================================================================
> compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
>
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
>
> commit:
> 91d7637912 ("sched/fair: reorder enqueue/dequeue_task_fair path")
> f6183ef98b ("sched/numa: replace runnable_load_avg by load_avg")
>
> 91d76379120e4640 f6183ef98bba39a5de563d4ab4b
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 0.04 -25.0% 0.03
phoronix-test-suite.aom-av1.0.frames_per_second
> 1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time
> 1648 +24.6% 2055
phoronix-test-suite.time.elapsed_time.max
> 1935 -19.6% 1556 meminfo.max_used_kB
> 1.518e+10 ± 29% +39.9% 2.124e+10 cpuidle.C1E.time
> 33131701 ± 34% +35.9% 45011103 ± 2% cpuidle.C1E.usage
> 672.00 -100.0% 0.00
slabinfo.dmaengine-unmap-16.active_objs
> 672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.num_objs
> 67.50 -2.2% 66.00 vmstat.cpu.id
> 31.00 +6.5% 33.00 vmstat.cpu.us
> 467.75 +5.0% 491.00 proc-vmstat.nr_mlock
> 484.75 +4.8% 508.00 proc-vmstat.nr_unevictable
> 484.75 +4.8% 508.00 proc-vmstat.nr_zone_unevictable
> 3492646 ± 7% +67.4% 5846384 ± 3% proc-vmstat.numa_hint_faults
> 2478506 ± 6% +127.8% 5646434 ± 4% proc-vmstat.numa_hint_faults_local
> 3650509 +13.5% 4141674 proc-vmstat.numa_hit
> 76734 +98.1% 151980 ± 2% proc-vmstat.numa_huge_pte_updates
> 3647139 +13.5% 4138249 proc-vmstat.numa_local
> 3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.numa_pages_migrated
> 42828088 +95.3% 83628988 ± 2% proc-vmstat.numa_pte_updates
> 7766914 ± 3% -25.6% 5776215 proc-vmstat.pgalloc_normal
> 7771023 ± 3% +43.5% 11147904 ± 2% proc-vmstat.pgfault
> 7716195 ± 3% -25.8% 5726088 proc-vmstat.pgfree
> 1152 ± 19% -88.9% 128.00 ±173% proc-vmstat.pgmigrate_fail
> 3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.pgmigrate_success
> 0.13 ±173% +0.5 0.63 ± 7%
perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
> 0.14 ±173% +0.5 0.67 ± 8%
perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 4.40 ± 47% +3.5 7.87 ± 16%
perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
> 4.40 ± 47% +3.5 7.87 ± 16%
perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
> 0.05 ± 67% +0.1 0.12 ± 17%
perf-profile.children.cycles-pp.ktime_get
> 0.01 ±173% +0.1 0.10 ± 27%
perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.1 0.10 ± 36%
perf-profile.children.cycles-pp.rcu_core
> 0.32 ± 86% +0.5 0.85 ± 54%
perf-profile.children.cycles-pp.__libc_start_main
> 0.60 ± 46% +0.6 1.19 ± 41%
perf-profile.children.cycles-pp.do_syscall_64
> 0.60 ± 46% +0.6 1.19 ± 41%
perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 0.34 ± 82% +0.6 0.94 ± 51%
perf-profile.children.cycles-pp.vfs_read
> 0.34 ± 82% +0.6 0.94 ± 51%
perf-profile.children.cycles-pp.ksys_read
> 0.29 ± 96% +0.6 0.90 ± 53%
perf-profile.children.cycles-pp.perf_event_read
> 0.29 ± 96% +0.6 0.90 ± 53%
perf-profile.children.cycles-pp.smp_call_function_single
> 0.30 ± 96% +0.6 0.91 ± 53%
perf-profile.children.cycles-pp.__libc_read
> 0.29 ± 96% +0.6 0.91 ± 52%
perf-profile.children.cycles-pp.perf_read
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.children.cycles-pp.start_kernel
> 0.01 ±173% +0.1 0.07 ± 19%
perf-profile.self.cycles-pp.cpuidle_enter_state
> 0.29 ± 98% +0.6 0.90 ± 53%
perf-profile.self.cycles-pp.smp_call_function_single
> 161465 ± 7% +66.5% 268796 ± 16% softirqs.CPU0.SCHED
> 154149 ± 4% +57.3% 242412 ± 5% softirqs.CPU1.SCHED
> 217783 ± 10% +17.4% 255655 ± 3% softirqs.CPU10.RCU
> 685538 ± 6% +16.3% 797133 ± 3% softirqs.CPU10.TIMER
> 647482 +29.6% 839256 ± 14% softirqs.CPU11.TIMER
> 215207 ± 9% +19.8% 257893 ± 5% softirqs.CPU12.RCU
> 676241 ± 4% +20.5% 814890 softirqs.CPU12.TIMER
> 218513 ± 8% +17.8% 257505 ± 4% softirqs.CPU14.RCU
> 665815 ± 5% +20.2% 800244 ± 3% softirqs.CPU14.TIMER
> 217764 ± 10% +16.0% 252579 ± 2% softirqs.CPU15.RCU
> 671996 ± 2% +18.0% 792743 ± 10% softirqs.CPU15.TIMER
> 216810 ± 10% +16.2% 251864 ± 3% softirqs.CPU2.RCU
> 168050 ± 8% +53.2% 257377 ± 4% softirqs.CPU2.SCHED
> 687608 ± 7% +13.9% 783359 ± 5% softirqs.CPU2.TIMER
> 214659 ± 10% +17.4% 252030 ± 2% softirqs.CPU3.RCU
> 201381 ± 8% +26.8% 255401 ± 10% softirqs.CPU3.SCHED
> 218567 ± 9% +15.1% 251588 ± 4% softirqs.CPU4.RCU
> 162228 ± 14% +39.7% 226690 ± 21% softirqs.CPU4.SCHED
> 686728 ± 7% +15.4% 792699 ± 4% softirqs.CPU4.TIMER
> 598763 ± 4% +33.1% 796983 ± 15% softirqs.CPU5.TIMER
> 219495 ± 10% +15.8% 254260 ± 4% softirqs.CPU6.RCU
> 685621 ± 6% +15.4% 790913 ± 4% softirqs.CPU6.TIMER
> 213675 ± 10% +18.1% 252314 ± 2% softirqs.CPU7.RCU
> 607580 ± 4% +27.7% 776035 ± 7% softirqs.CPU7.TIMER
> 217328 ± 10% +17.9% 256261 ± 4% softirqs.CPU8.RCU
> 672277 ± 8% +18.4% 795798 ± 3% softirqs.CPU8.TIMER
> 662696 ± 15% +22.2% 809936 ± 6% softirqs.CPU9.TIMER
> 3471815 ± 10% +16.7% 4051741 ± 2% softirqs.RCU
> 2788089 +21.0% 3372673 ± 2% softirqs.SCHED
> 10668573 ± 2% +17.5% 12539415 ± 5% softirqs.TIMER
> 250541 +31.8% 330137 ± 3% sched_debug.cfs_rq:/.exec_clock.avg
> 446877 ± 7% +68.2% 751493 ± 18% sched_debug.cfs_rq:/.exec_clock.max
> 112331 ± 17% +122.7% 250151 ± 35% sched_debug.cfs_rq:/.exec_clock.stddev
> 1517645 +35.0% 2049553 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg
> 2633991 ± 6% +65.2% 4352247 ± 18% sched_debug.cfs_rq:/.min_vruntime.max
> 684251 ± 18% +118.2% 1492888 ± 37%
sched_debug.cfs_rq:/.min_vruntime.stddev
> 0.46 ± 2% +8.1% 0.50 ± 4% sched_debug.cfs_rq:/.nr_running.avg
> -292907 -321.0% 647293 ± 49% sched_debug.cfs_rq:/.spread0.avg
> 823435 ± 35% +258.3% 2949987 ± 32% sched_debug.cfs_rq:/.spread0.max
> 684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.spread0.stddev
> 389.08 +10.3% 429.08 ± 3% sched_debug.cfs_rq:/.util_avg.avg
> 282.99 ± 2% +13.1% 320.08 ± 6%
sched_debug.cfs_rq:/.util_est_enqueued.avg
> 383.98 ± 2% +8.6% 416.84 ± 3%
sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 842328 +25.8% 1059503 sched_debug.cpu.clock.avg
> 842329 +25.8% 1059503 sched_debug.cpu.clock.max
> 842327 +25.8% 1059502 sched_debug.cpu.clock.min
> 1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock.stddev
> 842328 +25.8% 1059503 sched_debug.cpu.clock_task.avg
> 842329 +25.8% 1059503 sched_debug.cpu.clock_task.max
> 842327 +25.8% 1059502 sched_debug.cpu.clock_task.min
> 1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock_task.stddev
> 6159 ± 3% +29.3% 7966 ± 6% sched_debug.cpu.curr->pid.avg
> 21434 +25.0% 26795 sched_debug.cpu.curr->pid.max
> 7691 ± 2% +26.8% 9755 sched_debug.cpu.curr->pid.stddev
> 57507 ± 2% +27.2% 73171 sched_debug.cpu.nr_switches.avg
> 98634 ± 8% +51.9% 149817 ± 22% sched_debug.cpu.nr_switches.max
> 21105 ± 18% +85.3% 39113 ± 41% sched_debug.cpu.nr_switches.stddev
> 52692 ± 2% +29.9% 68458 sched_debug.cpu.sched_count.avg
> 94966 ± 8% +50.3% 142769 ± 25% sched_debug.cpu.sched_count.max
> 25787 ± 2% +27.6% 32917 sched_debug.cpu.sched_goidle.avg
> 47100 ± 8% +48.8% 70064 ± 26% sched_debug.cpu.sched_goidle.max
> 25109 ± 2% +28.2% 32200 sched_debug.cpu.ttwu_count.avg
> 47736 ± 4% +55.9% 74430 ± 9% sched_debug.cpu.ttwu_count.max
> 10778 ± 13% +80.8% 19490 ± 29% sched_debug.cpu.ttwu_count.stddev
> 11888 ± 4% +30.1% 15462 sched_debug.cpu.ttwu_local.avg
> 842327 +25.8% 1059502 sched_debug.cpu_clk
> 839741 +25.9% 1056916 sched_debug.ktime
> 843284 +25.8% 1060461 sched_debug.sched_clk
> 1307 +27.3% 1663 ± 10%
interrupts.35:PCI-MSI.524289-edge.eth0-rx-0
> 884.50 ± 2% +45.9% 1290 ± 26%
interrupts.42:PCI-MSI.524296-edge.eth0-tx-3
> 1810554 ± 9% -24.9% 1359040
interrupts.CAL:Function_call_interrupts
> 128994 ± 13% -54.5% 58688 ± 11%
interrupts.CPU0.CAL:Function_call_interrupts
> 3136833 ± 9% +30.6% 4097106
interrupts.CPU0.LOC:Local_timer_interrupts
> 147624 ± 12% -63.5% 53903 ± 7% interrupts.CPU0.TLB:TLB_shootdowns
> 131316 ± 12% -58.5% 54545 ± 3%
interrupts.CPU1.CAL:Function_call_interrupts
> 3196967 ± 6% +28.8% 4117877
interrupts.CPU1.LOC:Local_timer_interrupts
> 1316 ± 44% +134.5% 3086 ± 54%
interrupts.CPU1.RES:Rescheduling_interrupts
> 150414 ± 17% -65.7% 51585 ± 14% interrupts.CPU1.TLB:TLB_shootdowns
> 3214233 ± 5% +28.6% 4132526
interrupts.CPU10.LOC:Local_timer_interrupts
> 9879 ± 80% -85.4% 1445 ± 45%
interrupts.CPU10.RES:Rescheduling_interrupts
> 3177700 ± 6% +29.8% 4126192
interrupts.CPU11.LOC:Local_timer_interrupts
> 559.50 ± 46% +253.3% 1976 ± 51%
interrupts.CPU11.RES:Rescheduling_interrupts
> 1307 +27.3% 1663 ± 10%
interrupts.CPU12.35:PCI-MSI.524289-edge.eth0-rx-0
> 3234073 ± 4% +27.7% 4130544
interrupts.CPU12.LOC:Local_timer_interrupts
> 9543 ± 47% -86.8% 1263 ± 55%
interrupts.CPU12.RES:Rescheduling_interrupts
> 3178037 ± 7% +29.8% 4124112
interrupts.CPU13.LOC:Local_timer_interrupts
> 3234950 ± 4% +27.7% 4130372
interrupts.CPU14.LOC:Local_timer_interrupts
> 7845 ± 61% -86.3% 1074 ± 77%
interrupts.CPU14.RES:Rescheduling_interrupts
> 3177518 ± 7% +29.8% 4123846
interrupts.CPU15.LOC:Local_timer_interrupts
> 115381 ± 14% -46.8% 61375 ± 38%
interrupts.CPU2.CAL:Function_call_interrupts
> 3211923 ± 4% +28.3% 4122113
interrupts.CPU2.LOC:Local_timer_interrupts
> 128739 ± 17% -57.7% 54470 ± 48% interrupts.CPU2.TLB:TLB_shootdowns
> 884.50 ± 2% +45.9% 1290 ± 26%
interrupts.CPU3.42:PCI-MSI.524296-edge.eth0-tx-3
> 3189990 ± 6% +29.1% 4119429
interrupts.CPU3.LOC:Local_timer_interrupts
> 1680 ± 89% +222.4% 5418 ± 50%
interrupts.CPU3.RES:Rescheduling_interrupts
> 3207702 ± 5% +28.7% 4126732
interrupts.CPU4.LOC:Local_timer_interrupts
> 132646 ± 21% -46.4% 71034 ± 60% interrupts.CPU4.TLB:TLB_shootdowns
> 3185259 ± 7% +29.4% 4121092
interrupts.CPU5.LOC:Local_timer_interrupts
> 1237 ± 56% +203.8% 3757 ± 37%
interrupts.CPU5.RES:Rescheduling_interrupts
> 3200005 ± 5% +29.0% 4128870
interrupts.CPU6.LOC:Local_timer_interrupts
> 3187267 ± 7% +29.4% 4125551
interrupts.CPU7.LOC:Local_timer_interrupts
> 802.00 ± 53% +280.5% 3051 ± 26%
interrupts.CPU7.RES:Rescheduling_interrupts
> 3266308 ± 2% +26.5% 4132734
interrupts.CPU8.LOC:Local_timer_interrupts
> 23706 ± 90% -82.0% 4262 ± 66%
interrupts.CPU8.RES:Rescheduling_interrupts
> 3163000 ± 8% +30.5% 4128371
interrupts.CPU9.LOC:Local_timer_interrupts
> 51161770 ± 6% +29.0% 65987472 interrupts.LOC:Local_timer_interrupts
> 1944653 ± 10% -31.9% 1325214 interrupts.TLB:TLB_shootdowns
> 4.79 +19.6% 5.73 perf-stat.i.MPKI
> 1.596e+09 -19.9% 1.278e+09 perf-stat.i.branch-instructions
> 1.66 +0.1 1.76 perf-stat.i.branch-miss-rate%
> 26901389 -14.4% 23036625 perf-stat.i.branch-misses
> 1.39e+08 -2.8% 1.351e+08 perf-stat.i.cache-references
> 0.61 ± 6% +17.9% 0.72 perf-stat.i.cpi
> 16.50 ± 5% +145.5% 40.52 ± 2% perf-stat.i.cpu-migrations
> 5919 -3.6% 5706 ± 2%
perf-stat.i.cycles-between-cache-misses
> 0.05 ± 2% +0.0 0.07 perf-stat.i.dTLB-load-miss-rate%
> 3285384 ± 3% +15.2% 3784496 perf-stat.i.dTLB-load-misses
> 7.201e+09 -19.5% 5.793e+09 perf-stat.i.dTLB-loads
> 0.20 ± 2% +0.0 0.22 ± 4% perf-stat.i.dTLB-store-miss-rate%
> 3641778 ± 2% -7.6% 3366197 ± 5% perf-stat.i.dTLB-store-misses
> 1.925e+09 -19.5% 1.549e+09 perf-stat.i.dTLB-stores
> 0.01 ± 5% +0.0 0.02 ± 2% perf-stat.i.iTLB-load-miss-rate%
> 2872660 ± 2% +25.9% 3617061 ± 2% perf-stat.i.iTLB-load-misses
> 2.816e+10 -19.7% 2.261e+10 perf-stat.i.iTLB-loads
> 2.819e+10 -19.8% 2.261e+10 perf-stat.i.instructions
> 10654 ± 3% -27.1% 7764 ± 2% perf-stat.i.instructions-per-iTLB-miss
> 1.70 ± 4% -14.9% 1.45 perf-stat.i.ipc
> 0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.i.major-faults
> 4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.minor-faults
> 4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.page-faults
> 4.93 +21.2% 5.97 perf-stat.overall.MPKI
> 1.69 +0.1 1.80 perf-stat.overall.branch-miss-rate%
> 2.48 ± 4% +0.2 2.66 ± 2% perf-stat.overall.cache-miss-rate%
> 0.58 ± 2% +25.2% 0.73 perf-stat.overall.cpi
> 0.05 ± 3% +0.0 0.07 perf-stat.overall.dTLB-load-miss-rate%
> 0.19 ± 2% +0.0 0.22 ± 5%
perf-stat.overall.dTLB-store-miss-rate%
> 0.01 +0.0 0.02 ± 2% perf-stat.overall.iTLB-load-miss-rate%
> 9815 ± 2% -36.3% 6253 ± 2%
perf-stat.overall.instructions-per-iTLB-miss
> 1.72 ± 2% -20.2% 1.37 perf-stat.overall.ipc
> 1.595e+09 -19.9% 1.278e+09 perf-stat.ps.branch-instructions
> 26891393 -14.3% 23033727 perf-stat.ps.branch-misses
> 1.389e+08 -2.8% 1.35e+08 perf-stat.ps.cache-references
> 16.50 ± 5% +145.4% 40.50 ± 2% perf-stat.ps.cpu-migrations
> 3283187 ± 3% +15.2% 3782278 perf-stat.ps.dTLB-load-misses
> 7.196e+09 -19.5% 5.79e+09 perf-stat.ps.dTLB-loads
> 3639422 ± 2% -7.6% 3364334 ± 5% perf-stat.ps.dTLB-store-misses
> 1.924e+09 -19.5% 1.548e+09 perf-stat.ps.dTLB-stores
> 2870820 ± 2% +25.9% 3614797 ± 2% perf-stat.ps.iTLB-load-misses
> 2.814e+10 -19.7% 2.26e+10 perf-stat.ps.iTLB-loads
> 2.817e+10 -19.8% 2.259e+10 perf-stat.ps.instructions
> 0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.ps.major-faults
> 4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.minor-faults
> 4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.page-faults
>
>
>
> phoronix-test-suite.aom-av1.0.frames_per_second
>
> 0.04 +-------------------------------------------------------------------+
> | |
> 0.035 |-+ |
> 0.03 |-O O O O O O O O O O O O O O O O O O O O O O O O O O O |
> | |
> 0.025 |-+ |
> | |
> 0.02 |-+ |
> | |
> 0.015 |-+ |
> 0.01 |-+ |
> | |
> 0.005 |-+ |
> | |
> 0 +-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>
8:33 a.m.
On Wed, Feb 26, 2020 at 09:03:26AM +0100, Vincent Guittot wrote:
Hi Rong,
On Wed, 26 Feb 2020 at 02:33, kernel test robot <rong.a.chen(a)intel.com> wrote:
>
> Greeting,
>
> FYI, we noticed a -25.0% regression of
phoronix-test-suite.aom-av1.0.frames_per_second due to commit:
>
>
> commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa: replace
runnable_load_avg by load_avg")
> https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git
sched-lbnuma-rewrite-v2r6
The following patches in the branch recover the perf regression. Do
you see regression on the whole branch ?
If not, include "sched/fair: fix statistics for find_idlest_group" on top,
particularly if the workload is configured to use threads >= nr_cpus.
--
Mel Gorman
SUSE Labs
5:25 a.m.
On 2/26/20 4:33 PM, Mel Gorman wrote:
On Wed, Feb 26, 2020 at 09:03:26AM +0100, Vincent Guittot wrote:
> Hi Rong,
>
> On Wed, 26 Feb 2020 at 02:33, kernel test robot <rong.a.chen(a)intel.com> wrote:
>> Greeting,
>>
>> FYI, we noticed a -25.0% regression of
phoronix-test-suite.aom-av1.0.frames_per_second due to commit:
>>
>>
>> commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa: replace
runnable_load_avg by load_avg")
>> https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git
sched-lbnuma-rewrite-v2r6
> The following patches in the branch recover the perf regression. Do
> you see regression on the whole branch ?
>
If not, include "sched/fair: fix statistics for find_idlest_group" on top,
particularly if the workload is configured to use threads >= nr_cpus.
Hi Gorman,
The regression is still exist on commit "sched/fair: fix statistics for
find_idlest_group".
$ git log --oneline -2 db79824c2038adc60526970552242420386ed5e2
db79824c2038a sched/fair: fix statistics for find_idlest_group()
b92430cf87ebb sched/numa: Stop an exhastive search if a reasonable swap
candidate or idle CPU is found
/result/phoronix-test-suite/performance-aom-av1-1.2.0-ucode=0x1d/lkp-nhm-2ep1/debian-x86_64-phoronix/x86_64-rhel-7.6/gcc-7/db79824c2038adc60526970552242420386ed5e2/0/stats.json:
"phoronix-test-suite.aom-av1.0.frames_per_second": 0.03,
/result/phoronix-test-suite/performance-aom-av1-1.2.0-ucode=0x1d/lkp-nhm-2ep1/debian-x86_64-phoronix/x86_64-rhel-7.6/gcc-7/db79824c2038adc60526970552242420386ed5e2/1/stats.json:
"phoronix-test-suite.aom-av1.0.frames_per_second": 0.03,
/result/phoronix-test-suite/performance-aom-av1-1.2.0-ucode=0x1d/lkp-nhm-2ep1/debian-x86_64-phoronix/x86_64-rhel-7.6/gcc-7/db79824c2038adc60526970552242420386ed5e2/2/stats.json:
"phoronix-test-suite.aom-av1.0.frames_per_second": 0.03,
Best Regards,
Rong Chen
9:06 a.m.
On Thu, Feb 27, 2020 at 01:25:27PM +0800, Rong Chen wrote:
On 2/26/20 4:33 PM, Mel Gorman wrote:
> On Wed, Feb 26, 2020 at 09:03:26AM +0100, Vincent Guittot wrote:
> > Hi Rong,
> >
> > On Wed, 26 Feb 2020 at 02:33, kernel test robot <rong.a.chen(a)intel.com>
wrote:
> > > Greeting,
> > >
> > > FYI, we noticed a -25.0% regression of
phoronix-test-suite.aom-av1.0.frames_per_second due to commit:
> > >
> > >
> > > commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa:
replace runnable_load_avg by load_avg")
> > > https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git
sched-lbnuma-rewrite-v2r6
> > The following patches in the branch recover the perf regression. Do
> > you see regression on the whole branch ?
> >
> If not, include "sched/fair: fix statistics for find_idlest_group" on
top,
> particularly if the workload is configured to use threads >= nr_cpus.
>
Hi Gorman,
The regression is still exist on commit "sched/fair: fix statistics for
find_idlest_group".
$ git log --oneline -2 db79824c2038adc60526970552242420386ed5e2
db79824c2038a sched/fair: fix statistics for find_idlest_group()
b92430cf87ebb sched/numa: Stop an exhastive search if a reasonable swap
candidate or idle CPU is found
I failed to notice that you're testing v2 of the patch series and
this is severely out of date -- the branch doesn't even exist on
git.kernel.org any more. It was v6 that was ultimately picked up but
even if I push the branch I tested, it's unique as it's a rebased
tip/sched/core. I think it's better to check what's currently in
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core.
Commit a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10 is the latest version
that is aiming for the next merge window but note that it's missing
"sched/fair: fix statistics for find_idlest_group".
--
Mel Gorman
SUSE Labs
2:58 a.m.
On Thu, Feb 27, 2020 at 09:06:40AM +0000, Mel Gorman wrote:
On Thu, Feb 27, 2020 at 01:25:27PM +0800, Rong Chen wrote:
>
>
> On 2/26/20 4:33 PM, Mel Gorman wrote:
> > On Wed, Feb 26, 2020 at 09:03:26AM +0100, Vincent Guittot wrote:
> > > Hi Rong,
> > >
> > > On Wed, 26 Feb 2020 at 02:33, kernel test robot
<rong.a.chen(a)intel.com> wrote:
> > > > Greeting,
> > > >
> > > > FYI, we noticed a -25.0% regression of
phoronix-test-suite.aom-av1.0.frames_per_second due to commit:
> > > >
> > > >
> > > > commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa:
replace runnable_load_avg by load_avg")
> > > > https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git
sched-lbnuma-rewrite-v2r6
> > > The following patches in the branch recover the perf regression. Do
> > > you see regression on the whole branch ?
> > >
> > If not, include "sched/fair: fix statistics for find_idlest_group" on
top,
> > particularly if the workload is configured to use threads >= nr_cpus.
> >
>
> Hi Gorman,
>
> The regression is still exist on commit "sched/fair: fix statistics for
> find_idlest_group".
>
> $ git log --oneline -2 db79824c2038adc60526970552242420386ed5e2
> db79824c2038a sched/fair: fix statistics for find_idlest_group()
> b92430cf87ebb sched/numa: Stop an exhastive search if a reasonable swap
> candidate or idle CPU is found
>
I failed to notice that you're testing v2 of the patch series and
this is severely out of date -- the branch doesn't even exist on
git.kernel.org any more. It was v6 that was ultimately picked up but
even if I push the branch I tested, it's unique as it's a rebased
tip/sched/core. I think it's better to check what's currently in
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core.
Commit a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10 is the latest version
that is aiming for the next merge window but note that it's missing
"sched/fair: fix statistics for find_idlest_group".
--
Mel Gorman
SUSE Labs
Hi,
I tested on branch tip/sched/core, the regression is still there.
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
commit:
6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace runnable_load_avg by
load_avg")
253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix statistics for
find_idlest_group()")
6d4d22468dae3d87 6499b1b2dd1b8d404a16b9fbbf1 253e2b69ef8fda4d9345ff496b1
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
0.04 -25.0% 0.03 -25.0% 0.03
phoronix-test-suite.aom-av1.0.frames_per_second
Best Regards,
Rong Chen
10:12 a.m.
On Thu, Mar 05, 2020 at 10:58:22AM +0800, Rong Chen wrote:
Hi,
I tested on branch tip/sched/core, the regression is still there.
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
commit:
6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace runnable_load_avg
by load_avg")
253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix statistics for
find_idlest_group()")
Are you sure? I ask because tip/sched/core does not have the
patch "sched/fair: fix statistics for find_idlest_group" in it
nor is commit 253e2b69ef8fda4d9345ff496b12058faaeeff6b part of the
tip/sched/core history. You'd need to test the full series in the current
tip/sched/core with minimally 289de3598481 ("sched/fair: Fix statistics
for find_idlest_group()") from tip/sched/urgent on top. That's still
missing two fixes but one is a build issue and the other is a missing
rcu_read_lock that is unlikely to cause corruption unless there is a
hotplug event during the test.
Also, can you tell me more about the hardware? The stats indicate it's NUMA
but it only has 16 threads which seems very low for a modern NUMA machine.
--
Mel Gorman
SUSE Labs
11:15 a.m.
On 3/5/2020 6:12 PM, Mel Gorman wrote:
On Thu, Mar 05, 2020 at 10:58:22AM +0800, Rong Chen wrote:
> Hi,
>
> I tested on branch tip/sched/core, the regression is still there.
>
>
>
=========================================================================================
> compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
>
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
>
> commit:
> 6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
> 6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace
runnable_load_avg by load_avg")
> 253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix statistics for
find_idlest_group()")
>
Are you sure? I ask because tip/sched/core does not have the
patch "sched/fair: fix statistics for find_idlest_group" in it
nor is commit 253e2b69ef8fda4d9345ff496b12058faaeeff6b part of the
tip/sched/core history. You'd need to test the full series in the current
tip/sched/core with minimally 289de3598481 ("sched/fair: Fix statistics
for find_idlest_group()") from tip/sched/urgent on top. That's still
missing two fixes but one is a build issue and the other is a missing
rcu_read_lock that is unlikely to cause corruption unless there is a
hotplug event during the test.
Yes, commit 6d4d22468dae3 is not from tip/sched/core, I created it
based on tip/sched/core.
$ git log --oneline
6d4d22468dae3d8757af9f8b81b848a76ef4409d~..253e2b69ef8fda4d9345ff496b12058faaeeff6b
253e2b69ef8fd sched/fair: fix statistics for find_idlest_group()
a0f03b617c3b2 sched/numa: Stop an exhastive search if a reasonable swap
candidate or idle CPU is found
88cca72c9673e sched/numa: Bias swapping tasks based on their preferred node
5fb52dd93a2fe sched/numa: Find an alternative idle CPU if the CPU is
part of an active NUMA balance
ff7db0bf24db9 sched/numa: Prefer using an idle CPU as a migration target
instead of comparing tasks
070f5e860ee2b sched/fair: Take into account runnable_avg to classify group
9f68395333ad7 sched/pelt: Add a new runnable average signal
0dacee1bfa70e sched/pelt: Remove unused runnable load average
fb86f5b211924 sched/numa: Use similar logic to the load balancer for
moving between domains with spare capacity
6499b1b2dd1b8 sched/numa: Replace runnable_load_avg by load_avg
6d4d22468dae3 sched/fair: Reorder enqueue/dequeue_task_fair path
Also, can you tell me more about the hardware? The stats indicate it's NUMA
but it only has 16 threads which seems very low for a modern NUMA machine.
root@lkp-nhm-2ep1:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Model name: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
Stepping: 5
CPU MHz: 1655.106
CPU max MHz: 2927.0000
CPU min MHz: 1596.0000
BogoMIPS: 5852.83
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe
syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est
tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm pti ssbd ibrs
ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida flush_l1d
Best Regards,
Rong Chen
12:39 p.m.
On Thu, Mar 05, 2020 at 07:15:40PM +0800, Chen, Rong A wrote:
On 3/5/2020 6:12 PM, Mel Gorman wrote:
> On Thu, Mar 05, 2020 at 10:58:22AM +0800, Rong Chen wrote:
> > Hi,
> >
> > I tested on branch tip/sched/core, the regression is still there.
> >
> >
> >
=========================================================================================
> > compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
> >
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
> >
> > commit:
> > 6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
> > 6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace
runnable_load_avg by load_avg")
> > 253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix statistics
for find_idlest_group()")
> >
> Are you sure? I ask because tip/sched/core does not have the
> patch "sched/fair: fix statistics for find_idlest_group" in it
> nor is commit 253e2b69ef8fda4d9345ff496b12058faaeeff6b part of the
> tip/sched/core history. You'd need to test the full series in the current
> tip/sched/core with minimally 289de3598481 ("sched/fair: Fix statistics
> for find_idlest_group()") from tip/sched/urgent on top. That's still
> missing two fixes but one is a build issue and the other is a missing
> rcu_read_lock that is unlikely to cause corruption unless there is a
> hotplug event during the test.
Yes, commit 6d4d22468dae3 is not from tip/sched/core, I created it based on
tip/sched/core.
Understood.
$ git log --oneline
6d4d22468dae3d8757af9f8b81b848a76ef4409d~..253e2b69ef8fda4d9345ff496b12058faaeeff6b
253e2b69ef8fd sched/fair: fix statistics for find_idlest_group()
a0f03b617c3b2 sched/numa: Stop an exhastive search if a reasonable swap
candidate or idle CPU is found
88cca72c9673e sched/numa: Bias swapping tasks based on their preferred node
5fb52dd93a2fe sched/numa: Find an alternative idle CPU if the CPU is part of
an active NUMA balance
ff7db0bf24db9 sched/numa: Prefer using an idle CPU as a migration target
instead of comparing tasks
070f5e860ee2b sched/fair: Take into account runnable_avg to classify group
9f68395333ad7 sched/pelt: Add a new runnable average signal
0dacee1bfa70e sched/pelt: Remove unused runnable load average
fb86f5b211924 sched/numa: Use similar logic to the load balancer for moving
between domains with spare capacity
6499b1b2dd1b8 sched/numa: Replace runnable_load_avg by load_avg
6d4d22468dae3 sched/fair: Reorder enqueue/dequeue_task_fair path
Excellent. Now in your previous mail, the regression was reported based
on
commit:
6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace runnable_load_avg by
load_avg")
253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix statistics for
find_idlest_group()")
Can you confirm whether the report is based on just those commits or the
entire series? If it's not the entire series, can you give me the report
for the full series please? We know for a fact that this series is not
bisection safe in terms of performance.
>
> Also, can you tell me more about the hardware? The stats indicate it's NUMA
> but it only has 16 threads which seems very low for a modern NUMA machine.
>
root@lkp-nhm-2ep1:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Model name: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
Stepping: 5
CPU MHz: 1655.106
CPU max MHz: 2927.0000
CPU min MHz: 1596.0000
BogoMIPS: 5852.83
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3
cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm pti ssbd ibrs ibpb stibp
tpr_shadow vnmi flexpriority ept vpid dtherm ida flush_l1d
Ok, that makes some sense. It's an 11 year old Nehalem machine that is
no longer manufactured. No wonder I did not catch anything on my own
tests.
Can you tell me if this regression is machine-specific or are you seeing
it on a range of machines, particularly newer ones?
--
Mel Gorman
SUSE Labs
8:43 a.m.
On 3/5/20 8:39 PM, Mel Gorman wrote:
On Thu, Mar 05, 2020 at 07:15:40PM +0800, Chen, Rong A wrote:
>
> On 3/5/2020 6:12 PM, Mel Gorman wrote:
>> On Thu, Mar 05, 2020 at 10:58:22AM +0800, Rong Chen wrote:
>>> Hi,
>>>
>>> I tested on branch tip/sched/core, the regression is still there.
>>>
>>>
>>>
=========================================================================================
>>> compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
>>>
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
>>>
>>> commit:
>>> 6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
>>> 6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace
runnable_load_avg by load_avg")
>>> 253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix
statistics for find_idlest_group()")
>>>
>> Are you sure? I ask because tip/sched/core does not have the
>> patch "sched/fair: fix statistics for find_idlest_group" in it
>> nor is commit 253e2b69ef8fda4d9345ff496b12058faaeeff6b part of the
>> tip/sched/core history. You'd need to test the full series in the current
>> tip/sched/core with minimally 289de3598481 ("sched/fair: Fix statistics
>> for find_idlest_group()") from tip/sched/urgent on top. That's still
>> missing two fixes but one is a build issue and the other is a missing
>> rcu_read_lock that is unlikely to cause corruption unless there is a
>> hotplug event during the test.
> Yes, commit 6d4d22468dae3 is not from tip/sched/core, I created it based on
> tip/sched/core.
>
Understood.
> $ git log --oneline
6d4d22468dae3d8757af9f8b81b848a76ef4409d~..253e2b69ef8fda4d9345ff496b12058faaeeff6b
> 253e2b69ef8fd sched/fair: fix statistics for find_idlest_group()
> a0f03b617c3b2 sched/numa: Stop an exhastive search if a reasonable swap
> candidate or idle CPU is found
> 88cca72c9673e sched/numa: Bias swapping tasks based on their preferred node
> 5fb52dd93a2fe sched/numa: Find an alternative idle CPU if the CPU is part of
> an active NUMA balance
> ff7db0bf24db9 sched/numa: Prefer using an idle CPU as a migration target
> instead of comparing tasks
> 070f5e860ee2b sched/fair: Take into account runnable_avg to classify group
> 9f68395333ad7 sched/pelt: Add a new runnable average signal
> 0dacee1bfa70e sched/pelt: Remove unused runnable load average
> fb86f5b211924 sched/numa: Use similar logic to the load balancer for moving
> between domains with spare capacity
> 6499b1b2dd1b8 sched/numa: Replace runnable_load_avg by load_avg
> 6d4d22468dae3 sched/fair: Reorder enqueue/dequeue_task_fair path
>
Excellent. Now in your previous mail, the regression was reported based
on
commit:
6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace runnable_load_avg
by load_avg")
253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix statistics for
find_idlest_group()")
Can you confirm whether the report is based on just those commits or the
entire series? If it's not the entire series, can you give me the report
for the full series please? We know for a fact that this series is not
bisection safe in terms of performance.
Hi,
Sorry for the late, yes the report is based on the entire series, the
result of head commit is 0.03:
commit:
a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10 ("sched/numa: Stop an
exhastive search if a reasonable swap candidate or idle CPU is found")
$ ag -A9 phoronix-test-suite.aom-av1.0.frames_per_second
/result/phoronix-test-suite/performance-aom-av1-1.2.0-ucode=0x1d/lkp-nhm-2ep1/debian-x86_64-phoronix/x86_64-rhel-7.6/gcc-7/a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10/matrix.json
2: "phoronix-test-suite.aom-av1.0.frames_per_second": [
3- 0.03,
4- 0.03,
5- 0.03,
6- 0.03,
7- 0.03,
8- 0.03,
9- 0.03,
10- 0.03
11- ],
>> Also, can you tell me more about the hardware? The stats indicate it's NUMA
>> but it only has 16 threads which seems very low for a modern NUMA machine.
>>
> root@lkp-nhm-2ep1:~# lscpu
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> CPU(s): 16
> On-line CPU(s) list: 0-15
> Thread(s) per core: 2
> Core(s) per socket: 4
> Socket(s): 2
> NUMA node(s): 2
> Vendor ID: GenuineIntel
> CPU family: 6
> Model: 26
> Model name: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
> Stepping: 5
> CPU MHz: 1655.106
> CPU max MHz: 2927.0000
> CPU min MHz: 1596.0000
> BogoMIPS: 5852.83
> Virtualization: VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 256K
> L3 cache: 8192K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15
> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx
> rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
> nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3
> cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm pti ssbd ibrs ibpb stibp
> tpr_shadow vnmi flexpriority ept vpid dtherm ida flush_l1d
>
Ok, that makes some sense. It's an 11 year old Nehalem machine that is
no longer manufactured. No wonder I did not catch anything on my own
tests.
Can you tell me if this regression is machine-specific or are you seeing
it on a range of machines, particularly newer ones?
I tested on a Cascade Lake machine and doesn't see the regression.
As you'd expect, it's machine-specific.
Best Regards,
Rong Chen
10:53 a.m.
> Excellent. Now in your previous mail, the regression was
reported based
> on
>
> commit:
> 6d4d22468dae3d8757af9f8b81b848a76ef4409d ("sched/fair: Reorder
enqueue/dequeue_task_fair path")
> 6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d ("sched/numa: Replace
runnable_load_avg by load_avg")
> 253e2b69ef8fda4d9345ff496b12058faaeeff6b ("sched/fair: fix statistics for
find_idlest_group()")
>
> Can you confirm whether the report is based on just those commits or the
> entire series? If it's not the entire series, can you give me the report
> for the full series please? We know for a fact that this series is not
> bisection safe in terms of performance.
Hi,
Sorry for the late, yes the report is based on the entire series, the result
of head commit is 0.03:
commit:
a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10 ("sched/numa: Stop an exhastive
search if a reasonable swap candidate or idle CPU is found")
$ ag -A9 phoronix-test-suite.aom-av1.0.frames_per_second
/result/phoronix-test-suite/performance-aom-av1-1.2.0-ucode=0x1d/lkp-nhm-2ep1/debian-x86_64-phoronix/x86_64-rhel-7.6/gcc-7/a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10/matrix.json
2: "phoronix-test-suite.aom-av1.0.frames_per_second": [
3- 0.03,
4- 0.03,
5- 0.03,
6- 0.03,
7- 0.03,
8- 0.03,
9- 0.03,
10- 0.03
11- ],
Ok, that is missing a bunch of fixes that have been merged since or are
in other branches of tip. I'll try reproduce this before the end of the
week but I need to get some other stuff off the table first.
--
Mel Gorman
SUSE Labs
2:57 a.m.
On 2/26/20 4:03 PM, Vincent Guittot wrote:
Hi Rong,
On Wed, 26 Feb 2020 at 02:33, kernel test robot <rong.a.chen(a)intel.com> wrote:
> Greeting,
>
> FYI, we noticed a -25.0% regression of
phoronix-test-suite.aom-av1.0.frames_per_second due to commit:
>
>
> commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa: replace
runnable_load_avg by load_avg")
> https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git sched-lbnuma-rewrite-v2r6
The following patches in the branch recover the perf regression. Do
you see regression on the whole branch ?
Hi Guittot,
I confirmed that the regression can be found on the head commit
(b92430cf87eb "sched/numa: Stop an exhastive search if a reasonable swap
candidate or idle CPU is found").
Best Regards,
Rong Chen
> in testcase: phoronix-test-suite
> on test machine: 16 threads Intel(R) Xeon(R) CPU X5570 @ 2.93GHz with 48G memory
> with following parameters:
>
> test: aom-av1-1.2.0
> cpufreq_governor: performance
> ucode: 0x1d
>
> test-description: The Phoronix Test Suite is the most comprehensive testing and
benchmarking platform available that provides an extensible framework for which new tests
can be easily added.
> test-url: http://www.phoronix-test-suite.com/
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <rong.a.chen(a)intel.com>
>
>
> Details are as below:
>
-------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
>
=========================================================================================
> compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
>
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
>
> commit:
> 91d7637912 ("sched/fair: reorder enqueue/dequeue_task_fair path")
> f6183ef98b ("sched/numa: replace runnable_load_avg by load_avg")
>
> 91d76379120e4640 f6183ef98bba39a5de563d4ab4b
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 0.04 -25.0% 0.03
phoronix-test-suite.aom-av1.0.frames_per_second
> 1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time
> 1648 +24.6% 2055
phoronix-test-suite.time.elapsed_time.max
> 1935 -19.6% 1556 meminfo.max_used_kB
> 1.518e+10 ± 29% +39.9% 2.124e+10 cpuidle.C1E.time
> 33131701 ± 34% +35.9% 45011103 ± 2% cpuidle.C1E.usage
> 672.00 -100.0% 0.00
slabinfo.dmaengine-unmap-16.active_objs
> 672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.num_objs
> 67.50 -2.2% 66.00 vmstat.cpu.id
> 31.00 +6.5% 33.00 vmstat.cpu.us
> 467.75 +5.0% 491.00 proc-vmstat.nr_mlock
> 484.75 +4.8% 508.00 proc-vmstat.nr_unevictable
> 484.75 +4.8% 508.00 proc-vmstat.nr_zone_unevictable
> 3492646 ± 7% +67.4% 5846384 ± 3% proc-vmstat.numa_hint_faults
> 2478506 ± 6% +127.8% 5646434 ± 4% proc-vmstat.numa_hint_faults_local
> 3650509 +13.5% 4141674 proc-vmstat.numa_hit
> 76734 +98.1% 151980 ± 2% proc-vmstat.numa_huge_pte_updates
> 3647139 +13.5% 4138249 proc-vmstat.numa_local
> 3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.numa_pages_migrated
> 42828088 +95.3% 83628988 ± 2% proc-vmstat.numa_pte_updates
> 7766914 ± 3% -25.6% 5776215 proc-vmstat.pgalloc_normal
> 7771023 ± 3% +43.5% 11147904 ± 2% proc-vmstat.pgfault
> 7716195 ± 3% -25.8% 5726088 proc-vmstat.pgfree
> 1152 ± 19% -88.9% 128.00 ±173% proc-vmstat.pgmigrate_fail
> 3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.pgmigrate_success
> 0.13 ±173% +0.5 0.63 ± 7%
perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
> 0.14 ±173% +0.5 0.67 ± 8%
perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 4.40 ± 47% +3.5 7.87 ± 16%
perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
> 4.40 ± 47% +3.5 7.87 ± 16%
perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
> 0.05 ± 67% +0.1 0.12 ± 17%
perf-profile.children.cycles-pp.ktime_get
> 0.01 ±173% +0.1 0.10 ± 27%
perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.1 0.10 ± 36%
perf-profile.children.cycles-pp.rcu_core
> 0.32 ± 86% +0.5 0.85 ± 54%
perf-profile.children.cycles-pp.__libc_start_main
> 0.60 ± 46% +0.6 1.19 ± 41%
perf-profile.children.cycles-pp.do_syscall_64
> 0.60 ± 46% +0.6 1.19 ± 41%
perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 0.34 ± 82% +0.6 0.94 ± 51%
perf-profile.children.cycles-pp.vfs_read
> 0.34 ± 82% +0.6 0.94 ± 51%
perf-profile.children.cycles-pp.ksys_read
> 0.29 ± 96% +0.6 0.90 ± 53%
perf-profile.children.cycles-pp.perf_event_read
> 0.29 ± 96% +0.6 0.90 ± 53%
perf-profile.children.cycles-pp.smp_call_function_single
> 0.30 ± 96% +0.6 0.91 ± 53%
perf-profile.children.cycles-pp.__libc_read
> 0.29 ± 96% +0.6 0.91 ± 52%
perf-profile.children.cycles-pp.perf_read
> 4.41 ± 47% +3.5 7.94 ± 16%
perf-profile.children.cycles-pp.start_kernel
> 0.01 ±173% +0.1 0.07 ± 19%
perf-profile.self.cycles-pp.cpuidle_enter_state
> 0.29 ± 98% +0.6 0.90 ± 53%
perf-profile.self.cycles-pp.smp_call_function_single
> 161465 ± 7% +66.5% 268796 ± 16% softirqs.CPU0.SCHED
> 154149 ± 4% +57.3% 242412 ± 5% softirqs.CPU1.SCHED
> 217783 ± 10% +17.4% 255655 ± 3% softirqs.CPU10.RCU
> 685538 ± 6% +16.3% 797133 ± 3% softirqs.CPU10.TIMER
> 647482 +29.6% 839256 ± 14% softirqs.CPU11.TIMER
> 215207 ± 9% +19.8% 257893 ± 5% softirqs.CPU12.RCU
> 676241 ± 4% +20.5% 814890 softirqs.CPU12.TIMER
> 218513 ± 8% +17.8% 257505 ± 4% softirqs.CPU14.RCU
> 665815 ± 5% +20.2% 800244 ± 3% softirqs.CPU14.TIMER
> 217764 ± 10% +16.0% 252579 ± 2% softirqs.CPU15.RCU
> 671996 ± 2% +18.0% 792743 ± 10% softirqs.CPU15.TIMER
> 216810 ± 10% +16.2% 251864 ± 3% softirqs.CPU2.RCU
> 168050 ± 8% +53.2% 257377 ± 4% softirqs.CPU2.SCHED
> 687608 ± 7% +13.9% 783359 ± 5% softirqs.CPU2.TIMER
> 214659 ± 10% +17.4% 252030 ± 2% softirqs.CPU3.RCU
> 201381 ± 8% +26.8% 255401 ± 10% softirqs.CPU3.SCHED
> 218567 ± 9% +15.1% 251588 ± 4% softirqs.CPU4.RCU
> 162228 ± 14% +39.7% 226690 ± 21% softirqs.CPU4.SCHED
> 686728 ± 7% +15.4% 792699 ± 4% softirqs.CPU4.TIMER
> 598763 ± 4% +33.1% 796983 ± 15% softirqs.CPU5.TIMER
> 219495 ± 10% +15.8% 254260 ± 4% softirqs.CPU6.RCU
> 685621 ± 6% +15.4% 790913 ± 4% softirqs.CPU6.TIMER
> 213675 ± 10% +18.1% 252314 ± 2% softirqs.CPU7.RCU
> 607580 ± 4% +27.7% 776035 ± 7% softirqs.CPU7.TIMER
> 217328 ± 10% +17.9% 256261 ± 4% softirqs.CPU8.RCU
> 672277 ± 8% +18.4% 795798 ± 3% softirqs.CPU8.TIMER
> 662696 ± 15% +22.2% 809936 ± 6% softirqs.CPU9.TIMER
> 3471815 ± 10% +16.7% 4051741 ± 2% softirqs.RCU
> 2788089 +21.0% 3372673 ± 2% softirqs.SCHED
> 10668573 ± 2% +17.5% 12539415 ± 5% softirqs.TIMER
> 250541 +31.8% 330137 ± 3% sched_debug.cfs_rq:/.exec_clock.avg
> 446877 ± 7% +68.2% 751493 ± 18% sched_debug.cfs_rq:/.exec_clock.max
> 112331 ± 17% +122.7% 250151 ± 35%
sched_debug.cfs_rq:/.exec_clock.stddev
> 1517645 +35.0% 2049553 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg
> 2633991 ± 6% +65.2% 4352247 ± 18% sched_debug.cfs_rq:/.min_vruntime.max
> 684251 ± 18% +118.2% 1492888 ± 37%
sched_debug.cfs_rq:/.min_vruntime.stddev
> 0.46 ± 2% +8.1% 0.50 ± 4% sched_debug.cfs_rq:/.nr_running.avg
> -292907 -321.0% 647293 ± 49% sched_debug.cfs_rq:/.spread0.avg
> 823435 ± 35% +258.3% 2949987 ± 32% sched_debug.cfs_rq:/.spread0.max
> 684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.spread0.stddev
> 389.08 +10.3% 429.08 ± 3% sched_debug.cfs_rq:/.util_avg.avg
> 282.99 ± 2% +13.1% 320.08 ± 6%
sched_debug.cfs_rq:/.util_est_enqueued.avg
> 383.98 ± 2% +8.6% 416.84 ± 3%
sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 842328 +25.8% 1059503 sched_debug.cpu.clock.avg
> 842329 +25.8% 1059503 sched_debug.cpu.clock.max
> 842327 +25.8% 1059502 sched_debug.cpu.clock.min
> 1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock.stddev
> 842328 +25.8% 1059503 sched_debug.cpu.clock_task.avg
> 842329 +25.8% 1059503 sched_debug.cpu.clock_task.max
> 842327 +25.8% 1059502 sched_debug.cpu.clock_task.min
> 1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock_task.stddev
> 6159 ± 3% +29.3% 7966 ± 6% sched_debug.cpu.curr->pid.avg
> 21434 +25.0% 26795 sched_debug.cpu.curr->pid.max
> 7691 ± 2% +26.8% 9755 sched_debug.cpu.curr->pid.stddev
> 57507 ± 2% +27.2% 73171 sched_debug.cpu.nr_switches.avg
> 98634 ± 8% +51.9% 149817 ± 22% sched_debug.cpu.nr_switches.max
> 21105 ± 18% +85.3% 39113 ± 41% sched_debug.cpu.nr_switches.stddev
> 52692 ± 2% +29.9% 68458 sched_debug.cpu.sched_count.avg
> 94966 ± 8% +50.3% 142769 ± 25% sched_debug.cpu.sched_count.max
> 25787 ± 2% +27.6% 32917 sched_debug.cpu.sched_goidle.avg
> 47100 ± 8% +48.8% 70064 ± 26% sched_debug.cpu.sched_goidle.max
> 25109 ± 2% +28.2% 32200 sched_debug.cpu.ttwu_count.avg
> 47736 ± 4% +55.9% 74430 ± 9% sched_debug.cpu.ttwu_count.max
> 10778 ± 13% +80.8% 19490 ± 29% sched_debug.cpu.ttwu_count.stddev
> 11888 ± 4% +30.1% 15462 sched_debug.cpu.ttwu_local.avg
> 842327 +25.8% 1059502 sched_debug.cpu_clk
> 839741 +25.9% 1056916 sched_debug.ktime
> 843284 +25.8% 1060461 sched_debug.sched_clk
> 1307 +27.3% 1663 ± 10%
interrupts.35:PCI-MSI.524289-edge.eth0-rx-0
> 884.50 ± 2% +45.9% 1290 ± 26%
interrupts.42:PCI-MSI.524296-edge.eth0-tx-3
> 1810554 ± 9% -24.9% 1359040
interrupts.CAL:Function_call_interrupts
> 128994 ± 13% -54.5% 58688 ± 11%
interrupts.CPU0.CAL:Function_call_interrupts
> 3136833 ± 9% +30.6% 4097106
interrupts.CPU0.LOC:Local_timer_interrupts
> 147624 ± 12% -63.5% 53903 ± 7% interrupts.CPU0.TLB:TLB_shootdowns
> 131316 ± 12% -58.5% 54545 ± 3%
interrupts.CPU1.CAL:Function_call_interrupts
> 3196967 ± 6% +28.8% 4117877
interrupts.CPU1.LOC:Local_timer_interrupts
> 1316 ± 44% +134.5% 3086 ± 54%
interrupts.CPU1.RES:Rescheduling_interrupts
> 150414 ± 17% -65.7% 51585 ± 14% interrupts.CPU1.TLB:TLB_shootdowns
> 3214233 ± 5% +28.6% 4132526
interrupts.CPU10.LOC:Local_timer_interrupts
> 9879 ± 80% -85.4% 1445 ± 45%
interrupts.CPU10.RES:Rescheduling_interrupts
> 3177700 ± 6% +29.8% 4126192
interrupts.CPU11.LOC:Local_timer_interrupts
> 559.50 ± 46% +253.3% 1976 ± 51%
interrupts.CPU11.RES:Rescheduling_interrupts
> 1307 +27.3% 1663 ± 10%
interrupts.CPU12.35:PCI-MSI.524289-edge.eth0-rx-0
> 3234073 ± 4% +27.7% 4130544
interrupts.CPU12.LOC:Local_timer_interrupts
> 9543 ± 47% -86.8% 1263 ± 55%
interrupts.CPU12.RES:Rescheduling_interrupts
> 3178037 ± 7% +29.8% 4124112
interrupts.CPU13.LOC:Local_timer_interrupts
> 3234950 ± 4% +27.7% 4130372
interrupts.CPU14.LOC:Local_timer_interrupts
> 7845 ± 61% -86.3% 1074 ± 77%
interrupts.CPU14.RES:Rescheduling_interrupts
> 3177518 ± 7% +29.8% 4123846
interrupts.CPU15.LOC:Local_timer_interrupts
> 115381 ± 14% -46.8% 61375 ± 38%
interrupts.CPU2.CAL:Function_call_interrupts
> 3211923 ± 4% +28.3% 4122113
interrupts.CPU2.LOC:Local_timer_interrupts
> 128739 ± 17% -57.7% 54470 ± 48% interrupts.CPU2.TLB:TLB_shootdowns
> 884.50 ± 2% +45.9% 1290 ± 26%
interrupts.CPU3.42:PCI-MSI.524296-edge.eth0-tx-3
> 3189990 ± 6% +29.1% 4119429
interrupts.CPU3.LOC:Local_timer_interrupts
> 1680 ± 89% +222.4% 5418 ± 50%
interrupts.CPU3.RES:Rescheduling_interrupts
> 3207702 ± 5% +28.7% 4126732
interrupts.CPU4.LOC:Local_timer_interrupts
> 132646 ± 21% -46.4% 71034 ± 60% interrupts.CPU4.TLB:TLB_shootdowns
> 3185259 ± 7% +29.4% 4121092
interrupts.CPU5.LOC:Local_timer_interrupts
> 1237 ± 56% +203.8% 3757 ± 37%
interrupts.CPU5.RES:Rescheduling_interrupts
> 3200005 ± 5% +29.0% 4128870
interrupts.CPU6.LOC:Local_timer_interrupts
> 3187267 ± 7% +29.4% 4125551
interrupts.CPU7.LOC:Local_timer_interrupts
> 802.00 ± 53% +280.5% 3051 ± 26%
interrupts.CPU7.RES:Rescheduling_interrupts
> 3266308 ± 2% +26.5% 4132734
interrupts.CPU8.LOC:Local_timer_interrupts
> 23706 ± 90% -82.0% 4262 ± 66%
interrupts.CPU8.RES:Rescheduling_interrupts
> 3163000 ± 8% +30.5% 4128371
interrupts.CPU9.LOC:Local_timer_interrupts
> 51161770 ± 6% +29.0% 65987472 interrupts.LOC:Local_timer_interrupts
> 1944653 ± 10% -31.9% 1325214 interrupts.TLB:TLB_shootdowns
> 4.79 +19.6% 5.73 perf-stat.i.MPKI
> 1.596e+09 -19.9% 1.278e+09 perf-stat.i.branch-instructions
> 1.66 +0.1 1.76 perf-stat.i.branch-miss-rate%
> 26901389 -14.4% 23036625 perf-stat.i.branch-misses
> 1.39e+08 -2.8% 1.351e+08 perf-stat.i.cache-references
> 0.61 ± 6% +17.9% 0.72 perf-stat.i.cpi
> 16.50 ± 5% +145.5% 40.52 ± 2% perf-stat.i.cpu-migrations
> 5919 -3.6% 5706 ± 2%
perf-stat.i.cycles-between-cache-misses
> 0.05 ± 2% +0.0 0.07 perf-stat.i.dTLB-load-miss-rate%
> 3285384 ± 3% +15.2% 3784496 perf-stat.i.dTLB-load-misses
> 7.201e+09 -19.5% 5.793e+09 perf-stat.i.dTLB-loads
> 0.20 ± 2% +0.0 0.22 ± 4% perf-stat.i.dTLB-store-miss-rate%
> 3641778 ± 2% -7.6% 3366197 ± 5% perf-stat.i.dTLB-store-misses
> 1.925e+09 -19.5% 1.549e+09 perf-stat.i.dTLB-stores
> 0.01 ± 5% +0.0 0.02 ± 2% perf-stat.i.iTLB-load-miss-rate%
> 2872660 ± 2% +25.9% 3617061 ± 2% perf-stat.i.iTLB-load-misses
> 2.816e+10 -19.7% 2.261e+10 perf-stat.i.iTLB-loads
> 2.819e+10 -19.8% 2.261e+10 perf-stat.i.instructions
> 10654 ± 3% -27.1% 7764 ± 2%
perf-stat.i.instructions-per-iTLB-miss
> 1.70 ± 4% -14.9% 1.45 perf-stat.i.ipc
> 0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.i.major-faults
> 4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.minor-faults
> 4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.page-faults
> 4.93 +21.2% 5.97 perf-stat.overall.MPKI
> 1.69 +0.1 1.80 perf-stat.overall.branch-miss-rate%
> 2.48 ± 4% +0.2 2.66 ± 2% perf-stat.overall.cache-miss-rate%
> 0.58 ± 2% +25.2% 0.73 perf-stat.overall.cpi
> 0.05 ± 3% +0.0 0.07
perf-stat.overall.dTLB-load-miss-rate%
> 0.19 ± 2% +0.0 0.22 ± 5%
perf-stat.overall.dTLB-store-miss-rate%
> 0.01 +0.0 0.02 ± 2%
perf-stat.overall.iTLB-load-miss-rate%
> 9815 ± 2% -36.3% 6253 ± 2%
perf-stat.overall.instructions-per-iTLB-miss
> 1.72 ± 2% -20.2% 1.37 perf-stat.overall.ipc
> 1.595e+09 -19.9% 1.278e+09 perf-stat.ps.branch-instructions
> 26891393 -14.3% 23033727 perf-stat.ps.branch-misses
> 1.389e+08 -2.8% 1.35e+08 perf-stat.ps.cache-references
> 16.50 ± 5% +145.4% 40.50 ± 2% perf-stat.ps.cpu-migrations
> 3283187 ± 3% +15.2% 3782278 perf-stat.ps.dTLB-load-misses
> 7.196e+09 -19.5% 5.79e+09 perf-stat.ps.dTLB-loads
> 3639422 ± 2% -7.6% 3364334 ± 5% perf-stat.ps.dTLB-store-misses
> 1.924e+09 -19.5% 1.548e+09 perf-stat.ps.dTLB-stores
> 2870820 ± 2% +25.9% 3614797 ± 2% perf-stat.ps.iTLB-load-misses
> 2.814e+10 -19.7% 2.26e+10 perf-stat.ps.iTLB-loads
> 2.817e+10 -19.8% 2.259e+10 perf-stat.ps.instructions
> 0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.ps.major-faults
> 4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.minor-faults
> 4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.page-faults
>
>
>
> phoronix-test-suite.aom-av1.0.frames_per_second
>
> 0.04 +-------------------------------------------------------------------+
> | |
> 0.035 |-+ |
> 0.03 |-O O O O O O O O O O O O O O O O O O O O O O O O O O O |
> | |
> 0.025 |-+ |
> | |
> 0.02 |-+ |
> | |
> 0.015 |-+ |
> 0.01 |-+ |
> | |
> 0.005 |-+ |
> | |
> 0 +-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>
106
days inactive
127
days old
11 comments
5 participants
participants (5)
-
Chen, Rong A -
kernel test robot
-
Mel Gorman -
Rong Chen
-
Vincent Guittot