June 2019 - LKP - Ml01.01.Org

[btrfs] 8d47a0d8f7: fio.write_bw_MBps -28.6% regression

by kernel test robot

Greeting, FYI, we noticed a -28.6% regression of fio.write_bw_MBps due to commit: commit: 8d47a0d8f7947422dd359ac8e462687f81a7a137 ("btrfs: Do mandatory tree block check before submitting bio") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: fio-basic on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory with following parameters: disk: 2pmem fs: btrfs runtime: 200s nr_task: 50% time_based: tb rw: randwrite bs: 4k ioengine: libaio test_size: 100G cpufreq_governor: performance ucode: 0x3d test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 4k/gcc-7/performance/2pmem/btrfs/libaio/x86_64-rhel-7.6/50%/debian-x86_64-2018-04-03.cgz/200s/randwrite/lkp-hsw-ep2/100G/fio-basic/tb/0x3d commit: ff2ac107fa ("btrfs: tree-checker: Remove comprehensive root owner check") 8d47a0d8f7 ("btrfs: Do mandatory tree block check before submitting bio") ff2ac107fae2440b 8d47a0d8f7947422dd359ac8e46 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.BUG:unable_to_handle_kernel :4 25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception_in_interrupt :4 25% 1:4 dmesg.Oops:#[##] :4 25% 1:4 dmesg.RIP:cpuidle_enter_state :4 25% 1:4 dmesg.RIP:native_write_msr :4 25% 1:4 dmesg.RIP:perf_prepare_sample 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 0.01 ± 21% +0.3 0.28 ± 20% fio.latency_1000ms% 4.14 ± 13% +1.8 5.95 ± 9% fio.latency_10ms% 8.73 ± 15% +3.0 11.68 ± 8% fio.latency_20ms% 1.91 ± 5% -0.4 1.52 ± 3% fio.latency_250ms% 66.81 -5.9 60.90 ± 2% fio.latency_4ms% 0.18 ± 15% +0.1 0.29 ± 16% fio.latency_750us% 1.017e+08 -28.5% 72701525 fio.time.file_system_outputs 109.75 -26.8% 80.33 ± 2% fio.time.percent_of_cpu_this_job_got 197.97 -26.4% 145.79 ± 2% fio.time.system_time 38.66 ± 4% -24.0% 29.39 ± 3% fio.time.user_time 12388015 -28.5% 8858878 fio.time.voluntary_context_switches 12711059 -28.5% 9083954 fio.workload 247.91 -28.6% 176.96 fio.write_bw_MBps 309248 ± 2% +71.7% 531114 ± 4% fio.write_clat_99%_us 17574 +40.0% 24609 fio.write_clat_mean_us 57989 ± 4% +82.2% 105661 fio.write_clat_stddev 63463 -28.6% 45302 fio.write_iops 564.61 +40.2% 791.54 fio.write_slat_mean_us 9375 ± 7% +96.3% 18406 fio.write_slat_stddev 21.25 +8.5% 23.06 ± 9% boot-time.dhcp 93.32 +1.4% 94.58 iostat.cpu.idle 6.43 -18.6% 5.23 ± 2% iostat.cpu.system 6.48 -1.2 5.27 ± 2% mpstat.cpu.all.sys% 0.24 ± 5% -0.1 0.18 ± 2% mpstat.cpu.all.usr% 93.00 +1.1% 94.00 vmstat.cpu.id 591200 -30.0% 413921 vmstat.io.bo 31653776 ± 2% -10.9% 28196588 ± 2% vmstat.memory.cache 11927341 ± 7% +29.8% 15486805 ± 4% vmstat.memory.free 5.00 -26.7% 3.67 ± 12% vmstat.procs.r 635910 ± 2% -27.6% 460384 vmstat.system.cs 7822384 ± 2% -25.0% 5870498 numa-numastat.node0.local_node 7839861 ± 2% -24.8% 5893785 numa-numastat.node0.numa_hit 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node0.numa_miss 242465 ± 44% -84.3% 37964 ± 18% numa-numastat.node0.other_node 7762623 ± 12% -26.8% 5681358 ± 6% numa-numastat.node1.local_node 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node1.numa_foreign 7768618 ± 12% -26.9% 5681573 ± 6% numa-numastat.node1.numa_hit 9.429e+08 ± 10% -35.4% 6.094e+08 cpuidle.C1.time 53504154 ± 6% -31.8% 36504731 cpuidle.C1.usage 1.093e+09 ± 22% -50.3% 5.435e+08 ± 3% cpuidle.C1E.time 16541925 ± 13% -45.0% 9103775 cpuidle.C1E.usage 5.366e+09 ± 36% +71.7% 9.214e+09 ± 18% cpuidle.C3.time 16163664 ± 33% +57.0% 25379738 ± 5% cpuidle.C3.usage 9697055 ± 16% -35.8% 6225168 ± 19% cpuidle.C6.usage 19313595 ± 17% -16.3% 16166123 cpuidle.POLL.time 10291838 ± 9% -26.8% 7537045 ± 7% meminfo.Active 10034782 ± 9% -27.4% 7280710 ± 7% meminfo.Active(file) 31524905 ± 2% -10.9% 28082249 ± 2% meminfo.Cached 115157 ± 4% +61.1% 185560 ± 3% meminfo.CmaFree 6430720 ± 8% +19.1% 7660885 ± 6% meminfo.DirectMap2M 534230 ± 5% -12.8% 465932 ± 6% meminfo.Dirty 11737322 ± 7% +30.2% 15281189 ± 4% meminfo.MemFree 37604344 ± 2% -9.4% 34060483 meminfo.Memused 806.50 ± 37% +137.4% 1914 ± 8% meminfo.Writeback 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.active_objs 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.num_objs 15147 ± 5% -8.3% 13897 ± 2% slabinfo.blkdev_ioc.active_objs 17839 ± 6% -11.7% 15745 ± 2% slabinfo.blkdev_ioc.num_objs 12178 ± 4% -7.4% 11275 ± 2% slabinfo.kmalloc-512.num_objs 7794 ± 3% +25.4% 9774 slabinfo.proc_inode_cache.active_objs 8017 ± 2% +23.4% 9895 slabinfo.proc_inode_cache.num_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.active_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.num_objs 263.75 ± 2% -15.1% 224.00 ± 2% turbostat.Avg_MHz 9.46 ± 2% -1.4 8.07 ± 2% turbostat.Busy% 53502248 ± 6% -31.8% 36503898 turbostat.C1 5.86 ± 8% -2.0 3.82 turbostat.C1% 16540396 ± 13% -45.0% 9102733 turbostat.C1E 6.80 ± 22% -3.4 3.40 ± 4% turbostat.C1E% 16163332 ± 33% +57.0% 25379351 ± 5% turbostat.C3 33.55 ± 36% +24.1 57.61 ± 18% turbostat.C3% 9689445 ± 16% -35.8% 6221033 ± 19% turbostat.C6 18.04 ± 26% +49.1% 26.90 ± 9% turbostat.CPU%c3 53.25 ± 3% +10.8% 59.00 ± 4% turbostat.CoreTmp 4.67 ± 14% +62.7% 7.59 ± 26% turbostat.Pkg%pc2 56.50 +9.7% 62.00 ± 2% turbostat.PkgTmp 11.84 -4.8% 11.28 turbostat.RAMWatt 7232478 ± 10% -37.2% 4543037 ± 22% numa-meminfo.node0.Active 7072564 ± 10% -37.6% 4410135 ± 23% numa-meminfo.node0.Active(file) 308167 ± 6% -24.7% 232152 ± 10% numa-meminfo.node0.Dirty 8784028 ± 13% +21.7% 10686327 ± 2% numa-meminfo.node0.Inactive 8669548 ± 12% +20.7% 10465444 ± 2% numa-meminfo.node0.Inactive(file) 572.50 ± 22% +115.9% 1236 ± 25% numa-meminfo.node0.Writeback 15098140 ± 3% -17.7% 12425670 ± 3% numa-meminfo.node1.FilePages 11599549 ± 7% -22.3% 9018051 ± 4% numa-meminfo.node1.Inactive 105067 ± 92% -99.3% 725.33 ± 51% numa-meminfo.node1.Inactive(anon) 11494482 ± 7% -21.6% 9017325 ± 4% numa-meminfo.node1.Inactive(file) 5504 ± 9% -12.5% 4819 ± 2% numa-meminfo.node1.KernelStack 6709938 ± 11% +45.8% 9781880 ± 5% numa-meminfo.node1.MemFree 18029136 ± 4% -17.0% 14957200 ± 3% numa-meminfo.node1.MemUsed 4427 ± 69% -77.5% 996.00 ± 8% numa-meminfo.node1.PageTables 857660 ± 32% -37.2% 538646 ± 8% numa-meminfo.node1.SUnreclaim 109232 ± 88% -96.1% 4274 ± 10% numa-meminfo.node1.Shmem 986740 ± 35% -40.0% 592452 ± 10% numa-meminfo.node1.Slab 575.21 ± 15% +30.1% 748.58 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.max 100.45 ± 15% +28.6% 129.19 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.stddev 830.19 ± 9% +15.7% 960.25 ± 6% sched_debug.cfs_rq:/.util_avg.max 29.97 ± 16% -25.2% 22.43 ± 31% sched_debug.cfs_rq:/.util_est_enqueued.avg 591.17 ± 13% -22.2% 459.83 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.max 107.52 ± 13% -23.0% 82.84 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock_task.stddev 24.51 ± 7% +21.5% 29.77 ± 8% sched_debug.cpu.cpu_load[0].avg 541.29 ± 15% +38.3% 748.58 ± 2% sched_debug.cpu.cpu_load[0].max 93.97 ± 7% +27.8% 120.09 ± 2% sched_debug.cpu.cpu_load[0].stddev 519.08 ± 6% +40.5% 729.42 ± 4% sched_debug.cpu.cpu_load[1].max 87.45 ± 5% +28.5% 112.39 ± 4% sched_debug.cpu.cpu_load[1].stddev 480.83 ± 7% +39.5% 670.92 ± 13% sched_debug.cpu.cpu_load[2].max 80.92 ± 5% +25.7% 101.75 ± 8% sched_debug.cpu.cpu_load[2].stddev 408.83 ± 4% +54.0% 629.58 ± 19% sched_debug.cpu.cpu_load[3].max 71.62 ± 2% +30.7% 93.61 ± 11% sched_debug.cpu.cpu_load[3].stddev 381.50 ± 8% +66.7% 636.00 ± 18% sched_debug.cpu.cpu_load[4].max 64.40 ± 6% +39.7% 89.96 ± 12% sched_debug.cpu.cpu_load[4].stddev 106.75 ± 20% -88.4% 12.33 ± 10% proc-vmstat.kswapd_high_wmark_hit_quickly 2508660 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_active_file 33315360 -30.4% 23192301 proc-vmstat.nr_dirtied 133447 ± 5% -12.6% 116580 ± 6% proc-vmstat.nr_dirty 7881043 ± 2% -10.9% 7019533 ± 2% proc-vmstat.nr_file_pages 28790 ± 4% +61.1% 46392 ± 3% proc-vmstat.nr_free_cma 2934503 ± 7% +30.2% 3821369 ± 4% proc-vmstat.nr_free_pages 5044168 -3.4% 4872965 proc-vmstat.nr_inactive_file 430282 -6.0% 404589 proc-vmstat.nr_slab_unreclaimable 197.75 ± 38% +136.5% 467.67 ± 10% proc-vmstat.nr_writeback 33243480 -30.4% 23143756 proc-vmstat.nr_written 2508665 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_zone_active_file 5044399 -3.4% 4873026 proc-vmstat.nr_zone_inactive_file 134972 ± 5% -12.4% 118262 ± 5% proc-vmstat.nr_zone_write_pending 15632301 ± 5% -25.8% 11598535 ± 3% proc-vmstat.numa_hit 15608826 ± 5% -25.8% 11575029 ± 3% proc-vmstat.numa_local 2300530 ± 10% -31.1% 1585363 ± 10% proc-vmstat.pgactivate 16955745 -26.3% 12499611 ± 2% proc-vmstat.pgalloc_normal 17488000 -25.9% 12959812 ± 2% proc-vmstat.pgfree 1.33e+08 -30.4% 92575440 proc-vmstat.pgpgout 5728177 ± 6% -78.5% 1232433 ± 25% proc-vmstat.pgscan_kswapd 5727967 ± 6% -78.5% 1232244 ± 25% proc-vmstat.pgsteal_kswapd 32760 -7.6% 30285 proc-vmstat.slabs_scanned 1768211 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_active_file 10410188 ± 7% -32.5% 7023939 ± 7% numa-vmstat.node0.nr_dirtied 76995 ± 6% -24.6% 58035 ± 10% numa-vmstat.node0.nr_dirty 2167457 ± 12% +20.7% 2616147 ± 2% numa-vmstat.node0.nr_inactive_file 142.00 ± 24% +120.4% 313.00 ± 24% numa-vmstat.node0.nr_writeback 10308758 ± 7% -32.6% 6949497 ± 7% numa-vmstat.node0.nr_written 1768216 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_zone_active_file 2167563 ± 12% +20.7% 2616189 ± 2% numa-vmstat.node0.nr_zone_inactive_file 77786 ± 7% -24.3% 58899 ± 10% numa-vmstat.node0.nr_zone_write_pending 5889240 ± 3% -14.3% 5046874 ± 3% numa-vmstat.node0.numa_hit 5837851 ± 4% -14.0% 5023120 ± 3% numa-vmstat.node0.numa_local 68376 ± 48% -96.9% 2149 ± 46% numa-vmstat.node0.numa_miss 119766 ± 42% -78.4% 25904 ± 3% numa-vmstat.node0.numa_other 30.50 ± 38% -100.0% 0.00 numa-vmstat.node0.workingset_nodes 6829126 ± 6% -22.8% 5269898 ± 10% numa-vmstat.node1.nr_dirtied 3774645 ± 3% -17.7% 3106081 ± 3% numa-vmstat.node1.nr_file_pages 28796 ± 4% +61.1% 46405 ± 3% numa-vmstat.node1.nr_free_cma 1677332 ± 11% +45.8% 2445778 ± 5% numa-vmstat.node1.nr_free_pages 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_inactive_anon 2873702 ± 7% -21.6% 2254116 ± 4% numa-vmstat.node1.nr_inactive_file 5506 ± 9% -12.5% 4820 ± 2% numa-vmstat.node1.nr_kernel_stack 29031 ± 85% -88.2% 3436 numa-vmstat.node1.nr_mapped 1106 ± 69% -77.5% 249.00 ± 8% numa-vmstat.node1.nr_page_table_pages 27308 ± 88% -96.1% 1068 ± 10% numa-vmstat.node1.nr_shmem 214418 ± 32% -37.2% 134663 ± 8% numa-vmstat.node1.nr_slab_unreclaimable 75.00 ± 35% +145.3% 184.00 ± 18% numa-vmstat.node1.nr_writeback 6761224 ± 6% -23.1% 5202586 ± 10% numa-vmstat.node1.nr_written 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_zone_inactive_anon 2873810 ± 7% -21.6% 2254135 ± 4% numa-vmstat.node1.nr_zone_inactive_file 68381 ± 48% -96.9% 2152 ± 46% numa-vmstat.node1.numa_foreign 5035283 ± 15% -26.9% 3679318 ± 7% numa-vmstat.node1.numa_hit 4905249 ± 15% -28.2% 3522229 ± 8% numa-vmstat.node1.numa_local 18.43 ± 6% -28.3% 13.21 ± 3% perf-stat.i.MPKI 2.205e+09 ± 3% +10.5% 2.437e+09 perf-stat.i.branch-instructions 1.83 ± 5% -0.6 1.24 ± 7% perf-stat.i.branch-miss-rate% 37019551 ± 3% -29.4% 26140844 ± 5% perf-stat.i.branch-misses 28932020 ± 6% -31.1% 19933819 ± 3% perf-stat.i.cache-misses 2.03e+08 ± 4% -24.8% 1.526e+08 perf-stat.i.cache-references 645248 ± 2% -27.4% 468301 perf-stat.i.context-switches 1.97 ± 7% -22.1% 1.53 ± 2% perf-stat.i.cpi 1.983e+10 ± 3% -15.1% 1.684e+10 perf-stat.i.cpu-cycles 1.03 ± 5% -0.4 0.64 ± 10% perf-stat.i.dTLB-load-miss-rate% 31879502 ± 6% -37.9% 19807598 ± 9% perf-stat.i.dTLB-load-misses 3.201e+09 ± 3% +18.8% 3.804e+09 perf-stat.i.dTLB-loads 0.13 ± 7% -0.0 0.09 ± 10% perf-stat.i.dTLB-store-miss-rate% 2516181 ± 10% -28.3% 1805272 ± 10% perf-stat.i.dTLB-store-misses 2.143e+09 ± 3% +19.8% 2.568e+09 ± 2% perf-stat.i.dTLB-stores 13781489 ± 3% -28.4% 9866239 ± 3% perf-stat.i.iTLB-loads 1.164e+10 ± 3% +21.6% 1.415e+10 perf-stat.i.instructions 8435 ± 17% +42.5% 12023 ± 14% perf-stat.i.instructions-per-iTLB-miss 0.57 ± 3% +43.6% 0.82 perf-stat.i.ipc 16050361 ± 9% -30.7% 11127721 ± 3% perf-stat.i.node-load-misses 8120610 ± 9% -31.3% 5580996 perf-stat.i.node-loads 2591649 ± 13% -31.1% 1785932 ± 6% perf-stat.i.node-store-misses 1809775 ± 12% -31.2% 1244360 ± 2% perf-stat.i.node-stores 17.45 ± 4% -38.2% 10.79 perf-stat.overall.MPKI 1.68 ± 3% -0.6 1.07 ± 5% perf-stat.overall.branch-miss-rate% 1.70 ± 3% -30.2% 1.19 perf-stat.overall.cpi 689.69 ± 9% +22.6% 845.48 ± 2% perf-stat.overall.cycles-between-cache-misses 0.99 ± 5% -0.5 0.52 ± 10% perf-stat.overall.dTLB-load-miss-rate% 0.12 ± 8% -0.0 0.07 ± 10% perf-stat.overall.dTLB-store-miss-rate% 10.81 ± 14% +2.3 13.09 ± 10% perf-stat.overall.iTLB-load-miss-rate% 7119 ± 14% +35.9% 9679 ± 13% perf-stat.overall.instructions-per-iTLB-miss 0.59 ± 3% +43.1% 0.84 perf-stat.overall.ipc 202912 ± 2% +68.8% 342615 ± 3% perf-stat.overall.path-length 2.195e+09 ± 3% +10.5% 2.425e+09 perf-stat.ps.branch-instructions 36847010 ± 3% -29.4% 26015686 ± 5% perf-stat.ps.branch-misses 28799092 ± 6% -31.1% 19837306 ± 3% perf-stat.ps.cache-misses 2.021e+08 ± 4% -24.8% 1.519e+08 perf-stat.ps.cache-references 642266 ± 2% -27.4% 465968 perf-stat.ps.context-switches 1.974e+10 ± 3% -15.1% 1.676e+10 perf-stat.ps.cpu-cycles 31729472 ± 6% -37.9% 19715594 ± 9% perf-stat.ps.dTLB-load-misses 3.186e+09 ± 3% +18.8% 3.785e+09 perf-stat.ps.dTLB-loads 2504443 ± 10% -28.2% 1797154 ± 10% perf-stat.ps.dTLB-store-misses 2.133e+09 ± 3% +19.8% 2.556e+09 ± 2% perf-stat.ps.dTLB-stores 13717699 ± 3% -28.4% 9816979 ± 3% perf-stat.ps.iTLB-loads 1.159e+10 ± 3% +21.5% 1.408e+10 perf-stat.ps.instructions 15979371 ± 9% -30.7% 11073468 ± 3% perf-stat.ps.node-load-misses 8081205 ± 9% -31.3% 5554114 perf-stat.ps.node-loads 2580198 ± 13% -31.1% 1777169 ± 6% perf-stat.ps.node-store-misses 1801590 ± 12% -31.3% 1238321 ± 2% perf-stat.ps.node-stores 2.579e+12 +20.6% 3.111e+12 ± 2% perf-stat.total.instructions 36149 ± 4% -26.2% 26686 ± 5% softirqs.CPU0.RCU 37255 ± 3% -26.8% 27256 ± 2% softirqs.CPU1.RCU 37834 ± 5% -26.5% 27801 ± 9% softirqs.CPU10.RCU 39075 ± 3% -24.5% 29494 ± 2% softirqs.CPU11.RCU 38137 ± 6% -23.1% 29340 ± 4% softirqs.CPU12.RCU 37480 ± 3% -20.5% 29797 ± 2% softirqs.CPU13.RCU 37690 ± 5% -28.7% 26861 ± 4% softirqs.CPU14.RCU 33863 ± 9% -29.5% 23885 ± 5% softirqs.CPU15.RCU 34306 ± 9% -31.1% 23641 ± 12% softirqs.CPU16.RCU 34374 ± 9% -29.0% 24402 ± 7% softirqs.CPU17.RCU 44013 ± 9% -31.7% 30059 softirqs.CPU18.RCU 43114 ± 8% -29.9% 30219 softirqs.CPU19.RCU 36276 ± 8% -23.3% 27825 ± 5% softirqs.CPU2.RCU 67049 ± 7% +39.1% 93267 ± 18% softirqs.CPU2.TIMER 39721 ± 13% -31.4% 27233 ± 3% softirqs.CPU20.RCU 42340 ± 5% -28.6% 30215 ± 6% softirqs.CPU21.RCU 40896 ± 7% -30.3% 28506 ± 2% softirqs.CPU22.RCU 42422 ± 7% -33.2% 28357 ± 2% softirqs.CPU23.RCU 42330 ± 9% -33.5% 28150 ± 5% softirqs.CPU24.RCU 41406 ± 9% -31.4% 28398 softirqs.CPU25.RCU 39754 ± 5% -29.4% 28063 softirqs.CPU26.RCU 40207 ± 6% -27.9% 28995 ± 12% softirqs.CPU27.RCU 40351 ± 9% -29.4% 28487 ± 3% softirqs.CPU28.RCU 41880 ± 5% -29.6% 29472 softirqs.CPU29.RCU 36293 ± 4% -17.7% 29864 ± 7% softirqs.CPU3.RCU 39341 ± 9% -34.3% 25847 softirqs.CPU30.RCU 39687 ± 9% -33.5% 26400 ± 2% softirqs.CPU31.RCU 38062 ± 14% -39.9% 22878 ± 10% softirqs.CPU32.RCU 39769 ± 13% -33.8% 26314 ± 5% softirqs.CPU33.RCU 39504 ± 10% -37.2% 24824 ± 6% softirqs.CPU34.RCU 39591 ± 11% -38.6% 24300 ± 5% softirqs.CPU35.RCU 40398 ± 8% -27.0% 29488 ± 12% softirqs.CPU36.RCU 39464 ± 12% -25.3% 29478 ± 10% softirqs.CPU37.RCU 41177 ± 9% -34.0% 27190 ± 19% softirqs.CPU38.RCU 38734 ± 13% -24.8% 29145 ± 5% softirqs.CPU39.RCU 36804 ± 5% -23.0% 28354 softirqs.CPU4.RCU 40421 ± 12% -28.8% 28776 ± 6% softirqs.CPU40.RCU 40371 ± 11% -31.1% 27834 ± 8% softirqs.CPU41.RCU 40633 ± 10% -29.9% 28469 ± 10% softirqs.CPU42.RCU 41018 ± 9% -33.6% 27252 ± 6% softirqs.CPU43.RCU 38181 ± 12% -27.0% 27856 ± 10% softirqs.CPU44.RCU 39605 ± 12% -32.3% 26803 ± 12% softirqs.CPU45.RCU 42686 ± 11% -29.7% 30016 ± 9% softirqs.CPU47.RCU 40433 ± 9% -31.8% 27586 ± 11% softirqs.CPU48.RCU 38371 ± 12% -23.0% 29528 ± 7% softirqs.CPU49.RCU 37172 ± 2% -25.9% 27562 ± 3% softirqs.CPU5.RCU 39660 ± 12% -27.5% 28752 ± 11% softirqs.CPU50.RCU 40375 ± 10% -26.7% 29582 ± 3% softirqs.CPU51.RCU 40568 ± 9% -28.8% 28886 ± 6% softirqs.CPU52.RCU 39967 ± 10% -27.3% 29073 ± 11% softirqs.CPU53.RCU 40874 ± 11% -37.1% 25725 ± 6% softirqs.CPU54.RCU 41227 ± 14% -37.9% 25606 ± 6% softirqs.CPU55.RCU 38856 ± 13% -34.7% 25380 ± 8% softirqs.CPU56.RCU 39618 ± 13% -39.1% 24119 ± 4% softirqs.CPU57.RCU 39412 ± 12% -36.6% 24970 ± 3% softirqs.CPU58.RCU 41572 ± 14% -41.3% 24399 ± 10% softirqs.CPU59.RCU 36927 ± 3% -24.4% 27923 ± 22% softirqs.CPU6.RCU 37057 ± 10% -31.8% 25261 ± 2% softirqs.CPU60.RCU 36584 ± 8% -27.8% 26419 ± 2% softirqs.CPU61.RCU 35859 ± 4% -28.2% 25729 ± 5% softirqs.CPU62.RCU 35849 ± 7% -33.2% 23933 ± 5% softirqs.CPU63.RCU 35379 ± 6% -26.4% 26041 ± 4% softirqs.CPU64.RCU 36908 ± 6% -26.5% 27110 ± 2% softirqs.CPU65.RCU 37595 ± 3% -26.1% 27783 ± 4% softirqs.CPU66.RCU 36327 ± 2% -27.0% 26533 ± 2% softirqs.CPU67.RCU 35136 ± 7% -31.7% 23998 ± 9% softirqs.CPU68.RCU 35638 ± 4% -28.6% 25435 ± 4% softirqs.CPU69.RCU 37268 ± 6% -25.4% 27786 ± 4% softirqs.CPU7.RCU 36693 ± 4% -29.2% 25986 ± 5% softirqs.CPU70.RCU 36859 ± 3% -28.8% 26260 ± 4% softirqs.CPU71.RCU 37539 ± 7% -27.7% 27144 ± 4% softirqs.CPU8.RCU 37584 ± 4% -26.0% 27802 ± 15% softirqs.CPU9.RCU 2795313 ± 7% -29.7% 1964363 ± 4% softirqs.RCU 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.41:PCI-MSI.1572870-edge.eth0-TxRx-6 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.42:PCI-MSI.1572871-edge.eth0-TxRx-7 150.50 ± 43% -28.0% 108.33 interrupts.84:PCI-MSI.1572913-edge.eth0-TxRx-49 10393 ± 10% -26.4% 7653 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts 12518 ± 15% -31.0% 8642 ± 21% interrupts.CPU10.RES:Rescheduling_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.NMI:Non-maskable_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.PMI:Performance_monitoring_interrupts 13068 ± 18% -30.6% 9075 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts 11949 ± 11% -49.8% 6001 ± 29% interrupts.CPU12.RES:Rescheduling_interrupts 13711 ± 8% -19.4% 11052 ± 24% interrupts.CPU13.RES:Rescheduling_interrupts 13361 ± 6% -38.8% 8170 ± 25% interrupts.CPU14.RES:Rescheduling_interrupts 7628 ± 12% -29.2% 5401 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 7909 ± 25% -30.4% 5501 ± 37% interrupts.CPU19.RES:Rescheduling_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 2116 +23.2% 2606 ± 10% interrupts.CPU20.CAL:Function_call_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.NMI:Non-maskable_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.PMI:Performance_monitoring_interrupts 2096 ± 4% +18.0% 2474 ± 9% interrupts.CPU21.CAL:Function_call_interrupts 11313 ± 20% -56.4% 4935 ± 23% interrupts.CPU21.RES:Rescheduling_interrupts 2111 ± 3% +20.1% 2534 ± 11% interrupts.CPU22.CAL:Function_call_interrupts 2109 ± 4% +22.6% 2585 ± 10% interrupts.CPU23.CAL:Function_call_interrupts 10305 ± 16% -40.7% 6115 ± 17% interrupts.CPU23.RES:Rescheduling_interrupts 2128 ± 2% +17.8% 2507 ± 10% interrupts.CPU24.CAL:Function_call_interrupts 12271 ± 20% -39.0% 7488 ± 29% interrupts.CPU24.RES:Rescheduling_interrupts 2123 +19.7% 2541 ± 7% interrupts.CPU25.CAL:Function_call_interrupts 9780 ± 21% -32.9% 6563 ± 25% interrupts.CPU25.RES:Rescheduling_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.NMI:Non-maskable_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.PMI:Performance_monitoring_interrupts 11163 ± 28% -49.8% 5605 ± 13% interrupts.CPU27.RES:Rescheduling_interrupts 11071 ± 12% -56.3% 4835 ± 26% interrupts.CPU29.RES:Rescheduling_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 10599 ± 10% -38.7% 6501 ± 16% interrupts.CPU31.RES:Rescheduling_interrupts 2213 ± 4% +14.2% 2527 ± 7% interrupts.CPU32.CAL:Function_call_interrupts 8862 ± 16% -39.6% 5354 ± 28% interrupts.CPU32.RES:Rescheduling_interrupts 2156 ± 6% +20.5% 2599 ± 9% interrupts.CPU33.CAL:Function_call_interrupts 2169 ± 5% +19.9% 2601 ± 9% interrupts.CPU34.CAL:Function_call_interrupts 10483 ± 18% -48.8% 5371 ± 33% interrupts.CPU34.RES:Rescheduling_interrupts 2049 +24.4% 2549 ± 8% interrupts.CPU35.CAL:Function_call_interrupts 10929 ± 15% -49.0% 5578 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts 1993 ± 4% +28.5% 2561 ± 12% interrupts.CPU36.CAL:Function_call_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.NMI:Non-maskable_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1987 ± 4% +30.5% 2594 ± 13% interrupts.CPU37.CAL:Function_call_interrupts 1863 ± 10% +35.7% 2529 ± 14% interrupts.CPU38.CAL:Function_call_interrupts 1984 ± 6% +26.5% 2509 ± 12% interrupts.CPU41.CAL:Function_call_interrupts 12830 ± 30% -26.4% 9445 ± 9% interrupts.CPU43.RES:Rescheduling_interrupts 2067 ± 2% +27.4% 2632 ± 9% interrupts.CPU44.CAL:Function_call_interrupts 2066 ± 3% +27.2% 2629 ± 9% interrupts.CPU45.CAL:Function_call_interrupts 2158 ± 3% +19.6% 2582 ± 9% interrupts.CPU46.CAL:Function_call_interrupts 2036 ± 6% +30.3% 2652 ± 10% interrupts.CPU47.CAL:Function_call_interrupts 150.50 ± 43% -28.0% 108.33 interrupts.CPU49.84:PCI-MSI.1572913-edge.eth0-TxRx-49 11823 ± 12% -36.9% 7460 ± 18% interrupts.CPU49.RES:Rescheduling_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.NMI:Non-maskable_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.PMI:Performance_monitoring_interrupts 11748 ± 22% -40.0% 7048 ± 27% interrupts.CPU5.RES:Rescheduling_interrupts 2102 ± 5% +19.6% 2515 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 2115 ± 6% +19.5% 2527 ± 7% interrupts.CPU53.CAL:Function_call_interrupts 11318 ± 17% -20.4% 9004 ± 26% interrupts.CPU53.RES:Rescheduling_interrupts 2177 ± 5% +17.4% 2556 ± 9% interrupts.CPU54.CAL:Function_call_interrupts 10968 ± 23% -62.3% 4134 ± 21% interrupts.CPU54.RES:Rescheduling_interrupts 2161 ± 5% +20.2% 2598 ± 9% interrupts.CPU55.CAL:Function_call_interrupts 9334 ± 19% -55.4% 4158 ± 34% interrupts.CPU55.RES:Rescheduling_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 11377 ± 31% -56.9% 4907 ± 43% interrupts.CPU56.RES:Rescheduling_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.NMI:Non-maskable_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.PMI:Performance_monitoring_interrupts 8295 ± 31% -47.7% 4340 ± 24% interrupts.CPU57.RES:Rescheduling_interrupts 9182 ± 37% -59.7% 3701 ± 46% interrupts.CPU58.RES:Rescheduling_interrupts 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.CPU6.41:PCI-MSI.1572870-edge.eth0-TxRx-6 12473 ± 27% -44.4% 6940 ± 13% interrupts.CPU6.RES:Rescheduling_interrupts 7886 ± 12% -42.2% 4555 ± 31% interrupts.CPU60.RES:Rescheduling_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.NMI:Non-maskable_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.PMI:Performance_monitoring_interrupts 9253 ± 26% -56.4% 4033 ± 16% interrupts.CPU61.RES:Rescheduling_interrupts 2103 ± 5% +19.1% 2505 ± 11% interrupts.CPU63.CAL:Function_call_interrupts 2059 ± 2% +20.6% 2483 ± 10% interrupts.CPU64.CAL:Function_call_interrupts 8545 ± 17% -41.2% 5025 ± 8% interrupts.CPU65.RES:Rescheduling_interrupts 7378 ± 23% -51.0% 3614 ± 30% interrupts.CPU68.RES:Rescheduling_interrupts 8573 ± 28% -50.1% 4277 ± 6% interrupts.CPU69.RES:Rescheduling_interrupts 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.CPU7.42:PCI-MSI.1572871-edge.eth0-TxRx-7 9910 ± 28% -51.2% 4839 ± 40% interrupts.CPU70.RES:Rescheduling_interrupts 8836 ± 26% -52.7% 4180 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.NMI:Non-maskable_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.PMI:Performance_monitoring_interrupts 12326 ± 15% -32.3% 8350 ± 10% interrupts.CPU8.RES:Rescheduling_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.NMI:Non-maskable_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.PMI:Performance_monitoring_interrupts 749419 -31.9% 510225 ± 3% interrupts.RES:Rescheduling_interrupts 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread 5.99 ± 8% -1.6 4.37 ± 6% perf-profile.calltrace.cycles-pp.io_submit 5.83 ± 8% -1.6 4.25 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.io_submit 5.82 ± 8% -1.6 4.24 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.65 ± 6% -1.6 4.08 ± 20% perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 5.58 ± 8% -1.5 4.07 ± 6% perf-profile.calltrace.cycles-pp.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.50 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64 6.63 ± 8% -1.5 5.17 ± 10% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit 5.55 ± 8% -1.2 4.32 ± 10% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 4.98 ± 8% -1.1 3.86 ± 10% perf-profile.calltrace.cycles-pp.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages 5.00 ± 8% -1.1 3.88 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages 2.80 ± 4% -1.0 1.78 ± 22% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 2.32 ± 3% -0.6 1.72 ± 18% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.91 ± 8% -0.5 0.42 ± 73% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.88 ± 8% -0.5 0.41 ± 74% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 0.83 ± 5% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums 1.90 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 1.89 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.82 ± 4% -0.4 0.42 ± 71% perf-profile.calltrace.cycles-pp.run_one_async_done.normal_work_helper.process_one_work.worker_thread.kthread 0.81 ± 4% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.btrfs_map_bio.run_one_async_done.normal_work_helper.process_one_work.worker_thread 0.74 ± 6% -0.4 0.36 ± 70% perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks 1.19 ± 3% -0.4 0.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.calltrace.cycles-pp.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.25 ± 6% -0.3 0.90 ± 7% perf-profile.calltrace.cycles-pp.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.93 ± 8% -0.3 0.59 ± 7% perf-profile.calltrace.cycles-pp.__set_extent_bit.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter 0.94 ± 7% -0.3 0.60 ± 6% perf-profile.calltrace.cycles-pp.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.70 ± 8% -0.3 0.37 ± 70% perf-profile.calltrace.cycles-pp.__lookup_extent_mapping.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter 0.79 ± 6% -0.3 0.47 ± 71% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_duplicate_item.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 0.71 ± 21% -0.3 0.43 ± 74% perf-profile.calltrace.cycles-pp.__btrfs_free_extent.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space 1.15 ± 11% -0.3 0.88 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.15 ± 11% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 1.43 ± 13% -0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.06 ± 5% -0.3 0.81 ± 7% perf-profile.calltrace.cycles-pp.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.30 ± 15% -0.2 1.06 ± 4% perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write 1.03 ± 13% -0.2 0.79 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range 1.20 ± 4% -0.2 0.96 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.24 ± 3% -0.2 1.01 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.05 ± 8% -0.2 0.82 ± 12% perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.calltrace.cycles-pp.wait_reserve_ticket.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 1.01 ± 9% -0.2 0.84 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.01 ± 10% -0.2 0.85 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.72 ± 7% -0.2 0.56 ± 6% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.75 ± 8% -0.1 0.60 ± 10% perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages 0.92 ± 15% +0.2 1.14 ± 3% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.01 ± 15% +0.2 1.25 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 1.15 ± 15% +0.3 1.44 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.27 ±100% +0.4 0.63 ± 4% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 1.74 ± 9% +0.4 2.11 ± 2% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 2.63 ± 10% +0.6 3.26 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.00 +0.8 0.75 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread 0.00 +0.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 4.71 ± 11% +1.0 5.74 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 5.04 ± 10% +1.2 6.19 perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 0.00 +1.3 1.30 ± 19% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range 0.00 +1.4 1.36 ± 19% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents 0.29 ±101% +1.4 1.67 ± 18% perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space 0.00 +1.5 1.52 ± 18% perf-profile.calltrace.cycles-pp.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 0.00 +2.2 2.20 ± 14% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work.worker_thread 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work 49.45 ± 2% +4.3 53.79 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64 1.07 ± 18% +5.0 6.06 ± 20% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 0.93 ± 18% +5.0 5.95 ± 20% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes 61.26 ± 2% +5.1 66.36 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 61.21 ± 2% +5.1 66.32 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.33 ±100% +5.3 5.68 ± 20% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode 55.45 ± 2% +5.4 60.88 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +6.4 6.39 ± 13% perf-profile.calltrace.cycles-pp.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page 0.00 +6.5 6.50 ± 13% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.children.cycles-pp.normal_work_helper 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.children.cycles-pp.btrfs_finish_ordered_io 7.84 ± 5% -2.4 5.39 ± 10% perf-profile.children.cycles-pp.btrfs_search_slot 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_writepages 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_write_cache_pages 7.03 ± 7% -1.6 5.38 ± 12% perf-profile.children.cycles-pp.__extent_writepage 6.19 ± 8% -1.6 4.56 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 6.17 ± 8% -1.6 4.55 ± 7% perf-profile.children.cycles-pp.do_syscall_64 6.01 ± 8% -1.6 4.39 ± 7% perf-profile.children.cycles-pp.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.children.cycles-pp.__x64_sys_io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.children.cycles-pp.io_submit_one 5.66 ± 5% -1.6 4.09 ± 20% perf-profile.children.cycles-pp.btrfs_mark_extent_written 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.add_pending_csums 5.59 ± 8% -1.5 4.07 ± 6% perf-profile.children.cycles-pp.aio_write 4.51 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.btrfs_csum_file_blocks 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.children.cycles-pp.btrfs_file_write_iter 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.children.cycles-pp.btrfs_buffered_write 5.88 ± 7% -1.4 4.47 ± 11% perf-profile.children.cycles-pp.writepage_delalloc 5.30 ± 8% -1.3 4.01 ± 11% perf-profile.children.cycles-pp.btrfs_run_delalloc_range 5.28 ± 8% -1.3 3.99 ± 11% perf-profile.children.cycles-pp.run_delalloc_nocow 2.55 ± 2% -0.9 1.67 ± 16% perf-profile.children.cycles-pp.btrfs_cow_block 2.54 ± 2% -0.9 1.66 ± 16% perf-profile.children.cycles-pp.__btrfs_cow_block 2.15 ± 13% -0.8 1.31 ± 8% perf-profile.children.cycles-pp.__lookup_extent_mapping 1.87 ± 12% -0.8 1.04 ± 12% perf-profile.children.cycles-pp.__etree_search 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.printk 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.vprintk_emit 2.60 ± 2% -0.7 1.92 ± 14% perf-profile.children.cycles-pp.btrfs_insert_empty_items 3.00 ± 5% -0.7 2.33 ± 6% perf-profile.children.cycles-pp._raw_spin_lock 1.89 ± 2% -0.7 1.23 ± 7% perf-profile.children.cycles-pp.__set_extent_bit 2.78 ± 4% -0.6 2.14 ± 14% perf-profile.children.cycles-pp.__schedule 1.57 ± 2% -0.5 1.02 ± 9% perf-profile.children.cycles-pp.lock_extent_bits 1.19 ± 14% -0.5 0.70 ± 11% perf-profile.children.cycles-pp.__clear_extent_bit 1.51 ± 7% -0.5 1.04 ± 9% perf-profile.children.cycles-pp.read_block_for_search 2.08 ± 5% -0.5 1.62 ± 18% perf-profile.children.cycles-pp.setup_items_for_insert 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.btrfs_run_delayed_refs 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs 1.60 ± 6% -0.4 1.20 ± 12% perf-profile.children.cycles-pp.schedule 1.95 ± 11% -0.4 1.58 ± 12% perf-profile.children.cycles-pp.try_to_wake_up 1.31 ± 6% -0.4 0.94 ± 9% perf-profile.children.cycles-pp.create_io_em 0.99 ± 4% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need 1.27 ± 9% -0.4 0.92 ± 11% perf-profile.children.cycles-pp.__wake_up_common_lock 0.98 ± 3% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.btrfs_alloc_tree_block 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.children.cycles-pp.btrfs_dirty_pages 1.34 ± 7% -0.3 0.99 ± 10% perf-profile.children.cycles-pp.btrfs_map_bio 1.11 ± 9% -0.3 0.77 ± 11% perf-profile.children.cycles-pp.generic_bin_search 1.16 ± 6% -0.3 0.82 ± 9% perf-profile.children.cycles-pp.find_extent_buffer 0.91 ± 7% -0.3 0.57 ± 21% perf-profile.children.cycles-pp.btrfs_lookup_csum 1.21 ± 11% -0.3 0.90 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_csums_range 1.21 ± 11% -0.3 0.91 ± 9% perf-profile.children.cycles-pp.csum_exist_in_range 1.10 ± 10% -0.3 0.80 ± 9% perf-profile.children.cycles-pp.__wake_up_common 0.37 ± 70% -0.3 0.07 ± 18% perf-profile.children.cycles-pp.test_range_bit 1.61 ± 4% -0.3 1.32 ± 15% perf-profile.children.cycles-pp.push_leaf_right 0.60 ± 4% -0.3 0.31 ± 25% perf-profile.children.cycles-pp.btrfs_extend_item 1.02 ± 9% -0.3 0.73 ± 10% perf-profile.children.cycles-pp.autoremove_wake_function 1.12 ± 4% -0.3 0.84 ± 9% perf-profile.children.cycles-pp.btrfs_drop_extent_cache 0.82 ± 6% -0.3 0.55 ± 4% perf-profile.children.cycles-pp.btrfs_release_path 0.85 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_extent_buffer_full 0.99 ± 4% -0.3 0.73 ± 13% perf-profile.children.cycles-pp.btrfs_set_token_32 1.44 ± 13% -0.3 1.18 ± 5% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 0.84 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_page 0.87 ± 11% -0.3 0.62 ± 11% perf-profile.children.cycles-pp.pagecache_get_page 0.84 ± 7% -0.2 0.60 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc 1.12 ± 7% -0.2 0.88 ± 17% perf-profile.children.cycles-pp.__extent_writepage_io 1.30 ± 15% -0.2 1.06 ± 5% perf-profile.children.cycles-pp.reserve_metadata_bytes 0.82 ± 4% -0.2 0.58 ± 13% perf-profile.children.cycles-pp.run_one_async_done 1.25 ± 3% -0.2 1.01 ± 15% perf-profile.children.cycles-pp.schedule_idle 0.34 ± 53% -0.2 0.10 ± 28% perf-profile.children.cycles-pp.delay_tsc 0.70 ± 13% -0.2 0.47 ± 11% perf-profile.children.cycles-pp.prepare_pages 0.50 ± 22% -0.2 0.28 ± 13% perf-profile.children.cycles-pp.clear_state_bit 0.84 ± 5% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.btrfs_get_extent 0.54 ± 11% -0.2 0.34 ± 4% perf-profile.children.cycles-pp.btrfs_free_path 1.16 ± 11% -0.2 0.96 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.53 ± 9% -0.2 0.33 ± 18% perf-profile.children.cycles-pp.unpin_extent_cache 1.07 ± 10% -0.2 0.88 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_file_extent 0.50 ± 4% -0.2 0.31 ± 15% perf-profile.children.cycles-pp.clear_extent_bit 0.45 ± 13% -0.2 0.26 ± 11% perf-profile.children.cycles-pp.kmem_cache_free 0.40 ± 14% -0.2 0.21 ± 50% perf-profile.children.cycles-pp.__writeback_inodes_wb 0.64 ± 7% -0.2 0.45 ± 10% perf-profile.children.cycles-pp.dequeue_task_fair 0.50 ± 8% -0.2 0.33 ± 23% perf-profile.children.cycles-pp.btrfs_add_delayed_tree_ref 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.children.cycles-pp.wait_reserve_ticket 0.52 ± 9% -0.2 0.35 ± 7% perf-profile.children.cycles-pp.__radix_tree_lookup 0.52 ± 8% -0.2 0.36 ± 8% perf-profile.children.cycles-pp.xas_load 0.78 ± 19% -0.2 0.62 ± 23% perf-profile.children.cycles-pp.__btrfs_free_extent 0.56 ± 17% -0.1 0.42 ± 15% perf-profile.children.cycles-pp.btrfs_del_items 0.51 ± 5% -0.1 0.36 ± 9% perf-profile.children.cycles-pp.dequeue_entity 0.39 ± 7% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.82 ± 14% -0.1 0.68 ± 18% perf-profile.children.cycles-pp.ttwu_do_activate 0.55 ± 8% -0.1 0.41 ± 8% perf-profile.children.cycles-pp.btrfs_submit_bio_hook 0.53 ± 5% -0.1 0.40 ± 13% perf-profile.children.cycles-pp.update_load_avg 0.40 ± 8% -0.1 0.27 ± 27% perf-profile.children.cycles-pp.add_delayed_ref_head 0.35 ± 13% -0.1 0.22 ± 3% perf-profile.children.cycles-pp.set_extent_bit 0.52 ± 9% -0.1 0.39 ± 7% perf-profile.children.cycles-pp.btrfs_wq_submit_bio 0.35 ± 3% -0.1 0.23 ± 32% perf-profile.children.cycles-pp.queued_write_lock_slowpath 0.33 ± 6% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.__slab_alloc 0.63 ± 19% -0.1 0.51 ± 13% perf-profile.children.cycles-pp.sched_ttwu_pending 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode_fallback 0.57 ± 5% -0.1 0.46 ± 10% perf-profile.children.cycles-pp.find_lock_delalloc_range 0.45 ± 5% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.btrfs_cross_ref_exist 0.29 ± 3% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.free_extent_buffer 0.38 ± 3% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty 0.46 ± 5% -0.1 0.35 ± 13% perf-profile.children.cycles-pp.find_delalloc_range 0.41 ± 9% -0.1 0.30 ± 2% perf-profile.children.cycles-pp.btrfs_copy_from_user 0.40 ± 9% -0.1 0.29 ± 2% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.38 ± 10% -0.1 0.27 perf-profile.children.cycles-pp.find_get_entry 0.31 ± 12% -0.1 0.20 ± 31% perf-profile.children.cycles-pp.btrfs_remove_ordered_extent 0.30 ± 6% -0.1 0.20 ± 14% perf-profile.children.cycles-pp.___slab_alloc 0.26 ± 8% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.merge_state 0.38 ± 7% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.copyin 0.26 ± 17% -0.1 0.15 ± 26% perf-profile.children.cycles-pp.btrfs_delayed_update_inode 0.25 ± 4% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.insert_state 0.40 ± 13% -0.1 0.30 ± 13% perf-profile.children.cycles-pp.extent_clear_unlock_delalloc 0.36 ± 9% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.alloc_extent_state 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.39 ± 3% -0.1 0.29 ± 14% perf-profile.children.cycles-pp.alloc_extent_buffer 0.31 ± 4% -0.1 0.22 ± 11% perf-profile.children.cycles-pp.set_extent_buffer_dirty 0.26 ± 11% -0.1 0.17 ± 23% perf-profile.children.cycles-pp.unlock_up 0.32 ± 3% -0.1 0.24 ± 13% perf-profile.children.cycles-pp.__btrfs_map_block 0.23 ± 13% -0.1 0.15 ± 28% perf-profile.children.cycles-pp.btrfs_add_delayed_data_ref 0.25 ± 12% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.__slab_free 0.47 ± 6% -0.1 0.39 ± 15% perf-profile.children.cycles-pp.__push_leaf_right 0.26 ± 9% -0.1 0.18 ± 19% perf-profile.children.cycles-pp.__test_set_page_writeback 0.32 ± 11% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.btrfs_add_ordered_extent 0.15 ± 8% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__xa_set_mark 0.23 ± 7% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.release_extent_buffer 0.18 ± 8% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.alloc_extent_map 0.32 ± 10% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.__btrfs_add_ordered_extent 0.32 ± 17% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_lock_atomic 0.20 ± 13% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.btrfs_inc_extent_ref 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.find_ref_head 0.13 ± 32% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.rb_erase_cached 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_submit_bio_start 0.53 ± 12% -0.1 0.46 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_lock 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_csum_one_bio 0.17 ± 16% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.rb_erase 0.21 ± 7% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.run_one_async_start 0.22 ± 3% -0.1 0.15 ± 22% perf-profile.children.cycles-pp.btrfs_try_tree_write_lock 0.21 ± 9% -0.1 0.15 ± 21% perf-profile.children.cycles-pp.btrfs_leaf_free_space 0.15 ± 23% -0.1 0.09 ± 35% perf-profile.children.cycles-pp.free_extent_map 0.20 ± 9% -0.1 0.14 ± 20% perf-profile.children.cycles-pp.leaf_space_used 0.19 ± 8% -0.1 0.13 ± 13% perf-profile.children.cycles-pp.check_delayed_ref 0.17 ± 13% -0.1 0.12 ± 31% perf-profile.children.cycles-pp.get_page_from_freelist 0.13 ± 8% -0.1 0.08 ± 26% perf-profile.children.cycles-pp.__update_load_avg_se 0.17 ± 2% -0.1 0.12 ± 18% perf-profile.children.cycles-pp.btrfs_set_range_writeback 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.children.cycles-pp.___might_sleep 0.20 ± 11% -0.1 0.15 ± 35% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_unlock 0.14 ± 11% -0.1 0.09 ± 28% perf-profile.children.cycles-pp.btrfs_reserve_extent 0.10 ± 33% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.__btrfs_release_delayed_node 0.22 ± 9% -0.0 0.17 ± 25% perf-profile.children.cycles-pp.mark_extent_buffer_accessed 0.09 ± 13% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking 0.18 ± 8% -0.0 0.14 ± 24% perf-profile.children.cycles-pp.__list_del_entry_valid 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.verify_parent_transid 0.11 ± 19% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.22 ± 5% -0.0 0.18 ± 12% perf-profile.children.cycles-pp.check_committed_ref 0.11 ± 19% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.select_idle_sibling 0.14 ± 11% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.do_io_getevents 0.15 ± 7% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__x64_sys_io_getevents 0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_release_extent_buffer_pages 0.17 ± 18% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.clear_page_dirty_for_io 0.14 ± 5% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.block_group_cache_tree_search 0.18 ± 4% -0.0 0.14 ± 11% perf-profile.children.cycles-pp.btrfs_get_chunk_map 0.11 ± 13% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.btrfs_buffer_uptodate 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.children.cycles-pp.xas_find_marked 0.12 ± 15% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.btrfs_unlock_up_safe 0.12 ± 14% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.btrfs_set_path_blocking 0.11 ± 11% -0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_free_extent 0.24 ± 5% -0.0 0.20 ± 6% perf-profile.children.cycles-pp.btrfs_root_node 0.07 ± 6% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.extent_mergeable 0.13 ± 12% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.__switch_to 0.10 ± 18% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.rb_insert_color_cached 0.07 ± 14% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.rb_prev 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.get_io_u 0.10 ± 14% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.available_idle_cpu 0.17 ± 10% -0.0 0.13 ± 3% perf-profile.children.cycles-pp._raw_write_lock 0.10 ± 11% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep 0.17 ± 7% -0.0 0.14 ± 12% perf-profile.children.cycles-pp._raw_read_lock 0.17 ± 10% -0.0 0.14 ± 12% perf-profile.children.cycles-pp.can_overcommit 0.10 ± 11% -0.0 0.07 perf-profile.children.cycles-pp.__list_add_valid 0.09 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.aio_read_events 0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.___perf_sw_event 0.11 ± 9% -0.0 0.09 ± 18% perf-profile.children.cycles-pp.get_alloc_profile 0.11 ± 12% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.pick_next_task_idle 0.10 ± 13% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.read_events 0.08 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.replace_extent_mapping 0.10 ± 5% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.menu_reflect 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.05 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rcu_irq_enter 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.children.cycles-pp.native_irq_return_iret 0.20 ± 14% +0.1 0.27 ± 8% perf-profile.children.cycles-pp.btrfs_comp_cpu_keys 0.38 ± 9% +0.1 0.46 perf-profile.children.cycles-pp.load_balance 0.30 ± 8% +0.1 0.38 ± 15% perf-profile.children.cycles-pp.irq_enter 0.21 ± 24% +0.1 0.30 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.22 ± 3% +0.1 0.32 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline 0.52 ± 12% +0.1 0.63 ± 3% perf-profile.children.cycles-pp.rebalance_domains 1.90 ± 10% +0.3 2.23 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.23 ± 12% +0.4 0.67 ± 15% perf-profile.children.cycles-pp.read_extent_buffer 2.88 ± 11% +0.5 3.41 perf-profile.children.cycles-pp.hrtimer_interrupt 0.09 ± 12% +0.7 0.83 ± 15% perf-profile.children.cycles-pp.btrfs_get_token_8 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.children.cycles-pp.flush_space 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_marked_extents 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 0.42 ± 43% +1.3 1.67 ± 18% perf-profile.children.cycles-pp.btrfs_commit_transaction 0.25 ± 5% +1.9 2.17 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64 1.71 ± 5% +2.1 3.79 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32 0.43 ± 8% +2.4 2.84 ± 10% perf-profile.children.cycles-pp.map_private_extent_buffer 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_writeback 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.writeback_sb_inodes 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.__writeback_single_inode 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_workfn 50.06 ± 2% +3.8 53.82 ± 4% perf-profile.children.cycles-pp.intel_idle 61.99 ± 2% +4.4 66.39 ± 3% perf-profile.children.cycles-pp.do_idle 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.secondary_startup_64 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 8.51 ± 7% +4.5 13.04 ± 12% perf-profile.children.cycles-pp.do_writepages 56.43 ± 3% +4.9 61.30 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.children.cycles-pp.start_secondary 1.43 ± 7% +6.2 7.64 ± 13% perf-profile.children.cycles-pp.submit_extent_page 1.20 ± 17% +6.2 7.45 ± 13% perf-profile.children.cycles-pp.btree_write_cache_pages 1.05 ± 17% +6.3 7.31 ± 13% perf-profile.children.cycles-pp.write_one_eb 1.11 ± 7% +6.3 7.37 ± 13% perf-profile.children.cycles-pp.submit_one_bio 0.55 ± 16% +6.4 6.95 ± 13% perf-profile.children.cycles-pp.btree_submit_bio_hook 0.00 +6.5 6.48 ± 13% perf-profile.children.cycles-pp.check_leaf 0.00 +6.5 6.53 ± 13% perf-profile.children.cycles-pp.btree_csum_one_bio 2.12 ± 11% -0.8 1.29 ± 7% perf-profile.self.cycles-pp.__lookup_extent_mapping 1.83 ± 12% -0.8 1.03 ± 13% perf-profile.self.cycles-pp.__etree_search 2.06 ± 5% -0.5 1.59 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.81 ± 9% -0.3 0.52 ± 10% perf-profile.self.cycles-pp.generic_bin_search 0.84 ± 7% -0.3 0.58 ± 17% perf-profile.self.cycles-pp.copy_page 0.33 ± 54% -0.2 0.10 ± 28% perf-profile.self.cycles-pp.delay_tsc 0.89 ± 9% -0.2 0.69 ± 14% perf-profile.self.cycles-pp.btrfs_set_token_32 0.81 ± 4% -0.2 0.63 ± 14% perf-profile.self.cycles-pp.__schedule 0.51 ± 9% -0.2 0.35 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup 0.45 ± 8% -0.1 0.32 ± 10% perf-profile.self.cycles-pp.xas_load 0.43 ± 4% -0.1 0.31 ± 6% perf-profile.self.cycles-pp.find_extent_buffer 0.77 ± 5% -0.1 0.66 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.31 ± 10% -0.1 0.20 ± 28% perf-profile.self.cycles-pp.add_delayed_ref_head 0.39 ± 16% -0.1 0.29 ± 16% perf-profile.self.cycles-pp.try_to_wake_up 0.31 ± 4% -0.1 0.20 ± 26% perf-profile.self.cycles-pp.queued_write_lock_slowpath 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.20 ± 17% -0.1 0.11 ± 14% perf-profile.self.cycles-pp.kmem_cache_free 0.25 ± 11% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.__slab_free 0.16 ± 9% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.free_extent_buffer 0.28 ± 17% -0.1 0.20 ± 11% perf-profile.self.cycles-pp.btrfs_search_slot 0.31 ± 4% -0.1 0.24 ± 20% perf-profile.self.cycles-pp.kmem_cache_alloc 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.find_ref_head 0.16 ± 10% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.insert_state 0.12 ± 8% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.__update_load_avg_se 0.16 ± 15% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.rb_erase 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_unlock 0.15 ± 13% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.___slab_alloc 0.28 ± 13% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.queued_read_lock_slowpath 0.15 ± 21% -0.1 0.09 ± 35% perf-profile.self.cycles-pp.free_extent_map 0.18 ± 6% -0.1 0.12 ± 11% perf-profile.self.cycles-pp.update_load_avg 0.10 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.btrfs_extend_item 0.15 ± 23% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.add_extent_mapping 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.self.cycles-pp.___might_sleep 0.18 ± 6% -0.0 0.13 ± 22% perf-profile.self.cycles-pp.__list_del_entry_valid 0.18 ± 4% -0.0 0.14 ± 22% perf-profile.self.cycles-pp.pick_next_task_fair 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.verify_parent_transid 0.09 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_read_unlock_blocking 0.16 ± 13% -0.0 0.12 ± 24% perf-profile.self.cycles-pp.mark_page_accessed 0.12 ± 14% -0.0 0.08 ± 20% perf-profile.self.cycles-pp.dequeue_task_fair 0.10 ± 19% -0.0 0.07 ± 18% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.08 ± 14% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.unlock_up 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.self.cycles-pp.xas_find_marked 0.24 ± 6% -0.0 0.20 ± 8% perf-profile.self.cycles-pp.btrfs_root_node 0.17 ± 6% -0.0 0.13 ± 10% perf-profile.self.cycles-pp._raw_read_lock 0.10 ± 13% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.btrfs_csum_one_bio 0.16 ± 7% -0.0 0.13 ± 9% perf-profile.self.cycles-pp.dequeue_entity 0.07 -0.0 0.04 ± 71% perf-profile.self.cycles-pp.io_u_queued_complete 0.08 -0.0 0.05 ± 70% perf-profile.self.cycles-pp.mutex_lock 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.get_io_u 0.16 ± 9% -0.0 0.13 perf-profile.self.cycles-pp._raw_write_lock 0.07 ± 17% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.btrfs_tree_read_lock 0.12 ± 8% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.__switch_to 0.10 ± 11% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.__clear_extent_bit 0.07 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.rb_prev 0.10 ± 15% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.find_get_pages_range_tag 0.10 ± 12% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.available_idle_cpu 0.10 ± 11% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__list_add_valid 0.08 ± 13% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__might_sleep 0.12 ± 17% -0.0 0.09 ± 15% perf-profile.self.cycles-pp.release_extent_buffer 0.09 ± 17% -0.0 0.07 ± 14% perf-profile.self.cycles-pp.___perf_sw_event 0.08 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.__push_leaf_right 0.11 ± 7% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.set_extent_buffer_dirty 0.11 ± 4% -0.0 0.08 ± 11% perf-profile.self.cycles-pp.__set_extent_bit 0.09 ± 7% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.hrtimer_interrupt 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.07 ± 7% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.load_balance 0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles-pp.lapic_next_deadline 0.00 +0.1 0.06 ± 23% perf-profile.self.cycles-pp.find_next_and_bit 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.self.cycles-pp.native_irq_return_iret 0.19 ± 24% +0.1 0.28 ± 7% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.50 ± 8% +0.2 0.69 ± 17% perf-profile.self.cycles-pp.cpuidle_enter_state 0.07 ± 11% +0.4 0.46 ± 17% perf-profile.self.cycles-pp.btrfs_get_token_8 0.22 ± 13% +0.4 0.66 ± 16% perf-profile.self.cycles-pp.read_extent_buffer 0.00 +0.6 0.65 ± 14% perf-profile.self.cycles-pp.check_leaf 0.15 ± 11% +1.0 1.13 ± 11% perf-profile.self.cycles-pp.btrfs_get_token_64 1.48 ± 7% +1.0 2.51 ± 5% perf-profile.self.cycles-pp.btrfs_get_token_32 0.41 ± 8% +2.2 2.61 ± 9% perf-profile.self.cycles-pp.map_private_extent_buffer 49.99 ± 2% +3.7 53.74 ± 4% perf-profile.self.cycles-pp.intel_idle fio.write_bw_MBps 300 +-+-------------------------------------------------------------------+ | | 250 +-++.+ +. .+.. .+..+..+.+.. .+..+.. .+ | | : : +. +..+ + +..+ +. + | | : : : : : : | 200 +-+ : : : : : : O O | O O O: O :O O O O O O O O O O: O:O O: O: O O O O O 150 +-+ : : : : : : | | : : : : : : | 100 +-+ : : : : : : | | : : : : : : | | : : : : : : | 50 +-+ :: :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_iops 70000 +-+-----------------------------------------------------------------+ |..+.+ +..+..+.+.. .+.+..+.+..+ +.. +..+.+..+..+ | 60000 +-+ : : +. : : + : | | : : : : : : | 50000 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 40000 +-+ : O: : : : : | | : : : : : : | 30000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 10000 +-+ : : : | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_mean_us 30000 +-+-----------------------------------------------------------------+ | | 25000 O-+O O O O O O O O O O O O O O O O O O O | | O O O O O | | 20000 +-+ | |..+.+ +..+..+.+..+..+.+..+.+..+ +..+ +..+.+..+..+ | 15000 +-+ : : : : : : | | : : : : : : | 10000 +-+ : : : : : : | | : : : : : : | | : : : : : : | 5000 +-+ : : : : :: | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_stddev 160000 +-+----------------------------------------------------------------+ | | 140000 O-+O O O O O O O O O O O O O O | 120000 +-+ O O | | | 100000 +-+ O O O O O O O O | | 80000 +-+ | | .+.. | 60000 +-++.+ +..+.+. +.+..+.+..+..+ +.+ +..+..+.+..+ | 40000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 0 +-+-------------------------------------------O------------O-------+ fio.write_slat_mean_us 900 +-+-------------------------------------------------------------------+ O O O O O O O O O O O O O O O O O O | 800 +-+O O O O O O O 700 +-+ | | | 600 +-++.+ +.+..+..+..+.+..+..+.+..+ +..+ +..+..+..+.+ | 500 +-+ : : : : : : | | : : : : : : | 400 +-+ : : : : : : | 300 +-+ : : : : : : | | : : : : : : | 200 +-+ : : : : : : | 100 +-+ : : :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_slat_stddev 25000 +-+-----O---------O-------------------------------------------------+ O O O O O O O O O O O O O O O | | | 20000 +-+ | | O O O O O O O O | | 15000 +-+ | | | 10000 +-++. .+.. .+.. .+..+ .+ .+..+ | |. + +..+..+ +..+ + : +. : +..+.+. | | : : : : : : | 5000 +-+ : : : : : : | | : : : : : : | | :: : :: | 0 +-+--------------------------------------------O------------O-------+ fio.latency_1000ms_ 0.4 +-+------------------------------------------------------------------+ | | 0.35 +-+ O | 0.3 +-+ | | O O | 0.25 +-+ O O | | O O O 0.2 O-+ O O O O O O O O O O | | O O | 0.15 +-+O O O O | 0.1 +-+ | | | 0.05 +-+ | | .+. .+..+..+.+..+.. .+. .+.+..+.. | 0 +-+---------------------------------------------O------------O-------+ fio.workload 1.4e+07 +-+---------------------------------------------------------------+ |..+.+ +..+.+..+.+..+.+..+.+..+ +.. +.+..+.+..+ | 1.2e+07 +-+ : : : : + : | | : : : : : : | 1e+07 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 8e+06 +-+ : O: : : : : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | :: :: :: | 2e+06 +-+ : : : | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.voluntary_context_switches 1.4e+07 +-+---------------------------------------------------------------+ | | 1.2e+07 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | 1e+07 +-+ : : : : : : | | O O: : O O O : : :O : O O O O O O O 8e+06 O-+ : O:O O O O O O O: O:O O: : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | : : : : :: | 2e+06 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.file_system_outputs 1.2e+08 +-+---------------------------------------------------------------+ | | 1e+08 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | | : : : : : : | 8e+07 +-+ : : : : : : O O O O O O O O: O:O O O O O O O O O O: O:O O:O : O O | 6e+07 +-+ : : : : : : | | : : : : : : | 4e+07 +-+ : : : : : : | | : : : : : : | | : : : : :: | 2e+07 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 2 months

4
4
0 / 0

Re: [LKP] [btrfs] c8eaeac7b7: aim7.jobs-per-min -11.7% regression

by Rong Chen

On 6/26/19 11:17 AM, Josef Bacik wrote: > On Wed, Jun 26, 2019 at 10:39:36AM +0800, Rong Chen wrote: >> On 6/25/19 10:22 PM, Josef Bacik wrote: >>> On Fri, Jun 21, 2019 at 08:48:03AM +0800, Huang, Ying wrote: >>>> "Huang, Ying" <ying.huang(a)intel.com> writes: >>>> >>>>> "Huang, Ying" <ying.huang(a)intel.com> writes: >>>>> >>>>>> Hi, Josef, >>>>>> >>>>>> kernel test robot <rong.a.chen(a)intel.com> writes: >>>>>> >>>>>>> Greeting, >>>>>>> >>>>>>> FYI, we noticed a -11.7% regression of aim7.jobs-per-min due to commit: >>>>>>> >>>>>>> >>>>>>> commit: c8eaeac7b734347c3afba7008b7af62f37b9c140 ("btrfs: reserve >>>>>>> delalloc metadata differently") >>>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master >>>>>>> >>>>>>> in testcase: aim7 >>>>>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory >>>>>>> with following parameters: >>>>>>> >>>>>>> disk: 4BRD_12G >>>>>>> md: RAID0 >>>>>>> fs: btrfs >>>>>>> test: disk_rr >>>>>>> load: 1500 >>>>>>> cpufreq_governor: performance >>>>>>> >>>>>>> test-description: AIM7 is a traditional UNIX system level benchmark >>>>>>> suite which is used to test and measure the performance of multiuser >>>>>>> system. >>>>>>> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ >>>>>> Here's another regression, do you have time to take a look at this? >>>>> Ping >>>> Ping again ... >>>> >>> Finally got time to look at this but I can't get the reproducer to work >>> >>> root@destiny ~/lkp-tests# bin/lkp run ~/job-aim.yaml >>> Traceback (most recent call last): >>> 11: from /root/lkp-tests/bin/run-local:18:in `<main>' >>> 10: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require' >>> 9: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require' >>> 8: from /root/lkp-tests/lib/yaml.rb:5:in `<top (required)>' >>> 7: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require' >>> 6: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require' >>> 5: from /root/lkp-tests/lib/common.rb:9:in `<top (required)>' >>> 4: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require' >>> 3: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require' >>> 2: from /root/lkp-tests/lib/array_ext.rb:3:in `<top (required)>' >>> 1: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require' >>> /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- active_support/core_ext/enumerable (LoadError) >> Hi Josef, >> >> I tried the latest lkp-tests, and didn't have the problem. Could you please >> update the lkp-tests repo and run "lkp install" again? >> > I updated it this morning, and I just updated it now, my tree is on > > 2c5b1a95b08dbe81bba64419c482a877a3b424ac > > lkp install says everything is installed except > > No match for argument: libipc-run-perl I've just fixed it. could you add "libipc-run-perl: perl-IPC-Run" to the end of distro/adaptation/fedora? Thanks, Rong Chen > > and it still doesn't run properly. Thanks, > > Josef

2 years, 11 months

2
2
0 / 0

Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression

by Xing Zhengjun

On 5/31/2019 3:10 AM, Trond Myklebust wrote: > On Thu, 2019-05-30 at 15:20 +0800, Xing Zhengjun wrote: >> >> On 5/30/2019 10:00 AM, Trond Myklebust wrote: >>> Hi Xing, >>> >>> On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote: >>>> Hi Trond, >>>> >>>> On 5/20/2019 1:54 PM, kernel test robot wrote: >>>>> Greeting, >>>>> >>>>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due >>>>> to >>>>> commit: >>>>> >>>>> >>>>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: >>>>> Convert >>>>> socket page send code to use iov_iter()") >>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git >>>>> master >>>>> >>>>> in testcase: fsmark >>>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ >>>>> 3.00GHz with 384G memory >>>>> with following parameters: >>>>> >>>>> iterations: 1x >>>>> nr_threads: 64t >>>>> disk: 1BRD_48G >>>>> fs: xfs >>>>> fs2: nfsv4 >>>>> filesize: 4M >>>>> test_size: 40G >>>>> sync_method: fsyncBeforeClose >>>>> cpufreq_governor: performance >>>>> >>>>> test-description: The fsmark is a file system benchmark to test >>>>> synchronous write workloads, for example, mail servers >>>>> workload. >>>>> test-url: https://sourceforge.net/projects/fsmark/ >>>>> >>>>> >>>>> >>>>> Details are as below: >>>>> ------------------------------------------------------------- >>>>> ---- >>>>> ---------------------------------> >>>>> >>>>> >>>>> To reproduce: >>>>> >>>>> git clone https://github.com/intel/lkp-tests.git >>>>> cd lkp-tests >>>>> bin/lkp install job.yaml # job file is attached in >>>>> this >>>>> email >>>>> bin/lkp run job.yaml >>>>> >>>>> =============================================================== >>>>> ==== >>>>> ====================== >>>>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconf >>>>> ig/n >>>>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase: >>>>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel- >>>>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb- >>>>> ep01/40G/fsmark >>>>> >>>>> commit: >>>>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use >>>>> iov_iter_kvec()") >>>>> 0472e47660 ("SUNRPC: Convert socket page send code to use >>>>> iov_iter()") >>>>> >>>>> e791f8e9380d945e 0472e476604998c127f3c80d291 >>>>> ---------------- --------------------------- >>>>> fail:runs %reproduction fail:runs >>>>> | | | >>>>> :4 50% 2:4 dmesg.WARNING:a >>>>> t#for >>>>> _ip_interrupt_entry/0x >>>>> %stddev %change %stddev >>>>> \ | \ >>>>> 15118573 >>>>> ± 2% +16.0% 17538083 fsmark.app_overhead >>>>> 510.93 - >>>>> 22.7% 395.12 fsmark.files_per_sec >>>>> 24.90 +22.8% 30.57 fsmark.time.ela >>>>> psed_ >>>>> time >>>>> 24.90 +22.8% 30.57 fsmark.time.ela >>>>> psed_ >>>>> time.max >>>>> 288.00 ± 2% - >>>>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got >>>>> 70.03 ± 2% - >>>>> 11.3% 62.14 fsmark.time.system_time >>>>> >>>> >>>> Do you have time to take a look at this regression? >>> >>> From your stats, it looks to me as if the problem is increased >>> NUMA >>> overhead. Pretty much everything else appears to be the same or >>> actually performing better than previously. Am I interpreting that >>> correctly? >> The real regression is the throughput(fsmark.files_per_sec) is >> decreased >> by 22.7%. > > Understood, but I'm trying to make sense of why. I'm not able to > reproduce this, so I have to rely on your performance stats to > understand where the 22.7% regression is coming from. As far as I can > see, the only numbers in the stats you published that are showing a > performance regression (other than the fsmark number itself), are the > NUMA numbers. Is that a correct interpretation? > We re-test the case yesterday, the test result almost is the same. we will do more test and also check the test case itself, if you need more information, please let me know, thanks. >>> If my interpretation above is correct, then I'm not seeing where >>> this >>> patch would be introducing new NUMA regressions. It is just >>> converting >>> from using one method of doing socket I/O to another. Could it >>> perhaps >>> be a memory artefact due to your running the NFS client and server >>> on >>> the same machine? >>> >>> Apologies for pushing back a little, but I just don't have the >>> hardware available to test NUMA configurations, so I'm relying on >>> external testing for the above kind of scenario. >>> >> Thanks for looking at this. If you need more information, please let >> me >> know. >>> Thanks >>> Trond >>> -- Zhengjun Xing

2 years, 12 months

1
1
0 / 0

07173c3ec2: stderr.fio:pid=#, err=#/file:filesetup.c:#, func=fsync, error=Input/output_error

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable multipage bvecs") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: fio-basic on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory with following parameters: runtime: 300s nr_task: 8t disk: 1SSD fs: xfs rw: randread bs: 2M ioengine: sync test_size: 512g caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error To reproduce: git clonehttps://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Thanks, Rong Chen

3 years

3
19
0 / 0

[x86/hotplug] e1056a25da: WARNING:at_arch/x86/kernel/apic/apic.c:#setup_local_APIC

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: e1056a25daa6460c95e92d7d6853d05ad62458f7 ("x86/hotplug: Silence APIC and NMI when CPU is dead") https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git WIP.x86/ipi in testcase: locktorture with following parameters: runtime: 300s test: cpuhotplug test-description: This torture test consists of creating a number of kernel threads which acquire the lock and hold it for specific amount of time, thus simulating different critical region behaviors. test-url: https://www.kernel.org/doc/Documentation/locking/locktorture.txt on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 2G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +----------------------------------------------------------+------------+------------+ | | c1257b49b9 | e1056a25da | +----------------------------------------------------------+------------+------------+ | boot_successes | 8 | 6 | | boot_failures | 51 | 48 | | BUG:kernel_reboot-without-warning_in_test_stage | 50 | 22 | | BUG:soft_lockup-CPU##stuck_for#s | 1 | | | RIP:__memcpy | 1 | | | Kernel_panic-not_syncing:softlockup:hung_tasks | 1 | | | WARNING:at_arch/x86/kernel/apic/apic.c:#setup_local_APIC | 0 | 26 | | RIP:setup_local_APIC | 0 | 26 | +----------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 216.682229] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1473 setup_local_APIC+0x250/0x3b0 [ 216.724190] Modules linked in: locktorture torture sr_mod cdrom ata_generic pata_acpi bochs_drm ttm ppdev drm_kms_helper ata_piix syscopyarea snd_pcm sysfillrect sysimgblt libata fb_sys_fops snd_timer snd drm crc32c_intel soundcore joydev serio_raw pcspkr i2c_piix4 parport_pc parport floppy [ 216.771348] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc5-00003-ge1056a2 #1 [ 216.785027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 216.800351] RIP: 0010:setup_local_APIC+0x250/0x3b0 [ 216.809104] Code: 30 8b 0d eb 78 7d 01 85 c9 74 26 0f 31 c1 e1 0a 48 c1 e2 20 41 89 ce 4c 03 34 24 48 09 d0 49 29 c6 4d 85 f6 0f 8f 74 fe ff ff <0f> 0b e9 22 ff ff ff 49 83 ee 01 eb ea a9 00 00 01 00 8b 15 2c ac [ 216.843098] RSP: 0000:ffffffff89a03ee0 EFLAGS: 00010082 [ 216.852617] RAX: 00000076a573f693 RBX: 0000000000000020 RCX: 0000000089ff6800 [ 216.865848] RDX: 0000007600000000 RSI: 0000000000000020 RDI: ffffffff89a03ef0 [ 216.878971] RBP: 0000000000000000 R08: ffffffff8a348de2 R09: 0000000000000032 [ 216.892080] R10: ffffffff89a03ee8 R11: 0000000000000000 R12: 00000000000000f0 [ 216.905019] R13: 0000000000001000 R14: fffffffffffc1cfb R15: 0000000000000000 [ 216.918205] FS: 0000000000000000(0000) GS:ffff89af74800000(0000) knlGS:0000000000000000 [ 216.941317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 216.951812] CR2: 0000000057567184 CR3: 000000000ba0a000 CR4: 00000000000006f0 [ 216.964997] Call Trace: [ 216.969667] apic_ap_setup+0xa/0x20 [ 216.976266] start_secondary+0x78/0x1e0 [ 216.983423] secondary_startup_64+0xb6/0xc0 [ 216.991271] ---[ end trace 06172ec0274b1dfa ]--- To reproduce: # build kernel cd linux cp config-5.2.0-rc5-00003-ge1056a2 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 prepare make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 modules_prepare make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 SHELL=/bin/bash make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests find lib/ | cpio -o -H newc --quiet | gzip > modules.cgz bin/lkp qemu -k <bzImage> -m modules.cgz job-script # job-script is attached in this email Thanks, Rong Chen

3 years

4
19
0 / 0

fcc784be83 [ 150.952780] WARNING: held lock freed!

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master commit fcc784be837714a9173b372ff9fb9b514590dad9 Author: Steven Rostedt (VMware) <rostedt(a)goodmis.org> AuthorDate: Wed Apr 4 14:06:30 2018 -0400 Commit: Ingo Molnar <mingo(a)kernel.org> CommitDate: Thu Jun 21 18:19:01 2018 +0200 locking/lockdep: Do not record IRQ state within lockdep code While debugging where things were going wrong with mapping enabling/disabling interrupts with the lockdep state and actual real enabling and disabling interrupts, I had to silent the IRQ disabling/enabling in debug_check_no_locks_freed() because it was always showing up as it was called before the splat was. Use raw_local_irq_save/restore() for not only debug_check_no_locks_freed() but for all internal lockdep functions, as they hide useful information about where interrupts were used incorrectly last. Signed-off-by: Steven Rostedt (VMware) <rostedt(a)goodmis.org> Cc: Andrew Morton <akpm(a)linux-foundation.org> Cc: Linus Torvalds <torvalds(a)linux-foundation.org> Cc: Paul E. McKenney <paulmck(a)linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Thomas Gleixner <tglx(a)linutronix.de> Cc: Will Deacon <will.deacon(a)arm.com> Link: https://lkml.kernel.org/lkml/20180404140630.3f4f4c7a@gandalf.local.home Signed-off-by: Ingo Molnar <mingo(a)kernel.org> 03eeafdd9a locking/rwsem: Fix up_read_non_owner() warning with DEBUG_RWSEMS fcc784be83 locking/lockdep: Do not record IRQ state within lockdep code bed3c0d84e Merge tag 'for-5.2-rc5-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux 1c6b40509d Add linux-next specific files for 20190618 +----------------------------------------------------------------------------------+------------+------------+------------+---------------+ | | 03eeafdd9a | fcc784be83 | bed3c0d84e | next-20190618 | +----------------------------------------------------------------------------------+------------+------------+------------+---------------+ | boot_successes | 874 | 277 | 276 | 21 | | boot_failures | 14 | 19 | 64 | 7 | | BUG:soft_lockup-CPU##stuck_for#s | 11 | 2 | 4 | | | EIP:smp_call_function_single | 1 | 1 | | | | Kernel_panic-not_syncing:softlockup:hung_tasks | 11 | 2 | 4 | | | EIP:_raw_spin_unlock_irqrestore | 3 | 0 | 1 | | | EIP:__copy_user_ll | 2 | 0 | 1 | | | invoked_oom-killer:gfp_mask=0x | 1 | 2 | 1 | | | Mem-Info | 1 | 2 | 2 | | | EIP:wp_page_copy | 3 | | | | | BUG:kernel_hang_in_early-boot_stage | 1 | | | | | EIP:shmem_getpage_gfp | 2 | 0 | 1 | | | BUG:workqueue_lockup-pool | 1 | 0 | 1 | | | BUG:kernel_hang_in_boot-around-mounting-root_stage | 0 | 1 | | | | Out_of_memory:Kill_process | 0 | 1 | | | | WARNING:held_lock_freed | 0 | 13 | 58 | 7 | | is_freeing_memory#-#,with_a_lock_still_held_there | 0 | 13 | 58 | 7 | | BUG:kernel_hang_in_early-boot_stage,last_printk:early_console_in_setup_code | 0 | 1 | | | | EIP:rcu_is_watching | 0 | 1 | | | | EIP:ring_buffer_consume | 0 | 0 | 1 | | | page_allocation_failure:order:#,mode:#(GFP_KERNEL|__GFP_NORETRY),nodemask=(null) | 0 | 0 | 1 | | | BUG:unable_to_handle_page_fault_for_address | 0 | 0 | 1 | | | Oops:#[##] | 0 | 0 | 1 | | | EIP:debug_check_no_locks_freed | 0 | 0 | 1 | | | Kernel_panic-not_syncing:Fatal_exception | 0 | 0 | 1 | | +----------------------------------------------------------------------------------+------------+------------+------------+---------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 147.470176] init: tty3 main process (364) terminated with status 1 [ 147.497211] init: tty3 main process ended, respawning [ 147.612415] init: tty6 main process (365) terminated with status 1 [ 147.687364] init: tty6 main process ended, respawning [ 150.937355] [ 150.941927] ========================= [ 150.952780] WARNING: held lock freed! [ 150.963993] 4.17.0-09747-gfcc784b #1 Tainted: G T [ 150.989489] ------------------------- [ 151.005112] trinity-main/356 is freeing memory ce93de20-ce93de47, with a lock still held there! [ 151.040984] (ptrval) (&wq#3){....}, at: __wake_up_common_lock+0x5a/0x140 [ 151.069147] 1 lock held by trinity-main/356: [ 151.087139] #0: (ptrval) (&sb->s_type->i_mutex_key){....}, at: iterate_dir+0x42/0x2a0 [ 151.117861] [ 151.117861] stack backtrace: [ 151.132116] CPU: 1 PID: 356 Comm: trinity-main Tainted: G T 4.17.0-09747-gfcc784b #1 [ 151.162025] Call Trace: [ 151.170423] dump_stack+0x2bc/0x41a [ 151.182441] debug_check_no_locks_freed+0x354/0x370 [ 151.199065] __raw_spin_lock_init+0x29/0x80 [ 151.214099] __init_waitqueue_head+0x2e/0x70 [ 151.228615] ? proc_tgid_base_lookup+0x40/0x40 [ 151.243819] proc_fill_cache+0x118/0x360 [ 151.256974] ? format_decode+0x1a6/0x890 [ 151.270037] ? format_decode+0x308/0x890 [ 151.283679] ? vsnprintf+0x622/0x860 [ 151.293724] proc_map_files_readdir+0x648/0x6a0 [ 151.306619] ? proc_tgid_base_lookup+0x40/0x40 [ 151.320722] iterate_dir+0x225/0x2a0 [ 151.331637] ? __fget_light+0x7f/0x100 [ 151.342010] ksys_getdents64+0x1ac/0x2f0 [ 151.353610] ? sys_old_readdir+0x160/0x160 [ 151.366017] sys_getdents64+0x24/0x40 [ 151.377653] do_fast_syscall_32+0x14b/0x780 [ 151.392209] entry_SYSENTER_32+0x53/0x86 [ 151.404786] EIP: 0xa7fc4cd9 [ 151.423581] Code: 08 8b 80 5c cd ff ff 85 d2 74 02 89 02 5d c3 8b 04 24 c3 8b 0c 24 c3 8b 1c 24 c3 8b 3c 24 c3 90 90 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 8d 76 [ 151.547420] EAX: ffffffda EBX: 0000002b ECX: 08aefedc EDX: 00008000 [ 151.586666] ESI: 08aefedc EDI: ffffffd0 EBP: 00000026 ESP: af88aac4 [ 151.605753] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000246 [main] Added 10680 filenames from /proc [main] Added 13148 filenames from /proc [main] Added 13151 filenames from /proc [ 156.784347] init: tty4 main process (366) terminated with status 1 [ 156.849880] init: tty4 main process ended, respawning # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start v4.18 v4.17 -- git bisect bad c81b995f00c7a1c2ca9ad67f5bb4a50d02f98f84 # 23:06 B 6 1 0 0 Merge branch 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 2a70ea5cda00214a1d573acf19fa0cd06d947e38 # 23:43 G 205 0 9 9 Merge tag 'hsi-for-4.18' of git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi git bisect good e7655d2b25466c534ed1f539367dae595bb0bd20 # 00:13 G 209 0 15 15 Merge tag 'for-4.18-part2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux git bisect good 6d90eb7ba341b3eb035121eff0b69d370cbc251e # 00:44 G 206 0 18 18 Merge tag 'dma-rename-4.18' of git://git.infradead.org/users/hch/dma-mapping git bisect good 5e2204832b20ef9db859dd6a2e955ac3e33eef27 # 01:07 G 212 0 9 9 Merge tag 'powerpc-4.18-2' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux git bisect good 6242258b6b472f8fdd8ed9b735cc1190c185d16d # 01:37 G 206 0 24 24 Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad d4e860eaf0584dfcc1375e06eeb34f85f43c8d34 # 02:10 B 36 1 4 4 Merge branch 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 2da2ca24a38f0200111e3b8823c08d02cb59d362 # 03:07 B 90 1 7 7 Merge branch 'locking-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good a43de489934cadcbc4cc08a6590fdcc833768461 # 04:05 G 302 0 21 21 Merge branch 'ras-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 7ea959c45769612aa92557fb6464679f5fec7d9e # 04:37 G 306 0 18 18 locking/refcounts: Implement refcount_dec_and_lock_irqsave() git bisect bad fcc784be837714a9173b372ff9fb9b514590dad9 # 06:19 B 19 1 3 3 locking/lockdep: Do not record IRQ state within lockdep code git bisect good 03eeafdd9ab06a770d42c2b264d50dff7e2f4eee # 06:47 G 305 0 18 18 locking/rwsem: Fix up_read_non_owner() warning with DEBUG_RWSEMS # first bad commit: [fcc784be837714a9173b372ff9fb9b514590dad9] locking/lockdep: Do not record IRQ state within lockdep code git bisect good 03eeafdd9ab06a770d42c2b264d50dff7e2f4eee # 07:23 G 904 0 40 58 locking/rwsem: Fix up_read_non_owner() warning with DEBUG_RWSEMS # extra tests on HEAD of internal-eywa/master git bisect bad b4c6b079156ebc029114a45812d5e5298f51fa01 # 07:23 B 8 3 0 2 Intel Next: Add release files for v5.2-rc5 2019-06-17 # extra tests on tree/branch linus/master git bisect bad bed3c0d84e7e25c8e0964d297794f4c215b01f33 # 08:52 B 5 2 0 0 Merge tag 'for-5.2-rc5-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux # extra tests with first bad commit reverted git bisect bad b0f031ad2fd0758b5bb7c58afe7f8bbaab65c265 # 09:42 B 126 2 10 10 Revert "locking/lockdep: Do not record IRQ state within lockdep code" # extra tests on tree/branch linux-next/master git bisect bad 1c6b40509daf5190b1fd2c758649f7df1da4827b # 10:20 B 9 2 0 0 Add linux-next specific files for 20190618 --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation

3 years

3
2
0 / 0

[locking/rwsem] 7d43f1ce9d: vm-scalability.median -16.5% regression

by kernel test robot

Greeting, FYI, we noticed a -16.5% regression of vm-scalability.median due to commit: commit: 7d43f1ce9dd075d8b2aa3ad1f3970ef386a5c358 ("locking/rwsem: Enable time-based spinning on reader-owned rwsem") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: vm-scalability on test machine: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory with following parameters: runtime: 300s size: 8T test: anon-w-seq-mt cpufreq_governor: performance test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us. test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/ In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | vm-scalability: vm-scalability.median 14.0% improvement | | test machine | 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | runtime=300s | | | test=small-allocs-mt | | | ucode=0x21 | +------------------+-----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/8T/lkp-csl-2ap1/anon-w-seq-mt/vm-scalability commit: 94a9717b3c ("locking/rwsem: Make rwsem->owner an atomic_long_t") 7d43f1ce9d ("locking/rwsem: Enable time-based spinning on reader-owned rwsem") 94a9717b3c40e77a 7d43f1ce9dd075d8b2aa3ad1f39 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 0.01 ± 2% +166.9% 0.02 ± 6% vm-scalability.free_time 191759 -16.5% 160104 vm-scalability.median 0.04 ± 4% +163.7% 0.11 ± 8% vm-scalability.median_stddev 39109084 -7.1% 36338578 vm-scalability.throughput 306656 ± 2% -64.3% 109562 ± 5% vm-scalability.time.involuntary_context_switches 89394993 -8.2% 82031710 ± 4% vm-scalability.time.maximum_resident_set_size 13786907 -23.1% 10603356 ± 2% vm-scalability.time.minor_page_faults 15225 -29.2% 10777 ± 2% vm-scalability.time.percent_of_cpu_this_job_got 22047 -23.6% 16835 vm-scalability.time.system_time 23929 -34.3% 15715 ± 2% vm-scalability.time.user_time 1555880 -47.0% 825313 ± 2% vm-scalability.time.voluntary_context_switches 1.037e+10 -23.1% 7.98e+09 ± 2% vm-scalability.workload 37.83 +1.0% 38.20 boot-time.boot 18197810 ±138% -83.6% 2988363 ± 7% cpuidle.C1.time 21000317 ± 14% +109.1% 43903618 ± 21% cpuidle.C1E.usage 119868 ± 42% +54.1% 184715 ± 19% cpuidle.POLL.usage 20.00 ± 3% +113.8% 42.75 ± 4% vmstat.cpu.id 40.75 -34.4% 26.75 ± 3% vmstat.cpu.us 158.50 -31.1% 109.25 vmstat.procs.r 15578 -34.5% 10207 vmstat.system.cs 19.72 ± 2% +23.2 42.90 ± 3% mpstat.cpu.all.idle% 0.07 ± 5% -0.1 0.01 ± 36% mpstat.cpu.all.iowait% 0.00 ± 9% -0.0 0.00 ± 12% mpstat.cpu.all.soft% 38.55 -9.0 29.59 mpstat.cpu.all.sys% 41.67 -14.2 27.49 ± 2% mpstat.cpu.all.usr% 3990675 -22.9% 3074963 ± 2% numa-numastat.node1.local_node 4021819 -23.1% 3092032 ± 2% numa-numastat.node1.numa_hit 31144 -45.2% 17068 ± 64% numa-numastat.node1.other_node 4722883 ± 29% -33.2% 3155351 ± 3% numa-numastat.node2.local_node 4747771 ± 29% -32.9% 3186380 ± 3% numa-numastat.node2.numa_hit 2404 -28.1% 1728 ± 2% turbostat.Avg_MHz 80.61 -22.4 58.22 ± 2% turbostat.Busy% 20996179 ± 15% +109.1% 43902033 ± 21% turbostat.C1E 18.85 ± 2% +115.1% 40.56 ± 3% turbostat.CPU%c1 477.24 -11.9% 420.57 turbostat.PkgWatt 17150 ± 3% +13.7% 19503 ± 3% slabinfo.cred_jar.active_objs 17150 ± 3% +13.7% 19503 ± 3% slabinfo.cred_jar.num_objs 1757 ± 5% -11.9% 1547 ± 4% slabinfo.kmalloc-rcl-96.active_objs 1757 ± 5% -11.9% 1547 ± 4% slabinfo.kmalloc-rcl-96.num_objs 17269 -11.2% 15337 ± 2% slabinfo.task_delay_info.active_objs 17269 -11.2% 15337 ± 2% slabinfo.task_delay_info.num_objs 45090908 -30.3% 31436724 ± 2% meminfo.Active 45090677 -30.3% 31436484 ± 2% meminfo.Active(anon) 33671307 -30.5% 23416058 meminfo.AnonHugePages 44973813 -30.4% 31321842 ± 2% meminfo.AnonPages 90656336 -19.4% 73097429 meminfo.Committed_AS 568845 ± 10% -23.9% 432780 ± 4% meminfo.DirectMap4k 47516120 -28.9% 33772543 ± 2% meminfo.Memused 79828 -28.6% 56964 ± 2% meminfo.PageTables 303260 ± 2% -10.9% 270236 ± 4% meminfo.max_used_kB 11010213 ± 3% -27.8% 7949775 ± 22% numa-meminfo.node0.Active 11010184 ± 3% -27.8% 7949699 ± 22% numa-meminfo.node0.Active(anon) 10955056 ± 3% -27.7% 7918969 ± 22% numa-meminfo.node0.AnonPages 11634902 ± 3% -26.3% 8570261 ± 21% numa-meminfo.node0.MemUsed 11216510 ± 3% -40.5% 6673714 ± 5% numa-meminfo.node1.Active 11216510 ± 3% -40.5% 6673685 ± 5% numa-meminfo.node1.Active(anon) 8326268 ± 4% -40.7% 4934615 ± 5% numa-meminfo.node1.AnonHugePages 11198925 ± 3% -40.6% 6650406 ± 5% numa-meminfo.node1.AnonPages 37759405 +12.1% 42320486 numa-meminfo.node1.MemFree 11767954 ± 3% -38.8% 7206873 ± 5% numa-meminfo.node1.MemUsed 18759 ± 2% -37.9% 11650 ± 5% numa-meminfo.node1.PageTables 11526969 ± 9% -41.4% 6752594 ± 9% numa-meminfo.node2.Active 11526927 ± 9% -41.4% 6752552 ± 9% numa-meminfo.node2.Active(anon) 8576117 ± 10% -41.9% 4982950 ± 10% numa-meminfo.node2.AnonHugePages 11505263 ± 9% -41.5% 6735650 ± 9% numa-meminfo.node2.AnonPages 37426528 ± 2% +12.9% 42249731 numa-meminfo.node2.MemFree 12100840 ± 9% -39.9% 7277636 ± 8% numa-meminfo.node2.MemUsed 19912 ± 13% -37.6% 12427 ± 13% numa-meminfo.node2.PageTables 7507 ± 25% -25.9% 5563 ± 5% numa-meminfo.node3.Mapped 2653570 ± 3% -25.9% 1965712 ± 22% numa-vmstat.node0.nr_active_anon 2650970 ± 3% -25.9% 1964507 ± 22% numa-vmstat.node0.nr_zone_active_anon 2682266 ± 4% -37.6% 1672693 ± 5% numa-vmstat.node1.nr_active_anon 2677270 ± 4% -37.8% 1666244 ± 5% numa-vmstat.node1.nr_anon_pages 3905 ± 6% -38.4% 2405 ± 5% numa-vmstat.node1.nr_anon_transparent_hugepages 9564136 +10.6% 10577037 numa-vmstat.node1.nr_free_pages 4608 ± 3% -37.0% 2901 ± 5% numa-vmstat.node1.nr_page_table_pages 2680195 ± 4% -37.6% 1671790 ± 5% numa-vmstat.node1.nr_zone_active_anon 2295337 ± 5% -20.7% 1821302 ± 5% numa-vmstat.node1.numa_hit 2180885 ± 5% -21.1% 1720231 ± 5% numa-vmstat.node1.numa_local 2839093 ± 10% -39.9% 1707571 ± 11% numa-vmstat.node2.nr_active_anon 2833805 ± 10% -39.9% 1703266 ± 11% numa-vmstat.node2.nr_anon_pages 4158 ± 12% -40.7% 2465 ± 12% numa-vmstat.node2.nr_anon_transparent_hugepages 25.50 ±116% +204.9% 77.75 ± 80% numa-vmstat.node2.nr_dirtied 9402050 ± 3% +12.1% 10544232 numa-vmstat.node2.nr_free_pages 34.50 ±131% +145.7% 84.75 ± 76% numa-vmstat.node2.nr_inactive_file 4980 ± 14% -37.3% 3120 ± 12% numa-vmstat.node2.nr_page_table_pages 24.25 ±123% +193.8% 71.25 ± 79% numa-vmstat.node2.nr_written 2836803 ± 10% -39.8% 1706487 ± 11% numa-vmstat.node2.nr_zone_active_anon 34.50 ±131% +145.7% 84.75 ± 76% numa-vmstat.node2.nr_zone_inactive_file 2612725 ± 30% -27.7% 1887879 ± 6% numa-vmstat.node2.numa_hit 2504430 ± 31% -29.2% 1773066 ± 7% numa-vmstat.node2.numa_local 11388953 -31.7% 7777830 proc-vmstat.nr_active_anon 11358111 -31.8% 7748673 proc-vmstat.nr_anon_pages 16635 -31.6% 11372 proc-vmstat.nr_anon_transparent_hugepages 441.75 +20.8% 533.75 proc-vmstat.nr_dirtied 3718102 +9.8% 4082210 proc-vmstat.nr_dirty_background_threshold 7448721 +9.8% 8175520 proc-vmstat.nr_dirty_threshold 37434915 +9.7% 41069758 proc-vmstat.nr_free_pages 529.50 +8.3% 573.25 proc-vmstat.nr_inactive_file 28930 -1.0% 28637 proc-vmstat.nr_kernel_stack 9789 ± 2% -3.2% 9472 proc-vmstat.nr_mapped 20077 -29.2% 14204 proc-vmstat.nr_page_table_pages 46878 -5.0% 44552 ± 2% proc-vmstat.nr_shmem 416.00 +20.5% 501.25 proc-vmstat.nr_written 11388927 -31.7% 7777820 proc-vmstat.nr_zone_active_anon 529.50 +8.3% 573.25 proc-vmstat.nr_zone_inactive_file 19364148 -21.9% 15132755 ± 2% proc-vmstat.numa_hit 14619 ± 5% -43.5% 8257 ± 10% proc-vmstat.numa_huge_pte_updates 19270325 -22.0% 15039253 ± 2% proc-vmstat.numa_local 7544881 ± 5% -43.3% 4275462 ± 10% proc-vmstat.numa_pte_updates 27144 ± 5% -30.7% 18821 ± 3% proc-vmstat.pgactivate 2.311e+09 -23.1% 1.776e+09 ± 2% proc-vmstat.pgalloc_normal 14816778 -21.5% 11633019 ± 2% proc-vmstat.pgfault 2.308e+09 -23.4% 1.767e+09 ± 2% proc-vmstat.pgfree 4478662 -23.4% 3428568 ± 2% proc-vmstat.thp_deferred_split_page 4483698 -23.1% 3446054 ± 2% proc-vmstat.thp_fault_alloc 108799 ± 11% -36.7% 68886 ± 3% sched_debug.cfs_rq:/.exec_clock.avg 117668 ± 11% -22.3% 91468 ± 2% sched_debug.cfs_rq:/.exec_clock.max 98854 ± 11% -48.6% 50806 ± 4% sched_debug.cfs_rq:/.exec_clock.min 4923 ± 19% +124.0% 11028 ± 11% sched_debug.cfs_rq:/.exec_clock.stddev 285.55 ± 16% -17.5% 235.65 ± 20% sched_debug.cfs_rq:/.load_avg.max 1.28 ± 35% -100.0% 0.00 sched_debug.cfs_rq:/.load_avg.min 20824703 ± 11% -49.5% 10507532 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg 22103859 ± 12% -43.6% 12472189 ± 3% sched_debug.cfs_rq:/.min_vruntime.max 18960410 ± 12% -58.0% 7969687 ± 5% sched_debug.cfs_rq:/.min_vruntime.min 649563 ± 30% +62.7% 1056596 ± 12% sched_debug.cfs_rq:/.min_vruntime.stddev 0.64 ± 13% -24.2% 0.48 ± 9% sched_debug.cfs_rq:/.nr_running.avg 0.31 ± 19% +28.6% 0.40 ± 8% sched_debug.cfs_rq:/.nr_running.stddev 9.18 ± 10% -33.1% 6.15 ± 4% sched_debug.cfs_rq:/.nr_spread_over.avg 65.36 ± 20% -35.4% 42.25 ± 33% sched_debug.cfs_rq:/.nr_spread_over.max 7.30 ± 15% -33.5% 4.85 ± 13% sched_debug.cfs_rq:/.nr_spread_over.stddev 1791102 ± 26% +35.2% 2420709 ± 10% sched_debug.cfs_rq:/.spread0.avg 3067181 ± 20% +42.7% 4377704 ± 7% sched_debug.cfs_rq:/.spread0.max 651962 ± 30% +62.0% 1055918 ± 12% sched_debug.cfs_rq:/.spread0.stddev 773.26 ± 7% -20.2% 617.26 ± 4% sched_debug.cfs_rq:/.util_avg.avg 188.92 ± 19% -99.4% 1.05 ±105% sched_debug.cfs_rq:/.util_avg.min 184.74 ± 23% +58.9% 293.65 ± 12% sched_debug.cfs_rq:/.util_avg.stddev 1166 ± 10% -17.1% 966.80 ± 8% sched_debug.cfs_rq:/.util_est_enqueued.max 126.46 ± 15% -46.2% 67.98 ± 29% sched_debug.cpu.clock.stddev 126.46 ± 15% -46.2% 67.98 ± 29% sched_debug.cpu.clock_task.stddev 10644 ± 12% -37.8% 6622 ± 7% sched_debug.cpu.curr->pid.avg 15239 ± 9% -20.2% 12163 sched_debug.cpu.curr->pid.max 0.00 ± 15% -49.1% 0.00 ± 28% sched_debug.cpu.next_balance.stddev 144767 ± 10% -11.1% 128756 sched_debug.cpu.nr_load_updates.avg 125750 ± 9% -17.2% 104090 ± 10% sched_debug.cpu.nr_load_updates.min 0.64 ± 13% -28.8% 0.45 ± 7% sched_debug.cpu.nr_running.avg 1.51 ± 7% -23.8% 1.15 ± 14% sched_debug.cpu.nr_running.max 11047 ± 12% -41.7% 6438 sched_debug.cpu.nr_switches.avg 6783 ± 10% -44.3% 3775 ± 3% sched_debug.cpu.nr_switches.min 0.16 ± 51% +91.3% 0.30 ± 25% sched_debug.cpu.nr_uninterruptible.avg -23.28 +27.6% -29.70 sched_debug.cpu.nr_uninterruptible.min 14.58 ± 2% +46.8% 21.41 sched_debug.cpu.nr_uninterruptible.stddev 10564 ± 12% -44.1% 5908 sched_debug.cpu.sched_count.avg 6543 ± 11% -45.8% 3546 ± 2% sched_debug.cpu.sched_count.min 4217 ± 11% -37.3% 2643 sched_debug.cpu.sched_goidle.avg 2673 ± 12% -47.3% 1409 ± 15% sched_debug.cpu.sched_goidle.min 5061 ± 12% -42.6% 2902 ± 2% sched_debug.cpu.ttwu_count.avg 2587 ± 14% -53.7% 1197 ± 3% sched_debug.cpu.ttwu_count.min 818.87 ± 18% -52.2% 391.10 ± 2% sched_debug.cpu.ttwu_local.avg 17144 ± 28% -64.4% 6106 ± 25% sched_debug.cpu.ttwu_local.max 228.22 ± 12% -20.1% 182.40 ± 2% sched_debug.cpu.ttwu_local.min 1796 ± 28% -72.7% 489.75 ± 17% sched_debug.cpu.ttwu_local.stddev 5.02 -4.6% 4.79 ± 2% perf-stat.i.MPKI 3.584e+10 -20.8% 2.839e+10 ± 2% perf-stat.i.branch-instructions 0.02 ± 5% +0.0 0.03 ± 25% perf-stat.i.branch-miss-rate% 70.15 -10.3 59.89 ± 2% perf-stat.i.cache-miss-rate% 3.988e+08 -30.7% 2.762e+08 ± 2% perf-stat.i.cache-misses 5.652e+08 -21.8% 4.419e+08 perf-stat.i.cache-references 15914 -35.2% 10304 perf-stat.i.context-switches 3.81 -14.1% 3.27 perf-stat.i.cpi 4.361e+11 -28.3% 3.126e+11 perf-stat.i.cpu-cycles 305.95 ± 3% -16.3% 255.98 ± 2% perf-stat.i.cpu-migrations 1080 +18.5% 1280 ± 6% perf-stat.i.cycles-between-cache-misses 2.472e+10 -18.0% 2.026e+10 perf-stat.i.dTLB-loads 0.00 ± 2% +0.0 0.00 ± 16% perf-stat.i.dTLB-store-miss-rate% 301820 ± 2% -15.5% 255096 ± 9% perf-stat.i.dTLB-store-misses 8.332e+09 -22.0% 6.497e+09 ± 2% perf-stat.i.dTLB-stores 77.38 -14.2 63.16 perf-stat.i.iTLB-load-miss-rate% 2037871 +6.8% 2177306 ± 2% perf-stat.i.iTLB-load-misses 762134 +85.5% 1413938 ± 4% perf-stat.i.iTLB-loads 1.129e+11 -19.1% 9.132e+10 ± 2% perf-stat.i.instructions 59923 -25.4% 44683 perf-stat.i.instructions-per-iTLB-miss 0.27 +22.8% 0.33 perf-stat.i.ipc 49321 -21.9% 38543 ± 2% perf-stat.i.minor-faults 18.01 +4.5 22.49 perf-stat.i.node-load-miss-rate% 1432415 -12.0% 1260792 perf-stat.i.node-load-misses 6775281 -28.9% 4818204 ± 2% perf-stat.i.node-loads 1.205e+08 -26.7% 88302418 ± 2% perf-stat.i.node-stores 49298 -21.8% 38548 ± 2% perf-stat.i.page-faults 5.00 -3.4% 4.84 perf-stat.overall.MPKI 0.02 ± 5% +0.0 0.03 ± 20% perf-stat.overall.branch-miss-rate% 70.65 -8.0 62.65 ± 2% perf-stat.overall.cache-miss-rate% 3.87 -11.1% 3.44 perf-stat.overall.cpi 1094 +3.8% 1135 perf-stat.overall.cycles-between-cache-misses 72.93 -12.1 60.85 perf-stat.overall.iTLB-load-miss-rate% 55490 -23.8% 42278 perf-stat.overall.instructions-per-iTLB-miss 0.26 +12.5% 0.29 perf-stat.overall.ipc 17.41 +3.4 20.83 perf-stat.overall.node-load-miss-rate% 3179 +6.5% 3387 perf-stat.overall.path-length 3.561e+10 -20.3% 2.836e+10 ± 2% perf-stat.ps.branch-instructions 3.966e+08 -30.3% 2.764e+08 ± 2% perf-stat.ps.cache-misses 5.614e+08 -21.4% 4.412e+08 perf-stat.ps.cache-references 15636 -35.3% 10116 perf-stat.ps.context-switches 4.339e+11 -27.7% 3.139e+11 perf-stat.ps.cpu-cycles 304.71 ± 3% -15.8% 256.64 ± 2% perf-stat.ps.cpu-migrations 2.457e+10 -17.6% 2.025e+10 perf-stat.ps.dTLB-loads 299980 ± 2% -15.6% 253225 ± 9% perf-stat.ps.dTLB-store-misses 8.285e+09 -21.6% 6.491e+09 ± 2% perf-stat.ps.dTLB-stores 2021880 +6.8% 2158636 ± 2% perf-stat.ps.iTLB-load-misses 750297 +85.3% 1389972 ± 4% perf-stat.ps.iTLB-loads 1.122e+11 -18.7% 9.126e+10 ± 2% perf-stat.ps.instructions 49039 -21.8% 38340 ± 2% perf-stat.ps.minor-faults 1411907 -11.6% 1248139 perf-stat.ps.node-load-misses 6699062 -29.2% 4743984 ± 2% perf-stat.ps.node-loads 1.2e+08 -26.2% 88495173 ± 2% perf-stat.ps.node-stores 49041 -21.8% 38339 ± 2% perf-stat.ps.page-faults 3.298e+13 -18.0% 2.704e+13 ± 2% perf-stat.total.instructions 69.87 ± 6% -11.9 57.98 ± 16% perf-profile.calltrace.cycles-pp.do_access 42.62 ± 6% -9.0 33.64 ± 17% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_access 30.80 ± 6% -7.6 23.21 ± 18% perf-profile.calltrace.cycles-pp.do_rw_once 1.48 ± 6% -0.4 1.07 ± 28% perf-profile.calltrace.cycles-pp.rcu_all_qs._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault 0.72 ± 4% -0.3 0.47 ± 59% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.do_rw_once 0.84 ± 6% -0.2 0.65 ± 11% perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 0.86 ± 5% -0.2 0.70 ± 18% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.do_rw_once 0.00 +0.7 0.72 ± 20% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 0.00 +0.8 0.79 ± 21% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 0.00 +4.4 4.43 ± 12% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault.do_access 0.00 +4.8 4.81 ± 22% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault 0.00 +5.0 4.98 ± 23% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault 2.90 ± 9% +7.8 10.71 ± 16% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 3.22 ± 9% +8.3 11.56 ± 16% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 3.24 ± 9% +8.4 11.64 ± 16% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 3.43 ± 9% +8.6 12.08 ± 16% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 3.43 ± 9% +8.6 12.08 ± 16% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 3.43 ± 9% +8.7 12.08 ± 16% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 3.47 ± 9% +8.7 12.21 ± 16% perf-profile.calltrace.cycles-pp.secondary_startup_64 40.01 ± 6% -9.8 30.18 ± 18% perf-profile.children.cycles-pp.do_rw_once 0.24 ± 5% -0.0 0.20 ± 10% perf-profile.children.cycles-pp.do_anonymous_page 0.19 ± 6% -0.0 0.14 ± 9% perf-profile.children.cycles-pp.alloc_pages_vma 0.07 ± 17% +0.0 0.09 ± 4% perf-profile.children.cycles-pp.update_sd_lb_stats 0.07 ± 17% +0.0 0.10 ± 5% perf-profile.children.cycles-pp.find_busiest_group 0.07 ± 10% +0.0 0.11 ± 28% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.08 ± 13% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.load_balance 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.unmap_vmas 0.01 ±173% +0.0 0.06 ± 9% perf-profile.children.cycles-pp.unmap_page_range 0.17 ± 14% +0.1 0.22 ± 21% perf-profile.children.cycles-pp.munmap 0.00 +0.1 0.06 ± 22% perf-profile.children.cycles-pp.lapic_next_deadline 0.06 ± 13% +0.1 0.12 ± 6% perf-profile.children.cycles-pp.rebalance_domains 0.22 ± 5% +0.1 0.30 ± 9% perf-profile.children.cycles-pp.__softirqentry_text_start 0.13 ± 14% +0.1 0.22 ± 7% perf-profile.children.cycles-pp.__free_pages_ok 0.06 ± 11% +0.1 0.16 ± 14% perf-profile.children.cycles-pp.tick_nohz_next_event 0.17 ± 13% +0.1 0.27 ± 5% perf-profile.children.cycles-pp.release_pages 0.27 ± 5% +0.1 0.38 ± 8% perf-profile.children.cycles-pp.irq_exit 0.17 ± 13% +0.1 0.27 ± 7% perf-profile.children.cycles-pp.tlb_finish_mmu 0.17 ± 13% +0.1 0.27 ± 7% perf-profile.children.cycles-pp.tlb_flush_mmu 0.07 ± 10% +0.1 0.18 ± 17% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.21 ± 14% +0.1 0.33 ± 8% perf-profile.children.cycles-pp.__do_munmap 0.21 ± 15% +0.1 0.34 ± 7% perf-profile.children.cycles-pp.__vm_munmap 0.19 ± 13% +0.1 0.32 ± 7% perf-profile.children.cycles-pp.unmap_region 0.21 ± 14% +0.1 0.34 ± 7% perf-profile.children.cycles-pp.__x64_sys_munmap 0.23 ± 19% +0.2 0.40 ± 16% perf-profile.children.cycles-pp.update_cfs_group 0.11 ± 12% +0.2 0.29 ± 22% perf-profile.children.cycles-pp.menu_select 0.81 ± 13% +0.2 1.00 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.69 ± 5% +0.2 0.92 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 0.69 ± 5% +0.2 0.92 ± 7% perf-profile.children.cycles-pp.do_syscall_64 0.00 +4.8 4.82 ± 21% perf-profile.children.cycles-pp.osq_lock 0.00 +5.0 5.00 ± 23% perf-profile.children.cycles-pp.rwsem_optimistic_spin 0.10 ± 13% +5.0 5.12 ± 22% perf-profile.children.cycles-pp.rwsem_down_read_slowpath 2.95 ± 10% +7.9 10.83 ± 16% perf-profile.children.cycles-pp.intel_idle 3.29 ± 9% +8.5 11.76 ± 16% perf-profile.children.cycles-pp.cpuidle_enter 3.29 ± 9% +8.5 11.76 ± 16% perf-profile.children.cycles-pp.cpuidle_enter_state 3.43 ± 9% +8.6 12.08 ± 16% perf-profile.children.cycles-pp.start_secondary 3.47 ± 9% +8.7 12.21 ± 16% perf-profile.children.cycles-pp.do_idle 3.47 ± 9% +8.7 12.21 ± 16% perf-profile.children.cycles-pp.secondary_startup_64 3.47 ± 9% +8.7 12.21 ± 16% perf-profile.children.cycles-pp.cpu_startup_entry 38.36 ± 6% -9.6 28.80 ± 18% perf-profile.self.cycles-pp.do_rw_once 7.60 ± 6% -2.5 5.07 ± 18% perf-profile.self.cycles-pp.do_access 0.10 ± 5% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.__handle_mm_fault 0.04 ± 59% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.update_sd_lb_stats 0.06 ± 13% +0.0 0.11 ± 25% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.11 ± 13% +0.1 0.19 ± 8% perf-profile.self.cycles-pp.__free_pages_ok 0.00 +0.1 0.09 ± 28% perf-profile.self.cycles-pp.menu_select 0.00 +0.1 0.10 ± 22% perf-profile.self.cycles-pp.cpuidle_enter_state 0.00 +0.2 0.15 ± 57% perf-profile.self.cycles-pp.rwsem_optimistic_spin 0.23 ± 19% +0.2 0.40 ± 16% perf-profile.self.cycles-pp.update_cfs_group 0.00 +4.8 4.77 ± 21% perf-profile.self.cycles-pp.osq_lock 2.94 ± 10% +7.9 10.83 ± 16% perf-profile.self.cycles-pp.intel_idle 30379 ± 4% -12.5% 26581 softirqs.CPU0.RCU 17842 ± 4% +62.7% 29021 ± 8% softirqs.CPU0.SCHED 12748 ± 10% +62.8% 20757 ± 13% softirqs.CPU1.SCHED 112729 ± 4% +8.4% 122207 softirqs.CPU1.TIMER 32056 ± 3% -14.0% 27582 ± 2% softirqs.CPU10.RCU 13372 ± 7% +48.7% 19891 ± 15% softirqs.CPU10.SCHED 13163 ± 13% +56.7% 20628 ± 17% softirqs.CPU100.SCHED 13220 ± 8% +56.5% 20689 ± 16% softirqs.CPU101.SCHED 111761 ± 5% +8.5% 121220 softirqs.CPU101.TIMER 13280 ± 10% +56.7% 20806 ± 14% softirqs.CPU102.SCHED 31322 ± 5% -11.2% 27817 softirqs.CPU103.RCU 13245 ± 8% +54.5% 20461 ± 14% softirqs.CPU103.SCHED 13470 ± 9% +55.2% 20910 ± 17% softirqs.CPU104.SCHED 31043 ± 5% -11.0% 27631 softirqs.CPU105.RCU 13171 ± 8% +57.6% 20753 ± 16% softirqs.CPU105.SCHED 32204 ± 5% -14.8% 27443 softirqs.CPU106.RCU 13560 ± 4% +51.8% 20590 ± 17% softirqs.CPU106.SCHED 31484 ± 5% -8.8% 28711 ± 4% softirqs.CPU107.RCU 13376 ± 9% +52.6% 20415 ± 17% softirqs.CPU107.SCHED 13179 ± 9% +53.2% 20196 ± 15% softirqs.CPU108.SCHED 30932 ± 5% -9.5% 28009 ± 3% softirqs.CPU109.RCU 13315 ± 9% +53.1% 20390 ± 16% softirqs.CPU109.SCHED 31762 ± 4% -10.3% 28485 softirqs.CPU11.RCU 13133 ± 8% +56.5% 20559 ± 15% softirqs.CPU11.SCHED 31055 ± 4% -11.0% 27630 softirqs.CPU110.RCU 13271 ± 9% +54.0% 20443 ± 17% softirqs.CPU110.SCHED 31417 ± 4% -12.9% 27357 softirqs.CPU111.RCU 13352 ± 9% +53.5% 20500 ± 17% softirqs.CPU111.SCHED 31743 ± 4% -14.0% 27295 softirqs.CPU112.RCU 13334 ± 9% +53.4% 20461 ± 16% softirqs.CPU112.SCHED 31649 ± 4% -12.2% 27796 softirqs.CPU113.RCU 13586 ± 9% +51.2% 20542 ± 16% softirqs.CPU113.SCHED 32529 ± 8% -13.3% 28213 ± 2% softirqs.CPU114.RCU 13423 ± 8% +51.9% 20388 ± 16% softirqs.CPU114.SCHED 31879 ± 6% -13.3% 27646 softirqs.CPU115.RCU 13357 ± 9% +53.0% 20442 ± 17% softirqs.CPU115.SCHED 31622 ± 4% -9.7% 28569 ± 5% softirqs.CPU116.RCU 13626 ± 9% +52.6% 20789 ± 14% softirqs.CPU116.SCHED 31089 ± 5% -11.8% 27412 softirqs.CPU117.RCU 12525 ± 14% +66.6% 20866 ± 17% softirqs.CPU117.SCHED 13348 ± 9% +55.9% 20815 ± 17% softirqs.CPU118.SCHED 112643 ± 4% +8.5% 122261 softirqs.CPU118.TIMER 31601 ± 4% -12.8% 27540 softirqs.CPU119.RCU 13363 ± 8% +49.6% 19995 ± 16% softirqs.CPU119.SCHED 33443 ± 12% -14.6% 28564 softirqs.CPU12.RCU 13720 ± 12% +49.4% 20492 ± 14% softirqs.CPU12.SCHED 31831 ± 5% -15.2% 26996 ± 2% softirqs.CPU120.RCU 12720 ± 9% +76.8% 22491 ± 2% softirqs.CPU120.SCHED 32010 ± 3% -15.3% 27101 ± 2% softirqs.CPU121.RCU 12802 ± 11% +75.2% 22431 ± 3% softirqs.CPU121.SCHED 30635 ± 5% -8.7% 27972 ± 6% softirqs.CPU122.RCU 12666 ± 12% +80.5% 22856 ± 4% softirqs.CPU122.SCHED 31969 ± 2% -15.4% 27041 ± 3% softirqs.CPU123.RCU 12565 ± 12% +79.2% 22518 ± 3% softirqs.CPU123.SCHED 31849 ± 3% -15.4% 26938 ± 2% softirqs.CPU124.RCU 12703 ± 10% +75.7% 22323 ± 3% softirqs.CPU124.SCHED 32084 ± 3% -15.6% 27087 ± 2% softirqs.CPU125.RCU 12639 ± 10% +76.3% 22280 ± 2% softirqs.CPU125.SCHED 31975 ± 3% -15.7% 26959 ± 2% softirqs.CPU126.RCU 12668 ± 9% +72.4% 21836 ± 5% softirqs.CPU126.SCHED 31527 ± 5% -14.3% 27020 ± 2% softirqs.CPU127.RCU 12542 ± 9% +79.4% 22495 ± 3% softirqs.CPU127.SCHED 31366 ± 2% -12.8% 27348 ± 2% softirqs.CPU128.RCU 12644 ± 11% +76.6% 22330 ± 3% softirqs.CPU128.SCHED 31189 ± 2% -11.8% 27499 ± 2% softirqs.CPU129.RCU 12696 ± 9% +77.1% 22480 softirqs.CPU129.SCHED 13020 ± 8% +56.6% 20385 ± 15% softirqs.CPU13.SCHED 31251 ± 3% -11.5% 27664 ± 2% softirqs.CPU130.RCU 12720 ± 9% +75.6% 22337 ± 3% softirqs.CPU130.SCHED 31795 -13.0% 27663 ± 2% softirqs.CPU131.RCU 12715 ± 11% +75.1% 22266 ± 2% softirqs.CPU131.SCHED 31456 ± 3% -12.1% 27662 ± 3% softirqs.CPU132.RCU 12661 ± 10% +76.7% 22370 ± 2% softirqs.CPU132.SCHED 30812 ± 4% -12.6% 26922 ± 3% softirqs.CPU133.RCU 12542 ± 12% +82.2% 22851 ± 5% softirqs.CPU133.SCHED 31150 ± 2% -12.7% 27197 ± 3% softirqs.CPU134.RCU 12669 ± 11% +74.1% 22063 ± 3% softirqs.CPU134.SCHED 31286 ± 3% -11.6% 27648 ± 2% softirqs.CPU135.RCU 12685 ± 9% +80.7% 22921 ± 3% softirqs.CPU135.SCHED 31466 ± 2% -12.3% 27610 ± 3% softirqs.CPU136.RCU 12643 ± 10% +78.4% 22560 ± 2% softirqs.CPU136.SCHED 31550 ± 2% -12.9% 27464 ± 2% softirqs.CPU137.RCU 12744 ± 9% +75.0% 22309 ± 2% softirqs.CPU137.SCHED 31343 ± 3% -12.5% 27418 ± 2% softirqs.CPU138.RCU 12741 ± 10% +68.8% 21506 ± 12% softirqs.CPU138.SCHED 31628 ± 2% -13.0% 27505 ± 2% softirqs.CPU139.RCU 12659 ± 11% +76.9% 22398 ± 6% softirqs.CPU139.SCHED 33226 ± 9% -16.0% 27898 softirqs.CPU14.RCU 13074 ± 7% +51.3% 19779 ± 16% softirqs.CPU14.SCHED 31554 ± 3% -11.3% 27993 ± 3% softirqs.CPU140.RCU 12730 ± 9% +78.0% 22663 ± 4% softirqs.CPU140.SCHED 12497 ± 13% +82.2% 22769 ± 3% softirqs.CPU141.SCHED 31195 ± 4% -12.3% 27362 ± 4% softirqs.CPU142.RCU 12640 ± 12% +81.9% 22989 ± 6% softirqs.CPU142.SCHED 31194 ± 4% -13.2% 27074 ± 4% softirqs.CPU143.RCU 12716 ± 11% +72.5% 21937 ± 13% softirqs.CPU143.SCHED 31035 ± 4% -9.9% 27958 ± 2% softirqs.CPU144.RCU 13145 ± 8% +72.0% 22605 ± 7% softirqs.CPU144.SCHED 31116 ± 3% -11.4% 27567 softirqs.CPU145.RCU 13086 ± 8% +70.4% 22297 ± 6% softirqs.CPU145.SCHED 30996 ± 3% -13.6% 26791 ± 2% softirqs.CPU146.RCU 13084 ± 8% +58.7% 20761 ± 19% softirqs.CPU146.SCHED 30902 ± 3% -10.1% 27769 softirqs.CPU147.RCU 13209 ± 8% +69.5% 22392 ± 6% softirqs.CPU147.SCHED 30816 ± 3% -9.3% 27938 softirqs.CPU148.RCU 13056 ± 7% +74.1% 22731 ± 7% softirqs.CPU148.SCHED 30876 ± 3% -9.6% 27926 softirqs.CPU149.RCU 13104 ± 7% +66.5% 21813 ± 11% softirqs.CPU149.SCHED 32201 ± 6% -13.2% 27952 ± 2% softirqs.CPU15.RCU 13135 ± 6% +57.2% 20646 ± 14% softirqs.CPU15.SCHED 30836 ± 3% -10.0% 27752 softirqs.CPU150.RCU 13288 ± 6% +65.0% 21919 ± 10% softirqs.CPU150.SCHED 31074 ± 3% -11.4% 27531 softirqs.CPU151.RCU 13178 ± 8% +70.8% 22506 ± 9% softirqs.CPU151.SCHED 30955 ± 3% -10.2% 27786 softirqs.CPU152.RCU 13182 ± 8% +75.6% 23155 ± 2% softirqs.CPU152.SCHED 31032 ± 3% -13.3% 26914 ± 4% softirqs.CPU153.RCU 13156 ± 8% +69.5% 22293 ± 7% softirqs.CPU153.SCHED 31190 ± 3% -11.5% 27609 softirqs.CPU154.RCU 13279 ± 7% +67.4% 22223 ± 7% softirqs.CPU154.SCHED 31230 ± 2% -10.7% 27889 softirqs.CPU155.RCU 13298 ± 5% +69.1% 22485 ± 6% softirqs.CPU155.SCHED 31098 ± 2% -10.9% 27722 softirqs.CPU156.RCU 13306 ± 6% +71.6% 22834 ± 7% softirqs.CPU156.SCHED 31711 ± 6% -16.7% 26402 ± 3% softirqs.CPU157.RCU 13212 ± 7% +74.3% 23031 ± 8% softirqs.CPU157.SCHED 30982 ± 3% -11.6% 27383 softirqs.CPU158.RCU 13159 ± 7% +71.6% 22585 ± 8% softirqs.CPU158.SCHED 30935 ± 3% -9.9% 27859 softirqs.CPU159.RCU 13305 ± 8% +71.5% 22816 ± 6% softirqs.CPU159.SCHED 32162 ± 4% -12.6% 28095 softirqs.CPU16.RCU 12923 ± 8% +54.4% 19953 ± 16% softirqs.CPU16.SCHED 31177 ± 3% -10.7% 27829 ± 4% softirqs.CPU160.RCU 13060 ± 8% +75.9% 22970 ± 7% softirqs.CPU160.SCHED 31499 ± 3% -12.5% 27562 ± 3% softirqs.CPU161.RCU 13111 ± 8% +76.9% 23190 ± 8% softirqs.CPU161.SCHED 30994 ± 2% -10.1% 27854 ± 4% softirqs.CPU162.RCU 13231 ± 9% +74.1% 23036 ± 8% softirqs.CPU162.SCHED 30973 ± 2% -11.4% 27428 ± 3% softirqs.CPU163.RCU 13239 ± 8% +73.0% 22902 ± 8% softirqs.CPU163.SCHED 30916 ± 3% -10.8% 27588 ± 4% softirqs.CPU164.RCU 13376 ± 9% +74.0% 23279 ± 8% softirqs.CPU164.SCHED 30922 ± 3% -11.9% 27245 ± 3% softirqs.CPU165.RCU 13124 ± 8% +72.9% 22694 ± 9% softirqs.CPU165.SCHED 30990 ± 2% -11.2% 27508 ± 3% softirqs.CPU166.RCU 13235 ± 8% +74.7% 23116 ± 8% softirqs.CPU166.SCHED 30940 ± 3% -11.4% 27423 ± 3% softirqs.CPU167.RCU 13241 ± 8% +68.6% 22319 ± 6% softirqs.CPU167.SCHED 31837 ± 7% -15.2% 27013 ± 2% softirqs.CPU168.RCU 11975 ± 8% +51.5% 18143 ± 19% softirqs.CPU168.SCHED 32905 ± 9% -17.5% 27162 ± 2% softirqs.CPU169.RCU 12075 ± 10% +46.4% 17674 ± 20% softirqs.CPU169.SCHED 32180 ± 4% -11.8% 28372 softirqs.CPU17.RCU 12701 ± 8% +58.9% 20186 ± 16% softirqs.CPU17.SCHED 32128 ± 8% -16.3% 26892 ± 3% softirqs.CPU170.RCU 12379 ± 11% +43.9% 17817 ± 19% softirqs.CPU170.SCHED 31840 ± 6% -15.0% 27057 ± 2% softirqs.CPU171.RCU 12030 ± 10% +48.8% 17901 ± 19% softirqs.CPU171.SCHED 31612 ± 7% -13.1% 27469 ± 4% softirqs.CPU172.RCU 12139 ± 10% +48.1% 17974 ± 20% softirqs.CPU172.SCHED 31799 ± 7% -14.4% 27217 ± 3% softirqs.CPU173.RCU 12137 ± 12% +47.7% 17923 ± 19% softirqs.CPU173.SCHED 31879 ± 6% -15.2% 27038 ± 3% softirqs.CPU174.RCU 12235 ± 12% +47.0% 17985 ± 20% softirqs.CPU174.SCHED 32162 ± 6% -14.4% 27533 ± 3% softirqs.CPU175.RCU 12298 ± 15% +46.7% 18043 ± 18% softirqs.CPU175.SCHED 30774 ± 6% -11.1% 27366 ± 2% softirqs.CPU176.RCU 12026 ± 11% +48.0% 17803 ± 19% softirqs.CPU176.SCHED 30679 ± 4% -11.5% 27156 ± 2% softirqs.CPU177.RCU 12176 ± 13% +45.5% 17714 ± 20% softirqs.CPU177.SCHED 12140 ± 11% +48.8% 18069 ± 19% softirqs.CPU178.SCHED 12211 ± 12% +47.4% 18003 ± 19% softirqs.CPU179.SCHED 32382 ± 4% -12.2% 28441 softirqs.CPU18.RCU 12902 ± 6% +57.4% 20303 ± 15% softirqs.CPU18.SCHED 31075 ± 5% -11.9% 27369 ± 3% softirqs.CPU180.RCU 12165 ± 12% +46.6% 17839 ± 20% softirqs.CPU180.SCHED 30964 ± 4% -12.0% 27238 ± 2% softirqs.CPU181.RCU 12248 ± 12% +47.8% 18098 ± 22% softirqs.CPU181.SCHED 31103 ± 4% -11.8% 27425 softirqs.CPU182.RCU 12424 ± 12% +43.3% 17805 ± 19% softirqs.CPU182.SCHED 30938 ± 5% -10.5% 27675 ± 2% softirqs.CPU183.RCU 12480 ± 16% +42.5% 17782 ± 18% softirqs.CPU183.SCHED 12169 ± 12% +47.7% 17972 ± 20% softirqs.CPU184.SCHED 30878 ± 5% -11.6% 27312 ± 2% softirqs.CPU185.RCU 12060 ± 13% +49.6% 18037 ± 20% softirqs.CPU185.SCHED 31185 ± 6% -13.0% 27132 ± 4% softirqs.CPU186.RCU 12442 ± 11% +44.3% 17958 ± 20% softirqs.CPU186.SCHED 30816 ± 6% -9.9% 27753 ± 2% softirqs.CPU187.RCU 11967 ± 13% +51.8% 18169 ± 19% softirqs.CPU187.SCHED 30741 ± 5% -10.3% 27590 softirqs.CPU188.RCU 12301 ± 11% +47.0% 18077 ± 18% softirqs.CPU188.SCHED 30878 ± 5% -11.1% 27437 softirqs.CPU189.RCU 12403 ± 11% +46.1% 18125 ± 19% softirqs.CPU189.SCHED 33549 ± 9% -15.5% 28337 softirqs.CPU19.RCU 13145 ± 8% +56.0% 20502 ± 15% softirqs.CPU19.SCHED 12217 ± 12% +45.2% 17734 ± 20% softirqs.CPU190.SCHED 31599 ± 4% -12.2% 27731 ± 2% softirqs.CPU191.RCU 12201 ± 13% +46.1% 17832 ± 20% softirqs.CPU191.SCHED 13107 ± 9% +57.7% 20673 ± 15% softirqs.CPU2.SCHED 32171 ± 4% -8.8% 29343 ± 4% softirqs.CPU20.RCU 13049 ± 7% +57.4% 20544 ± 15% softirqs.CPU20.SCHED 31950 ± 4% -12.1% 28099 softirqs.CPU21.RCU 13103 ± 7% +56.7% 20536 ± 16% softirqs.CPU21.SCHED 112050 ± 5% +9.1% 122248 softirqs.CPU21.TIMER 32375 ± 8% -12.1% 28462 softirqs.CPU22.RCU 13126 ± 8% +57.0% 20615 ± 15% softirqs.CPU22.SCHED 32092 ± 4% -11.6% 28371 softirqs.CPU23.RCU 12677 ± 12% +65.3% 20955 ± 17% softirqs.CPU23.SCHED 32554 ± 7% -12.0% 28659 softirqs.CPU24.RCU 12382 ± 10% +80.6% 22358 ± 3% softirqs.CPU24.SCHED 32092 ± 3% -13.9% 27628 ± 2% softirqs.CPU25.RCU 12271 ± 10% +81.6% 22280 ± 5% softirqs.CPU25.SCHED 31747 ± 4% -14.1% 27267 ± 4% softirqs.CPU26.RCU 12617 ± 7% +76.2% 22233 ± 4% softirqs.CPU26.SCHED 32013 ± 3% -13.6% 27673 ± 2% softirqs.CPU27.RCU 12501 ± 9% +80.0% 22496 ± 3% softirqs.CPU27.SCHED 32129 ± 3% -12.1% 28232 ± 2% softirqs.CPU28.RCU 12719 ± 9% +76.8% 22482 ± 4% softirqs.CPU28.SCHED 32415 ± 3% -14.2% 27797 ± 2% softirqs.CPU29.RCU 12529 ± 9% +77.8% 22284 ± 3% softirqs.CPU29.SCHED 31703 ± 5% -9.7% 28624 softirqs.CPU3.RCU 13071 ± 7% +56.5% 20460 ± 18% softirqs.CPU3.SCHED 32136 ± 3% -13.5% 27802 softirqs.CPU30.RCU 12408 ± 11% +82.4% 22632 ± 3% softirqs.CPU30.SCHED 32766 ± 2% -14.7% 27942 ± 2% softirqs.CPU31.RCU 12782 ± 10% +69.8% 21700 ± 3% softirqs.CPU31.SCHED 32114 ± 2% -13.2% 27885 ± 2% softirqs.CPU32.RCU 12565 ± 10% +76.9% 22227 ± 3% softirqs.CPU32.SCHED 31908 -10.9% 28428 ± 5% softirqs.CPU33.RCU 12470 ± 11% +81.9% 22679 softirqs.CPU33.SCHED 31862 ± 2% -12.2% 27972 ± 3% softirqs.CPU34.RCU 12441 ± 11% +79.4% 22324 ± 3% softirqs.CPU34.SCHED 32334 ± 2% -13.0% 28127 ± 2% softirqs.CPU35.RCU 12571 ± 10% +78.0% 22380 ± 2% softirqs.CPU35.SCHED 31808 ± 3% -11.3% 28213 ± 3% softirqs.CPU36.RCU 12634 ± 9% +76.1% 22245 ± 2% softirqs.CPU36.SCHED 31475 ± 4% -12.0% 27691 ± 2% softirqs.CPU37.RCU 12674 ± 9% +75.3% 22217 ± 2% softirqs.CPU37.SCHED 31654 -11.4% 28031 ± 3% softirqs.CPU38.RCU 12380 ± 9% +77.6% 21990 ± 2% softirqs.CPU38.SCHED 31846 ± 3% -11.1% 28313 ± 3% softirqs.CPU39.RCU 12426 ± 9% +80.6% 22446 ± 3% softirqs.CPU39.SCHED 12972 ± 6% +53.7% 19940 ± 17% softirqs.CPU4.SCHED 112980 ± 5% +7.3% 121215 ± 4% softirqs.CPU4.TIMER 31924 ± 2% -12.5% 27919 ± 3% softirqs.CPU40.RCU 12607 ± 8% +76.4% 22236 ± 4% softirqs.CPU40.SCHED 31979 ± 2% -13.2% 27768 ± 2% softirqs.CPU41.RCU 12515 ± 9% +83.2% 22925 ± 6% softirqs.CPU41.SCHED 31649 ± 2% -11.4% 28056 ± 2% softirqs.CPU42.RCU 12409 ± 10% +82.5% 22643 ± 2% softirqs.CPU42.SCHED 31892 ± 2% -12.3% 27980 ± 3% softirqs.CPU43.RCU 12459 ± 9% +77.6% 22123 ± 5% softirqs.CPU43.SCHED 31735 ± 2% -11.7% 28017 ± 2% softirqs.CPU44.RCU 12510 ± 9% +79.4% 22438 ± 2% softirqs.CPU44.SCHED 31322 ± 2% -11.7% 27656 ± 3% softirqs.CPU45.RCU 12601 ± 9% +73.3% 21837 ± 2% softirqs.CPU45.SCHED 31661 ± 3% -11.6% 27979 ± 4% softirqs.CPU46.RCU 12508 ± 9% +79.2% 22411 ± 3% softirqs.CPU46.SCHED 31532 ± 3% -11.8% 27802 ± 2% softirqs.CPU47.RCU 12568 ± 8% +78.5% 22440 ± 2% softirqs.CPU47.SCHED 32597 ± 3% -13.9% 28075 softirqs.CPU48.RCU 12838 ± 6% +74.5% 22404 ± 6% softirqs.CPU48.SCHED 31583 ± 3% -11.8% 27858 softirqs.CPU49.RCU 12826 ± 6% +74.7% 22407 ± 4% softirqs.CPU49.SCHED 32127 ± 7% -11.5% 28427 softirqs.CPU5.RCU 13560 ± 10% +50.6% 20420 ± 15% softirqs.CPU5.SCHED 31596 ± 3% -12.9% 27527 softirqs.CPU50.RCU 12964 ± 7% +73.7% 22515 ± 4% softirqs.CPU50.SCHED 31453 ± 3% -10.5% 28160 softirqs.CPU51.RCU 12739 ± 6% +73.9% 22153 ± 7% softirqs.CPU51.SCHED 12804 ± 6% +75.9% 22528 ± 6% softirqs.CPU52.SCHED 31450 ± 2% -10.4% 28167 softirqs.CPU53.RCU 12942 ± 6% +75.0% 22651 ± 7% softirqs.CPU53.SCHED 31441 ± 3% -10.1% 28251 softirqs.CPU54.RCU 13105 ± 3% +71.3% 22453 ± 7% softirqs.CPU54.SCHED 31603 ± 3% -11.0% 28132 softirqs.CPU55.RCU 12777 ± 7% +77.2% 22637 ± 8% softirqs.CPU55.SCHED 31555 ± 3% -11.1% 28066 softirqs.CPU56.RCU 12777 ± 6% +75.1% 22376 ± 6% softirqs.CPU56.SCHED 31436 ± 3% -12.0% 27671 ± 3% softirqs.CPU57.RCU 12781 ± 6% +77.2% 22649 ± 6% softirqs.CPU57.SCHED 31503 ± 3% -10.7% 28139 softirqs.CPU58.RCU 12954 ± 6% +75.3% 22709 ± 7% softirqs.CPU58.SCHED 31839 ± 3% -10.9% 28371 ± 2% softirqs.CPU59.RCU 13236 ± 7% +71.1% 22653 ± 6% softirqs.CPU59.SCHED 13223 ± 8% +52.4% 20148 ± 16% softirqs.CPU6.SCHED 32616 -13.7% 28163 softirqs.CPU60.RCU 13139 +73.8% 22841 ± 7% softirqs.CPU60.SCHED 31250 ± 3% -11.5% 27660 ± 3% softirqs.CPU61.RCU 12835 ± 6% +76.8% 22694 ± 7% softirqs.CPU61.SCHED 31480 ± 3% -11.3% 27922 softirqs.CPU62.RCU 12962 ± 6% +72.4% 22345 ± 6% softirqs.CPU62.SCHED 31567 ± 3% -10.2% 28342 ± 2% softirqs.CPU63.RCU 12909 ± 7% +74.7% 22556 ± 6% softirqs.CPU63.SCHED 30655 ± 5% -14.7% 26163 softirqs.CPU64.RCU 12793 ± 6% +75.5% 22452 ± 7% softirqs.CPU64.SCHED 30643 ± 5% -13.8% 26417 softirqs.CPU65.RCU 12839 ± 7% +73.3% 22251 ± 7% softirqs.CPU65.SCHED 30720 ± 4% -14.6% 26249 softirqs.CPU66.RCU 12754 ± 6% +75.7% 22415 ± 7% softirqs.CPU66.SCHED 30650 ± 4% -14.2% 26294 softirqs.CPU67.RCU 12760 ± 6% +77.6% 22659 ± 8% softirqs.CPU67.SCHED 30537 ± 5% -13.3% 26480 ± 2% softirqs.CPU68.RCU 12927 ± 7% +75.6% 22699 ± 7% softirqs.CPU68.SCHED 30702 ± 5% -14.5% 26253 ± 2% softirqs.CPU69.RCU 12814 ± 7% +79.8% 23040 ± 9% softirqs.CPU69.SCHED 12743 ± 10% +61.6% 20591 ± 13% softirqs.CPU7.SCHED 30578 ± 4% -13.8% 26345 softirqs.CPU70.RCU 12985 ± 8% +74.9% 22711 ± 8% softirqs.CPU70.SCHED 30596 ± 4% -14.2% 26259 ± 2% softirqs.CPU71.RCU 12903 ± 7% +75.1% 22588 ± 7% softirqs.CPU71.SCHED 31963 ± 10% -17.8% 26279 softirqs.CPU72.RCU 12353 ± 6% +52.5% 18835 ± 10% softirqs.CPU72.SCHED 32117 ± 12% -18.9% 26039 softirqs.CPU73.RCU 12342 ± 9% +45.4% 17944 ± 16% softirqs.CPU73.SCHED 32473 ± 12% -19.9% 26000 ± 2% softirqs.CPU74.RCU 12308 ± 8% +47.5% 18160 ± 16% softirqs.CPU74.SCHED 31632 ± 10% -16.9% 26275 ± 2% softirqs.CPU75.RCU 12134 ± 9% +46.8% 17808 ± 17% softirqs.CPU75.SCHED 31589 ± 10% -17.2% 26141 ± 2% softirqs.CPU76.RCU 11922 ± 11% +65.2% 19694 ± 33% softirqs.CPU76.SCHED 31630 ± 11% -17.2% 26191 ± 2% softirqs.CPU77.RCU 12062 ± 9% +48.6% 17923 ± 15% softirqs.CPU77.SCHED 32390 ± 9% -19.5% 26088 ± 2% softirqs.CPU78.RCU 12168 ± 9% +47.0% 17883 ± 16% softirqs.CPU78.SCHED 31792 ± 10% -16.7% 26488 ± 4% softirqs.CPU79.RCU 12295 ± 11% +44.8% 17807 ± 17% softirqs.CPU79.SCHED 31541 ± 5% -10.3% 28297 softirqs.CPU8.RCU 13076 ± 7% +58.5% 20721 ± 15% softirqs.CPU8.SCHED 12180 ± 10% +45.3% 17693 ± 17% softirqs.CPU80.SCHED 31450 ± 7% -12.7% 27447 ± 2% softirqs.CPU81.RCU 12339 ± 12% +42.2% 17544 ± 17% softirqs.CPU81.SCHED 12207 ± 9% +45.6% 17777 ± 18% softirqs.CPU82.SCHED 12064 ± 9% +48.7% 17943 ± 16% softirqs.CPU83.SCHED 12084 ± 8% +46.6% 17712 ± 18% softirqs.CPU84.SCHED 31146 ± 4% -10.9% 27756 ± 2% softirqs.CPU85.RCU 12156 ± 8% +47.3% 17911 ± 16% softirqs.CPU85.SCHED 31536 ± 4% -12.1% 27718 ± 3% softirqs.CPU86.RCU 12371 ± 8% +42.9% 17673 ± 18% softirqs.CPU86.SCHED 12263 ± 10% +44.0% 17662 ± 18% softirqs.CPU87.SCHED 12216 ± 9% +45.7% 17804 ± 18% softirqs.CPU88.SCHED 31424 ± 5% -11.8% 27724 ± 2% softirqs.CPU89.RCU 12379 ± 8% +44.2% 17853 ± 16% softirqs.CPU89.SCHED 31423 ± 5% -10.3% 28191 softirqs.CPU9.RCU 13188 ± 5% +58.2% 20864 ± 14% softirqs.CPU9.SCHED 113341 ± 4% +8.2% 122664 softirqs.CPU9.TIMER 31525 ± 5% -12.5% 27585 ± 4% softirqs.CPU90.RCU 12345 ± 8% +45.0% 17895 ± 20% softirqs.CPU90.SCHED 11971 ± 8% +49.7% 17922 ± 18% softirqs.CPU91.SCHED 12212 ± 9% +47.2% 17979 ± 16% softirqs.CPU92.SCHED 12305 ± 10% +46.2% 17988 ± 18% softirqs.CPU93.SCHED 31246 ± 5% -11.2% 27734 ± 2% softirqs.CPU94.RCU 12173 ± 9% +48.9% 18124 ± 19% softirqs.CPU94.SCHED 31258 ± 5% -11.0% 27824 ± 2% softirqs.CPU95.RCU 12187 ± 11% +47.6% 17988 ± 19% softirqs.CPU95.SCHED 31677 ± 4% -15.7% 26719 ± 2% softirqs.CPU96.RCU 13001 ± 7% +57.1% 20420 ± 15% softirqs.CPU96.SCHED 31379 ± 6% -11.8% 27678 ± 2% softirqs.CPU97.RCU 12869 ± 9% +62.5% 20909 ± 14% softirqs.CPU97.SCHED 110944 ± 5% +10.3% 122392 softirqs.CPU97.TIMER 33271 ± 7% -16.0% 27945 softirqs.CPU98.RCU 13545 ± 7% +56.5% 21193 ± 15% softirqs.CPU98.SCHED 31522 ± 5% -10.9% 28072 softirqs.CPU99.RCU 13760 ± 7% +50.1% 20650 ± 16% softirqs.CPU99.SCHED 6052777 ± 4% -12.3% 5310987 softirqs.RCU 2454586 +63.5% 4012658 ± 2% softirqs.SCHED 5164839 ± 4% -44.0% 2891677 ± 4% interrupts.CAL:Function_call_interrupts 23211 ± 5% -49.8% 11662 ± 14% interrupts.CPU0.CAL:Function_call_interrupts 19793 ± 4% -56.4% 8622 ± 20% interrupts.CPU0.TLB:TLB_shootdowns 25756 ± 6% -48.9% 13159 ± 6% interrupts.CPU1.CAL:Function_call_interrupts 6216 ± 6% -29.9% 4356 ± 21% interrupts.CPU1.NMI:Non-maskable_interrupts 6216 ± 6% -29.9% 4356 ± 21% interrupts.CPU1.PMI:Performance_monitoring_interrupts 22359 ± 5% -54.3% 10221 ± 8% interrupts.CPU1.TLB:TLB_shootdowns 25979 ± 5% -44.8% 14345 ± 10% interrupts.CPU10.CAL:Function_call_interrupts 6067 ± 5% -29.7% 4265 ± 36% interrupts.CPU10.NMI:Non-maskable_interrupts 6067 ± 5% -29.7% 4265 ± 36% interrupts.CPU10.PMI:Performance_monitoring_interrupts 22666 ± 4% -49.6% 11419 ± 11% interrupts.CPU10.TLB:TLB_shootdowns 26575 ± 9% -45.8% 14399 ± 6% interrupts.CPU100.CAL:Function_call_interrupts 23233 ± 11% -50.7% 11457 ± 8% interrupts.CPU100.TLB:TLB_shootdowns 27400 ± 6% -45.7% 14890 ± 8% interrupts.CPU101.CAL:Function_call_interrupts 24034 ± 7% -50.1% 11999 ± 10% interrupts.CPU101.TLB:TLB_shootdowns 27415 ± 9% -44.7% 15151 ± 6% interrupts.CPU102.CAL:Function_call_interrupts 6287 ± 4% -26.4% 4625 ± 16% interrupts.CPU102.NMI:Non-maskable_interrupts 6287 ± 4% -26.4% 4625 ± 16% interrupts.CPU102.PMI:Performance_monitoring_interrupts 24027 ± 11% -49.2% 12215 ± 8% interrupts.CPU102.TLB:TLB_shootdowns 27495 ± 7% -47.1% 14548 ± 7% interrupts.CPU103.CAL:Function_call_interrupts 24192 ± 6% -52.3% 11541 ± 9% interrupts.CPU103.TLB:TLB_shootdowns 27461 ± 7% -44.9% 15142 ± 9% interrupts.CPU104.CAL:Function_call_interrupts 6105 ± 6% -43.1% 3471 ± 39% interrupts.CPU104.NMI:Non-maskable_interrupts 6105 ± 6% -43.1% 3471 ± 39% interrupts.CPU104.PMI:Performance_monitoring_interrupts 24138 ± 6% -49.7% 12153 ± 11% interrupts.CPU104.TLB:TLB_shootdowns 26900 ± 7% -45.0% 14796 ± 7% interrupts.CPU105.CAL:Function_call_interrupts 23655 ± 7% -50.1% 11793 ± 8% interrupts.CPU105.TLB:TLB_shootdowns 26032 ± 9% -42.6% 14934 ± 10% interrupts.CPU106.CAL:Function_call_interrupts 6015 ± 6% -33.6% 3994 ± 18% interrupts.CPU106.NMI:Non-maskable_interrupts 6015 ± 6% -33.6% 3994 ± 18% interrupts.CPU106.PMI:Performance_monitoring_interrupts 22884 ± 8% -47.4% 12037 ± 12% interrupts.CPU106.TLB:TLB_shootdowns 27375 ± 5% -43.6% 15441 ± 9% interrupts.CPU107.CAL:Function_call_interrupts 24034 ± 6% -48.1% 12472 ± 12% interrupts.CPU107.TLB:TLB_shootdowns 27473 ± 6% -45.5% 14969 ± 8% interrupts.CPU108.CAL:Function_call_interrupts 24119 ± 6% -50.4% 11962 ± 10% interrupts.CPU108.TLB:TLB_shootdowns 27681 ± 9% -43.6% 15612 ± 8% interrupts.CPU109.CAL:Function_call_interrupts 24298 ± 9% -48.0% 12638 ± 10% interrupts.CPU109.TLB:TLB_shootdowns 27016 ± 7% -44.6% 14956 ± 10% interrupts.CPU11.CAL:Function_call_interrupts 6075 ± 6% -26.1% 4487 ± 22% interrupts.CPU11.NMI:Non-maskable_interrupts 6075 ± 6% -26.1% 4487 ± 22% interrupts.CPU11.PMI:Performance_monitoring_interrupts 23697 ± 8% -49.4% 12001 ± 13% interrupts.CPU11.TLB:TLB_shootdowns 27375 ± 6% -44.3% 15248 ± 9% interrupts.CPU110.CAL:Function_call_interrupts 6121 ± 5% -28.7% 4364 ± 34% interrupts.CPU110.NMI:Non-maskable_interrupts 6121 ± 5% -28.7% 4364 ± 34% interrupts.CPU110.PMI:Performance_monitoring_interrupts 24007 ± 7% -48.9% 12263 ± 12% interrupts.CPU110.TLB:TLB_shootdowns 27096 ± 5% -45.0% 14899 ± 9% interrupts.CPU111.CAL:Function_call_interrupts 23791 ± 5% -49.9% 11920 ± 12% interrupts.CPU111.TLB:TLB_shootdowns 27578 ± 7% -46.7% 14694 ± 10% interrupts.CPU112.CAL:Function_call_interrupts 6310 ± 7% -29.2% 4468 ± 17% interrupts.CPU112.NMI:Non-maskable_interrupts 6310 ± 7% -29.2% 4468 ± 17% interrupts.CPU112.PMI:Performance_monitoring_interrupts 24277 ± 7% -51.7% 11726 ± 12% interrupts.CPU112.TLB:TLB_shootdowns 28568 ± 4% -48.0% 14870 ± 7% interrupts.CPU113.CAL:Function_call_interrupts 6171 ± 7% -43.9% 3464 ± 39% interrupts.CPU113.NMI:Non-maskable_interrupts 6171 ± 7% -43.9% 3464 ± 39% interrupts.CPU113.PMI:Performance_monitoring_interrupts 25339 ± 5% -53.0% 11908 ± 9% interrupts.CPU113.TLB:TLB_shootdowns 27869 ± 7% -47.3% 14690 ± 11% interrupts.CPU114.CAL:Function_call_interrupts 24553 ± 6% -52.5% 11664 ± 14% interrupts.CPU114.TLB:TLB_shootdowns 27168 ± 5% -44.2% 15162 ± 9% interrupts.CPU115.CAL:Function_call_interrupts 23784 ± 5% -48.7% 12203 ± 10% interrupts.CPU115.TLB:TLB_shootdowns 27674 ± 7% -44.4% 15395 ± 10% interrupts.CPU116.CAL:Function_call_interrupts 24359 ± 7% -49.2% 12386 ± 13% interrupts.CPU116.TLB:TLB_shootdowns 25881 ± 15% -41.8% 15054 ± 11% interrupts.CPU117.CAL:Function_call_interrupts 22516 ± 18% -46.2% 12105 ± 13% interrupts.CPU117.TLB:TLB_shootdowns 27172 ± 6% -43.8% 15279 ± 7% interrupts.CPU118.CAL:Function_call_interrupts 23916 ± 6% -48.6% 12298 ± 9% interrupts.CPU118.TLB:TLB_shootdowns 27108 ± 7% -46.3% 14548 ± 13% interrupts.CPU119.CAL:Function_call_interrupts 6164 ± 4% -28.8% 4388 ± 21% interrupts.CPU119.NMI:Non-maskable_interrupts 6164 ± 4% -28.8% 4388 ± 21% interrupts.CPU119.PMI:Performance_monitoring_interrupts 23830 ± 7% -51.4% 11571 ± 17% interrupts.CPU119.TLB:TLB_shootdowns 26907 ± 5% -43.4% 15217 ± 8% interrupts.CPU12.CAL:Function_call_interrupts 6205 ± 5% -32.9% 4164 ± 34% interrupts.CPU12.NMI:Non-maskable_interrupts 6205 ± 5% -32.9% 4164 ± 34% interrupts.CPU12.PMI:Performance_monitoring_interrupts 23637 ± 6% -48.1% 12262 ± 11% interrupts.CPU12.TLB:TLB_shootdowns 25698 ± 9% -36.6% 16298 ± 5% interrupts.CPU120.CAL:Function_call_interrupts 22429 ± 9% -40.2% 13422 ± 6% interrupts.CPU120.TLB:TLB_shootdowns 25824 ± 9% -38.1% 15984 ± 2% interrupts.CPU121.CAL:Function_call_interrupts 6254 ± 4% -34.0% 4127 ± 24% interrupts.CPU121.NMI:Non-maskable_interrupts 6254 ± 4% -34.0% 4127 ± 24% interrupts.CPU121.PMI:Performance_monitoring_interrupts 22548 ± 8% -42.3% 13012 ± 3% interrupts.CPU121.TLB:TLB_shootdowns 25386 ± 12% -36.5% 16116 ± 4% interrupts.CPU122.CAL:Function_call_interrupts 21996 ± 11% -39.5% 13312 ± 4% interrupts.CPU122.TLB:TLB_shootdowns 25944 ± 11% -37.5% 16217 ± 5% interrupts.CPU123.CAL:Function_call_interrupts 6261 ± 3% -49.5% 3161 ± 42% interrupts.CPU123.NMI:Non-maskable_interrupts 6261 ± 3% -49.5% 3161 ± 42% interrupts.CPU123.PMI:Performance_monitoring_interrupts 22614 ± 10% -41.4% 13258 ± 7% interrupts.CPU123.TLB:TLB_shootdowns 26203 ± 9% -41.2% 15417 ± 5% interrupts.CPU124.CAL:Function_call_interrupts 22819 ± 9% -45.7% 12387 ± 6% interrupts.CPU124.TLB:TLB_shootdowns 26658 ± 9% -37.8% 16571 ± 3% interrupts.CPU125.CAL:Function_call_interrupts 6317 ± 4% -38.0% 3914 ± 35% interrupts.CPU125.NMI:Non-maskable_interrupts 6317 ± 4% -38.0% 3914 ± 35% interrupts.CPU125.PMI:Performance_monitoring_interrupts 23411 ± 9% -42.1% 13551 ± 4% interrupts.CPU125.TLB:TLB_shootdowns 26357 ± 11% -39.0% 16090 interrupts.CPU126.CAL:Function_call_interrupts 22964 ± 11% -42.7% 13155 ± 2% interrupts.CPU126.TLB:TLB_shootdowns 26279 ± 10% -37.8% 16357 ± 2% interrupts.CPU127.CAL:Function_call_interrupts 6414 ± 4% -31.6% 4389 ± 19% interrupts.CPU127.NMI:Non-maskable_interrupts 6414 ± 4% -31.6% 4389 ± 19% interrupts.CPU127.PMI:Performance_monitoring_interrupts 22928 ± 9% -42.0% 13304 ± 3% interrupts.CPU127.TLB:TLB_shootdowns 26403 ± 7% -38.7% 16195 ± 5% interrupts.CPU128.CAL:Function_call_interrupts 6320 ± 3% -48.7% 3241 ± 41% interrupts.CPU128.NMI:Non-maskable_interrupts 6320 ± 3% -48.7% 3241 ± 41% interrupts.CPU128.PMI:Performance_monitoring_interrupts 23017 ± 6% -42.9% 13151 ± 7% interrupts.CPU128.TLB:TLB_shootdowns 26062 ± 11% -38.4% 16041 ± 2% interrupts.CPU129.CAL:Function_call_interrupts 22712 ± 11% -42.6% 13045 ± 2% interrupts.CPU129.TLB:TLB_shootdowns 27023 ± 8% -46.6% 14438 ± 9% interrupts.CPU13.CAL:Function_call_interrupts 23836 ± 9% -51.8% 11478 ± 11% interrupts.CPU13.TLB:TLB_shootdowns 25782 ± 11% -37.9% 16023 ± 4% interrupts.CPU130.CAL:Function_call_interrupts 22383 ± 11% -41.5% 13102 ± 6% interrupts.CPU130.TLB:TLB_shootdowns 25769 ± 10% -37.1% 16197 ± 3% interrupts.CPU131.CAL:Function_call_interrupts 22457 ± 9% -41.1% 13226 ± 3% interrupts.CPU131.TLB:TLB_shootdowns 25939 ± 8% -36.8% 16394 ± 3% interrupts.CPU132.CAL:Function_call_interrupts 6245 ± 3% -39.8% 3761 ± 35% interrupts.CPU132.NMI:Non-maskable_interrupts 6245 ± 3% -39.8% 3761 ± 35% interrupts.CPU132.PMI:Performance_monitoring_interrupts 22535 ± 8% -40.5% 13418 ± 5% interrupts.CPU132.TLB:TLB_shootdowns 24855 ± 17% -37.7% 15479 ± 8% interrupts.CPU133.CAL:Function_call_interrupts 21434 ± 17% -41.9% 12446 ± 10% interrupts.CPU133.TLB:TLB_shootdowns 26463 ± 12% -39.6% 15994 ± 5% interrupts.CPU134.CAL:Function_call_interrupts 6319 ± 2% -31.5% 4328 ± 18% interrupts.CPU134.NMI:Non-maskable_interrupts 6319 ± 2% -31.5% 4328 ± 18% interrupts.CPU134.PMI:Performance_monitoring_interrupts 23156 ± 11% -44.1% 12953 ± 7% interrupts.CPU134.TLB:TLB_shootdowns 25511 ± 10% -38.3% 15740 ± 5% interrupts.CPU135.CAL:Function_call_interrupts 22206 ± 9% -42.3% 12803 ± 6% interrupts.CPU135.TLB:TLB_shootdowns 26114 ± 12% -37.6% 16304 ± 2% interrupts.CPU136.CAL:Function_call_interrupts 6348 ± 4% -35.5% 4096 ± 17% interrupts.CPU136.NMI:Non-maskable_interrupts 6348 ± 4% -35.5% 4096 ± 17% interrupts.CPU136.PMI:Performance_monitoring_interrupts 22716 ± 12% -41.4% 13306 ± 3% interrupts.CPU136.TLB:TLB_shootdowns 25715 ± 11% -38.7% 15752 ± 4% interrupts.CPU137.CAL:Function_call_interrupts 6314 ± 3% -43.2% 3586 ± 38% interrupts.CPU137.NMI:Non-maskable_interrupts 6314 ± 3% -43.2% 3586 ± 38% interrupts.CPU137.PMI:Performance_monitoring_interrupts 22325 ± 10% -43.0% 12718 ± 5% interrupts.CPU137.TLB:TLB_shootdowns 25832 ± 14% -40.7% 15324 ± 10% interrupts.CPU138.CAL:Function_call_interrupts 6262 ± 5% -57.8% 2644 ± 36% interrupts.CPU138.NMI:Non-maskable_interrupts 6262 ± 5% -57.8% 2644 ± 36% interrupts.CPU138.PMI:Performance_monitoring_interrupts 22456 ± 14% -45.4% 12269 ± 13% interrupts.CPU138.TLB:TLB_shootdowns 26008 ± 10% -39.9% 15625 ± 7% interrupts.CPU139.CAL:Function_call_interrupts 22688 ± 10% -44.4% 12617 ± 9% interrupts.CPU139.TLB:TLB_shootdowns 27079 ± 7% -47.2% 14294 ± 6% interrupts.CPU14.CAL:Function_call_interrupts 23830 ± 7% -52.5% 11308 ± 7% interrupts.CPU14.TLB:TLB_shootdowns 25986 ± 9% -38.9% 15865 ± 5% interrupts.CPU140.CAL:Function_call_interrupts 6388 ± 4% -41.6% 3733 ± 34% interrupts.CPU140.NMI:Non-maskable_interrupts 6388 ± 4% -41.6% 3733 ± 34% interrupts.CPU140.PMI:Performance_monitoring_interrupts 22603 ± 9% -43.3% 12818 ± 6% interrupts.CPU140.TLB:TLB_shootdowns 25088 ± 17% -37.9% 15590 ± 5% interrupts.CPU141.CAL:Function_call_interrupts 6327 ± 4% -34.5% 4144 ± 34% interrupts.CPU141.NMI:Non-maskable_interrupts 6327 ± 4% -34.5% 4144 ± 34% interrupts.CPU141.PMI:Performance_monitoring_interrupts 21633 ± 17% -41.9% 12562 ± 6% interrupts.CPU141.TLB:TLB_shootdowns 25905 ± 15% -40.1% 15514 ± 8% interrupts.CPU142.CAL:Function_call_interrupts 22513 ± 16% -44.6% 12479 ± 10% interrupts.CPU142.TLB:TLB_shootdowns 26096 ± 12% -42.5% 15005 ± 5% interrupts.CPU143.CAL:Function_call_interrupts 22777 ± 11% -47.1% 12059 ± 6% interrupts.CPU143.TLB:TLB_shootdowns 27603 ± 3% -46.2% 14854 ± 8% interrupts.CPU144.CAL:Function_call_interrupts 24314 ± 5% -51.0% 11902 ± 10% interrupts.CPU144.TLB:TLB_shootdowns 28108 ± 3% -45.8% 15236 ± 6% interrupts.CPU145.CAL:Function_call_interrupts 24783 ± 5% -50.8% 12200 ± 8% interrupts.CPU145.TLB:TLB_shootdowns 27705 ± 2% -48.2% 14342 ± 7% interrupts.CPU146.CAL:Function_call_interrupts 24528 ± 4% -54.1% 11263 ± 10% interrupts.CPU146.TLB:TLB_shootdowns 27894 -44.5% 15469 ± 5% interrupts.CPU147.CAL:Function_call_interrupts 6127 ± 7% -37.3% 3843 ± 20% interrupts.CPU147.NMI:Non-maskable_interrupts 6127 ± 7% -37.3% 3843 ± 20% interrupts.CPU147.PMI:Performance_monitoring_interrupts 24626 ± 4% -49.5% 12436 ± 6% interrupts.CPU147.TLB:TLB_shootdowns 28319 ± 4% -44.3% 15773 ± 7% interrupts.CPU148.CAL:Function_call_interrupts 25004 ± 5% -48.9% 12776 ± 9% interrupts.CPU148.TLB:TLB_shootdowns 27991 ± 2% -44.4% 15576 ± 3% interrupts.CPU149.CAL:Function_call_interrupts 6236 ± 3% -30.5% 4336 ± 24% interrupts.CPU149.NMI:Non-maskable_interrupts 6236 ± 3% -30.5% 4336 ± 24% interrupts.CPU149.PMI:Performance_monitoring_interrupts 24647 ± 3% -49.2% 12518 ± 4% interrupts.CPU149.TLB:TLB_shootdowns 26702 ± 8% -43.9% 14975 ± 9% interrupts.CPU15.CAL:Function_call_interrupts 6223 ± 4% -26.3% 4587 ± 17% interrupts.CPU15.NMI:Non-maskable_interrupts 6223 ± 4% -26.3% 4587 ± 17% interrupts.CPU15.PMI:Performance_monitoring_interrupts 23511 ± 9% -48.8% 12038 ± 11% interrupts.CPU15.TLB:TLB_shootdowns 27200 ± 4% -42.7% 15580 ± 7% interrupts.CPU150.CAL:Function_call_interrupts 23912 ± 4% -47.7% 12512 ± 10% interrupts.CPU150.TLB:TLB_shootdowns 28217 ± 3% -45.3% 15433 ± 8% interrupts.CPU151.CAL:Function_call_interrupts 6289 ± 2% -34.3% 4132 ± 29% interrupts.CPU151.NMI:Non-maskable_interrupts 6289 ± 2% -34.3% 4132 ± 29% interrupts.CPU151.PMI:Performance_monitoring_interrupts 24912 ± 3% -50.3% 12379 ± 11% interrupts.CPU151.TLB:TLB_shootdowns 27892 ± 4% -43.5% 15747 ± 5% interrupts.CPU152.CAL:Function_call_interrupts 24655 ± 5% -48.4% 12730 ± 7% interrupts.CPU152.TLB:TLB_shootdowns 27985 ± 3% -43.6% 15781 ± 3% interrupts.CPU153.CAL:Function_call_interrupts 24700 ± 5% -48.2% 12785 ± 4% interrupts.CPU153.TLB:TLB_shootdowns 28258 -45.4% 15424 ± 6% interrupts.CPU154.CAL:Function_call_interrupts 24984 ± 2% -50.4% 12395 ± 9% interrupts.CPU154.TLB:TLB_shootdowns 28729 ± 4% -46.4% 15412 ± 5% interrupts.CPU155.CAL:Function_call_interrupts 25466 ± 3% -51.5% 12348 ± 7% interrupts.CPU155.TLB:TLB_shootdowns 28471 ± 2% -46.4% 15270 ± 8% interrupts.CPU156.CAL:Function_call_interrupts 25189 ± 4% -51.5% 12218 ± 10% interrupts.CPU156.TLB:TLB_shootdowns 28339 ± 2% -46.9% 15043 ± 2% interrupts.CPU157.CAL:Function_call_interrupts 25019 -51.6% 12097 interrupts.CPU157.TLB:TLB_shootdowns 27386 ± 3% -42.3% 15802 ± 3% interrupts.CPU158.CAL:Function_call_interrupts 24077 ± 6% -47.1% 12748 ± 3% interrupts.CPU158.TLB:TLB_shootdowns 28242 ± 3% -43.6% 15939 ± 4% interrupts.CPU159.CAL:Function_call_interrupts 6371 ± 2% -31.3% 4379 ± 22% interrupts.CPU159.NMI:Non-maskable_interrupts 6371 ± 2% -31.3% 4379 ± 22% interrupts.CPU159.PMI:Performance_monitoring_interrupts 24953 ± 5% -48.4% 12885 ± 6% interrupts.CPU159.TLB:TLB_shootdowns 27056 ± 7% -46.3% 14540 ± 7% interrupts.CPU16.CAL:Function_call_interrupts 23823 ± 7% -51.5% 11562 ± 10% interrupts.CPU16.TLB:TLB_shootdowns 28362 -43.8% 15926 ± 6% interrupts.CPU160.CAL:Function_call_interrupts 6236 ± 4% -39.2% 3788 ± 38% interrupts.CPU160.NMI:Non-maskable_interrupts 6236 ± 4% -39.2% 3788 ± 38% interrupts.CPU160.PMI:Performance_monitoring_interrupts 25119 ± 4% -48.9% 12826 ± 8% interrupts.CPU160.TLB:TLB_shootdowns 28858 ± 3% -45.7% 15668 ± 9% interrupts.CPU161.CAL:Function_call_interrupts 6292 ± 3% -43.9% 3530 ± 21% interrupts.CPU161.NMI:Non-maskable_interrupts 6292 ± 3% -43.9% 3530 ± 21% interrupts.CPU161.PMI:Performance_monitoring_interrupts 25535 ± 4% -50.5% 12638 ± 11% interrupts.CPU161.TLB:TLB_shootdowns 28903 -47.3% 15233 ± 11% interrupts.CPU162.CAL:Function_call_interrupts 6140 ± 6% -41.1% 3616 ± 16% interrupts.CPU162.NMI:Non-maskable_interrupts 6140 ± 6% -41.1% 3616 ± 16% interrupts.CPU162.PMI:Performance_monitoring_interrupts 25603 ± 3% -52.5% 12159 ± 13% interrupts.CPU162.TLB:TLB_shootdowns 28874 ± 2% -48.4% 14894 ± 7% interrupts.CPU163.CAL:Function_call_interrupts 25490 ± 3% -53.5% 11857 ± 9% interrupts.CPU163.TLB:TLB_shootdowns 29304 ± 3% -48.4% 15117 ± 9% interrupts.CPU164.CAL:Function_call_interrupts 25985 ± 5% -53.5% 12092 ± 12% interrupts.CPU164.TLB:TLB_shootdowns 28915 ± 2% -49.4% 14641 ± 7% interrupts.CPU165.CAL:Function_call_interrupts 6260 ± 3% -37.2% 3933 ± 36% interrupts.CPU165.NMI:Non-maskable_interrupts 6260 ± 3% -37.2% 3933 ± 36% interrupts.CPU165.PMI:Performance_monitoring_interrupts 25555 ± 4% -54.8% 11548 ± 9% interrupts.CPU165.TLB:TLB_shootdowns 28013 ± 3% -47.5% 14719 ± 7% interrupts.CPU166.CAL:Function_call_interrupts 6299 -38.1% 3896 ± 37% interrupts.CPU166.NMI:Non-maskable_interrupts 6299 -38.1% 3896 ± 37% interrupts.CPU166.PMI:Performance_monitoring_interrupts 24656 ± 7% -52.4% 11733 ± 10% interrupts.CPU166.TLB:TLB_shootdowns 29233 ± 5% -47.6% 15321 ± 6% interrupts.CPU167.CAL:Function_call_interrupts 6154 ± 4% -27.6% 4454 ± 14% interrupts.CPU167.NMI:Non-maskable_interrupts 6154 ± 4% -27.6% 4454 ± 14% interrupts.CPU167.PMI:Performance_monitoring_interrupts 25980 ± 7% -52.6% 12311 ± 7% interrupts.CPU167.TLB:TLB_shootdowns 26911 ± 10% -45.7% 14624 ± 10% interrupts.CPU168.CAL:Function_call_interrupts 6336 ± 5% -32.3% 4287 ± 27% interrupts.CPU168.NMI:Non-maskable_interrupts 6336 ± 5% -32.3% 4287 ± 27% interrupts.CPU168.PMI:Performance_monitoring_interrupts 23567 ± 10% -50.9% 11568 ± 13% interrupts.CPU168.TLB:TLB_shootdowns 27296 ± 9% -46.2% 14687 ± 11% interrupts.CPU169.CAL:Function_call_interrupts 6293 ± 5% -41.3% 3694 ± 36% interrupts.CPU169.NMI:Non-maskable_interrupts 6293 ± 5% -41.3% 3694 ± 36% interrupts.CPU169.PMI:Performance_monitoring_interrupts 24002 ± 9% -51.4% 11676 ± 15% interrupts.CPU169.TLB:TLB_shootdowns 26859 ± 5% -46.1% 14479 ± 7% interrupts.CPU17.CAL:Function_call_interrupts 6134 ± 6% -26.5% 4509 ± 18% interrupts.CPU17.NMI:Non-maskable_interrupts 6134 ± 6% -26.5% 4509 ± 18% interrupts.CPU17.PMI:Performance_monitoring_interrupts 23653 ± 6% -51.2% 11535 ± 9% interrupts.CPU17.TLB:TLB_shootdowns 27483 ± 10% -45.7% 14933 ± 13% interrupts.CPU170.CAL:Function_call_interrupts 24190 ± 10% -51.1% 11827 ± 16% interrupts.CPU170.TLB:TLB_shootdowns 27788 ± 8% -46.7% 14812 ± 12% interrupts.CPU171.CAL:Function_call_interrupts 24457 ± 8% -51.9% 11762 ± 15% interrupts.CPU171.TLB:TLB_shootdowns 27544 ± 9% -45.5% 15011 ± 8% interrupts.CPU172.CAL:Function_call_interrupts 6348 ± 7% -33.9% 4194 ± 26% interrupts.CPU172.NMI:Non-maskable_interrupts 6348 ± 7% -33.9% 4194 ± 26% interrupts.CPU172.PMI:Performance_monitoring_interrupts 24149 ± 9% -50.5% 11961 ± 10% interrupts.CPU172.TLB:TLB_shootdowns 27362 ± 9% -45.6% 14880 ± 8% interrupts.CPU173.CAL:Function_call_interrupts 23971 ± 10% -50.8% 11805 ± 10% interrupts.CPU173.TLB:TLB_shootdowns 27355 ± 10% -46.2% 14722 ± 8% interrupts.CPU174.CAL:Function_call_interrupts 6334 ± 6% -22.9% 4882 ± 26% interrupts.CPU174.NMI:Non-maskable_interrupts 6334 ± 6% -22.9% 4882 ± 26% interrupts.CPU174.PMI:Performance_monitoring_interrupts 24098 ± 10% -51.5% 11690 ± 10% interrupts.CPU174.TLB:TLB_shootdowns 27909 ± 10% -44.5% 15487 ± 9% interrupts.CPU175.CAL:Function_call_interrupts 6305 ± 6% -33.4% 4196 ± 31% interrupts.CPU175.NMI:Non-maskable_interrupts 6305 ± 6% -33.4% 4196 ± 31% interrupts.CPU175.PMI:Performance_monitoring_interrupts 24615 ± 10% -49.7% 12380 ± 12% interrupts.CPU175.TLB:TLB_shootdowns 27866 ± 8% -46.0% 15044 ± 8% interrupts.CPU176.CAL:Function_call_interrupts 24526 ± 8% -51.4% 11927 ± 11% interrupts.CPU176.TLB:TLB_shootdowns 27709 ± 11% -43.8% 15585 ± 11% interrupts.CPU177.CAL:Function_call_interrupts 24404 ± 12% -48.7% 12521 ± 15% interrupts.CPU177.TLB:TLB_shootdowns 28578 ± 8% -47.0% 15143 ± 9% interrupts.CPU178.CAL:Function_call_interrupts 25259 ± 8% -52.2% 12066 ± 11% interrupts.CPU178.TLB:TLB_shootdowns 27908 ± 8% -46.2% 15014 ± 11% interrupts.CPU179.CAL:Function_call_interrupts 24608 ± 8% -51.5% 11937 ± 15% interrupts.CPU179.TLB:TLB_shootdowns 27059 ± 7% -45.1% 14845 ± 9% interrupts.CPU18.CAL:Function_call_interrupts 23765 ± 6% -50.0% 11880 ± 11% interrupts.CPU18.TLB:TLB_shootdowns 27930 ± 9% -45.6% 15199 ± 13% interrupts.CPU180.CAL:Function_call_interrupts 24730 ± 7% -50.9% 12139 ± 16% interrupts.CPU180.TLB:TLB_shootdowns 27442 ± 6% -48.0% 14279 ± 11% interrupts.CPU181.CAL:Function_call_interrupts 24097 ± 6% -52.7% 11387 ± 15% interrupts.CPU181.TLB:TLB_shootdowns 27579 ± 7% -46.7% 14708 ± 10% interrupts.CPU182.CAL:Function_call_interrupts 24311 ± 6% -52.1% 11633 ± 14% interrupts.CPU182.TLB:TLB_shootdowns 27735 ± 9% -46.0% 14966 ± 11% interrupts.CPU183.CAL:Function_call_interrupts 24351 ± 9% -51.1% 11909 ± 13% interrupts.CPU183.TLB:TLB_shootdowns 27962 ± 10% -45.6% 15200 ± 12% interrupts.CPU184.CAL:Function_call_interrupts 24675 ± 10% -50.5% 12209 ± 15% interrupts.CPU184.TLB:TLB_shootdowns 27476 ± 6% -45.1% 15087 ± 12% interrupts.CPU185.CAL:Function_call_interrupts 24052 ± 7% -49.6% 12116 ± 15% interrupts.CPU185.TLB:TLB_shootdowns 27700 ± 7% -46.8% 14723 ± 10% interrupts.CPU186.CAL:Function_call_interrupts 24339 ± 8% -52.1% 11654 ± 13% interrupts.CPU186.TLB:TLB_shootdowns 27587 ± 8% -45.8% 14949 ± 11% interrupts.CPU187.CAL:Function_call_interrupts 6344 ± 5% -23.0% 4888 ± 25% interrupts.CPU187.NMI:Non-maskable_interrupts 6344 ± 5% -23.0% 4888 ± 25% interrupts.CPU187.PMI:Performance_monitoring_interrupts 24155 ± 10% -51.0% 11829 ± 15% interrupts.CPU187.TLB:TLB_shootdowns 28173 ± 10% -48.0% 14658 ± 12% interrupts.CPU188.CAL:Function_call_interrupts 24829 ± 10% -53.6% 11517 ± 15% interrupts.CPU188.TLB:TLB_shootdowns 28397 ± 8% -49.6% 14307 ± 13% interrupts.CPU189.CAL:Function_call_interrupts 25047 ± 8% -55.2% 11217 ± 17% interrupts.CPU189.TLB:TLB_shootdowns 27106 ± 8% -44.4% 15067 ± 9% interrupts.CPU19.CAL:Function_call_interrupts 23886 ± 7% -49.2% 12128 ± 11% interrupts.CPU19.TLB:TLB_shootdowns 28041 ± 10% -47.2% 14810 ± 12% interrupts.CPU190.CAL:Function_call_interrupts 24695 ± 10% -52.7% 11687 ± 16% interrupts.CPU190.TLB:TLB_shootdowns 28185 ± 9% -47.3% 14851 ± 10% interrupts.CPU191.CAL:Function_call_interrupts 6298 ± 6% -50.1% 3141 ± 49% interrupts.CPU191.NMI:Non-maskable_interrupts 6298 ± 6% -50.1% 3141 ± 49% interrupts.CPU191.PMI:Performance_monitoring_interrupts 24829 ± 9% -52.7% 11748 ± 13% interrupts.CPU191.TLB:TLB_shootdowns 25802 ± 6% -49.2% 13105 ± 5% interrupts.CPU2.CAL:Function_call_interrupts 22493 ± 6% -55.1% 10089 ± 7% interrupts.CPU2.TLB:TLB_shootdowns 26384 ± 9% -43.1% 15007 ± 8% interrupts.CPU20.CAL:Function_call_interrupts 23142 ± 8% -48.0% 12028 ± 10% interrupts.CPU20.TLB:TLB_shootdowns 26128 ± 9% -43.2% 14833 ± 9% interrupts.CPU21.CAL:Function_call_interrupts 6251 ± 5% -33.4% 4162 ± 33% interrupts.CPU21.NMI:Non-maskable_interrupts 6251 ± 5% -33.4% 4162 ± 33% interrupts.CPU21.PMI:Performance_monitoring_interrupts 22823 ± 10% -48.1% 11854 ± 11% interrupts.CPU21.TLB:TLB_shootdowns 26900 ± 8% -45.7% 14604 ± 6% interrupts.CPU22.CAL:Function_call_interrupts 6279 ± 6% -26.0% 4643 ± 17% interrupts.CPU22.NMI:Non-maskable_interrupts 6279 ± 6% -26.0% 4643 ± 17% interrupts.CPU22.PMI:Performance_monitoring_interrupts 23712 ± 10% -50.8% 11667 ± 7% interrupts.CPU22.TLB:TLB_shootdowns 26228 ± 9% -44.3% 14618 ± 7% interrupts.CPU23.CAL:Function_call_interrupts 6251 ± 4% -40.9% 3697 ± 45% interrupts.CPU23.NMI:Non-maskable_interrupts 6251 ± 4% -40.9% 3697 ± 45% interrupts.CPU23.PMI:Performance_monitoring_interrupts 22955 ± 10% -48.5% 11824 ± 9% interrupts.CPU23.TLB:TLB_shootdowns 24749 ± 12% -42.9% 14142 ± 5% interrupts.CPU24.CAL:Function_call_interrupts 6406 ± 2% -39.9% 3847 ± 34% interrupts.CPU24.NMI:Non-maskable_interrupts 6406 ± 2% -39.9% 3847 ± 34% interrupts.CPU24.PMI:Performance_monitoring_interrupts 21366 ± 12% -47.8% 11161 ± 7% interrupts.CPU24.TLB:TLB_shootdowns 24735 ± 9% -41.2% 14545 ± 5% interrupts.CPU25.CAL:Function_call_interrupts 6006 ± 5% -32.6% 4045 ± 19% interrupts.CPU25.NMI:Non-maskable_interrupts 6006 ± 5% -32.6% 4045 ± 19% interrupts.CPU25.PMI:Performance_monitoring_interrupts 21376 ± 8% -45.8% 11588 ± 6% interrupts.CPU25.TLB:TLB_shootdowns 24910 ± 9% -39.7% 15012 ± 5% interrupts.CPU26.CAL:Function_call_interrupts 6311 ± 3% -34.7% 4123 ± 23% interrupts.CPU26.NMI:Non-maskable_interrupts 6311 ± 3% -34.7% 4123 ± 23% interrupts.CPU26.PMI:Performance_monitoring_interrupts 21534 ± 9% -43.7% 12131 ± 8% interrupts.CPU26.TLB:TLB_shootdowns 25255 ± 8% -41.0% 14892 ± 4% interrupts.CPU27.CAL:Function_call_interrupts 21957 ± 7% -45.5% 11959 ± 5% interrupts.CPU27.TLB:TLB_shootdowns 25496 ± 8% -41.6% 14890 ± 2% interrupts.CPU28.CAL:Function_call_interrupts 6340 ± 2% -44.1% 3546 ± 37% interrupts.CPU28.NMI:Non-maskable_interrupts 6340 ± 2% -44.1% 3546 ± 37% interrupts.CPU28.PMI:Performance_monitoring_interrupts 22200 ± 7% -46.0% 11986 ± 3% interrupts.CPU28.TLB:TLB_shootdowns 25768 ± 8% -41.9% 14977 ± 4% interrupts.CPU29.CAL:Function_call_interrupts 22356 ± 8% -46.5% 11961 ± 4% interrupts.CPU29.TLB:TLB_shootdowns 26674 ± 7% -46.9% 14173 ± 8% interrupts.CPU3.CAL:Function_call_interrupts 6329 ± 5% -26.1% 4680 ± 20% interrupts.CPU3.NMI:Non-maskable_interrupts 6329 ± 5% -26.1% 4680 ± 20% interrupts.CPU3.PMI:Performance_monitoring_interrupts 23313 ± 7% -52.0% 11198 ± 11% interrupts.CPU3.TLB:TLB_shootdowns 24951 ± 7% -37.8% 15520 ± 3% interrupts.CPU30.CAL:Function_call_interrupts 21567 ± 6% -41.9% 12538 ± 3% interrupts.CPU30.TLB:TLB_shootdowns 25942 ± 10% -38.6% 15925 ± 2% interrupts.CPU31.CAL:Function_call_interrupts 6355 ± 4% -31.7% 4343 ± 18% interrupts.CPU31.NMI:Non-maskable_interrupts 6355 ± 4% -31.7% 4343 ± 18% interrupts.CPU31.PMI:Performance_monitoring_interrupts 22629 ± 10% -42.8% 12954 ± 2% interrupts.CPU31.TLB:TLB_shootdowns 25946 ± 6% -40.2% 15516 ± 2% interrupts.CPU32.CAL:Function_call_interrupts 6418 ± 4% -41.7% 3744 ± 31% interrupts.CPU32.NMI:Non-maskable_interrupts 6418 ± 4% -41.7% 3744 ± 31% interrupts.CPU32.PMI:Performance_monitoring_interrupts 22602 ± 5% -44.1% 12631 ± 2% interrupts.CPU32.TLB:TLB_shootdowns 25218 ± 8% -40.5% 15014 interrupts.CPU33.CAL:Function_call_interrupts 21993 ± 6% -44.1% 12292 interrupts.CPU33.TLB:TLB_shootdowns 24915 ± 10% -36.8% 15737 ± 6% interrupts.CPU34.CAL:Function_call_interrupts 21629 ± 9% -41.0% 12751 ± 7% interrupts.CPU34.TLB:TLB_shootdowns 25492 ± 10% -39.2% 15504 ± 4% interrupts.CPU35.CAL:Function_call_interrupts 22134 ± 9% -43.2% 12579 ± 5% interrupts.CPU35.TLB:TLB_shootdowns 25612 ± 7% -38.6% 15723 ± 5% interrupts.CPU36.CAL:Function_call_interrupts 6202 ± 5% -56.3% 2713 ± 47% interrupts.CPU36.NMI:Non-maskable_interrupts 6202 ± 5% -56.3% 2713 ± 47% interrupts.CPU36.PMI:Performance_monitoring_interrupts 22216 ± 6% -42.6% 12741 ± 7% interrupts.CPU36.TLB:TLB_shootdowns 24492 ± 14% -35.7% 15750 ± 4% interrupts.CPU37.CAL:Function_call_interrupts 5455 ± 24% -46.0% 2945 ± 35% interrupts.CPU37.NMI:Non-maskable_interrupts 5455 ± 24% -46.0% 2945 ± 35% interrupts.CPU37.PMI:Performance_monitoring_interrupts 21124 ± 15% -39.2% 12849 ± 5% interrupts.CPU37.TLB:TLB_shootdowns 25436 ± 9% -37.7% 15841 ± 3% interrupts.CPU38.CAL:Function_call_interrupts 6297 ± 3% -31.3% 4327 ± 21% interrupts.CPU38.NMI:Non-maskable_interrupts 6297 ± 3% -31.3% 4327 ± 21% interrupts.CPU38.PMI:Performance_monitoring_interrupts 22093 ± 8% -41.7% 12888 ± 3% interrupts.CPU38.TLB:TLB_shootdowns 24543 ± 9% -35.4% 15843 ± 5% interrupts.CPU39.CAL:Function_call_interrupts 6360 ± 5% -43.0% 3623 ± 32% interrupts.CPU39.NMI:Non-maskable_interrupts 6360 ± 5% -43.0% 3623 ± 32% interrupts.CPU39.PMI:Performance_monitoring_interrupts 21208 ± 8% -39.1% 12907 ± 6% interrupts.CPU39.TLB:TLB_shootdowns 26453 ± 8% -47.2% 13959 ± 7% interrupts.CPU4.CAL:Function_call_interrupts 23163 ± 8% -52.3% 11052 ± 10% interrupts.CPU4.TLB:TLB_shootdowns 24773 ± 12% -36.3% 15769 ± 3% interrupts.CPU40.CAL:Function_call_interrupts 21535 ± 11% -40.5% 12806 ± 4% interrupts.CPU40.TLB:TLB_shootdowns 25594 ± 10% -39.8% 15414 ± 2% interrupts.CPU41.CAL:Function_call_interrupts 22313 ± 8% -44.4% 12401 ± 2% interrupts.CPU41.TLB:TLB_shootdowns 25198 ± 13% -38.7% 15458 ± 4% interrupts.CPU42.CAL:Function_call_interrupts 21796 ± 13% -42.4% 12565 ± 5% interrupts.CPU42.TLB:TLB_shootdowns 25525 ± 7% -38.7% 15639 ± 6% interrupts.CPU43.CAL:Function_call_interrupts 6398 ± 3% -32.6% 4310 ± 21% interrupts.CPU43.NMI:Non-maskable_interrupts 6398 ± 3% -32.6% 4310 ± 21% interrupts.CPU43.PMI:Performance_monitoring_interrupts 22240 ± 6% -43.1% 12660 ± 7% interrupts.CPU43.TLB:TLB_shootdowns 25386 ± 12% -38.1% 15723 ± 5% interrupts.CPU44.CAL:Function_call_interrupts 6338 ± 5% -44.1% 3545 ± 24% interrupts.CPU44.NMI:Non-maskable_interrupts 6338 ± 5% -44.1% 3545 ± 24% interrupts.CPU44.PMI:Performance_monitoring_interrupts 22061 ± 11% -42.0% 12799 ± 7% interrupts.CPU44.TLB:TLB_shootdowns 24287 ± 13% -33.6% 16117 ± 2% interrupts.CPU45.CAL:Function_call_interrupts 6357 ± 5% -27.5% 4607 ± 21% interrupts.CPU45.NMI:Non-maskable_interrupts 6357 ± 5% -27.5% 4607 ± 21% interrupts.CPU45.PMI:Performance_monitoring_interrupts 20932 ± 12% -37.4% 13099 ± 2% interrupts.CPU45.TLB:TLB_shootdowns 25436 ± 10% -37.1% 15993 interrupts.CPU46.CAL:Function_call_interrupts 6270 ± 5% -30.6% 4350 ± 23% interrupts.CPU46.NMI:Non-maskable_interrupts 6270 ± 5% -30.6% 4350 ± 23% interrupts.CPU46.PMI:Performance_monitoring_interrupts 747.75 ± 30% -38.1% 462.50 ± 25% interrupts.CPU46.RES:Rescheduling_interrupts 22069 ± 9% -41.1% 12997 ± 2% interrupts.CPU46.TLB:TLB_shootdowns 25670 ± 10% -38.7% 15735 ± 8% interrupts.CPU47.CAL:Function_call_interrupts 22454 ± 10% -43.3% 12733 ± 10% interrupts.CPU47.TLB:TLB_shootdowns 27053 ± 3% -47.8% 14113 ± 6% interrupts.CPU48.CAL:Function_call_interrupts 6279 ± 2% -34.2% 4134 ± 25% interrupts.CPU48.NMI:Non-maskable_interrupts 6279 ± 2% -34.2% 4134 ± 25% interrupts.CPU48.PMI:Performance_monitoring_interrupts 23836 ± 4% -53.3% 11123 ± 7% interrupts.CPU48.TLB:TLB_shootdowns 26789 -49.1% 13642 ± 7% interrupts.CPU49.CAL:Function_call_interrupts 6202 -35.4% 4006 ± 38% interrupts.CPU49.NMI:Non-maskable_interrupts 6202 -35.4% 4006 ± 38% interrupts.CPU49.PMI:Performance_monitoring_interrupts 23620 ± 4% -54.9% 10645 ± 9% interrupts.CPU49.TLB:TLB_shootdowns 26950 ± 4% -45.1% 14808 ± 7% interrupts.CPU5.CAL:Function_call_interrupts 6268 ± 5% -35.6% 4034 ± 32% interrupts.CPU5.NMI:Non-maskable_interrupts 6268 ± 5% -35.6% 4034 ± 32% interrupts.CPU5.PMI:Performance_monitoring_interrupts 23633 ± 5% -49.9% 11849 ± 9% interrupts.CPU5.TLB:TLB_shootdowns 26621 -46.0% 14377 ± 2% interrupts.CPU50.CAL:Function_call_interrupts 6283 ± 3% -31.5% 4304 ± 20% interrupts.CPU50.NMI:Non-maskable_interrupts 6283 ± 3% -31.5% 4304 ± 20% interrupts.CPU50.PMI:Performance_monitoring_interrupts 23351 ± 3% -51.4% 11359 ± 3% interrupts.CPU50.TLB:TLB_shootdowns 26585 -44.9% 14661 ± 5% interrupts.CPU51.CAL:Function_call_interrupts 6309 ± 4% -36.1% 4031 ± 38% interrupts.CPU51.NMI:Non-maskable_interrupts 6309 ± 4% -36.1% 4031 ± 38% interrupts.CPU51.PMI:Performance_monitoring_interrupts 23286 ± 5% -49.8% 11685 ± 6% interrupts.CPU51.TLB:TLB_shootdowns 26514 ± 4% -42.4% 15272 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 6147 ± 5% -48.6% 3159 ± 33% interrupts.CPU52.NMI:Non-maskable_interrupts 6147 ± 5% -48.6% 3159 ± 33% interrupts.CPU52.PMI:Performance_monitoring_interrupts 23414 ± 7% -47.6% 12270 ± 9% interrupts.CPU52.TLB:TLB_shootdowns 27328 ± 3% -46.5% 14624 ± 5% interrupts.CPU53.CAL:Function_call_interrupts 24033 ± 7% -51.6% 11632 ± 6% interrupts.CPU53.TLB:TLB_shootdowns 26150 ± 3% -43.3% 14824 ± 8% interrupts.CPU54.CAL:Function_call_interrupts 22887 ± 3% -48.1% 11879 ± 10% interrupts.CPU54.TLB:TLB_shootdowns 27196 ± 2% -45.0% 14952 ± 6% interrupts.CPU55.CAL:Function_call_interrupts 6356 ± 2% -32.9% 4263 ± 23% interrupts.CPU55.NMI:Non-maskable_interrupts 6356 ± 2% -32.9% 4263 ± 23% interrupts.CPU55.PMI:Performance_monitoring_interrupts 23944 ± 4% -49.7% 12051 ± 7% interrupts.CPU55.TLB:TLB_shootdowns 26840 ± 2% -43.1% 15283 ± 6% interrupts.CPU56.CAL:Function_call_interrupts 23570 ± 5% -47.7% 12319 ± 7% interrupts.CPU56.TLB:TLB_shootdowns 27239 ± 2% -43.1% 15511 ± 8% interrupts.CPU57.CAL:Function_call_interrupts 393.00 ± 12% +21.9% 479.25 ± 13% interrupts.CPU57.RES:Rescheduling_interrupts 23900 ± 5% -47.6% 12525 ± 10% interrupts.CPU57.TLB:TLB_shootdowns 27064 ± 2% -43.0% 15437 ± 5% interrupts.CPU58.CAL:Function_call_interrupts 23828 ± 4% -47.7% 12455 ± 7% interrupts.CPU58.TLB:TLB_shootdowns 27483 -43.8% 15433 ± 10% interrupts.CPU59.CAL:Function_call_interrupts 24266 ± 3% -48.6% 12463 ± 12% interrupts.CPU59.TLB:TLB_shootdowns 26361 ± 5% -46.2% 14179 ± 7% interrupts.CPU6.CAL:Function_call_interrupts 6265 ± 4% -24.6% 4723 ± 18% interrupts.CPU6.NMI:Non-maskable_interrupts 6265 ± 4% -24.6% 4723 ± 18% interrupts.CPU6.PMI:Performance_monitoring_interrupts 23127 ± 6% -51.5% 11222 ± 9% interrupts.CPU6.TLB:TLB_shootdowns 26757 ± 4% -43.3% 15169 ± 7% interrupts.CPU60.CAL:Function_call_interrupts 24077 ± 4% -49.3% 12205 ± 9% interrupts.CPU60.TLB:TLB_shootdowns 27530 -43.7% 15489 ± 6% interrupts.CPU61.CAL:Function_call_interrupts 24260 ± 2% -48.4% 12528 ± 9% interrupts.CPU61.TLB:TLB_shootdowns 27161 -44.6% 15044 ± 5% interrupts.CPU62.CAL:Function_call_interrupts 23894 ± 3% -49.2% 12132 ± 7% interrupts.CPU62.TLB:TLB_shootdowns 26635 ± 3% -42.9% 15214 ± 5% interrupts.CPU63.CAL:Function_call_interrupts 23534 ± 3% -47.9% 12263 ± 6% interrupts.CPU63.TLB:TLB_shootdowns 27482 ± 3% -45.4% 15004 ± 4% interrupts.CPU64.CAL:Function_call_interrupts 24240 ± 5% -50.5% 12010 ± 5% interrupts.CPU64.TLB:TLB_shootdowns 27551 ± 2% -43.2% 15661 ± 4% interrupts.CPU65.CAL:Function_call_interrupts 24288 ± 3% -47.8% 12668 ± 6% interrupts.CPU65.TLB:TLB_shootdowns 27620 ± 4% -44.8% 15257 ± 5% interrupts.CPU66.CAL:Function_call_interrupts 24313 ± 5% -49.6% 12248 ± 7% interrupts.CPU66.TLB:TLB_shootdowns 27420 ± 2% -43.7% 15438 ± 7% interrupts.CPU67.CAL:Function_call_interrupts 6064 ± 8% -27.8% 4378 ± 18% interrupts.CPU67.NMI:Non-maskable_interrupts 6064 ± 8% -27.8% 4378 ± 18% interrupts.CPU67.PMI:Performance_monitoring_interrupts 24187 ± 4% -48.5% 12455 ± 9% interrupts.CPU67.TLB:TLB_shootdowns 27980 ± 3% -46.0% 15105 ± 6% interrupts.CPU68.CAL:Function_call_interrupts 6223 ± 4% -30.8% 4303 ± 20% interrupts.CPU68.NMI:Non-maskable_interrupts 6223 ± 4% -30.8% 4303 ± 20% interrupts.CPU68.PMI:Performance_monitoring_interrupts 24702 ± 6% -50.9% 12120 ± 8% interrupts.CPU68.TLB:TLB_shootdowns 27578 ± 5% -45.0% 15181 ± 7% interrupts.CPU69.CAL:Function_call_interrupts 6112 ± 4% -40.5% 3638 ± 16% interrupts.CPU69.NMI:Non-maskable_interrupts 6112 ± 4% -40.5% 3638 ± 16% interrupts.CPU69.PMI:Performance_monitoring_interrupts 24269 ± 6% -49.7% 12205 ± 8% interrupts.CPU69.TLB:TLB_shootdowns 26146 ± 7% -44.8% 14424 ± 7% interrupts.CPU7.CAL:Function_call_interrupts 22825 ± 6% -49.7% 11483 ± 10% interrupts.CPU7.TLB:TLB_shootdowns 27118 ± 5% -46.2% 14601 ± 8% interrupts.CPU70.CAL:Function_call_interrupts 6198 ± 4% -39.0% 3780 ± 32% interrupts.CPU70.NMI:Non-maskable_interrupts 6198 ± 4% -39.0% 3780 ± 32% interrupts.CPU70.PMI:Performance_monitoring_interrupts 23866 ± 8% -51.3% 11619 ± 10% interrupts.CPU70.TLB:TLB_shootdowns 27347 ± 6% -45.7% 14855 ± 7% interrupts.CPU71.CAL:Function_call_interrupts 24244 ± 10% -51.0% 11883 ± 9% interrupts.CPU71.TLB:TLB_shootdowns 26505 ± 9% -45.8% 14367 ± 6% interrupts.CPU72.CAL:Function_call_interrupts 23166 ± 9% -50.8% 11391 ± 8% interrupts.CPU72.TLB:TLB_shootdowns 27462 ± 8% -47.5% 14410 ± 13% interrupts.CPU73.CAL:Function_call_interrupts 6234 ± 3% -32.7% 4197 ± 25% interrupts.CPU73.NMI:Non-maskable_interrupts 6234 ± 3% -32.7% 4197 ± 25% interrupts.CPU73.PMI:Performance_monitoring_interrupts 24197 ± 8% -52.9% 11396 ± 17% interrupts.CPU73.TLB:TLB_shootdowns 26916 ± 8% -47.1% 14237 ± 9% interrupts.CPU74.CAL:Function_call_interrupts 23636 ± 8% -52.1% 11321 ± 11% interrupts.CPU74.TLB:TLB_shootdowns 27194 ± 8% -46.6% 14528 ± 8% interrupts.CPU75.CAL:Function_call_interrupts 6285 ± 6% -33.9% 4153 ± 31% interrupts.CPU75.NMI:Non-maskable_interrupts 6285 ± 6% -33.9% 4153 ± 31% interrupts.CPU75.PMI:Performance_monitoring_interrupts 23889 ± 8% -51.9% 11497 ± 11% interrupts.CPU75.TLB:TLB_shootdowns 27127 ± 9% -45.3% 14833 ± 11% interrupts.CPU76.CAL:Function_call_interrupts 23874 ± 11% -50.4% 11841 ± 14% interrupts.CPU76.TLB:TLB_shootdowns 27266 ± 9% -46.1% 14706 ± 10% interrupts.CPU77.CAL:Function_call_interrupts 23940 ± 10% -51.3% 11653 ± 13% interrupts.CPU77.TLB:TLB_shootdowns 27483 ± 9% -46.9% 14595 ± 11% interrupts.CPU78.CAL:Function_call_interrupts 24199 ± 9% -52.0% 11605 ± 15% interrupts.CPU78.TLB:TLB_shootdowns 27843 ± 11% -48.8% 14261 ± 9% interrupts.CPU79.CAL:Function_call_interrupts 6445 ± 4% -24.5% 4868 ± 22% interrupts.CPU79.NMI:Non-maskable_interrupts 6445 ± 4% -24.5% 4868 ± 22% interrupts.CPU79.PMI:Performance_monitoring_interrupts 24643 ± 12% -54.0% 11330 ± 10% interrupts.CPU79.TLB:TLB_shootdowns 26776 ± 11% -46.1% 14430 ± 6% interrupts.CPU8.CAL:Function_call_interrupts 23541 ± 11% -51.4% 11431 ± 8% interrupts.CPU8.TLB:TLB_shootdowns 27139 ± 10% -45.4% 14826 ± 10% interrupts.CPU80.CAL:Function_call_interrupts 23925 ± 10% -50.4% 11866 ± 13% interrupts.CPU80.TLB:TLB_shootdowns 26416 ± 12% -45.7% 14354 ± 10% interrupts.CPU81.CAL:Function_call_interrupts 23185 ± 12% -50.8% 11399 ± 13% interrupts.CPU81.TLB:TLB_shootdowns 28161 ± 11% -47.9% 14665 ± 10% interrupts.CPU82.CAL:Function_call_interrupts 6338 ± 5% -34.3% 4164 ± 29% interrupts.CPU82.NMI:Non-maskable_interrupts 6338 ± 5% -34.3% 4164 ± 29% interrupts.CPU82.PMI:Performance_monitoring_interrupts 24870 ± 11% -52.7% 11756 ± 13% interrupts.CPU82.TLB:TLB_shootdowns 27751 ± 11% -46.6% 14821 ± 10% interrupts.CPU83.CAL:Function_call_interrupts 24497 ± 11% -51.7% 11823 ± 13% interrupts.CPU83.TLB:TLB_shootdowns 27812 ± 9% -47.2% 14675 ± 11% interrupts.CPU84.CAL:Function_call_interrupts 24536 ± 9% -52.5% 11650 ± 14% interrupts.CPU84.TLB:TLB_shootdowns 27537 ± 8% -44.6% 15254 ± 10% interrupts.CPU85.CAL:Function_call_interrupts 24260 ± 8% -49.0% 12380 ± 13% interrupts.CPU85.TLB:TLB_shootdowns 27193 ± 11% -47.4% 14313 ± 12% interrupts.CPU86.CAL:Function_call_interrupts 6416 ± 4% -45.7% 3485 ± 27% interrupts.CPU86.NMI:Non-maskable_interrupts 6416 ± 4% -45.7% 3485 ± 27% interrupts.CPU86.PMI:Performance_monitoring_interrupts 24032 ± 11% -52.7% 11370 ± 15% interrupts.CPU86.TLB:TLB_shootdowns 27092 ± 9% -45.6% 14750 ± 7% interrupts.CPU87.CAL:Function_call_interrupts 23820 ± 10% -50.7% 11754 ± 10% interrupts.CPU87.TLB:TLB_shootdowns 27593 ± 9% -45.8% 14961 ± 9% interrupts.CPU88.CAL:Function_call_interrupts 24289 ± 9% -50.8% 11950 ± 11% interrupts.CPU88.TLB:TLB_shootdowns 27852 ± 9% -47.1% 14746 ± 8% interrupts.CPU89.CAL:Function_call_interrupts 24606 ± 10% -52.4% 11718 ± 11% interrupts.CPU89.TLB:TLB_shootdowns 27251 ± 4% -46.2% 14670 ± 2% interrupts.CPU9.CAL:Function_call_interrupts 6241 ± 4% -40.6% 3710 ± 30% interrupts.CPU9.NMI:Non-maskable_interrupts 6241 ± 4% -40.6% 3710 ± 30% interrupts.CPU9.PMI:Performance_monitoring_interrupts 23911 ± 5% -50.8% 11756 ± 2% interrupts.CPU9.TLB:TLB_shootdowns 26653 ± 8% -44.2% 14876 ± 10% interrupts.CPU90.CAL:Function_call_interrupts 6351 ± 3% -31.2% 4370 ± 32% interrupts.CPU90.NMI:Non-maskable_interrupts 6351 ± 3% -31.2% 4370 ± 32% interrupts.CPU90.PMI:Performance_monitoring_interrupts 23402 ± 8% -49.3% 11858 ± 13% interrupts.CPU90.TLB:TLB_shootdowns 27116 ± 10% -46.6% 14491 ± 10% interrupts.CPU91.CAL:Function_call_interrupts 6467 ± 4% -33.5% 4301 ± 34% interrupts.CPU91.NMI:Non-maskable_interrupts 6467 ± 4% -33.5% 4301 ± 34% interrupts.CPU91.PMI:Performance_monitoring_interrupts 23824 ± 11% -51.6% 11536 ± 12% interrupts.CPU91.TLB:TLB_shootdowns 26961 ± 11% -46.1% 14532 ± 8% interrupts.CPU92.CAL:Function_call_interrupts 23679 ± 13% -51.4% 11497 ± 11% interrupts.CPU92.TLB:TLB_shootdowns 27548 ± 10% -46.2% 14817 ± 11% interrupts.CPU93.CAL:Function_call_interrupts 6512 ± 5% -30.4% 4531 ± 38% interrupts.CPU93.NMI:Non-maskable_interrupts 6512 ± 5% -30.4% 4531 ± 38% interrupts.CPU93.PMI:Performance_monitoring_interrupts 24280 ± 11% -51.2% 11841 ± 14% interrupts.CPU93.TLB:TLB_shootdowns 27800 ± 11% -47.4% 14610 ± 9% interrupts.CPU94.CAL:Function_call_interrupts 6433 ± 4% -41.1% 3789 ± 45% interrupts.CPU94.NMI:Non-maskable_interrupts 6433 ± 4% -41.1% 3789 ± 45% interrupts.CPU94.PMI:Performance_monitoring_interrupts 24567 ± 11% -52.6% 11637 ± 11% interrupts.CPU94.TLB:TLB_shootdowns 27488 ± 9% -44.3% 15305 ± 7% interrupts.CPU95.CAL:Function_call_interrupts 24215 ± 10% -48.8% 12388 ± 9% interrupts.CPU95.TLB:TLB_shootdowns 24706 ± 3% -47.9% 12861 ± 10% interrupts.CPU96.CAL:Function_call_interrupts 21352 ± 3% -54.2% 9779 ± 12% interrupts.CPU96.TLB:TLB_shootdowns 26692 ± 6% -46.5% 14274 ± 6% interrupts.CPU97.CAL:Function_call_interrupts 6129 ± 4% -35.1% 3975 ± 30% interrupts.CPU97.NMI:Non-maskable_interrupts 6129 ± 4% -35.1% 3975 ± 30% interrupts.CPU97.PMI:Performance_monitoring_interrupts 23379 ± 4% -51.5% 11344 ± 8% interrupts.CPU97.TLB:TLB_shootdowns 26419 ± 6% -46.1% 14244 ± 7% interrupts.CPU98.CAL:Function_call_interrupts 23076 ± 7% -50.9% 11339 ± 10% interrupts.CPU98.TLB:TLB_shootdowns 26972 ± 7% -44.4% 14989 ± 12% interrupts.CPU99.CAL:Function_call_interrupts 6269 ± 5% -46.7% 3344 ± 27% interrupts.CPU99.NMI:Non-maskable_interrupts 6269 ± 5% -46.7% 3344 ± 27% interrupts.CPU99.PMI:Performance_monitoring_interrupts 23671 ± 8% -49.2% 12023 ± 16% interrupts.CPU99.TLB:TLB_shootdowns 170.25 -44.5% 94.50 ± 47% interrupts.IWI:IRQ_work_interrupts 1103722 -28.1% 794110 ± 14% interrupts.NMI:Non-maskable_interrupts 1103722 -28.1% 794110 ± 14% interrupts.PMI:Performance_monitoring_interrupts 4531323 ± 2% -48.8% 2317840 ± 6% interrupts.TLB:TLB_shootdowns vm-scalability.time.user_time 25000 +-+-----------------------------------------------------------------+ | +.+.+.+.+ +.+.+.+.+ +.+.+.+.+.+.+.+ +.+ +.+.+.+.+.+.| | O | 20000 O-+ O O O O O O O | | | | O O O O | 15000 +-+ | | | 10000 +-+ | | | | | 5000 +-+ | | | | | 0 +-O-O-O-O-----------------------------------------------------------+ vm-scalability.time.system_time 25000 +-+-----------------------------------------------------------------+ |. .+.+.+.+. .+. .+. .+. | | + +.+.+.+.+.+.+.+ +.+.+.+.+.+.+ +.+.+ +.+.+.+.+.+.+.| 20000 O-+ O O O O O O O O | | O O | | O O | 15000 +-+ | | | 10000 +-+ | | | | | 5000 +-+ | | | | | 0 +-O-O-O-O-----------------------------------------------------------+ vm-scalability.time.percent_of_cpu_this_job_got 16000 +-+-----------------------------------------------------------------+ |.+.+.+.+.+.+.+.+.+.+.+.+.+ +.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.| 14000 O-+ O O | 12000 +-+ O O O O O O | | O O | 10000 +-+ O O | | | 8000 +-+ | | | 6000 +-+ | 4000 +-+ | | | 2000 +-+ | | | 0 +-O-O-O-O-----------------------------------------------------------+ vm-scalability.time.minor_page_faults 1.6e+07 +-+---------------------------------------------------------------+ | | 1.4e+07 +-+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.| 1.2e+07 O-+ O O O OO O O O | | | 1e+07 +-+ O O O O | | | 8e+06 +-+ | | | 6e+06 +-+ | 4e+06 +-+ | | | 2e+06 +-+ | | | 0 +-O-O-O-O---------------------------------------------------------+ vm-scalability.time.voluntary_context_switches 1.6e+06 +-+---------------------------------------------------------------+ |.+. .+. .+. .++. .+. .+.+. .+. .+.+. .+ .+. + + | 1.4e+06 O-+ + +.O O O O + O O +.+.+ + +.+.+ + + | | O O | 1.2e+06 +-+ | 1e+06 +-+ | | | 800000 +-+ O O O O | | | 600000 +-+ | 400000 +-+ | | | 200000 +-+ | | | 0 +-O-O-O-O---------------------------------------------------------+ vm-scalability.time.involuntary_context_switches 350000 +-+----------------------------------------------------------------+ | .+.+. .+. +. .+. .+.+. .+.+. .+. | 300000 +-+.+.+.+.+.+ +.+ +.+.+.+.+.+ + +.+.+ +.+.+ + +.| | | 250000 +-+ | | | 200000 +-+ | O O O | 150000 +-+ O O O O O O | | O | 100000 +-+ O O O | | | 50000 +-+ | | | 0 +-O-O-O-O----------------------------------------------------------+ vm-scalability.free_time 0.025 +-+-----------------------O-----O-----------------------------------+ | | | O O | 0.02 +-+ | | | | | 0.015 +-+ | | O O O O O | 0.01 O-+ O O O | |.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.| | | 0.005 +-+ | | | | | 0 +-O-O-O-O-----------------------------------------------------------+ vm-scalability.median 200000 +-+----------------------------------------------------------------+ 180000 +-+ + +.+.O O O O + + + +.+ + + + | O O O O O | 160000 +-+ O O O O | 140000 +-+ | | | 120000 +-+ | 100000 +-+ | 80000 +-+ | | | 60000 +-+ | 40000 +-+ | | | 20000 +-+ | 0 +-O-O-O-O----------------------------------------------------------+ vm-scalability.median_stddev 0.12 +-+------------------------------------------------------------------+ | O O | 0.1 +-+ O | | O | O O O O O | 0.08 +-+ O O O | | O | 0.06 +-+ | | +. | 0.04 +-+ .+.+. .+.+. .+. .+.+.+. .+.. .+. .+. .+.+. .+. + +. .+.| |.+.+ + + + +.+ + + + + +.+.+.+ + | | | 0.02 +-+ | | | 0 +-O-O-O-O------------------------------------------------------------+ vm-scalability.workload 1.2e+10 +-+---------------------------------------------------------------+ | | 1e+10 +-+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.| O O O O OO O O O | | | 8e+09 +-+ O O O O | | | 6e+09 +-+ | | | 4e+09 +-+ | | | | | 2e+09 +-+ | | | 0 +-O-O-O-O---------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase: gcc-7/performance/1BRD_48G/f2fs/x86_64-rhel-7.6/3000/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_src/aim7 commit: 94a9717b3c ("locking/rwsem: Make rwsem->owner an atomic_long_t") 7d43f1ce9d ("locking/rwsem: Enable time-based spinning on reader-owned rwsem") 94a9717b3c40e77a 7d43f1ce9dd075d8b2aa3ad1f39 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 50% 1:2 dmesg.WARNING:at_ip_do_filp_open/0x *************************************************************************************************** lkp-ivb-d02: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-d02/small-allocs-mt/vm-scalability/0x21 commit: 94a9717b3c ("locking/rwsem: Make rwsem->owner an atomic_long_t") 7d43f1ce9d ("locking/rwsem: Enable time-based spinning on reader-owned rwsem") 94a9717b3c40e77a 7d43f1ce9dd075d8b2aa3ad1f39 ---------------- --------------------------- %stddev %change %stddev \ | \ 793660 +14.0% 904895 vm-scalability.median 3173995 +14.0% 3617058 vm-scalability.throughput 151.84 -11.9% 133.72 vm-scalability.time.elapsed_time 151.84 -11.9% 133.72 vm-scalability.time.elapsed_time.max 713096 ± 2% +20.1% 856760 ± 2% vm-scalability.time.involuntary_context_switches 360.25 +4.9% 378.00 vm-scalability.time.percent_of_cpu_this_job_got 200.83 -19.5% 161.71 vm-scalability.time.system_time 7120853 ± 3% -97.5% 180425 vm-scalability.time.voluntary_context_switches 21.06 +5.8% 22.29 ± 4% boot-time.boot 54.91 +9.2% 59.95 ± 7% boot-time.idle 124534 ± 3% -9.1% 113147 meminfo.AnonHugePages 28262 +14.7% 32431 meminfo.max_used_kB 8.74 ± 4% -5.4 3.37 ± 3% mpstat.cpu.all.idle% 58.13 +7.7 65.86 mpstat.cpu.all.usr% 1207 ± 4% +14.8% 1385 ± 2% slabinfo.kmalloc-96.active_objs 1207 ± 4% +14.8% 1385 ± 2% slabinfo.kmalloc-96.num_objs 9.25 ± 4% -56.8% 4.00 vmstat.cpu.id 57.25 +12.7% 64.50 vmstat.cpu.us 101684 ± 2% -83.4% 16863 vmstat.system.cs 13871 +1.5% 14082 vmstat.system.in 5053 -1.2% 4991 proc-vmstat.nr_inactive_anon 5512 +1.4% 5588 proc-vmstat.nr_mapped 260063 +1.6% 264256 proc-vmstat.nr_slab_unreclaimable 5053 -1.2% 4991 proc-vmstat.nr_zone_inactive_anon 1113 -1.6% 1095 proc-vmstat.pgactivate 62845 ± 2% -74.7% 15909 ± 10% softirqs.CPU0.SCHED 61686 -72.7% 16834 ± 6% softirqs.CPU1.SCHED 62727 -76.0% 15049 ± 9% softirqs.CPU2.SCHED 60562 ± 2% -72.5% 16673 ± 4% softirqs.CPU3.SCHED 247822 -74.0% 64466 ± 2% softirqs.SCHED 30358831 ± 4% -94.8% 1581973 ± 4% cpuidle.C1.time 6764050 ± 3% -97.5% 169367 cpuidle.C1.usage 1648723 ± 27% +212.2% 5147756 ± 11% cpuidle.C1E.time 37741 ± 13% +132.6% 87804 ± 18% cpuidle.C1E.usage 104315 ± 7% +653.2% 785684 ± 76% cpuidle.C3.time 9526 ± 4% -71.0% 2761 ± 64% cpuidle.C3.usage 196095 ± 2% -98.8% 2432 ± 3% cpuidle.POLL.time 88406 ± 2% -97.4% 2325 ± 4% cpuidle.POLL.usage 3059 +3.8% 3176 turbostat.Avg_MHz 6764093 ± 3% -97.5% 169364 turbostat.C1 4.96 ± 3% -4.7 0.29 ± 3% turbostat.C1% 37886 ± 13% +131.7% 87799 ± 18% turbostat.C1E 0.27 ± 27% +0.7 0.95 ± 10% turbostat.C1E% 9529 ± 4% -71.1% 2754 ± 64% turbostat.C3 0.02 ± 24% +0.1 0.14 ± 72% turbostat.C3% 1.83 ± 3% +0.3 2.15 ± 2% turbostat.C6% 6.00 ± 3% -61.8% 2.29 ± 6% turbostat.CPU%c1 0.00 ±173% +25100.0% 0.63 ± 98% turbostat.CPU%c3 2138754 -10.3% 1918451 turbostat.IRQ 4014 -10.5% 3593 ± 2% interrupts.CAL:Function_call_interrupts 18427 ± 6% -95.1% 895.00 ± 11% interrupts.CPU0.RES:Rescheduling_interrupts 895.75 ±144% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts 895.75 ±144% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts 19299 ± 11% -94.1% 1139 ± 20% interrupts.CPU1.RES:Rescheduling_interrupts 58.00 ± 73% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 58.00 ± 73% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 19013 ± 8% -94.4% 1062 ± 24% interrupts.CPU2.RES:Rescheduling_interrupts 509962 ± 3% -20.9% 403302 ± 4% interrupts.CPU3.LOC:Local_timer_interrupts 631.75 ±163% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 631.75 ±163% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 18040 ± 12% -93.7% 1129 ± 14% interrupts.CPU3.RES:Rescheduling_interrupts 1607 ± 81% -99.6% 6.00 interrupts.NMI:Non-maskable_interrupts 1607 ± 81% -99.6% 6.00 interrupts.PMI:Performance_monitoring_interrupts 74780 ± 4% -94.3% 4226 ± 13% interrupts.RES:Rescheduling_interrupts 2.25 ± 23% +75.9% 3.96 ± 5% sched_debug.cfs_rq:/.nr_spread_over.avg 4.50 ± 44% +103.7% 9.17 ± 5% sched_debug.cfs_rq:/.nr_spread_over.max 1.64 ± 49% +101.5% 3.31 ± 3% sched_debug.cfs_rq:/.nr_spread_over.stddev 422.17 ± 14% +29.7% 547.50 ± 3% sched_debug.cfs_rq:/.runnable_load_avg.max 135.56 ± 13% +34.4% 182.15 ± 8% sched_debug.cfs_rq:/.runnable_load_avg.stddev 1047 ± 12% +33.7% 1400 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.max 416.51 ± 11% +24.1% 516.73 sched_debug.cfs_rq:/.util_est_enqueued.stddev 273573 ± 4% +14.6% 313548 ± 9% sched_debug.cpu.avg_idle.avg 0.68 ± 22% +155.0% 1.73 ± 14% sched_debug.cpu.clock.stddev 0.68 ± 22% +155.0% 1.73 ± 14% sched_debug.cpu.clock_task.stddev 216.10 ± 10% +34.3% 290.33 ± 12% sched_debug.cpu.cpu_load[0].avg 401.50 ± 23% +40.5% 564.17 ± 4% sched_debug.cpu.cpu_load[0].max 112.08 ± 39% +65.0% 184.96 ± 7% sched_debug.cpu.cpu_load[0].stddev 211.88 ± 9% +17.9% 249.83 ± 4% sched_debug.cpu.cpu_load[1].avg 325.50 ± 15% +47.4% 479.83 ± 7% sched_debug.cpu.cpu_load[1].max 72.16 ± 25% +102.9% 146.40 ± 12% sched_debug.cpu.cpu_load[1].stddev 205.62 ± 6% +11.2% 228.71 ± 3% sched_debug.cpu.cpu_load[2].avg 273.50 ± 9% +47.5% 403.33 sched_debug.cpu.cpu_load[2].max 47.18 ± 17% +128.6% 107.86 sched_debug.cpu.cpu_load[2].stddev 202.62 ± 4% +7.5% 217.75 ± 3% sched_debug.cpu.cpu_load[3].avg 246.67 ± 5% +38.0% 340.50 ± 7% sched_debug.cpu.cpu_load[3].max 32.83 ± 7% +127.3% 74.63 ± 18% sched_debug.cpu.cpu_load[3].stddev 236.42 ± 4% +26.2% 298.33 ± 11% sched_debug.cpu.cpu_load[4].max 25.55 ± 13% +105.2% 52.42 ± 30% sched_debug.cpu.cpu_load[4].stddev 0.00 ± 14% -50.1% 0.00 sched_debug.cpu.next_balance.stddev 1905769 ± 2% -79.6% 387826 sched_debug.cpu.nr_switches.avg 1972844 ± 2% -73.6% 520051 ± 5% sched_debug.cpu.nr_switches.max 1744240 -84.6% 269070 ± 3% sched_debug.cpu.nr_switches.min 23.92 ± 55% +130.7% 55.17 ± 45% sched_debug.cpu.nr_uninterruptible.max 18.08 ± 60% +118.2% 39.45 ± 26% sched_debug.cpu.nr_uninterruptible.stddev 1895971 ± 2% -80.1% 377492 sched_debug.cpu.sched_count.avg 1962479 ± 2% -73.9% 512285 ± 5% sched_debug.cpu.sched_count.max 1734135 -84.9% 261092 ± 2% sched_debug.cpu.sched_count.min 794818 ± 2% -94.8% 41267 ± 8% sched_debug.cpu.sched_goidle.avg 825986 ± 3% -93.5% 54059 ± 12% sched_debug.cpu.sched_goidle.max 738516 ± 2% -96.2% 28020 sched_debug.cpu.sched_goidle.min 33586 ± 11% -71.6% 9536 ± 24% sched_debug.cpu.sched_goidle.stddev 986934 ± 2% -80.6% 191360 sched_debug.cpu.ttwu_count.avg 1048238 ± 2% -74.3% 269323 ± 5% sched_debug.cpu.ttwu_count.max 882027 ± 2% -86.0% 123850 ± 5% sched_debug.cpu.ttwu_count.min 179043 ± 2% +29.0% 230973 ± 6% sched_debug.cpu.ttwu_local.max 130127 ± 4% -22.1% 101364 ± 6% sched_debug.cpu.ttwu_local.min 18651 ± 12% +174.2% 51141 ± 9% sched_debug.cpu.ttwu_local.stddev 2.32 ± 16% -62.1% 0.88 perf-stat.i.MPKI 3.793e+09 +10.4% 4.187e+09 perf-stat.i.branch-instructions 2.64 ± 14% +2.8 5.39 ± 6% perf-stat.i.cache-miss-rate% 499435 ± 17% +37.6% 687301 ± 9% perf-stat.i.cache-misses 24288477 ± 3% -51.9% 11692134 perf-stat.i.cache-references 103479 ± 3% -83.4% 17136 perf-stat.i.context-switches 1.02 -6.7% 0.95 perf-stat.i.cpi 1.23e+10 +3.9% 1.278e+10 perf-stat.i.cpu-cycles 1998 ± 2% -96.2% 76.32 ± 10% perf-stat.i.cpu-migrations 3.219e+09 +9.8% 3.533e+09 perf-stat.i.dTLB-loads 0.28 +0.0 0.29 perf-stat.i.dTLB-store-miss-rate% 2078000 +4.5% 2171902 perf-stat.i.dTLB-store-misses 7.347e+08 +2.0% 7.492e+08 perf-stat.i.dTLB-stores 84.00 +14.2 98.21 perf-stat.i.iTLB-load-miss-rate% 1805387 ± 6% -31.7% 1232340 perf-stat.i.iTLB-load-misses 353772 ± 4% -95.3% 16649 perf-stat.i.iTLB-loads 1.226e+10 +9.6% 1.344e+10 perf-stat.i.instructions 8575 ± 12% +69.5% 14535 ± 3% perf-stat.i.instructions-per-iTLB-miss 1.00 +6.2% 1.06 perf-stat.i.ipc 690868 +13.2% 781827 perf-stat.i.minor-faults 690868 +13.2% 781827 perf-stat.i.page-faults 1.98 ± 3% -56.1% 0.87 perf-stat.overall.MPKI 2.06 ± 18% +3.8 5.89 ± 11% perf-stat.overall.cache-miss-rate% 1.00 -5.2% 0.95 perf-stat.overall.cpi 0.28 +0.0 0.29 perf-stat.overall.dTLB-store-miss-rate% 83.57 +15.1 98.67 perf-stat.overall.iTLB-load-miss-rate% 6818 ± 6% +59.9% 10905 perf-stat.overall.instructions-per-iTLB-miss 1.00 +5.5% 1.05 perf-stat.overall.ipc 3951 -3.2% 3826 perf-stat.overall.path-length 3.771e+09 +10.4% 4.164e+09 perf-stat.ps.branch-instructions 496503 ± 17% +37.7% 683817 ± 9% perf-stat.ps.cache-misses 24145521 ± 3% -51.8% 11629801 perf-stat.ps.cache-references 102871 ± 3% -83.4% 17045 perf-stat.ps.context-switches 1.223e+10 +3.9% 1.271e+10 perf-stat.ps.cpu-cycles 1986 ± 2% -96.2% 75.93 ± 10% perf-stat.ps.cpu-migrations 3.2e+09 +9.8% 3.514e+09 perf-stat.ps.dTLB-loads 2065804 +4.6% 2160396 perf-stat.ps.dTLB-store-misses 7.304e+08 +2.0% 7.453e+08 perf-stat.ps.dTLB-stores 1794803 ± 6% -31.7% 1225789 perf-stat.ps.iTLB-load-misses 351694 ± 4% -95.3% 16561 perf-stat.ps.iTLB-loads 1.219e+10 +9.7% 1.337e+10 perf-stat.ps.instructions 686822 +13.2% 777664 perf-stat.ps.minor-faults 686822 +13.2% 777664 perf-stat.ps.page-faults 1.861e+12 -3.2% 1.801e+12 perf-stat.total.instructions 56.68 ± 69% -56.7 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 56.68 ± 69% -56.7 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 56.32 ± 70% -56.3 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 56.32 ± 70% -56.3 0.00 perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 56.32 ± 70% -56.3 0.00 perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 56.11 ± 71% -56.1 0.00 perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 56.07 ± 71% -56.1 0.00 perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 36.23 ±101% -36.2 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64 35.47 ±105% -35.5 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 35.47 ±105% -35.5 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 35.47 ±105% -35.5 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 34.78 ±106% -34.8 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 34.75 ±106% -34.7 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 29.00 ±110% -29.0 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 19.43 ± 97% -19.4 0.00 perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit 18.70 ± 59% -18.7 0.00 perf-profile.calltrace.cycles-pp.remove_vma.exit_mmap.mmput.do_exit.do_group_exit 14.99 ± 96% -15.0 0.00 perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit 14.52 ±108% -14.5 0.00 perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit 12.42 ±108% -12.4 0.00 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit 10.42 ± 59% -10.4 0.00 perf-profile.calltrace.cycles-pp.kmem_cache_free.remove_vma.exit_mmap.mmput.do_exit 7.54 ± 95% -7.5 0.00 perf-profile.calltrace.cycles-pp.__rb_erase_color.unlink_file_vma.free_pgtables.exit_mmap.mmput 6.58 ± 61% -6.6 0.00 perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.remove_vma.exit_mmap.mmput 6.29 ±111% -6.3 0.00 perf-profile.calltrace.cycles-pp.con_scroll.lf.vt_console_print.console_unlock.vprintk_emit 6.29 ±111% -6.3 0.00 perf-profile.calltrace.cycles-pp.fbcon_scroll.con_scroll.lf.vt_console_print.console_unlock 6.29 ±111% -6.3 0.00 perf-profile.calltrace.cycles-pp.fbcon_redraw.fbcon_scroll.con_scroll.lf.vt_console_print 6.12 ±113% -6.1 0.00 perf-profile.calltrace.cycles-pp.fbcon_putcs.fbcon_redraw.fbcon_scroll.con_scroll.lf 6.12 ±113% -6.1 0.00 perf-profile.calltrace.cycles-pp.bit_putcs.fbcon_putcs.fbcon_redraw.fbcon_scroll.con_scroll 5.22 ±113% -5.2 0.00 perf-profile.calltrace.cycles-pp.drm_fb_helper_cfb_imageblit.bit_putcs.fbcon_putcs.fbcon_redraw.fbcon_scroll 5.22 ±113% -5.2 0.00 perf-profile.calltrace.cycles-pp.cfb_imageblit.drm_fb_helper_cfb_imageblit.bit_putcs.fbcon_putcs.fbcon_redraw 3.95 ±103% -3.9 0.00 perf-profile.calltrace.cycles-pp.vprintk_emit.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt 3.95 ±103% -3.9 0.00 perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.irq_work_run_list.irq_work_run 3.95 ±103% -3.9 0.00 perf-profile.calltrace.cycles-pp.vt_console_print.console_unlock.vprintk_emit.printk.irq_work_run_list 3.95 ±103% -3.9 0.00 perf-profile.calltrace.cycles-pp.lf.vt_console_print.console_unlock.vprintk_emit.printk 3.95 ±103% -3.9 0.00 perf-profile.calltrace.cycles-pp.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt 3.76 ± 95% -3.8 0.00 perf-profile.calltrace.cycles-pp.vma_interval_tree_remove.unlink_file_vma.free_pgtables.exit_mmap.mmput 3.00 ±101% -3.0 0.00 perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput 61.00 ± 61% -61.0 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 60.99 ± 61% -61.0 0.00 perf-profile.children.cycles-pp.do_syscall_64 56.33 ± 70% -56.3 0.00 perf-profile.children.cycles-pp.do_group_exit 56.33 ± 70% -56.3 0.00 perf-profile.children.cycles-pp.do_exit 56.32 ± 70% -56.3 0.00 perf-profile.children.cycles-pp.__x64_sys_exit_group 56.12 ± 71% -56.1 0.00 perf-profile.children.cycles-pp.mmput 56.12 ± 71% -56.1 0.00 perf-profile.children.cycles-pp.exit_mmap 36.23 ±101% -36.2 0.00 perf-profile.children.cycles-pp.secondary_startup_64 36.23 ±101% -36.2 0.00 perf-profile.children.cycles-pp.cpu_startup_entry 36.23 ±101% -36.2 0.00 perf-profile.children.cycles-pp.do_idle 35.48 ±102% -35.5 0.00 perf-profile.children.cycles-pp.cpuidle_enter 35.48 ±102% -35.5 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state 35.47 ±105% -35.5 0.00 perf-profile.children.cycles-pp.start_secondary 29.21 ±110% -29.2 0.00 perf-profile.children.cycles-pp.intel_idle 19.51 ± 97% -19.5 0.00 perf-profile.children.cycles-pp.free_pgtables 19.14 ± 59% -19.1 0.00 perf-profile.children.cycles-pp.remove_vma 15.08 ± 96% -15.1 0.00 perf-profile.children.cycles-pp.unlink_file_vma 14.52 ±108% -14.5 0.00 perf-profile.children.cycles-pp.unmap_vmas 12.67 ±107% -12.7 0.00 perf-profile.children.cycles-pp.unmap_page_range 10.58 ± 58% -10.6 0.00 perf-profile.children.cycles-pp.kmem_cache_free 7.55 ± 95% -7.6 0.00 perf-profile.children.cycles-pp.__rb_erase_color 7.39 ±102% -7.4 0.00 perf-profile.children.cycles-pp.console_unlock 7.36 ±103% -7.4 0.00 perf-profile.children.cycles-pp.vprintk_emit 7.17 ±101% -7.2 0.00 perf-profile.children.cycles-pp.vt_console_print 6.99 ±103% -7.0 0.00 perf-profile.children.cycles-pp.fbcon_putcs 6.99 ±103% -7.0 0.00 perf-profile.children.cycles-pp.bit_putcs 6.75 ± 97% -6.7 0.00 perf-profile.children.cycles-pp.lf 6.75 ± 97% -6.7 0.00 perf-profile.children.cycles-pp.con_scroll 6.75 ± 97% -6.7 0.00 perf-profile.children.cycles-pp.fbcon_scroll 6.74 ± 98% -6.7 0.00 perf-profile.children.cycles-pp.fbcon_redraw 6.63 ± 60% -6.6 0.00 perf-profile.children.cycles-pp.__slab_free 5.98 ±105% -6.0 0.00 perf-profile.children.cycles-pp.drm_fb_helper_cfb_imageblit 5.98 ±105% -6.0 0.00 perf-profile.children.cycles-pp.cfb_imageblit 4.29 ± 87% -4.3 0.00 perf-profile.children.cycles-pp.irq_work_interrupt 4.29 ± 87% -4.3 0.00 perf-profile.children.cycles-pp.smp_irq_work_interrupt 4.29 ± 87% -4.3 0.00 perf-profile.children.cycles-pp.irq_work_run_list 4.29 ± 87% -4.3 0.00 perf-profile.children.cycles-pp.irq_work_run 4.29 ± 87% -4.3 0.00 perf-profile.children.cycles-pp.printk 4.05 ± 55% -4.0 0.00 perf-profile.children.cycles-pp.___might_sleep 3.82 ± 95% -3.8 0.00 perf-profile.children.cycles-pp.vma_interval_tree_remove 3.19 ± 98% -3.2 0.00 perf-profile.children.cycles-pp.down_write 29.21 ±110% -29.2 0.00 perf-profile.self.cycles-pp.intel_idle 7.47 ±108% -7.5 0.00 perf-profile.self.cycles-pp.unmap_page_range 6.47 ± 59% -6.5 0.00 perf-profile.self.cycles-pp.__slab_free 5.98 ±105% -6.0 0.00 perf-profile.self.cycles-pp.cfb_imageblit 5.36 ± 97% -5.4 0.00 perf-profile.self.cycles-pp.__rb_erase_color 3.97 ± 55% -4.0 0.00 perf-profile.self.cycles-pp.___might_sleep 3.73 ± 95% -3.7 0.00 perf-profile.self.cycles-pp.vma_interval_tree_remove Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

3 years

1
0
0 / 0

9a23f5dbd7 ("Merge 'arm-soc/tegra/dt64' into .."): BUG: kernel reboot-without-warning in boot stage

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is git://internal_merge_and_test_tree devel-catchup-201906252112 commit 9a23f5dbd7bf84223f6afe70bd50c854c54ad3bc Merge: 8569ea7437777 a586c88eab656 Author: 0day robot <lkp(a)intel.com> AuthorDate: Tue Jun 25 21:12:57 2019 +0800 Commit: 0day robot <lkp(a)intel.com> CommitDate: Tue Jun 25 21:12:57 2019 +0800 Merge 'arm-soc/tegra/dt64' into devel-catchup-201906252112 8569ea7437 Merge 'arm-soc/sunxi/dt' into devel-catchup-201906252112 a586c88eab arm64: tegra: Enable PCIe slots in P2972-0000 board 9a23f5dbd7 Merge 'arm-soc/tegra/dt64' into devel-catchup-201906252112 b65ee3b5ec 0day head guard for 'devel-catchup-201906252112' +-------------------------------------------------------+------------+------------+------------+------------+ | | 8569ea7437 | a586c88eab | 9a23f5dbd7 | b65ee3b5ec | +-------------------------------------------------------+------------+------------+------------+------------+ | boot_successes | 30 | 30 | 0 | 0 | | boot_failures | 9 | 3 | 17 | 23 | | BUG:kernel_timeout_in_boot_stage | 5 | | | | | BUG:kernel_timeout_in_boot-around-mounting-root_stage | 2 | | | | | BUG:kernel_hang_in_boot-around-mounting-root_stage | 2 | | | | | invoked_oom-killer:gfp_mask=0x | 0 | 2 | | | | Mem-Info | 0 | 3 | | | | RIP:copy_user_generic_unrolled | 0 | 1 | | | | BUG:kernel_reboot-without-warning_in_boot_stage | 0 | 0 | 17 | 23 | +-------------------------------------------------------+------------+------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 0.024775] check: Scanning 1 areas for low memory corruption [ 0.025752] BRK [0x05201000, 0x05201fff] PGTABLE [ 0.026538] BRK [0x05202000, 0x05202fff] PGTABLE [ 0.027311] BRK [0x05203000, 0x05203fff] PGTABLE [ 0.030561] BRK [0x05204000, 0x05204fff] PGTABLE BUG: kernel reboot-without-warning in boot stage # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start b65ee3b5ecf5f1387642db259b7f39c279678315 9e0babf2c06c73cda2c0cd37a1653d823adb40ec -- git bisect good a81bdf82c8af97246edca3c5c4f4d023c914a6f5 # 12:48 G 10 0 1 1 Merge 'arm-soc/qcom/dt64' into devel-catchup-201906252112 git bisect bad 9a23f5dbd7bf84223f6afe70bd50c854c54ad3bc # 12:57 B 0 7 22 0 Merge 'arm-soc/tegra/dt64' into devel-catchup-201906252112 git bisect good 8569ea7437777addc37cd5e64f8f8f4d5dda7de6 # 13:47 G 10 0 1 1 Merge 'arm-soc/sunxi/dt' into devel-catchup-201906252112 git bisect good a4ff413b714d5b627bb81f5aca6d30e4714260e4 # 14:30 G 10 0 1 1 arm64: tegra: Add ID EEPROM for Jetson TX2 module git bisect good ece6031ece2dd64d63708cfe1088016cee5b10c0 # 14:55 G 11 0 1 1 arm64: tegra: Update Jetson TX1 GPU regulator timings git bisect good 9a182db4576d0f5d729a6900f16e4430f8fb1890 # 15:49 G 10 0 0 0 arm64: tegra: Enable ACONNECT, ADMA and AGIC git bisect good 2602c32f15e71729f3d638da5ad1692099cacf97 # 16:35 G 10 0 1 1 arm64: tegra: Add P2U and PCIe controller nodes to Tegra194 DT git bisect good a586c88eab65619f3654194dc90d46c98e712af2 # 17:04 G 10 0 3 3 arm64: tegra: Enable PCIe slots in P2972-0000 board # first bad commit: [9a23f5dbd7bf84223f6afe70bd50c854c54ad3bc] Merge 'arm-soc/tegra/dt64' into devel-catchup-201906252112 git bisect good 8569ea7437777addc37cd5e64f8f8f4d5dda7de6 # 17:17 G 30 0 7 9 Merge 'arm-soc/sunxi/dt' into devel-catchup-201906252112 git bisect good a586c88eab65619f3654194dc90d46c98e712af2 # 17:36 G 30 0 2 5 arm64: tegra: Enable PCIe slots in P2972-0000 board # extra tests on HEAD of linux-devel/devel-catchup-201906252112 git bisect bad b65ee3b5ecf5f1387642db259b7f39c279678315 # 17:37 B 0 23 51 0 0day head guard for 'devel-catchup-201906252112' # extra tests on tree/branch linux-devel/devel-catchup-201906252112 git bisect bad b65ee3b5ecf5f1387642db259b7f39c279678315 # 17:37 B 0 23 51 0 0day head guard for 'devel-catchup-201906252112' --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation

3 years

1
0
0 / 0

[bpf/tools] cd17d77705: kernel_selftests.bpf.test_sock_addr.sh.fail

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: cd17d77705780e2270937fb3cbd2b985adab3edc ("bpf/tools: sync bpf.h") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: kernel_selftests with following parameters: group: kselftests-00 test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> # selftests: bpf: test_sock_addr.sh # Wait for testing IPv4/IPv6 to become available ... OK # libbpf: load bpf program failed: Permission denied # libbpf: -- BEGIN DUMP LOG --- # libbpf: # ; int connect_v4_prog(struct bpf_sock_addr *ctx) # 0: (bf) r6 = r1 # 1: (b7) r1 = 1544617984 # ; memset(&tuple.ipv4.sport, 0, sizeof(tuple.ipv4.sport)); # 2: (7b) *(u64 *)(r10 -32) = r1 # 3: (18) r1 = 0x100007f00000000 # ; memset(&tuple.ipv4.saddr, 0, sizeof(tuple.ipv4.saddr)); # 5: (7b) *(u64 *)(r10 -40) = r1 # 6: (b7) r7 = 0 # ; struct bpf_sock_tuple tuple = {}; # 7: (63) *(u32 *)(r10 -8) = r7 # 8: (63) *(u32 *)(r10 -12) = r7 # 9: (63) *(u32 *)(r10 -16) = r7 # 10: (63) *(u32 *)(r10 -20) = r7 # 11: (63) *(u32 *)(r10 -24) = r7 # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 12: (61) r1 = *(u32 *)(r6 +32) # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 13: (bf) r2 = r1 # 14: (07) r2 += -1 # 15: (67) r2 <<= 32 # 16: (77) r2 >>= 32 # 17: (25) if r2 > 0x1 goto pc+24 # R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????0000 fp-16=00000000 fp-24=00000000 fp-32=mmmmmmmm fp-40=mmmmmmmm # ; else if (ctx->type == SOCK_STREAM) # 18: (55) if r1 != 0x1 goto pc+8 # R1=inv1 R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????0000 fp-16=00000000 fp-24=00000000 fp-32=mmmmmmmm fp-40=mmmmmmmm # 19: (bf) r2 = r10 # ; sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof(tuple.ipv4), # 20: (07) r2 += -40 # 21: (bf) r1 = r6 # 22: (b7) r3 = 12 # 23: (b7) r4 = -1 # 24: (b7) r5 = 0 # 25: (85) call bpf_sk_lookup_tcp#84 # 26: (05) goto pc+7 # ; if (!sk) # 34: (15) if r0 == 0x0 goto pc+7 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????0000 fp-16=00000000 fp-24=00000000 fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; if (sk->src_ip4 != tuple.ipv4.daddr || # 35: (61) r1 = *(u32 *)(r0 +24) # ; if (sk->src_ip4 != tuple.ipv4.daddr || # 36: (61) r2 = *(u32 *)(r10 -36) # ; if (sk->src_ip4 != tuple.ipv4.daddr || # 37: (5d) if r1 != r2 goto pc+2 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????0000 fp-16=00000000 fp-24=00000000 fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_port != DST_REWRITE_PORT4) { # 38: (61) r1 = *(u32 *)(r0 +44) # ; if (sk->src_ip4 != tuple.ipv4.daddr || # 39: (15) if r1 == 0x115c goto pc+4 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????0000 fp-16=00000000 fp-24=00000000 fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 40: (bf) r1 = r0 # 41: (85) call bpf_sk_release#86 # ; } # 42: (bf) r0 = r7 # 43: (95) exit # # from 39 to 44: R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv4444 R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????0000 fp-16=00000000 fp-24=00000000 fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 44: (bf) r1 = r0 # 45: (85) call bpf_sk_release#86 # 46: (b7) r1 = 23569 # ; ctx->user_port = bpf_htons(DST_REWRITE_PORT4); # 47: (63) *(u32 *)(r6 +24) = r1 # 48: (b7) r1 = 16777343 # ; ctx->user_ip4 = bpf_htonl(DST_REWRITE_IP4); # 49: (63) *(u32 *)(r6 +4) = r1 # invalid bpf_context access off=4 size=4 # processed 42 insns (limit 1000000) max_states_per_insn 0 total_states 10 peak_states 10 mark_read 8 # # libbpf: -- END LOG -- # libbpf: failed to load program 'cgroup/connect4' # libbpf: failed to load object './connect4_prog.o' # libbpf: load bpf program failed: Permission denied # libbpf: -- BEGIN DUMP LOG --- # libbpf: # ; int connect_v6_prog(struct bpf_sock_addr *ctx) # 0: (bf) r6 = r1 # 1: (18) r1 = 0x100000000000000 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 3: (7b) *(u64 *)(r10 -16) = r1 # 4: (b7) r1 = 169476096 # ; memset(&tuple.ipv6.sport, 0, sizeof(tuple.ipv6.sport)); # 5: (63) *(u32 *)(r10 -8) = r1 # 6: (b7) r7 = 0 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 7: (7b) *(u64 *)(r10 -24) = r7 # 8: (7b) *(u64 *)(r10 -32) = r7 # 9: (7b) *(u64 *)(r10 -40) = r7 # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 10: (61) r1 = *(u32 *)(r6 +32) # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 11: (bf) r2 = r1 # 12: (07) r2 += -1 # 13: (67) r2 <<= 32 # 14: (77) r2 >>= 32 # 15: (25) if r2 > 0x1 goto pc+33 # R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # ; else if (ctx->type == SOCK_STREAM) # 16: (55) if r1 != 0x1 goto pc+8 # R1=inv1 R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # 17: (bf) r2 = r10 # ; sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof(tuple.ipv6), # 18: (07) r2 += -40 # 19: (bf) r1 = r6 # 20: (b7) r3 = 36 # 21: (b7) r4 = -1 # 22: (b7) r5 = 0 # 23: (85) call bpf_sk_lookup_tcp#84 # 24: (05) goto pc+7 # ; if (!sk) # 32: (15) if r0 == 0x0 goto pc+16 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 33: (61) r1 = *(u32 *)(r0 +28) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 34: (61) r2 = *(u32 *)(r10 -24) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 35: (5d) if r1 != r2 goto pc+11 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 36: (61) r1 = *(u32 *)(r0 +32) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 37: (61) r2 = *(u32 *)(r10 -20) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 38: (5d) if r1 != r2 goto pc+8 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 39: (61) r1 = *(u32 *)(r0 +36) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 40: (61) r2 = *(u32 *)(r10 -16) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 41: (5d) if r1 != r2 goto pc+5 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 42: (61) r1 = *(u32 *)(r0 +40) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 43: (61) r2 = *(u32 *)(r10 -12) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 44: (5d) if r1 != r2 goto pc+2 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_port != DST_REWRITE_PORT6) { # 45: (61) r1 = *(u32 *)(r0 +44) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 46: (15) if r1 == 0x1a0a goto pc+4 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 47: (bf) r1 = r0 # 48: (85) call bpf_sk_release#86 # ; } # 49: (bf) r0 = r7 # 50: (95) exit # # from 46 to 51: R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv6666 R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 51: (bf) r1 = r0 # 52: (85) call bpf_sk_release#86 # 53: (b7) r1 = 2586 # ; ctx->user_port = bpf_htons(DST_REWRITE_PORT6); # 54: (63) *(u32 *)(r6 +24) = r1 # 55: (18) r1 = 0x100000000000000 # ; ctx->user_ip6[2] = bpf_htonl(DST_REWRITE_IP6_2); # 57: (7b) *(u64 *)(r6 +16) = r1 # invalid bpf_context access off=16 size=8 # processed 49 insns (limit 1000000) max_states_per_insn 0 total_states 13 peak_states 13 mark_read 11 # # libbpf: -- END LOG -- # libbpf: failed to load program 'cgroup/connect6' # libbpf: failed to load object './connect6_prog.o' # libbpf: load bpf program failed: Permission denied # libbpf: -- BEGIN DUMP LOG --- # libbpf: # ; int connect_v6_prog(struct bpf_sock_addr *ctx) # 0: (bf) r6 = r1 # 1: (18) r1 = 0x100000000000000 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 3: (7b) *(u64 *)(r10 -16) = r1 # 4: (b7) r1 = 169476096 # ; memset(&tuple.ipv6.sport, 0, sizeof(tuple.ipv6.sport)); # 5: (63) *(u32 *)(r10 -8) = r1 # 6: (b7) r7 = 0 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 7: (7b) *(u64 *)(r10 -24) = r7 # 8: (7b) *(u64 *)(r10 -32) = r7 # 9: (7b) *(u64 *)(r10 -40) = r7 # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 10: (61) r1 = *(u32 *)(r6 +32) # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 11: (bf) r2 = r1 # 12: (07) r2 += -1 # 13: (67) r2 <<= 32 # 14: (77) r2 >>= 32 # 15: (25) if r2 > 0x1 goto pc+33 # R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # ; else if (ctx->type == SOCK_STREAM) # 16: (55) if r1 != 0x1 goto pc+8 # R1=inv1 R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # 17: (bf) r2 = r10 # ; sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof(tuple.ipv6), # 18: (07) r2 += -40 # 19: (bf) r1 = r6 # 20: (b7) r3 = 36 # 21: (b7) r4 = -1 # 22: (b7) r5 = 0 # 23: (85) call bpf_sk_lookup_tcp#84 # 24: (05) goto pc+7 # ; if (!sk) # 32: (15) if r0 == 0x0 goto pc+16 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 33: (61) r1 = *(u32 *)(r0 +28) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 34: (61) r2 = *(u32 *)(r10 -24) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 35: (5d) if r1 != r2 goto pc+11 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 36: (61) r1 = *(u32 *)(r0 +32) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 37: (61) r2 = *(u32 *)(r10 -20) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 38: (5d) if r1 != r2 goto pc+8 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 39: (61) r1 = *(u32 *)(r0 +36) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 40: (61) r2 = *(u32 *)(r10 -16) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 41: (5d) if r1 != r2 goto pc+5 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 42: (61) r1 = *(u32 *)(r0 +40) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 43: (61) r2 = *(u32 *)(r10 -12) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 44: (5d) if r1 != r2 goto pc+2 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_port != DST_REWRITE_PORT6) { # 45: (61) r1 = *(u32 *)(r0 +44) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 46: (15) if r1 == 0x1a0a goto pc+4 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 47: (bf) r1 = r0 # 48: (85) call bpf_sk_release#86 # ; } # 49: (bf) r0 = r7 # 50: (95) exit # # from 46 to 51: R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv6666 R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 51: (bf) r1 = r0 # 52: (85) call bpf_sk_release#86 # 53: (b7) r1 = 2586 # ; ctx->user_port = bpf_htons(DST_REWRITE_PORT6); # 54: (63) *(u32 *)(r6 +24) = r1 # 55: (18) r1 = 0x100000000000000 # ; ctx->user_ip6[2] = bpf_htonl(DST_REWRITE_IP6_2); # 57: (7b) *(u64 *)(r6 +16) = r1 # invalid bpf_context access off=16 size=8 # processed 49 insns (limit 1000000) max_states_per_insn 0 total_states 13 peak_states 13 mark_read 11 # # libbpf: -- END LOG -- # libbpf: failed to load program 'cgroup/connect6' # libbpf: failed to load object './connect6_prog.o' # (test_sock_addr.c:752: errno: Bad file descriptor) >>> Loading program (./connect6_prog.o) error. # # libbpf: load bpf program failed: Permission denied # libbpf: -- BEGIN DUMP LOG --- # libbpf: # ; int connect_v6_prog(struct bpf_sock_addr *ctx) # 0: (bf) r6 = r1 # 1: (18) r1 = 0x100000000000000 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 3: (7b) *(u64 *)(r10 -16) = r1 # 4: (b7) r1 = 169476096 # ; memset(&tuple.ipv6.sport, 0, sizeof(tuple.ipv6.sport)); # 5: (63) *(u32 *)(r10 -8) = r1 # 6: (b7) r7 = 0 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 7: (7b) *(u64 *)(r10 -24) = r7 # 8: (7b) *(u64 *)(r10 -32) = r7 # 9: (7b) *(u64 *)(r10 -40) = r7 # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 10: (61) r1 = *(u32 *)(r6 +32) # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 11: (bf) r2 = r1 # 12: (07) r2 += -1 # 13: (67) r2 <<= 32 # 14: (77) r2 >>= 32 # 15: (25) if r2 > 0x1 goto pc+33 # R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # ; else if (ctx->type == SOCK_STREAM) # 16: (55) if r1 != 0x1 goto pc+8 # R1=inv1 R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # 17: (bf) r2 = r10 # ; sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof(tuple.ipv6), # 18: (07) r2 += -40 # 19: (bf) r1 = r6 # 20: (b7) r3 = 36 # 21: (b7) r4 = -1 # 22: (b7) r5 = 0 # 23: (85) call bpf_sk_lookup_tcp#84 # 24: (05) goto pc+7 # ; if (!sk) # 32: (15) if r0 == 0x0 goto pc+16 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 33: (61) r1 = *(u32 *)(r0 +28) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 34: (61) r2 = *(u32 *)(r10 -24) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 35: (5d) if r1 != r2 goto pc+11 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 36: (61) r1 = *(u32 *)(r0 +32) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 37: (61) r2 = *(u32 *)(r10 -20) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 38: (5d) if r1 != r2 goto pc+8 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 39: (61) r1 = *(u32 *)(r0 +36) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 40: (61) r2 = *(u32 *)(r10 -16) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 41: (5d) if r1 != r2 goto pc+5 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 42: (61) r1 = *(u32 *)(r0 +40) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 43: (61) r2 = *(u32 *)(r10 -12) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 44: (5d) if r1 != r2 goto pc+2 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_port != DST_REWRITE_PORT6) { # 45: (61) r1 = *(u32 *)(r0 +44) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 46: (15) if r1 == 0x1a0a goto pc+4 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 47: (bf) r1 = r0 # 48: (85) call bpf_sk_release#86 # ; } # 49: (bf) r0 = r7 # 50: (95) exit # # from 46 to 51: R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv6666 R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 51: (bf) r1 = r0 # 52: (85) call bpf_sk_release#86 # 53: (b7) r1 = 2586 # ; ctx->user_port = bpf_htons(DST_REWRITE_PORT6); # 54: (63) *(u32 *)(r6 +24) = r1 # 55: (18) r1 = 0x100000000000000 # ; ctx->user_ip6[2] = bpf_htonl(DST_REWRITE_IP6_2); # 57: (7b) *(u64 *)(r6 +16) = r1 # invalid bpf_context access off=16 size=8 # processed 49 insns (limit 1000000) max_states_per_insn 0 total_states 13 peak_states 13 mark_read 11 # # libbpf: -- END LOG -- # libbpf: failed to load program 'cgroup/connect6' # libbpf: failed to load object './connect6_prog.o' # (test_sock_addr.c:752: errno: Bad file descriptor) >>> Loading program (./connect6_prog.o) error. # # libbpf: load bpf program failed: Permission denied # libbpf: -- BEGIN DUMP LOG --- # libbpf: # ; int connect_v6_prog(struct bpf_sock_addr *ctx) # 0: (bf) r6 = r1 # 1: (18) r1 = 0x100000000000000 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 3: (7b) *(u64 *)(r10 -16) = r1 # 4: (b7) r1 = 169476096 # ; memset(&tuple.ipv6.sport, 0, sizeof(tuple.ipv6.sport)); # 5: (63) *(u32 *)(r10 -8) = r1 # 6: (b7) r7 = 0 # ; tuple.ipv6.daddr[0] = bpf_htonl(DST_REWRITE_IP6_0); # 7: (7b) *(u64 *)(r10 -24) = r7 # 8: (7b) *(u64 *)(r10 -32) = r7 # 9: (7b) *(u64 *)(r10 -40) = r7 # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 10: (61) r1 = *(u32 *)(r6 +32) # ; if (ctx->type != SOCK_STREAM && ctx->type != SOCK_DGRAM) # 11: (bf) r2 = r1 # 12: (07) r2 += -1 # 13: (67) r2 <<= 32 # 14: (77) r2 >>= 32 # 15: (25) if r2 > 0x1 goto pc+33 # R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # ; else if (ctx->type == SOCK_STREAM) # 16: (55) if r1 != 0x1 goto pc+8 # R1=inv1 R2=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=00000000 fp-32=00000000 fp-40=00000000 # 17: (bf) r2 = r10 # ; sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof(tuple.ipv6), # 18: (07) r2 += -40 # 19: (bf) r1 = r6 # 20: (b7) r3 = 36 # 21: (b7) r4 = -1 # 22: (b7) r5 = 0 # 23: (85) call bpf_sk_lookup_tcp#84 # 24: (05) goto pc+7 # ; if (!sk) # 32: (15) if r0 == 0x0 goto pc+16 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 33: (61) r1 = *(u32 *)(r0 +28) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 34: (61) r2 = *(u32 *)(r10 -24) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 35: (5d) if r1 != r2 goto pc+11 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 36: (61) r1 = *(u32 *)(r0 +32) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 37: (61) r2 = *(u32 *)(r10 -20) # ; sk->src_ip6[1] != tuple.ipv6.daddr[1] || # 38: (5d) if r1 != r2 goto pc+8 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 39: (61) r1 = *(u32 *)(r0 +36) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 40: (61) r2 = *(u32 *)(r10 -16) # ; sk->src_ip6[2] != tuple.ipv6.daddr[2] || # 41: (5d) if r1 != r2 goto pc+5 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 42: (61) r1 = *(u32 *)(r0 +40) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 43: (61) r2 = *(u32 *)(r10 -12) # ; sk->src_ip6[3] != tuple.ipv6.daddr[3] || # 44: (5d) if r1 != r2 goto pc+2 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; sk->src_port != DST_REWRITE_PORT6) { # 45: (61) r1 = *(u32 *)(r0 +44) # ; if (sk->src_ip6[0] != tuple.ipv6.daddr[0] || # 46: (15) if r1 == 0x1a0a goto pc+4 # R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 47: (bf) r1 = r0 # 48: (85) call bpf_sk_release#86 # ; } # 49: (bf) r0 = r7 # 50: (95) exit # # from 46 to 51: R0=sock(id=0,ref_obj_id=2,off=0,imm=0) R1=inv6666 R2=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv0 R10=fp0,call_-1 fp-8=????mmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm refs=2 # ; bpf_sk_release(sk); # 51: (bf) r1 = r0 # 52: (85) call bpf_sk_release#86 # 53: (b7) r1 = 2586 # ; ctx->user_port = bpf_htons(DST_REWRITE_PORT6); # 54: (63) *(u32 *)(r6 +24) = r1 # 55: (18) r1 = 0x100000000000000 # ; ctx->user_ip6[2] = bpf_htonl(DST_REWRITE_IP6_2); # 57: (7b) *(u64 *)(r6 +16) = r1 # invalid bpf_context access off=16 size=8 # processed 49 insns (limit 1000000) max_states_per_insn 0 total_states 13 peak_states 13 mark_read 11 # # libbpf: -- END LOG -- # libbpf: failed to load program 'cgroup/connect6' # libbpf: failed to load object './connect6_prog.o' # (test_sock_addr.c:752: errno: Bad file descriptor) >>> Loading program (./connect6_prog.o) error. # # (test_sock_addr.c:1122: errno: Operation not permitted) Fail to send message to server # libbpf: load bpf program failed: Permission denied # libbpf: -- BEGIN DUMP LOG --- # libbpf: # ; int sendmsg_v6_prog(struct bpf_sock_addr *ctx) # 0: (b7) r0 = 0 # ; if (ctx->type != SOCK_DGRAM) # 1: (61) r2 = *(u32 *)(r1 +32) # ; if (ctx->type != SOCK_DGRAM) # 2: (55) if r2 != 0x2 goto pc+20 # R0=inv0 R1=ctx(id=0,off=0,imm=0) R2=inv2 R10=fp0,call_-1 # ; if (ctx->msg_src_ip6[3] == bpf_htonl(1) || # 3: (61) r2 = *(u32 *)(r1 +56) # ; if (ctx->msg_src_ip6[3] == bpf_htonl(1) || # 4: (47) r2 |= 16777216 # 5: (15) if r2 == 0x1000000 goto pc+1 # R0=inv0 R1=ctx(id=0,off=0,imm=0) R2=inv(id=0,umin_value=16777216,umax_value=4294967295,var_off=(0x1000000; 0xfeffffff)) R10=fp0,call_-1 # 6: (05) goto pc+16 # ; } # 23: (95) exit # # from 5 to 7: R0=inv0 R1=ctx(id=0,off=0,imm=0) R2=inv16777216 R10=fp0,call_-1 # ; if (ctx->msg_src_ip6[3] == bpf_htonl(1) || # 7: (b7) r2 = 100663296 # ; ctx->msg_src_ip6[3] = bpf_htonl(SRC_REWRITE_IP6_3); # 8: (63) *(u32 *)(r1 +56) = r2 # 9: (b7) r0 = 0 # ; ctx->msg_src_ip6[2] = bpf_htonl(SRC_REWRITE_IP6_2); # 10: (63) *(u32 *)(r1 +52) = r0 # ; ctx->msg_src_ip6[1] = bpf_htonl(SRC_REWRITE_IP6_1); # 11: (63) *(u32 *)(r1 +48) = r0 # ; ctx->msg_src_ip6[0] = bpf_htonl(SRC_REWRITE_IP6_0); # 12: (63) *(u32 *)(r1 +44) = r0 # ; if ((ctx->user_ip6[0] & 0xFFFF) == bpf_htons(0xFACE) && # 13: (61) r2 = *(u32 *)(r1 +8) # ; if ((ctx->user_ip6[0] & 0xFFFF) == bpf_htons(0xFACE) && # 14: (55) if r2 != 0xcb0cefa goto pc+8 # R0=inv0 R1=ctx(id=0,off=0,imm=0) R2=inv212913914 R10=fp0,call_-1 # 15: (b7) r2 = 2586 # ; ctx->user_port = bpf_htons(DST_REWRITE_PORT6); # 16: (63) *(u32 *)(r1 +24) = r2 # 17: (18) r2 = 0x100000000000000 # ; ctx->user_ip6[2] = bpf_htonl(DST_REWRITE_IP6_2); # 19: (7b) *(u64 *)(r1 +16) = r2 # invalid bpf_context access off=16 size=8 # processed 20 insns (limit 1000000) max_states_per_insn 0 total_states 5 peak_states 5 mark_read 3 # # libbpf: -- END LOG -- # libbpf: failed to load program 'cgroup/sendmsg6' # libbpf: failed to load object './sendmsg6_prog.o' # (test_sock_addr.c:752: errno: Bad file descriptor) >>> Loading program (./sendmsg6_prog.o) error. # # (test_sock_addr.c:1122: errno: Unknown error 524) Fail to send message to server # (test_sock_addr.c:1122: errno: Operation not permitted) Fail to send message to server # Test case: bind4: load prog with wrong expected attach type .. [PASS] # Test case: bind4: attach prog with wrong attach type .. [PASS] # Test case: bind4: rewrite IP & TCP port in .. [PASS] # Test case: bind4: rewrite IP & UDP port in .. [PASS] # Test case: bind6: load prog with wrong expected attach type .. [PASS] # Test case: bind6: attach prog with wrong attach type .. [PASS] # Test case: bind6: rewrite IP & TCP port in .. [PASS] # Test case: bind6: rewrite IP & UDP port in .. [PASS] # Test case: connect4: load prog with wrong expected attach type .. [PASS] # Test case: connect4: attach prog with wrong attach type .. [PASS] # Test case: connect4: rewrite IP & TCP port .. [PASS] # Test case: connect4: rewrite IP & UDP port .. [PASS] # Test case: connect6: load prog with wrong expected attach type .. [PASS] # Test case: connect6: attach prog with wrong attach type .. [FAIL] # Test case: connect6: rewrite IP & TCP port .. [FAIL] # Test case: connect6: rewrite IP & UDP port .. [FAIL] # Test case: sendmsg4: load prog with wrong expected attach type .. [PASS] # Test case: sendmsg4: attach prog with wrong attach type .. [PASS] # Test case: sendmsg4: rewrite IP & port (asm) .. [PASS] # Test case: sendmsg4: rewrite IP & port (C) .. [PASS] # Test case: sendmsg4: deny call .. [PASS] # Test case: sendmsg6: load prog with wrong expected attach type .. [PASS] # Test case: sendmsg6: attach prog with wrong attach type .. [PASS] # Test case: sendmsg6: rewrite IP & port (asm) .. [PASS] # Test case: sendmsg6: rewrite IP & port (C) .. [FAIL] # Test case: sendmsg6: IPv4-mapped IPv6 .. [PASS] # Test case: sendmsg6: set dst IP = [::] (BSD'ism) .. [PASS] # Test case: sendmsg6: preserve dst IP = [::] (BSD'ism) .. [PASS] # Test case: sendmsg6: deny call .. [PASS] # Summary: 25 PASSED, 4 FAILED not ok 34 selftests: bpf: test_sock_addr.sh To reproduce: # build kernel cd linux cp config-5.2.0-rc2-00597-gcd17d77 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 prepare make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 modules_prepare make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 SHELL=/bin/bash make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

3 years

1
0
0 / 0

[nl80211] d8572a564d: assertion_failed

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: d8572a564d8aa17cb726a2aed04d0bbe52403249 ("[PATCH 2/3] nl80211: Limit certain commands to interface owner") url: https://github.com/0day-ci/linux/commits/Denis-Kenzior/nl80211-Update-uap... base: https://git.kernel.org/cgit/linux/kernel/git/jberg/mac80211-next.git master in testcase: hwsim with following parameters: group: hwsim-13 on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 4G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +---------------------------------------------------------------------------------+------------+------------+ | | 97b91db67a | d8572a564d | +---------------------------------------------------------------------------------+------------+------------+ | boot_successes | 2 | 0 | | boot_failures | 18 | 20 | | BUG:kernel_reboot-without-warning_in_test_stage | 18 | 4 | | assertion_failed | 0 | 16 | | WARNING:at_net/wireless/reg.c:#reset_regdomains[cfg80211] | 0 | 16 | | RIP:reset_regdomains[cfg80211] | 0 | 16 | | WARNING:at_net/wireless/reg.c:#update_all_wiphy_regulatory[cfg80211] | 0 | 16 | | RIP:update_all_wiphy_regulatory[cfg80211] | 0 | 16 | | WARNING:at_net/wireless/mlme.c:#cfg80211_process_mlme_unregistrations[cfg80211] | 0 | 16 | | RIP:cfg80211_process_mlme_unregistrations[cfg80211] | 0 | 16 | | WARNING:at_net/core/dev.c:#call_netdevice_notifiers_info | 0 | 11 | | RIP:call_netdevice_notifiers_info | 0 | 11 | | WARNING:at_net/wireless/nl80211.c:#__cfg80211_wdev_from_attrs[cfg80211] | 0 | 16 | | RIP:__cfg80211_wdev_from_attrs[cfg80211] | 0 | 16 | | WARNING:at_net/sched/sch_api.c:#qdisc_hash_add | 0 | 10 | | RIP:qdisc_hash_add | 0 | 10 | | WARNING:at_net/core/rtnetlink.c:#rtnl_fill_ifinfo | 0 | 12 | | RIP:rtnl_fill_ifinfo | 0 | 12 | | WARNING:at_net/core/devlink.c:#devlink_compat_phys_port_name_get | 0 | 12 | | RIP:devlink_compat_phys_port_name_get | 0 | 12 | | WARNING:at_net/core/devlink.c:#devlink_compat_switch_id_get | 0 | 12 | | RIP:devlink_compat_switch_id_get | 0 | 12 | | WARNING:at_net/core/rtnetlink.c:#rtnl_xdp_prog_skb | 0 | 12 | | RIP:rtnl_xdp_prog_skb | 0 | 12 | | WARNING:at_net/core/fib_rules.c:#fib_rules_event | 0 | 11 | | RIP:fib_rules_event | 0 | 11 | | WARNING:at_net/ipv4/devinet.c:#inetdev_event | 0 | 11 | | RIP:inetdev_event | 0 | 11 | | WARNING:at_net/wireless/nl80211.c:#__cfg80211_rdev_from_attrs[cfg80211] | 0 | 16 | | RIP:__cfg80211_rdev_from_attrs[cfg80211] | 0 | 16 | | WARNING:at_net/wireless/core.c:#cfg80211_rdev_by_wiphy_idx[cfg80211] | 0 | 16 | | RIP:cfg80211_rdev_by_wiphy_idx[cfg80211] | 0 | 16 | | WARNING:at_net/wireless/util.c:#cfg80211_change_iface[cfg80211] | 0 | 16 | | RIP:cfg80211_change_iface[cfg80211] | 0 | 16 | | WARNING:at_net/wireless/util.c:#cfg80211_process_rdev_events[cfg80211] | 0 | 16 | | RIP:cfg80211_process_rdev_events[cfg80211] | 0 | 16 | | WARNING:at_net/mac80211/iface.c:#ieee80211_if_change_type[mac80211] | 0 | 16 | | RIP:ieee80211_if_change_type[mac80211] | 0 | 16 | | WARNING:at_net/wireless/nl80211.c:#nl80211_set_wiphy[cfg80211] | 0 | 16 | | RIP:nl80211_set_wiphy[cfg80211] | 0 | 16 | | WARNING:at_net/wireless/chan.c:#cfg80211_reg_can_beacon_relax[cfg80211] | 0 | 16 | | RIP:cfg80211_reg_can_beacon_relax[cfg80211] | 0 | 16 | | WARNING:at_net/wireless/reg.c:#restore_regulatory_settings[cfg80211] | 0 | 2 | | RIP:restore_regulatory_settings[cfg80211] | 0 | 2 | | WARNING:at_net/wireless/core.c:#wiphy_idx_to_wiphy[cfg80211] | 0 | 8 | | RIP:wiphy_idx_to_wiphy[cfg80211] | 0 | 8 | | WARNING:at_net/wireless/core.c:#__cfg80211_leave[cfg80211] | 0 | 3 | | RIP:__cfg80211_leave[cfg80211] | 0 | 3 | | BUG:unable_to_handle_page_fault_for_address | 0 | 1 | | Oops:#[##] | 0 | 1 | | RIP:netpoll_poll_disable | 0 | 1 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 2 | | WARNING:at_net/ipv4/igmp.c:#ip_mc_up | 0 | 6 | | RIP:ip_mc_up | 0 | 6 | | WARNING:at_net/ipv4/igmp.c:#____ip_mc_inc_group | 0 | 6 | | RIP:____ip_mc_inc_group | 0 | 6 | | WARNING:at_net/wireless/core.c:#cfg80211_update_iface_num[cfg80211] | 0 | 7 | | RIP:cfg80211_update_iface_num[cfg80211] | 0 | 7 | | WARNING:at_net/wireless/wext-sme.c:#cfg80211_mgd_wext_connect[cfg80211] | 0 | 6 | | RIP:cfg80211_mgd_wext_connect[cfg80211] | 0 | 6 | | WARNING:at_net/ipv6/addrconf.c:#addrconf_dev_config | 0 | 4 | | RIP:addrconf_dev_config | 0 | 4 | | WARNING:at_net/ipv6/addrconf.c:#addrconf_add_dev | 0 | 4 | | RIP:addrconf_add_dev | 0 | 4 | | WARNING:at_net/ipv6/addrconf.c:#ipv6_find_idev | 0 | 4 | | RIP:ipv6_find_idev | 0 | 4 | | WARNING:at_net/mac80211/iface.c:#ieee80211_check_concurrent_iface[mac80211] | 0 | 4 | | RIP:ieee80211_check_concurrent_iface[mac80211] | 0 | 4 | | WARNING:at_net/wireless/reg.c:#set_regdom[cfg80211] | 0 | 7 | | RIP:set_regdom[cfg80211] | 0 | 7 | | general_protection_fault:#[##] | 0 | 1 | | RIP:dev_deactivate_many | 0 | 1 | | WARNING:at_net/wireless/chan.c:#cfg80211_any_wiphy_oper_chan[cfg80211] | 0 | 1 | | RIP:cfg80211_any_wiphy_oper_chan[cfg80211] | 0 | 1 | | WARNING:at_net/wireless/core.c:#cfg80211_destroy_ifaces[cfg80211] | 0 | 2 | | RIP:cfg80211_destroy_ifaces[cfg80211] | 0 | 2 | | WARNING:at_net/mac80211/iface.c:#ieee80211_if_add[mac80211] | 0 | 2 | | RIP:ieee80211_if_add[mac80211] | 0 | 2 | | WARNING:at_net/core/dev.c:#register_netdevice | 0 | 2 | | RIP:register_netdevice | 0 | 2 | | WARNING:at_net/core/dev.c:#__netdev_update_features | 0 | 2 | | RIP:__netdev_update_features | 0 | 2 | | WARNING:at_net/core/dev.c:#list_netdevice | 0 | 2 | | RIP:list_netdevice | 0 | 2 | | WARNING:at_net/ipv4/devinet.c:#inetdev_init | 0 | 2 | | RIP:inetdev_init | 0 | 2 | | WARNING:at_net/ipv4/igmp.c:#ip_mc_init_dev | 0 | 2 | | RIP:ip_mc_init_dev | 0 | 2 | | WARNING:at_net/ipv6/addrconf.c:#ipv6_add_dev | 0 | 2 | | RIP:ipv6_add_dev | 0 | 2 | | WARNING:at_net/ipv6/mcast.c:#__ipv6_dev_mc_inc | 0 | 2 | | RIP:__ipv6_dev_mc_inc | 0 | 2 | | WARNING:at_net/mac80211/iface.c:#ieee80211_if_remove[mac80211] | 0 | 2 | | RIP:ieee80211_if_remove[mac80211] | 0 | 2 | | WARNING:at_net/core/dev.c:#unregister_netdevice_queue | 0 | 2 | | RIP:unregister_netdevice_queue | 0 | 2 | | WARNING:at_net/ipv4/igmp.c:#ip_mc_down | 0 | 2 | | RIP:ip_mc_down | 0 | 2 | | WARNING:at_net/ipv4/igmp.c:#__ip_mc_dec_group | 0 | 2 | | RIP:__ip_mc_dec_group | 0 | 2 | | WARNING:at_net/ipv6/addrconf.c:#addrconf_ifdown | 0 | 2 | | RIP:addrconf_ifdown | 0 | 2 | | WARNING:at_net/ipv6/addrconf.c:#__ipv6_ifa_notify | 0 | 3 | | RIP:__ipv6_ifa_notify | 0 | 3 | | WARNING:at_net/ipv6/mcast.c:#__ipv6_dev_mc_dec | 0 | 2 | | RIP:__ipv6_dev_mc_dec | 0 | 2 | | WARNING:at_net/core/dev.c:#unlist_netdevice | 0 | 1 | | RIP:unlist_netdevice | 0 | 1 | | WARNING:at_net/sched/sch_api.c:#qdisc_hash_del | 0 | 1 | | RIP:qdisc_hash_del | 0 | 1 | | WARNING:at_net/ipv4/igmp.c:#ip_mc_destroy_dev | 0 | 1 | | RIP:ip_mc_destroy_dev | 0 | 1 | | WARNING:at_net/wireless/core.c:#__cfg80211_unregister_wdev[cfg80211] | 0 | 1 | | RIP:__cfg80211_unregister_wdev[cfg80211] | 0 | 1 | | WARNING:at_net/core/dev.c:#netdev_has_any_upper_dev | 0 | 1 | | RIP:netdev_has_any_upper_dev | 0 | 1 | | WARNING:at_net/core/dev.c:#rollback_registered_many | 0 | 2 | | RIP:rollback_registered_many | 0 | 2 | | WARNING:at_net/core/dev.c:#__dev_close_many | 0 | 2 | | RIP:__dev_close_many | 0 | 2 | | BUG:soft_lockup-CPU##stuck_for#s | 0 | 1 | | RIP:find_next_bit | 0 | 1 | | Kernel_panic-not_syncing:softlockup:hung_tasks | 0 | 1 | +---------------------------------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 186.333346] cfg80211: Loading compiled-in X.509 certificates for regulatory database [ 186.336318] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' [ 186.339612] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2 [ 186.343633] cfg80211: failed to load regulatory.db [ 186.359277] ------------[ cut here ]------------ [ 186.361527] RTNL: assertion failed at net/wireless/reg.c (306) [ 186.364166] WARNING: CPU: 0 PID: 4804 at net/wireless/reg.c:306 reset_regdomains+0xd5/0xe0 [cfg80211] [ 186.367388] Modules linked in: cfg80211 rfkill bochs_drm ttm sr_mod cdrom crct10dif_pclmul drm_kms_helper crc32_pclmul sg crc32c_intel ata_generic pata_acpi ghash_clmulni_intel ppdev syscopyarea sysfillrect sysimgblt fb_sys_fops snd_pcm drm ata_piix aesni_intel snd_timer snd crypto_simd libata cryptd glue_helper soundcore joydev pcspkr serio_raw parport_pc i2c_piix4 floppy parport ip_tables [ 186.378827] CPU: 0 PID: 4804 Comm: crda Not tainted 5.2.0-rc3-01111-gd8572a5 #1 [ 186.381529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 186.384383] RIP: 0010:reset_regdomains+0xd5/0xe0 [cfg80211] [ 186.386829] Code: 33 07 09 00 00 0f 85 63 ff ff ff ba 32 01 00 00 48 c7 c6 40 9d 55 c0 48 c7 c7 68 9d 55 c0 c6 05 13 07 09 00 01 e8 eb fa 79 ea <0f> 0b e9 3d ff ff ff 0f 1f 40 00 66 66 66 66 90 48 8b 05 d4 26 08 [ 186.392729] RSP: 0018:ffffb103c09bba38 EFLAGS: 00010282 [ 186.395104] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 186.397792] RDX: ffff95233fc26d80 RSI: ffff95233fc17778 RDI: ffff95233fc17778 [ 186.400389] RBP: ffff95227e894400 R08: 0000000000000459 R09: 0000000000aaaaaa [ 186.403046] R10: 0000000000000000 R11: ffff95227dd39e30 R12: ffff95227e894414 [ 186.405696] R13: ffffffffc0556140 R14: 0000000000000000 R15: ffff95227e7ef9c8 [ 186.408337] FS: 00007f157ef5cb40(0000) GS:ffff95233fc00000(0000) knlGS:0000000000000000 [ 186.411470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 186.413856] CR2: 00005623bc744028 CR3: 000000008861e000 CR4: 00000000000406f0 [ 186.416457] Call Trace: [ 186.418174] set_regdom+0x2b0/0x430 [cfg80211] [ 186.420255] nl80211_set_reg+0x290/0x2c0 [cfg80211] [ 186.422395] genl_family_rcv_msg+0x203/0x400 [ 186.424986] ? __alloc_pages_nodemask+0x157/0x340 [ 186.427159] genl_rcv_msg+0x47/0x90 [ 186.429001] ? __kmalloc_node_track_caller+0x59/0x2a0 [ 186.431140] ? genl_family_rcv_msg+0x400/0x400 [ 186.433207] netlink_rcv_skb+0x4a/0x110 [ 186.435092] genl_rcv+0x24/0x40 [ 186.436918] netlink_unicast+0x193/0x230 [ 186.438784] netlink_sendmsg+0x2c1/0x3c0 [ 186.440809] sock_sendmsg+0x4c/0x50 [ 186.442756] ___sys_sendmsg+0x289/0x300 [ 186.444825] ? mem_cgroup_throttle_swaprate+0x20/0x160 [ 186.447196] ? do_anonymous_page+0x2f3/0x3f0 [ 186.449197] ? __seccomp_filter+0x96/0x6c0 [ 186.451348] ? __sys_sendmsg+0x5e/0xa0 [ 186.453442] __sys_sendmsg+0x5e/0xa0 [ 186.455394] do_syscall_64+0x5b/0x1e0 [ 186.457454] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 186.459677] RIP: 0033:0x7f157e463dc7 [ 186.461625] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb cd 66 0f 1f 44 00 00 8b 05 4a 49 2b 00 85 c0 75 2e 48 63 ff 48 63 d2 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 a1 f0 2a 00 f7 d8 64 89 02 48 [ 186.467465] RSP: 002b:00007ffd7f9d92d8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 186.470182] RAX: ffffffffffffffda RBX: 00005623bc756010 RCX: 00007f157e463dc7 [ 186.472725] RDX: 0000000000000000 RSI: 00007ffd7f9d9360 RDI: 0000000000000000 [ 186.475275] RBP: 00005623bc755ef0 R08: 0000000000000000 R09: 00000000000000f0 [ 186.477814] R10: 00000000000000ea R11: 0000000000000246 R12: 00005623bc756060 [ 186.480547] R13: 00007ffd7f9d9360 R14: 00005623bc755c20 R15: 0000000000000008 [ 186.484264] ---[ end trace 9a39044671a6e95d ]--- [ 186.486521] ------------[ cut here ]------------ [ 186.488449] RTNL: assertion failed at net/wireless/reg.c (2252) [ 186.491060] WARNING: CPU: 0 PID: 4804 at net/wireless/reg.c:2252 update_all_wiphy_regulatory+0x8a/0x90 [cfg80211] [ 186.495546] Modules linked in: cfg80211 rfkill bochs_drm ttm sr_mod cdrom crct10dif_pclmul drm_kms_helper crc32_pclmul sg crc32c_intel ata_generic pata_acpi ghash_clmulni_intel ppdev syscopyarea sysfillrect sysimgblt fb_sys_fops snd_pcm drm ata_piix aesni_intel snd_timer snd crypto_simd libata cryptd glue_helper soundcore joydev pcspkr serio_raw parport_pc i2c_piix4 floppy parport ip_tables [ 186.506468] CPU: 0 PID: 4804 Comm: crda Tainted: G W 5.2.0-rc3-01111-gd8572a5 #1 [ 186.509258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 186.512042] RIP: 0010:update_all_wiphy_regulatory+0x8a/0x90 [cfg80211] [ 186.514585] Code: 7b ea 80 3d d5 e3 08 00 00 75 a7 ba cc 08 00 00 48 c7 c6 40 9d 55 c0 48 c7 c7 68 9d 55 c0 c6 05 b9 e3 08 00 01 e8 96 d7 79 ea <0f> 0b eb 84 66 90 66 66 66 66 90 41 55 41 54 55 53 48 89 fb 8b 6f [ 186.520426] RSP: 0018:ffffb103c09bba38 EFLAGS: 00010282 [ 186.522651] RAX: 0000000000000000 RBX: ffff95227e894400 RCX: 0000000000000000 [ 186.525202] RDX: ffff95233fc26d80 RSI: ffff95233fc17778 RDI: ffff95233fc17778 [ 186.527808] RBP: 0000000000000000 R08: 0000000000000488 R09: 0000000000aaaaaa [ 186.530318] R10: 0000000000000000 R11: ffff95227dd38b70 R12: 0000000000000000 [ 186.533247] R13: ffffffffc0556140 R14: 0000000000000000 R15: ffff95227e7ef9c8 [ 186.535818] FS: 00007f157ef5cb40(0000) GS:ffff95233fc00000(0000) knlGS:0000000000000000 [ 186.538591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 186.540959] CR2: 00005623bc744028 CR3: 000000008861e000 CR4: 00000000000406f0 [ 186.543642] Call Trace: [ 186.545460] set_regdom+0x149/0x430 [cfg80211] [ 186.547596] nl80211_set_reg+0x290/0x2c0 [cfg80211] [ 186.549758] genl_family_rcv_msg+0x203/0x400 [ 186.551913] ? __alloc_pages_nodemask+0x157/0x340 [ 186.554053] genl_rcv_msg+0x47/0x90 [ 186.556085] ? __kmalloc_node_track_caller+0x59/0x2a0 [ 186.558417] ? genl_family_rcv_msg+0x400/0x400 [ 186.560609] netlink_rcv_skb+0x4a/0x110 [ 186.562710] genl_rcv+0x24/0x40 [ 186.564712] netlink_unicast+0x193/0x230 [ 186.566835] netlink_sendmsg+0x2c1/0x3c0 [ 186.568847] sock_sendmsg+0x4c/0x50 [ 186.570935] ___sys_sendmsg+0x289/0x300 [ 186.572994] ? mem_cgroup_throttle_swaprate+0x20/0x160 [ 186.575196] ? do_anonymous_page+0x2f3/0x3f0 [ 186.577419] ? __seccomp_filter+0x96/0x6c0 [ 186.579439] ? __sys_sendmsg+0x5e/0xa0 [ 186.581528] __sys_sendmsg+0x5e/0xa0 [ 186.583526] do_syscall_64+0x5b/0x1e0 [ 186.585418] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 186.587625] RIP: 0033:0x7f157e463dc7 [ 186.589610] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb cd 66 0f 1f 44 00 00 8b 05 4a 49 2b 00 85 c0 75 2e 48 63 ff 48 63 d2 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 a1 f0 2a 00 f7 d8 64 89 02 48 [ 186.595368] RSP: 002b:00007ffd7f9d92d8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 186.598376] RAX: ffffffffffffffda RBX: 00005623bc756010 RCX: 00007f157e463dc7 [ 186.601001] RDX: 0000000000000000 RSI: 00007ffd7f9d9360 RDI: 0000000000000000 [ 186.603656] RBP: 00005623bc755ef0 R08: 0000000000000000 R09: 00000000000000f0 [ 186.606129] R10: 00000000000000ea R11: 0000000000000246 R12: 00005623bc756060 [ 186.608590] R13: 00007ffd7f9d9360 R14: 00005623bc755c20 R15: 0000000000000008 [ 186.614598] ---[ end trace 9a39044671a6e95e ]--- [ 186.749954] mac80211_hwsim: initializing netlink [ 186.777079] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht' To reproduce: # build kernel cd linux cp config-5.2.0-rc3-01111-gd8572a5 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 prepare make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 modules_prepare make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 SHELL=/bin/bash make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

3 years

1
0
0 / 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP June 2019