June 2021 - LKP - Ml01.01.Org

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 18bb7d74fbd6886c19068d0f6eea3d983ebf4c23 ("xfs: allow queued AG intents to drain before scrubbing") https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git scrub-drain-intents in testcase: xfstests version: xfstests-x86_64-820b5e3-1_20210609 with following parameters: disk: 4HDD fs: xfs test: xfs-reflink-15 ucode: 0x28 test-description: xfstests is a regression test suite for xfs and other files ystems. test-url: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git on test machine: 8 threads 1 sockets Intel(R) Core(TM) i7-4790 v3 @ 3.60GHz with 6G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> [ 67.979803] XFS (sdb4): Unmounting Filesystem [ 68.939007] XFS (sdb4): Mounting V5 Filesystem [ 68.983028] XFS (sdb4): Ending clean mount [ 69.017237] xfs filesystem being mounted at /fs/scratch supports timestamps until 2038 (0x7fffffff) [ 69.257224] XFS (sdb4): Unmounting Filesystem [ 69.321334] XFS: Assertion failed: atomic_read(&pag->pag_intents) == 0, file: fs/xfs/xfs_mount.c, line: 152 [ 69.331066] ------------[ cut here ]------------ [ 69.335675] kernel BUG at fs/xfs/xfs_message.c:110! [ 69.340564] invalid opcode: 0000 [#1] SMP PTI [ 69.344918] CPU: 3 PID: 12251 Comm: umount Not tainted 5.13.0-rc6-00320-g18bb7d74fbd6 #1 [ 69.352995] Hardware name: Dell Inc. OptiPlex 9020/03CPWF, BIOS A11 04/01/2015 [ 69.360198] RIP: 0010:assfail (kbuild/src/consumer/fs/xfs/./xfs_trace.h:164) xfs [ 69.364634] Code: 67 fc ff ff 0f 0b c3 0f 1f 44 00 00 41 89 c8 48 89 d1 48 89 f2 48 c7 c6 80 48 81 c0 e8 82 f9 ff ff 80 3d 24 df 0c 00 00 74 02 <0f> 0b 0f 0b c3 48 8d 45 10 48 89 e2 4c 89 e6 48 89 1c 24 48 89 44 All code ======== 0: 67 fc addr32 cld 2: ff (bad) 3: ff 0f decl (%rdi) 5: 0b c3 or %ebx,%eax 7: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) c: 41 89 c8 mov %ecx,%r8d f: 48 89 d1 mov %rdx,%rcx 12: 48 89 f2 mov %rsi,%rdx 15: 48 c7 c6 80 48 81 c0 mov $0xffffffffc0814880,%rsi 1c: e8 82 f9 ff ff callq 0xfffffffffffff9a3 21: 80 3d 24 df 0c 00 00 cmpb $0x0,0xcdf24(%rip) # 0xcdf4c 28: 74 02 je 0x2c 2a:* 0f 0b ud2 <-- trapping instruction 2c: 0f 0b ud2 2e: c3 retq 2f: 48 8d 45 10 lea 0x10(%rbp),%rax 33: 48 89 e2 mov %rsp,%rdx 36: 4c 89 e6 mov %r12,%rsi 39: 48 89 1c 24 mov %rbx,(%rsp) 3d: 48 rex.W 3e: 89 .byte 0x89 3f: 44 rex.R Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 0f 0b ud2 4: c3 retq 5: 48 8d 45 10 lea 0x10(%rbp),%rax 9: 48 89 e2 mov %rsp,%rdx c: 4c 89 e6 mov %r12,%rsi f: 48 89 1c 24 mov %rbx,(%rsp) 13: 48 rex.W 14: 89 .byte 0x89 15: 44 rex.R [ 69.383359] RSP: 0018:ffffc9000996fdf0 EFLAGS: 00010202 [ 69.388568] RAX: 0000000000000000 RBX: ffff888191d86000 RCX: 0000000000000000 [ 69.395683] RDX: 00000000ffffffc0 RSI: 000000000000000a RDI: ffffffffc0805241 [ 69.402799] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 69.409921] R10: 000000000000000a R11: f000000000000000 R12: ffff888192010580 [ 69.417045] R13: ffff888192010000 R14: ffff888192010570 R15: ffff888191774f80 [ 69.424161] FS: 00007f9f57447080(0000) GS:ffff88819eac0000(0000) knlGS:0000000000000000 [ 69.432229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 69.437958] CR2: 000055acc59e3ae8 CR3: 0000000191c4e003 CR4: 00000000001706e0 [ 69.445072] Call Trace: [ 69.447513] xfs_free_perag (kbuild/src/consumer/fs/xfs/xfs_mount.c:152 (discriminator 1)) xfs [ 69.451917] xfs_unmountfs (kbuild/src/consumer/fs/xfs/xfs_mount.c:1239) xfs [ 69.456152] xfs_fs_put_super (kbuild/src/consumer/fs/xfs/xfs_super.c:1099) xfs [ 69.460553] generic_shutdown_super (kbuild/src/consumer/include/linux/list.h:282 kbuild/src/consumer/fs/super.c:467) [ 69.464987] kill_block_super (kbuild/src/consumer/fs/super.c:1396) To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

1
0
0 / 0

[mm/mremap] ecf8443e51: vm-scalability.throughput -29.4% regression

by kernel test robot

Greeting, FYI, we noticed a -29.4% regression of vm-scalability.throughput due to commit: commit: ecf8443e51a862b261313c2319ab4e4aed9e6b7e ("[PATCH v7 02/11] mm/mremap: Fix race between MOVE_PUD mremap and pageout") url: https://github.com/0day-ci/linux/commits/Aneesh-Kumar-K-V/Speedup-mremap-... base: https://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git next in testcase: vm-scalability on test machine: 192 threads 4 sockets Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory with following parameters: runtime: 300 thp_enabled: never thp_defrag: always nr_task: 8 nr_ssd: 1 test: swap-w-seq-mt cpufreq_governor: performance ucode: 0x4003006 test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us. test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/ If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file ========================================================================================= compiler/cpufreq_governor/kconfig/nr_ssd/nr_task/rootfs/runtime/tbox_group/test/testcase/thp_defrag/thp_enabled/ucode: gcc-9/performance/x86_64-rhel-8.3/1/8/debian-10.4-x86_64-20200603.cgz/300/lkp-csl-2ap1/swap-w-seq-mt/vm-scalability/always/never/0x4003006 commit: 5f80ee2fc0 ("mm/mremap: Fix race between MOVE_PMD mremap and pageout") ecf8443e51 ("mm/mremap: Fix race between MOVE_PUD mremap and pageout") 5f80ee2fc08b3613 ecf8443e51a862b261313c2319a ---------------- --------------------------- %stddev %change %stddev \ | \ 7.93 ± 4% +14.6% 9.08 ± 5% vm-scalability.free_time 373210 -29.4% 263582 ± 3% vm-scalability.median 43.45 ± 5% -8.1 35.34 ± 7% vm-scalability.stddev% 2954757 -29.4% 2085715 ± 3% vm-scalability.throughput 220.03 +38.3% 304.37 ± 2% vm-scalability.time.elapsed_time 220.03 +38.3% 304.37 ± 2% vm-scalability.time.elapsed_time.max 32410 ± 2% +29.9% 42117 ± 10% vm-scalability.time.involuntary_context_switches 528.33 +13.1% 597.67 vm-scalability.time.percent_of_cpu_this_job_got 913.19 ± 2% +70.7% 1558 ± 3% vm-scalability.time.system_time 250.22 +4.7% 261.96 vm-scalability.time.user_time 34211 ± 11% -95.9% 1400 ± 38% vm-scalability.time.voluntary_context_switches 4.82 ± 3% +15.8% 5.57 ± 3% iostat.cpu.system 3.77e+08 ± 24% +46.4% 5.52e+08 ± 19% turbostat.IRQ 271.03 +30.2% 352.88 ± 2% uptime.boot 48496 +30.1% 63095 ± 2% uptime.idle 0.66 ± 12% -0.7 0.00 ± 57% mpstat.cpu.all.iowait% 0.06 ± 4% -0.0 0.06 ± 4% mpstat.cpu.all.soft% 3.09 ± 2% +0.9 4.03 ± 3% mpstat.cpu.all.sys% 0.60 -0.1 0.48 ± 4% mpstat.cpu.all.usr% 28255 ± 5% +77.2% 50067 ± 6% meminfo.Active 27866 ± 5% +78.2% 49655 ± 6% meminfo.Active(anon) 28180965 -20.0% 22548765 ± 3% meminfo.MemAvailable 28675444 -19.7% 23016466 ± 3% meminfo.MemFree 38693 ± 5% +56.1% 60412 ± 2% meminfo.Shmem 496106 ± 7% +2180.0% 11311305 ± 12% meminfo.SwapCached 60640 ± 3% -97.9% 1245 ± 46% meminfo.Writeback 87575 ± 3% -43.8% 49246 ± 8% slabinfo.dmaengine-unmap-16.active_objs 2123 ± 3% -44.5% 1179 ± 8% slabinfo.dmaengine-unmap-16.active_slabs 89210 ± 3% -44.5% 49532 ± 8% slabinfo.dmaengine-unmap-16.num_objs 2123 ± 3% -44.5% 1179 ± 8% slabinfo.dmaengine-unmap-16.num_slabs 4866 -20.4% 3874 ± 3% slabinfo.kmalloc-4k.active_objs 623.83 -20.9% 493.50 ± 3% slabinfo.kmalloc-4k.active_slabs 4992 -20.8% 3952 ± 3% slabinfo.kmalloc-4k.num_objs 623.83 -20.9% 493.50 ± 3% slabinfo.kmalloc-4k.num_slabs 374.17 ± 2% -47.8% 195.17 ± 53% vmstat.io.bi 1341570 -23.6% 1025527 ± 2% vmstat.io.bo 28897122 -21.2% 22757749 ± 2% vmstat.memory.free 1.238e+08 +8.1% 1.339e+08 ± 2% vmstat.memory.swpd 1.00 -100.0% 0.00 vmstat.procs.b 8.00 +29.2% 10.33 ± 7% vmstat.procs.r 364.17 ± 2% -48.3% 188.17 ± 55% vmstat.swap.si 1341561 -23.6% 1025520 ± 2% vmstat.swap.so 3328 ± 2% -19.9% 2666 vmstat.system.cs 1511984 +17.7% 1780129 vmstat.system.in 14215 ± 15% -98.5% 209.67 ± 84% numa-meminfo.node0.Writeback 308351 ±116% +1024.4% 3467064 ± 11% numa-meminfo.node1.FilePages 168771 ± 6% -15.1% 143367 ± 12% numa-meminfo.node1.KReclaimable 168771 ± 6% -15.1% 143367 ± 12% numa-meminfo.node1.SReclaimable 127034 ± 32% +2618.6% 3453577 ± 10% numa-meminfo.node1.SwapCached 14382 ± 9% -97.7% 332.00 ± 34% numa-meminfo.node1.Writeback 124583 ± 35% +2602.3% 3366554 ± 14% numa-meminfo.node2.FilePages 77286 ± 26% +55.8% 120430 ± 18% numa-meminfo.node2.KReclaimable 77286 ± 26% +55.8% 120430 ± 18% numa-meminfo.node2.SReclaimable 138348 ± 20% +28.1% 177225 ± 13% numa-meminfo.node2.Slab 113067 ± 23% +2874.7% 3363404 ± 14% numa-meminfo.node2.SwapCached 14263 ± 5% -97.4% 373.17 ± 37% numa-meminfo.node2.Writeback 13835 ± 13% +155.7% 35375 ± 18% numa-meminfo.node3.Active 13835 ± 13% +155.2% 35308 ± 18% numa-meminfo.node3.Active(anon) 261289 ±137% +1236.3% 3491534 ± 7% numa-meminfo.node3.FilePages 16541 ± 12% +130.9% 38192 ± 21% numa-meminfo.node3.Shmem 81973 ± 30% +3903.0% 3281379 ± 9% numa-meminfo.node3.SwapCached 16321 ± 6% -97.7% 372.33 ± 42% numa-meminfo.node3.Writeback 3643 ± 14% -98.5% 55.00 ±102% numa-vmstat.node0.nr_writeback 3619 ± 14% -98.6% 49.67 ±107% numa-vmstat.node0.nr_zone_write_pending 77012 ±116% +1039.9% 877855 ± 10% numa-vmstat.node1.nr_file_pages 42097 ± 6% -14.0% 36208 ± 10% numa-vmstat.node1.nr_slab_reclaimable 31679 ± 32% +2660.4% 874478 ± 10% numa-vmstat.node1.nr_swapcached 11014726 ± 2% +12.6% 12401236 ± 4% numa-vmstat.node1.nr_vmscan_write 3577 ± 10% -97.5% 89.83 ± 41% numa-vmstat.node1.nr_writeback 11011253 ± 2% +12.6% 12401174 ± 4% numa-vmstat.node1.nr_written 3502 ± 10% -97.7% 82.17 ± 49% numa-vmstat.node1.nr_zone_write_pending 31302 ± 35% +2617.2% 850555 ± 13% numa-vmstat.node2.nr_file_pages 19271 ± 26% +57.8% 30407 ± 17% numa-vmstat.node2.nr_slab_reclaimable 28421 ± 24% +2889.9% 849766 ± 12% numa-vmstat.node2.nr_swapcached 11041559 ± 3% +19.1% 13149436 ± 10% numa-vmstat.node2.nr_vmscan_write 3782 ± 4% -97.6% 92.00 ± 34% numa-vmstat.node2.nr_writeback 11037863 ± 3% +19.1% 13149374 ± 10% numa-vmstat.node2.nr_written 3724 ± 4% -97.9% 79.00 ± 39% numa-vmstat.node2.nr_zone_write_pending 3443 ± 14% +160.0% 8954 ± 20% numa-vmstat.node3.nr_active_anon 65304 ±137% +1252.7% 883364 ± 6% numa-vmstat.node3.nr_file_pages 4129 ± 12% +134.5% 9682 ± 23% numa-vmstat.node3.nr_shmem 20479 ± 30% +3956.2% 830690 ± 8% numa-vmstat.node3.nr_swapcached 10829696 ± 5% +14.0% 12346461 ± 4% numa-vmstat.node3.nr_vmscan_write 4124 ± 4% -97.6% 97.00 ± 39% numa-vmstat.node3.nr_writeback 10825671 ± 5% +14.0% 12346386 ± 4% numa-vmstat.node3.nr_written 3444 ± 14% +160.0% 8955 ± 20% numa-vmstat.node3.nr_zone_active_anon 4055 ± 4% -97.9% 83.67 ± 43% numa-vmstat.node3.nr_zone_write_pending 8412 ± 3% +28.0% 10768 ± 13% proc-vmstat.allocstall_movable 5996587 ± 46% +183.4% 16993010 ± 32% proc-vmstat.compact_free_scanned 3451053 ± 29% +75.7% 6064315 ± 15% proc-vmstat.compact_isolated 3259087 ± 27% +89.7% 6184085 ± 15% proc-vmstat.compact_migrate_scanned 6961 ± 5% +78.5% 12425 ± 6% proc-vmstat.nr_active_anon 40993758 -3.1% 39718983 proc-vmstat.nr_anon_pages 698799 -19.8% 560310 ± 3% proc-vmstat.nr_dirty_background_threshold 1399308 -19.8% 1121992 ± 3% proc-vmstat.nr_dirty_threshold 388625 ± 2% +695.9% 3092997 ± 11% proc-vmstat.nr_file_pages 7165507 -19.3% 5779541 ± 3% proc-vmstat.nr_free_pages 41119695 +3.5% 42542747 proc-vmstat.nr_inactive_anon 145036 +3.1% 149568 proc-vmstat.nr_page_table_pages 9675 ± 5% +56.0% 15094 ± 2% proc-vmstat.nr_shmem 123879 ± 8% +2178.6% 2822769 ± 12% proc-vmstat.nr_swapcached 38677277 +7.7% 41656286 ± 3% proc-vmstat.nr_vmscan_write 14819 ± 3% -97.9% 307.33 ± 50% proc-vmstat.nr_writeback 74789620 +5.5% 78882933 ± 2% proc-vmstat.nr_written 6962 ± 5% +78.5% 12426 ± 6% proc-vmstat.nr_zone_active_anon 41119448 +3.5% 42542544 proc-vmstat.nr_zone_inactive_anon 14582 ± 3% -98.1% 274.50 ± 47% proc-vmstat.nr_zone_write_pending 20967960 ± 16% +27.7% 26777776 ± 10% proc-vmstat.numa_pte_updates 16061 ± 13% +34.4% 21592 ± 7% proc-vmstat.pgactivate 1726355 ± 29% +75.7% 3033559 ± 15% proc-vmstat.pgmigrate_success 2.992e+08 +5.5% 3.155e+08 ± 2% proc-vmstat.pgpgout 48003 ± 2% +32.3% 63514 ± 3% proc-vmstat.pgreuse 74393768 -12.6% 65057070 ± 4% proc-vmstat.pgrotated 2.36e+08 +8.4% 2.558e+08 proc-vmstat.pgscan_anon 1.048e+08 ± 2% +24.6% 1.305e+08 ± 6% proc-vmstat.pgscan_direct 74792004 +5.5% 78884465 ± 2% proc-vmstat.pswpout 50067 ± 7% +17.0% 58567 ± 4% proc-vmstat.slabs_scanned 4.724e+09 -18.5% 3.85e+09 perf-stat.i.branch-instructions 3231 ± 2% -19.3% 2608 perf-stat.i.context-switches 1.86 ± 2% +47.3% 2.74 ± 5% perf-stat.i.cpi 3.281e+10 ± 2% +15.9% 3.803e+10 ± 4% perf-stat.i.cpu-cycles 206.24 -1.9% 202.23 perf-stat.i.cpu-migrations 642.02 ± 6% +14.2% 732.98 ± 8% perf-stat.i.cycles-between-cache-misses 4.851e+09 -17.3% 4.014e+09 perf-stat.i.dTLB-loads 3304908 -19.0% 2677570 ± 4% perf-stat.i.dTLB-store-misses 2.027e+09 -19.4% 1.633e+09 perf-stat.i.dTLB-stores 5484115 ± 3% -14.5% 4691022 ± 3% perf-stat.i.iTLB-load-misses 1.857e+10 -17.5% 1.533e+10 perf-stat.i.instructions 5181 ± 3% -16.9% 4306 ± 2% perf-stat.i.instructions-per-iTLB-miss 0.59 ± 3% -24.6% 0.44 ± 3% perf-stat.i.ipc 23.06 ± 6% -49.2% 11.72 ± 54% perf-stat.i.major-faults 0.17 ± 2% +15.9% 0.20 ± 4% perf-stat.i.metric.GHz 60.43 -18.1% 49.46 perf-stat.i.metric.M/sec 585912 -26.9% 428491 ± 2% perf-stat.i.minor-faults 87.62 +3.0 90.65 perf-stat.i.node-load-miss-rate% 1450164 ± 6% -29.1% 1027918 ± 3% perf-stat.i.node-loads 83.34 +4.1 87.48 perf-stat.i.node-store-miss-rate% 1130049 ± 10% -24.5% 853398 ± 11% perf-stat.i.node-stores 585935 -26.9% 428503 ± 2% perf-stat.i.page-faults 1.77 ± 2% +42.5% 2.52 ± 5% perf-stat.overall.cpi 560.05 ± 6% +22.0% 683.28 ± 11% perf-stat.overall.cycles-between-cache-misses 0.56 ± 2% -29.7% 0.40 ± 5% perf-stat.overall.ipc 91.72 +2.0 93.70 perf-stat.overall.node-load-miss-rate% 86.56 +3.1 89.65 perf-stat.overall.node-store-miss-rate% 7112 +13.7% 8086 perf-stat.overall.path-length 4.703e+09 -18.7% 3.821e+09 ± 2% perf-stat.ps.branch-instructions 3220 ± 2% -19.7% 2587 perf-stat.ps.context-switches 3.277e+10 ± 2% +17.3% 3.843e+10 ± 5% perf-stat.ps.cpu-cycles 205.01 -2.7% 199.55 ± 2% perf-stat.ps.cpu-migrations 4.834e+09 -17.5% 3.987e+09 perf-stat.ps.dTLB-loads 3283502 -19.6% 2639256 ± 4% perf-stat.ps.dTLB-store-misses 2.02e+09 -19.9% 1.618e+09 perf-stat.ps.dTLB-stores 5471166 ± 3% -15.0% 4652576 ± 2% perf-stat.ps.iTLB-load-misses 1.85e+10 -17.7% 1.523e+10 perf-stat.ps.instructions 22.79 ± 6% -49.2% 11.57 ± 54% perf-stat.ps.major-faults 581983 -27.5% 422213 ± 2% perf-stat.ps.minor-faults 1442428 ± 6% -29.8% 1013002 ± 3% perf-stat.ps.node-loads 1123088 ± 10% -25.3% 839225 ± 10% perf-stat.ps.node-stores 582006 -27.5% 422225 ± 2% perf-stat.ps.page-faults 4.085e+12 +13.6% 4.642e+12 perf-stat.total.instructions 0.01 ± 23% -49.2% 0.01 ± 13% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.55 ±221% -99.5% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.02 ± 32% -100.0% 0.00 perf-sched.sch_delay.avg.ms.kswapd.kthread.ret_from_fork 0.01 ± 62% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap 0.01 ± 17% -47.2% 0.00 ± 10% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.02 ± 21% -49.1% 0.01 ± 22% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.02 ± 37% -60.0% 0.01 ± 34% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 166.76 ±223% -100.0% 0.01 ± 10% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.03 ± 67% -62.5% 0.01 ± 51% perf-sched.sch_delay.max.ms.io_schedule.__lock_page_or_retry.do_swap_page.__handle_mm_fault 0.03 ±130% -77.5% 0.01 ± 61% perf-sched.sch_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead 0.02 ± 39% -100.0% 0.00 perf-sched.sch_delay.max.ms.kswapd.kthread.ret_from_fork 0.09 ± 59% -85.5% 0.01 ± 10% perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.05 ± 77% -89.3% 0.01 ± 56% perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork 0.02 ± 96% -100.0% 0.00 perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap 0.02 ± 36% -62.8% 0.01 ± 21% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 1018 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 1018 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read 15.36 ± 84% -97.3% 0.42 ± 78% perf-sched.wait_and_delay.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead 2042 ± 37% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 8.00 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64 8.00 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read 93.67 ±133% +261.4% 338.50 ± 12% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__alloc_pages_slowpath.constprop.0 20.17 ±223% +547.1% 130.50 ± 10% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.mempool_alloc.bio_alloc_bioset.__swap_writepage 346.50 ± 74% +194.0% 1018 ± 10% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.shrink_node.do_try_to_free_pages.try_to_free_pages 9.00 ± 19% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 23.00 ± 5% +24.6% 28.67 ± 10% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 5996 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 5996 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read 4492 ± 79% -99.9% 3.21 ± 71% perf-sched.wait_and_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead 6025 ± 32% -82.0% 1082 ± 4% perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 6024 ± 32% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 1018 ± 14% -99.9% 1.33 ± 15% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 1018 ± 14% -99.8% 1.60 ± 57% perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read 309.68 ±143% -100.0% 0.01 ± 27% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 224.46 ±171% -99.9% 0.13 ±110% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 378.18 ± 74% -99.8% 0.69 ± 48% perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead 15.36 ± 84% -97.3% 0.42 ± 78% perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead 105.18 -100.0% 0.00 perf-sched.wait_time.avg.ms.kswapd.kthread.ret_from_fork 57.26 ± 71% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas 2.81 ± 22% -68.2% 0.89 ± 45% perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork 754.86 ±100% -100.0% 0.00 perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 298.39 ±141% -100.0% 0.00 perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap 2042 ± 37% -100.0% 0.37 ± 99% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.00 ± 10% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kswapd.kthread.ret_from_fork 5996 ± 31% -100.0% 2.66 ± 15% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 5996 ± 31% -100.0% 2.56 ± 22% perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read 1179 ±142% -100.0% 0.03 ± 89% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 818.62 ±149% -100.0% 0.13 ±110% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 4236 ± 54% -99.9% 3.55 ± 58% perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead 4492 ± 79% -99.9% 3.21 ± 71% perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead 107.44 -100.0% 0.00 perf-sched.wait_time.max.ms.kswapd.kthread.ret_from_fork 6025 ± 32% -82.0% 1082 ± 4% perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 2367 ± 71% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas 5.00 -56.8% 2.16 ± 93% perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork 1888 ±101% -100.0% 0.00 perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 1240 ±140% -100.0% 0.00 perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap 6024 ± 32% -100.0% 0.74 ±107% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.01 ± 13% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kswapd.kthread.ret_from_fork 443.33 +38.2% 612.67 ± 2% interrupts.9:IO-APIC.9-fasteoi.acpi 2.506e+08 ± 2% +71.0% 4.285e+08 ± 2% interrupts.CAL:Function_call_interrupts 436770 +39.6% 609678 ± 3% interrupts.CPU0.LOC:Local_timer_interrupts 443.33 +38.2% 612.67 ± 2% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi 436473 +39.6% 609534 ± 3% interrupts.CPU1.LOC:Local_timer_interrupts 436112 +39.7% 609460 ± 3% interrupts.CPU10.LOC:Local_timer_interrupts 435885 +39.8% 609476 ± 3% interrupts.CPU100.LOC:Local_timer_interrupts 435912 +39.8% 609519 ± 3% interrupts.CPU101.LOC:Local_timer_interrupts 435983 +39.8% 609496 ± 3% interrupts.CPU102.LOC:Local_timer_interrupts 436229 +39.7% 609544 ± 3% interrupts.CPU103.LOC:Local_timer_interrupts 152.33 ± 49% +944.5% 1591 ± 79% interrupts.CPU103.NMI:Non-maskable_interrupts 152.33 ± 49% +944.5% 1591 ± 79% interrupts.CPU103.PMI:Performance_monitoring_interrupts 436056 +39.8% 609461 ± 3% interrupts.CPU104.LOC:Local_timer_interrupts 435861 +39.8% 609478 ± 3% interrupts.CPU105.LOC:Local_timer_interrupts 436021 +39.8% 609379 ± 3% interrupts.CPU106.LOC:Local_timer_interrupts 435843 +39.9% 609554 ± 3% interrupts.CPU107.LOC:Local_timer_interrupts 435999 +39.8% 609466 ± 3% interrupts.CPU108.LOC:Local_timer_interrupts 436372 +39.7% 609463 ± 3% interrupts.CPU109.LOC:Local_timer_interrupts 435946 +39.8% 609652 ± 3% interrupts.CPU11.LOC:Local_timer_interrupts 436096 +39.8% 609463 ± 3% interrupts.CPU110.LOC:Local_timer_interrupts 435856 +39.8% 609436 ± 3% interrupts.CPU111.LOC:Local_timer_interrupts 436180 +39.7% 609470 ± 3% interrupts.CPU112.LOC:Local_timer_interrupts 435902 +39.8% 609466 ± 3% interrupts.CPU113.LOC:Local_timer_interrupts 435823 +39.9% 609521 ± 3% interrupts.CPU114.LOC:Local_timer_interrupts 435831 +39.8% 609416 ± 3% interrupts.CPU115.LOC:Local_timer_interrupts 435716 +39.9% 609492 ± 3% interrupts.CPU116.LOC:Local_timer_interrupts 435934 +39.8% 609518 ± 3% interrupts.CPU117.LOC:Local_timer_interrupts 435946 +39.8% 609459 ± 3% interrupts.CPU118.LOC:Local_timer_interrupts 435735 +39.9% 609492 ± 3% interrupts.CPU119.LOC:Local_timer_interrupts 436018 +39.8% 609544 ± 3% interrupts.CPU12.LOC:Local_timer_interrupts 1730294 ± 54% +231.1% 5729689 ± 73% interrupts.CPU120.CAL:Function_call_interrupts 436513 +38.6% 605176 ± 4% interrupts.CPU120.LOC:Local_timer_interrupts 437125 +38.4% 605185 ± 4% interrupts.CPU121.LOC:Local_timer_interrupts 119.00 ± 49% +2582.6% 3192 ± 78% interrupts.CPU121.NMI:Non-maskable_interrupts 119.00 ± 49% +2582.6% 3192 ± 78% interrupts.CPU121.PMI:Performance_monitoring_interrupts 436783 +38.6% 605196 ± 4% interrupts.CPU122.LOC:Local_timer_interrupts 436134 +38.8% 605202 ± 4% interrupts.CPU123.LOC:Local_timer_interrupts 436402 +38.7% 605173 ± 4% interrupts.CPU124.LOC:Local_timer_interrupts 436347 +38.7% 605194 ± 4% interrupts.CPU125.LOC:Local_timer_interrupts 436510 +38.7% 605240 ± 4% interrupts.CPU126.LOC:Local_timer_interrupts 436805 +38.5% 605169 ± 4% interrupts.CPU127.LOC:Local_timer_interrupts 436659 +38.6% 605187 ± 4% interrupts.CPU128.LOC:Local_timer_interrupts 436231 +38.7% 605179 ± 4% interrupts.CPU129.LOC:Local_timer_interrupts 436500 +39.6% 609536 ± 3% interrupts.CPU13.LOC:Local_timer_interrupts 436424 +38.7% 605184 ± 4% interrupts.CPU130.LOC:Local_timer_interrupts 436297 +38.7% 605194 ± 4% interrupts.CPU131.LOC:Local_timer_interrupts 436562 +38.6% 605189 ± 4% interrupts.CPU132.LOC:Local_timer_interrupts 436922 +38.5% 605198 ± 4% interrupts.CPU133.LOC:Local_timer_interrupts 436581 +38.6% 605204 ± 4% interrupts.CPU134.LOC:Local_timer_interrupts 436103 +38.8% 605178 ± 4% interrupts.CPU135.LOC:Local_timer_interrupts 436818 +38.5% 605171 ± 4% interrupts.CPU136.LOC:Local_timer_interrupts 436327 +38.7% 605173 ± 4% interrupts.CPU137.LOC:Local_timer_interrupts 436190 +38.7% 605185 ± 4% interrupts.CPU138.LOC:Local_timer_interrupts 436439 +38.7% 605194 ± 4% interrupts.CPU139.LOC:Local_timer_interrupts 436084 +39.8% 609534 ± 3% interrupts.CPU14.LOC:Local_timer_interrupts 436238 +38.7% 605188 ± 4% interrupts.CPU140.LOC:Local_timer_interrupts 436692 +38.6% 605188 ± 4% interrupts.CPU141.LOC:Local_timer_interrupts 436428 +38.7% 605187 ± 4% interrupts.CPU142.LOC:Local_timer_interrupts 436618 +38.6% 605181 ± 4% interrupts.CPU143.LOC:Local_timer_interrupts 2395060 ± 21% +210.7% 7440320 ± 40% interrupts.CPU144.CAL:Function_call_interrupts 437076 ± 2% +38.3% 604600 ± 4% interrupts.CPU144.LOC:Local_timer_interrupts 4849080 ± 21% +113.2% 10339751 ± 39% interrupts.CPU144.TLB:TLB_shootdowns 437303 ± 2% +38.3% 604631 ± 4% interrupts.CPU145.LOC:Local_timer_interrupts 437337 ± 2% +38.3% 604650 ± 4% interrupts.CPU146.LOC:Local_timer_interrupts 1628946 ± 35% +144.0% 3974501 ± 26% interrupts.CPU147.CAL:Function_call_interrupts 436987 ± 2% +38.4% 604641 ± 4% interrupts.CPU147.LOC:Local_timer_interrupts 3313139 ± 38% +62.6% 5387495 ± 26% interrupts.CPU147.TLB:TLB_shootdowns 437013 ± 2% +38.4% 604647 ± 4% interrupts.CPU148.LOC:Local_timer_interrupts 437047 ± 2% +38.3% 604641 ± 4% interrupts.CPU149.LOC:Local_timer_interrupts 435764 +39.9% 609486 ± 3% interrupts.CPU15.LOC:Local_timer_interrupts 437104 ± 2% +38.3% 604641 ± 4% interrupts.CPU150.LOC:Local_timer_interrupts 437148 ± 2% +38.3% 604674 ± 4% interrupts.CPU151.LOC:Local_timer_interrupts 437106 ± 2% +38.3% 604644 ± 4% interrupts.CPU152.LOC:Local_timer_interrupts 437065 ± 2% +38.4% 604688 ± 4% interrupts.CPU153.LOC:Local_timer_interrupts 437182 ± 2% +38.3% 604648 ± 4% interrupts.CPU154.LOC:Local_timer_interrupts 437068 ± 2% +38.3% 604649 ± 4% interrupts.CPU155.LOC:Local_timer_interrupts 437126 ± 2% +38.3% 604638 ± 4% interrupts.CPU156.LOC:Local_timer_interrupts 437340 ± 2% +38.3% 604639 ± 4% interrupts.CPU157.LOC:Local_timer_interrupts 437229 ± 2% +38.3% 604631 ± 4% interrupts.CPU158.LOC:Local_timer_interrupts 437031 ± 2% +38.4% 604646 ± 4% interrupts.CPU159.LOC:Local_timer_interrupts 436217 +39.7% 609502 ± 3% interrupts.CPU16.LOC:Local_timer_interrupts 437281 ± 2% +38.3% 604671 ± 4% interrupts.CPU160.LOC:Local_timer_interrupts 437174 ± 2% +38.3% 604664 ± 4% interrupts.CPU161.LOC:Local_timer_interrupts 437093 ± 2% +38.3% 604644 ± 4% interrupts.CPU162.LOC:Local_timer_interrupts 437080 ± 2% +38.3% 604617 ± 4% interrupts.CPU163.LOC:Local_timer_interrupts 437014 ± 2% +38.4% 604623 ± 4% interrupts.CPU164.LOC:Local_timer_interrupts 437064 ± 2% +38.3% 604664 ± 4% interrupts.CPU165.LOC:Local_timer_interrupts 437060 ± 2% +38.3% 604645 ± 4% interrupts.CPU166.LOC:Local_timer_interrupts 437133 ± 2% +38.3% 604647 ± 4% interrupts.CPU167.LOC:Local_timer_interrupts 437691 +38.8% 607710 ± 3% interrupts.CPU168.LOC:Local_timer_interrupts 437778 +38.8% 607719 ± 3% interrupts.CPU169.LOC:Local_timer_interrupts 333.83 ±130% +630.9% 2440 ±113% interrupts.CPU169.NMI:Non-maskable_interrupts 333.83 ±130% +630.9% 2440 ±113% interrupts.CPU169.PMI:Performance_monitoring_interrupts 435893 +39.8% 609457 ± 3% interrupts.CPU17.LOC:Local_timer_interrupts 437845 +38.8% 607745 ± 3% interrupts.CPU170.LOC:Local_timer_interrupts 972.00 ±199% +287.4% 3765 ± 75% interrupts.CPU170.NMI:Non-maskable_interrupts 972.00 ±199% +287.4% 3765 ± 75% interrupts.CPU170.PMI:Performance_monitoring_interrupts 437428 +38.9% 607740 ± 3% interrupts.CPU171.LOC:Local_timer_interrupts 437494 +38.9% 607739 ± 3% interrupts.CPU172.LOC:Local_timer_interrupts 1590512 ± 27% +139.8% 3814104 ± 44% interrupts.CPU173.CAL:Function_call_interrupts 437559 +38.9% 607732 ± 3% interrupts.CPU173.LOC:Local_timer_interrupts 437721 +38.8% 607747 ± 3% interrupts.CPU174.LOC:Local_timer_interrupts 437316 +39.0% 607744 ± 3% interrupts.CPU175.LOC:Local_timer_interrupts 437481 +38.9% 607717 ± 3% interrupts.CPU176.LOC:Local_timer_interrupts 437671 +38.9% 607816 ± 3% interrupts.CPU177.LOC:Local_timer_interrupts 437549 +38.9% 607776 ± 3% interrupts.CPU178.LOC:Local_timer_interrupts 437791 +38.8% 607753 ± 3% interrupts.CPU179.LOC:Local_timer_interrupts 435869 +39.8% 609482 ± 3% interrupts.CPU18.LOC:Local_timer_interrupts 437851 +38.8% 607737 ± 3% interrupts.CPU180.LOC:Local_timer_interrupts 437762 +38.8% 607765 ± 3% interrupts.CPU181.LOC:Local_timer_interrupts 437506 +38.9% 607755 ± 3% interrupts.CPU182.LOC:Local_timer_interrupts 437825 +38.8% 607732 ± 3% interrupts.CPU183.LOC:Local_timer_interrupts 437482 +38.9% 607739 ± 3% interrupts.CPU184.LOC:Local_timer_interrupts 437502 +38.9% 607760 ± 3% interrupts.CPU185.LOC:Local_timer_interrupts 437904 +38.8% 607748 ± 3% interrupts.CPU186.LOC:Local_timer_interrupts 437601 +38.9% 607731 ± 3% interrupts.CPU187.LOC:Local_timer_interrupts 437425 +38.9% 607746 ± 3% interrupts.CPU188.LOC:Local_timer_interrupts 437660 +38.9% 607742 ± 3% interrupts.CPU189.LOC:Local_timer_interrupts 435885 +39.8% 609492 ± 3% interrupts.CPU19.LOC:Local_timer_interrupts 658863 ± 99% +449.3% 3618986 ± 71% interrupts.CPU190.CAL:Function_call_interrupts 437525 +38.9% 607724 ± 3% interrupts.CPU190.LOC:Local_timer_interrupts 437714 +38.9% 607775 ± 3% interrupts.CPU191.LOC:Local_timer_interrupts 436340 +39.7% 609513 ± 3% interrupts.CPU2.LOC:Local_timer_interrupts 435793 +39.9% 609516 ± 3% interrupts.CPU20.LOC:Local_timer_interrupts 435929 +39.8% 609516 ± 3% interrupts.CPU21.LOC:Local_timer_interrupts 435897 +39.8% 609589 ± 3% interrupts.CPU22.LOC:Local_timer_interrupts 436105 +39.7% 609457 ± 3% interrupts.CPU23.LOC:Local_timer_interrupts 436542 +38.6% 605219 ± 4% interrupts.CPU24.LOC:Local_timer_interrupts 437153 +38.4% 605215 ± 4% interrupts.CPU25.LOC:Local_timer_interrupts 436932 +38.5% 605126 ± 4% interrupts.CPU26.LOC:Local_timer_interrupts 436176 +38.8% 605266 ± 4% interrupts.CPU27.LOC:Local_timer_interrupts 436415 +38.7% 605223 ± 4% interrupts.CPU28.LOC:Local_timer_interrupts 436413 +38.7% 605249 ± 4% interrupts.CPU29.LOC:Local_timer_interrupts 435892 +39.8% 609496 ± 3% interrupts.CPU3.LOC:Local_timer_interrupts 1884 ± 93% -90.6% 177.17 ± 79% interrupts.CPU3.NMI:Non-maskable_interrupts 1884 ± 93% -90.6% 177.17 ± 79% interrupts.CPU3.PMI:Performance_monitoring_interrupts 3717304 ± 27% -51.0% 1822336 ± 39% interrupts.CPU3.TLB:TLB_shootdowns 436566 +38.6% 605260 ± 4% interrupts.CPU30.LOC:Local_timer_interrupts 436820 +38.5% 605209 ± 4% interrupts.CPU31.LOC:Local_timer_interrupts 436693 +38.6% 605210 ± 4% interrupts.CPU32.LOC:Local_timer_interrupts 436239 +38.7% 605207 ± 4% interrupts.CPU33.LOC:Local_timer_interrupts 436425 +38.7% 605210 ± 4% interrupts.CPU34.LOC:Local_timer_interrupts 436281 +38.7% 605214 ± 4% interrupts.CPU35.LOC:Local_timer_interrupts 436621 +38.6% 605203 ± 4% interrupts.CPU36.LOC:Local_timer_interrupts 436949 +38.5% 605220 ± 4% interrupts.CPU37.LOC:Local_timer_interrupts 436637 +38.6% 605224 ± 4% interrupts.CPU38.LOC:Local_timer_interrupts 436124 +38.8% 605199 ± 4% interrupts.CPU39.LOC:Local_timer_interrupts 435874 +39.8% 609500 ± 3% interrupts.CPU4.LOC:Local_timer_interrupts 436826 +38.5% 605202 ± 4% interrupts.CPU40.LOC:Local_timer_interrupts 436386 +38.7% 605204 ± 4% interrupts.CPU41.LOC:Local_timer_interrupts 436202 +38.7% 605125 ± 4% interrupts.CPU42.LOC:Local_timer_interrupts 436444 +38.7% 605206 ± 4% interrupts.CPU43.LOC:Local_timer_interrupts 436315 +38.7% 605211 ± 4% interrupts.CPU44.LOC:Local_timer_interrupts 436710 +38.6% 605216 ± 4% interrupts.CPU45.LOC:Local_timer_interrupts 436448 +38.7% 605224 ± 4% interrupts.CPU46.LOC:Local_timer_interrupts 436683 +38.6% 605205 ± 4% interrupts.CPU47.LOC:Local_timer_interrupts 437175 ± 2% +38.3% 604605 ± 4% interrupts.CPU48.LOC:Local_timer_interrupts 428230 ± 65% +243.5% 1470847 ± 61% interrupts.CPU49.CAL:Function_call_interrupts 437324 ± 2% +38.3% 604629 ± 4% interrupts.CPU49.LOC:Local_timer_interrupts 435982 +39.8% 609551 ± 3% interrupts.CPU5.LOC:Local_timer_interrupts 437347 ± 2% +38.3% 604679 ± 4% interrupts.CPU50.LOC:Local_timer_interrupts 436995 ± 2% +38.4% 604678 ± 4% interrupts.CPU51.LOC:Local_timer_interrupts 437070 ± 2% +38.3% 604662 ± 4% interrupts.CPU52.LOC:Local_timer_interrupts 437067 ± 2% +38.3% 604680 ± 4% interrupts.CPU53.LOC:Local_timer_interrupts 437130 ± 2% +38.3% 604671 ± 4% interrupts.CPU54.LOC:Local_timer_interrupts 437155 ± 2% +38.3% 604700 ± 4% interrupts.CPU55.LOC:Local_timer_interrupts 437141 ± 2% +38.3% 604682 ± 4% interrupts.CPU56.LOC:Local_timer_interrupts 437048 ± 2% +38.4% 604671 ± 4% interrupts.CPU57.LOC:Local_timer_interrupts 437189 ± 2% +38.3% 604701 ± 4% interrupts.CPU58.LOC:Local_timer_interrupts 437069 ± 2% +38.4% 604688 ± 4% interrupts.CPU59.LOC:Local_timer_interrupts 436032 +39.8% 609555 ± 3% interrupts.CPU6.LOC:Local_timer_interrupts 437190 ± 2% +38.3% 604670 ± 4% interrupts.CPU60.LOC:Local_timer_interrupts 437388 ± 2% +38.2% 604673 ± 4% interrupts.CPU61.LOC:Local_timer_interrupts 437227 ± 2% +38.3% 604666 ± 4% interrupts.CPU62.LOC:Local_timer_interrupts 437036 ± 2% +38.4% 604662 ± 4% interrupts.CPU63.LOC:Local_timer_interrupts 437304 ± 2% +38.3% 604662 ± 4% interrupts.CPU64.LOC:Local_timer_interrupts 437177 ± 2% +38.3% 604654 ± 4% interrupts.CPU65.LOC:Local_timer_interrupts 437090 ± 2% +38.3% 604659 ± 4% interrupts.CPU66.LOC:Local_timer_interrupts 437100 ± 2% +38.3% 604602 ± 4% interrupts.CPU67.LOC:Local_timer_interrupts 437019 ± 2% +38.4% 604618 ± 4% interrupts.CPU68.LOC:Local_timer_interrupts 437048 ± 2% +38.4% 604675 ± 4% interrupts.CPU69.LOC:Local_timer_interrupts 436254 +39.7% 609555 ± 3% interrupts.CPU7.LOC:Local_timer_interrupts 140.83 ± 25% +500.0% 845.00 ± 93% interrupts.CPU7.NMI:Non-maskable_interrupts 140.83 ± 25% +500.0% 845.00 ± 93% interrupts.CPU7.PMI:Performance_monitoring_interrupts 437047 ± 2% +38.4% 604663 ± 4% interrupts.CPU70.LOC:Local_timer_interrupts 437164 ± 2% +38.3% 604658 ± 4% interrupts.CPU71.LOC:Local_timer_interrupts 437731 +38.8% 607745 ± 3% interrupts.CPU72.LOC:Local_timer_interrupts 437796 +38.8% 607772 ± 3% interrupts.CPU73.LOC:Local_timer_interrupts 437891 +38.8% 607769 ± 3% interrupts.CPU74.LOC:Local_timer_interrupts 488.67 ±180% +653.0% 3679 ± 75% interrupts.CPU74.NMI:Non-maskable_interrupts 488.67 ±180% +653.0% 3679 ± 75% interrupts.CPU74.PMI:Performance_monitoring_interrupts 456635 ± 60% +233.5% 1522817 ± 41% interrupts.CPU75.CAL:Function_call_interrupts 437451 +38.9% 607764 ± 3% interrupts.CPU75.LOC:Local_timer_interrupts 437531 +38.9% 607754 ± 3% interrupts.CPU76.LOC:Local_timer_interrupts 437603 +38.9% 607768 ± 3% interrupts.CPU77.LOC:Local_timer_interrupts 437752 +38.8% 607791 ± 3% interrupts.CPU78.LOC:Local_timer_interrupts 437335 +39.0% 607782 ± 3% interrupts.CPU79.LOC:Local_timer_interrupts 436062 +39.8% 609477 ± 3% interrupts.CPU8.LOC:Local_timer_interrupts 437520 +38.9% 607783 ± 3% interrupts.CPU80.LOC:Local_timer_interrupts 437687 +38.9% 607803 ± 3% interrupts.CPU81.LOC:Local_timer_interrupts 437567 +38.9% 607820 ± 3% interrupts.CPU82.LOC:Local_timer_interrupts 437815 +38.8% 607773 ± 3% interrupts.CPU83.LOC:Local_timer_interrupts 437892 +38.8% 607756 ± 3% interrupts.CPU84.LOC:Local_timer_interrupts 437781 +38.8% 607806 ± 3% interrupts.CPU85.LOC:Local_timer_interrupts 437537 +38.9% 607769 ± 3% interrupts.CPU86.LOC:Local_timer_interrupts 437848 +38.8% 607766 ± 3% interrupts.CPU87.LOC:Local_timer_interrupts 437543 +38.9% 607809 ± 3% interrupts.CPU88.LOC:Local_timer_interrupts 437531 +38.9% 607776 ± 3% interrupts.CPU89.LOC:Local_timer_interrupts 435850 +39.8% 609496 ± 3% interrupts.CPU9.LOC:Local_timer_interrupts 437953 +38.8% 607759 ± 3% interrupts.CPU90.LOC:Local_timer_interrupts 437616 +38.9% 607766 ± 3% interrupts.CPU91.LOC:Local_timer_interrupts 437458 +38.9% 607762 ± 3% interrupts.CPU92.LOC:Local_timer_interrupts 437697 +38.9% 607787 ± 3% interrupts.CPU93.LOC:Local_timer_interrupts 437556 +38.9% 607804 ± 3% interrupts.CPU94.LOC:Local_timer_interrupts 437772 +38.8% 607836 ± 3% interrupts.CPU95.LOC:Local_timer_interrupts 436537 +39.6% 609467 ± 3% interrupts.CPU96.LOC:Local_timer_interrupts 436496 +39.6% 609495 ± 3% interrupts.CPU97.LOC:Local_timer_interrupts 436305 +39.7% 609468 ± 3% interrupts.CPU98.LOC:Local_timer_interrupts 435736 +39.9% 609454 ± 3% interrupts.CPU99.LOC:Local_timer_interrupts 83872531 +38.9% 1.165e+08 ± 4% interrupts.LOC:Local_timer_interrupts 0.00 +1.9e+104% 192.00 interrupts.MCP:Machine_check_polls 122140 ± 6% +15.2% 140671 ± 5% interrupts.NMI:Non-maskable_interrupts 122140 ± 6% +15.2% 140671 ± 5% interrupts.PMI:Performance_monitoring_interrupts 5.166e+08 ± 2% +15.1% 5.948e+08 ± 2% interrupts.TLB:TLB_shootdowns 17.91 ± 6% -7.3 10.57 ± 2% perf-profile.calltrace.cycles-pp.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 17.90 ± 6% -7.3 10.56 ± 2% perf-profile.calltrace.cycles-pp.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec 17.22 ± 6% -7.0 10.22 ± 2% perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list 16.87 ± 6% -6.8 10.07 ± 2% perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list 40.74 ± 7% -4.6 36.13 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 40.30 ± 8% -4.5 35.84 ± 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 40.30 ± 8% -4.5 35.84 ± 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 40.30 ± 8% -4.5 35.83 ± 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 38.05 ± 9% -4.4 33.60 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 37.65 ± 9% -4.4 33.26 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 4.33 ± 9% -3.2 1.13 ± 22% perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 5.57 ± 6% -2.7 2.85 ± 4% perf-profile.calltrace.cycles-pp.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 3.96 ± 5% -2.2 1.76 ± 7% perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 3.72 ± 6% -1.8 1.90 ± 8% perf-profile.calltrace.cycles-pp.__swap_writepage.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec 3.91 ± 6% -1.8 2.13 ± 3% perf-profile.calltrace.cycles-pp.llist_add_batch.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty 1.83 ± 6% -1.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty 3.04 ± 11% -0.9 2.17 ± 11% perf-profile.calltrace.cycles-pp.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 5.24 ± 7% -0.8 4.43 ± 4% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 3.40 ± 6% -0.8 2.62 ± 10% perf-profile.calltrace.cycles-pp.do_rw_once 4.81 ± 7% -0.7 4.06 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 1.94 ± 11% -0.7 1.19 ± 9% perf-profile.calltrace.cycles-pp.__mem_cgroup_charge.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault 3.31 ± 8% -0.5 2.78 ± 5% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 3.29 ± 8% -0.5 2.76 ± 5% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 0.61 ± 5% -0.3 0.28 ±100% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 1.15 ± 17% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_slowpath.__alloc_pages.alloc_pages_vma.do_anonymous_page 0.73 ± 6% -0.1 0.62 ± 6% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.46 ± 44% +0.4 0.83 ± 17% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.do_access 1.58 ± 8% +0.5 2.07 ± 6% perf-profile.calltrace.cycles-pp.__default_send_IPI_dest_field.default_send_IPI_mask_sequence_phys.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush 0.00 +2.2 2.25 ± 4% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one 0.00 +2.4 2.38 ± 8% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one 0.00 +3.4 3.37 ± 9% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath 0.00 +3.4 3.41 ± 9% perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock 0.00 +3.7 3.72 ± 6% perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk 5.56 ± 8% +10.7 16.22 ± 3% perf-profile.calltrace.cycles-pp.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec 5.84 ± 8% +10.8 16.68 ± 3% perf-profile.calltrace.cycles-pp.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 4.96 ± 8% +10.9 15.85 ± 3% perf-profile.calltrace.cycles-pp.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list 0.10 ±223% +12.3 12.40 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon 0.10 ±223% +12.5 12.62 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon 0.34 ±109% +13.2 13.56 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap 2.00 ± 12% +13.4 15.36 ± 3% perf-profile.calltrace.cycles-pp.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec 2.13 ± 12% +13.4 15.48 ± 3% perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 0.32 ±105% +13.4 13.74 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced 0.50 ± 77% +13.5 14.04 ± 3% perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list 0.51 ± 76% +13.9 14.39 ± 3% perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list 0.87 ± 22% +14.0 14.83 ± 3% perf-profile.calltrace.cycles-pp.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list 18.00 ± 6% -7.4 10.64 ± 2% perf-profile.children.cycles-pp.try_to_unmap_flush_dirty 17.99 ± 6% -7.4 10.64 ± 2% perf-profile.children.cycles-pp.arch_tlbbatch_flush 17.31 ± 6% -7.0 10.29 ± 2% perf-profile.children.cycles-pp.on_each_cpu_cond_mask 17.30 ± 6% -7.0 10.29 ± 3% perf-profile.children.cycles-pp.smp_call_function_many_cond 40.74 ± 7% -4.6 36.13 ± 4% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 40.74 ± 7% -4.6 36.13 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 40.74 ± 7% -4.6 36.13 ± 4% perf-profile.children.cycles-pp.do_idle 38.48 ± 8% -4.6 33.90 ± 4% perf-profile.children.cycles-pp.cpuidle_enter 38.46 ± 8% -4.6 33.89 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state 40.30 ± 8% -4.5 35.84 ± 4% perf-profile.children.cycles-pp.start_secondary 4.34 ± 9% -3.0 1.33 ± 8% perf-profile.children.cycles-pp.__remove_mapping 5.60 ± 6% -2.7 2.87 ± 4% perf-profile.children.cycles-pp.pageout 3.99 ± 5% -2.1 1.84 ± 3% perf-profile.children.cycles-pp.add_to_swap 4.12 ± 5% -1.8 2.29 ± 6% perf-profile.children.cycles-pp.llist_add_batch 3.74 ± 6% -1.8 1.98 ± 4% perf-profile.children.cycles-pp.__swap_writepage 1.78 ± 12% -1.4 0.38 ± 13% perf-profile.children.cycles-pp.mem_cgroup_swapout 1.64 ± 6% -1.1 0.55 ± 4% perf-profile.children.cycles-pp.get_swap_page 3.25 ± 11% -1.0 2.28 ± 11% perf-profile.children.cycles-pp.mem_cgroup_charge 1.57 ± 6% -0.9 0.66 ± 6% perf-profile.children.cycles-pp.swap_writepage 3.90 ± 6% -0.9 3.01 ± 11% perf-profile.children.cycles-pp.do_rw_once 5.60 ± 7% -0.8 4.78 ± 4% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 1.09 ± 19% -0.8 0.29 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 1.35 ± 6% -0.8 0.57 ± 5% perf-profile.children.cycles-pp._swap_info_get 1.16 ± 5% -0.8 0.39 ± 8% perf-profile.children.cycles-pp.page_swap_info 5.12 ± 7% -0.7 4.37 ± 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 1.14 ± 5% -0.7 0.39 ± 8% perf-profile.children.cycles-pp.swap_set_page_dirty 2.02 ± 11% -0.7 1.30 ± 9% perf-profile.children.cycles-pp.__mem_cgroup_charge 0.88 ± 13% -0.7 0.19 ± 18% perf-profile.children.cycles-pp.page_counter_uncharge 2.00 ± 10% -0.6 1.36 ± 11% perf-profile.children.cycles-pp.get_page_from_freelist 0.79 ± 13% -0.6 0.16 ± 18% perf-profile.children.cycles-pp.page_counter_cancel 1.26 ± 2% -0.6 0.64 ± 7% perf-profile.children.cycles-pp.handle_irq_event 1.27 ± 2% -0.6 0.65 ± 7% perf-profile.children.cycles-pp.__common_interrupt 1.25 ± 2% -0.6 0.64 ± 7% perf-profile.children.cycles-pp.handle_irq_event_percpu 1.26 ± 2% -0.6 0.65 ± 7% perf-profile.children.cycles-pp.handle_edge_irq 1.25 ± 2% -0.6 0.64 ± 7% perf-profile.children.cycles-pp.__handle_irq_event_percpu 1.24 ± 2% -0.6 0.63 ± 7% perf-profile.children.cycles-pp.nvme_irq 1.28 ± 2% -0.6 0.67 ± 6% perf-profile.children.cycles-pp.common_interrupt 1.28 ± 2% -0.6 0.67 ± 6% perf-profile.children.cycles-pp.asm_common_interrupt 0.93 ± 5% -0.6 0.34 ± 5% perf-profile.children.cycles-pp.swap_page_sector 1.05 ± 7% -0.6 0.48 ± 4% perf-profile.children.cycles-pp.try_to_free_swap 1.02 ± 7% -0.6 0.46 ± 5% perf-profile.children.cycles-pp.page_swapcount 3.59 ± 8% -0.5 3.05 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 3.56 ± 8% -0.5 3.04 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt 1.10 ± 2% -0.5 0.58 ± 8% perf-profile.children.cycles-pp.blk_update_request 1.11 ± 2% -0.5 0.59 ± 7% perf-profile.children.cycles-pp.blk_mq_end_request 0.70 ± 9% -0.5 0.20 ± 10% perf-profile.children.cycles-pp.swap_duplicate 0.70 ± 10% -0.5 0.19 ± 11% perf-profile.children.cycles-pp.__swap_duplicate 1.32 ± 5% -0.5 0.83 ± 2% perf-profile.children.cycles-pp.add_to_swap_cache 0.69 ± 9% -0.5 0.23 ± 5% perf-profile.children.cycles-pp.get_swap_pages 1.21 ± 11% -0.4 0.77 ± 12% perf-profile.children.cycles-pp.prep_new_page 1.13 ± 11% -0.4 0.71 ± 12% perf-profile.children.cycles-pp.clear_page_erms 0.93 ± 12% -0.4 0.52 ± 11% perf-profile.children.cycles-pp.try_charge 0.84 ± 5% -0.4 0.44 ± 4% perf-profile.children.cycles-pp.bio_alloc_bioset 1.12 ± 8% -0.4 0.74 ± 8% perf-profile.children.cycles-pp.submit_bio 0.45 ± 22% -0.4 0.06 ± 11% perf-profile.children.cycles-pp.swap_cgroup_record 1.11 ± 7% -0.4 0.72 ± 8% perf-profile.children.cycles-pp.submit_bio_noacct 0.66 ± 9% -0.4 0.29 ± 8% perf-profile.children.cycles-pp.put_swap_page 0.67 ± 6% -0.4 0.30 ± 7% perf-profile.children.cycles-pp.__delete_from_swap_cache 0.84 ± 8% -0.4 0.47 ± 8% perf-profile.children.cycles-pp.page_lock_anon_vma_read 0.57 ± 8% -0.4 0.20 ± 5% perf-profile.children.cycles-pp.scan_swap_map_slots 0.51 ± 3% -0.3 0.18 ± 11% perf-profile.children.cycles-pp.__frontswap_store 0.65 ± 3% -0.3 0.32 ± 9% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.72 ± 15% -0.3 0.39 ± 11% perf-profile.children.cycles-pp._find_next_bit 0.59 ± 8% -0.3 0.28 ± 7% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 0.80 ± 4% -0.3 0.48 ± 8% perf-profile.children.cycles-pp.end_swap_bio_write 0.79 ± 4% -0.3 0.48 ± 8% perf-profile.children.cycles-pp.end_page_writeback 0.61 ± 5% -0.3 0.31 ± 7% perf-profile.children.cycles-pp.native_flush_tlb_multi 0.38 ± 8% -0.3 0.09 ± 12% perf-profile.children.cycles-pp.mem_cgroup_id_get_online 0.72 ± 7% -0.3 0.43 ± 5% perf-profile.children.cycles-pp.blk_mq_submit_bio 0.64 ± 11% -0.3 0.35 ± 7% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics 0.47 ± 8% -0.3 0.21 ± 7% perf-profile.children.cycles-pp.sysvec_call_function_single 0.44 ± 7% -0.3 0.19 ± 7% perf-profile.children.cycles-pp.__sysvec_call_function_single 0.52 ± 9% -0.2 0.27 ± 4% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.56 ± 4% -0.2 0.33 ± 6% perf-profile.children.cycles-pp.bio_init 0.28 ± 13% -0.2 0.06 ± 13% perf-profile.children.cycles-pp.get_swap_device 0.36 ± 26% -0.2 0.15 ± 26% perf-profile.children.cycles-pp.cpumask_any_but 0.54 ± 5% -0.2 0.36 ± 4% perf-profile.children.cycles-pp.__mod_lruvec_page_state 0.27 ± 11% -0.2 0.09 ± 12% perf-profile.children.cycles-pp.scan_swap_map_try_ssd_cluster 0.44 ± 8% -0.2 0.26 ± 4% perf-profile.children.cycles-pp.isolate_lru_pages 0.38 ± 8% -0.2 0.20 ± 8% perf-profile.children.cycles-pp.__set_page_dirty_no_writeback 0.42 ± 7% -0.2 0.25 ± 10% perf-profile.children.cycles-pp.__list_del_entry_valid 0.27 ± 8% -0.2 0.11 ± 10% perf-profile.children.cycles-pp.mempool_alloc 0.43 ± 6% -0.2 0.26 ± 10% perf-profile.children.cycles-pp.rotate_reclaimable_page 0.30 ± 8% -0.2 0.14 ± 8% perf-profile.children.cycles-pp.blk_mq_flush_plug_list 0.29 ± 8% -0.2 0.13 ± 5% perf-profile.children.cycles-pp.blk_mq_sched_insert_requests 0.39 ± 10% -0.2 0.23 ± 4% perf-profile.children.cycles-pp.__test_set_page_writeback 0.30 ± 8% -0.2 0.14 ± 6% perf-profile.children.cycles-pp.blk_flush_plug_list 0.29 ± 9% -0.2 0.13 ± 6% perf-profile.children.cycles-pp.nvme_queue_rq 0.28 ± 8% -0.2 0.13 ± 5% perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly 0.28 ± 8% -0.2 0.13 ± 6% perf-profile.children.cycles-pp.__blk_mq_try_issue_directly 0.28 ± 9% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.blk_mq_request_issue_directly 0.48 ± 7% -0.1 0.33 ± 4% perf-profile.children.cycles-pp.cpumask_next 0.25 ± 8% -0.1 0.10 ± 9% perf-profile.children.cycles-pp.nvme_map_data 0.38 ± 5% -0.1 0.24 ± 9% perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.40 ± 7% -0.1 0.26 ± 5% perf-profile.children.cycles-pp.rmqueue 0.21 ± 8% -0.1 0.08 ± 20% perf-profile.children.cycles-pp.kmem_cache_alloc 0.34 ± 7% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.free_unref_page_list 0.34 ± 5% -0.1 0.23 ± 7% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.46 ± 11% -0.1 0.35 ± 9% perf-profile.children.cycles-pp.lru_cache_add 0.30 ± 13% -0.1 0.20 ± 10% perf-profile.children.cycles-pp.rcu_read_unlock_strict 0.23 ± 6% -0.1 0.12 ± 9% perf-profile.children.cycles-pp.rmqueue_bulk 0.42 ± 10% -0.1 0.32 ± 10% perf-profile.children.cycles-pp.__pagevec_lru_add 0.62 ± 5% -0.1 0.53 ± 7% perf-profile.children.cycles-pp.__softirqentry_text_start 0.12 ± 9% -0.1 0.03 ±100% perf-profile.children.cycles-pp.kmem_cache_free 0.21 ± 5% -0.1 0.13 ± 8% perf-profile.children.cycles-pp.xas_store 0.23 ± 7% -0.1 0.14 ± 14% perf-profile.children.cycles-pp.pagevec_move_tail_fn 0.22 ± 5% -0.1 0.14 ± 3% perf-profile.children.cycles-pp.__mod_node_page_state 0.24 ± 3% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.xas_create 0.23 ± 10% -0.1 0.15 ± 4% perf-profile.children.cycles-pp.free_pcppages_bulk 0.33 ± 8% -0.1 0.25 ± 13% perf-profile.children.cycles-pp.submit_bio_checks 0.24 ± 8% -0.1 0.17 ± 9% perf-profile.children.cycles-pp.blk_attempt_plug_merge 0.21 ± 5% -0.1 0.14 ± 9% perf-profile.children.cycles-pp.test_clear_page_writeback 0.20 ± 6% -0.1 0.14 ± 8% perf-profile.children.cycles-pp.__mod_lruvec_state 0.18 ± 2% -0.1 0.11 ± 4% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.17 ± 8% -0.1 0.11 ± 11% perf-profile.children.cycles-pp.page_add_new_anon_rmap 0.09 ± 14% -0.1 0.03 ±101% perf-profile.children.cycles-pp.__count_memcg_events 0.19 ± 11% -0.1 0.13 ± 6% perf-profile.children.cycles-pp.page_mapping 0.20 ± 6% -0.1 0.14 ± 11% perf-profile.children.cycles-pp.load_balance 0.15 ± 10% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.page_remove_rmap 0.16 ± 2% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.update_sd_lb_stats 0.18 ± 11% -0.1 0.12 ± 10% perf-profile.children.cycles-pp.bio_attempt_back_merge 0.16 ± 4% -0.0 0.11 ± 14% perf-profile.children.cycles-pp.find_busiest_group 0.13 ± 8% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.mutex_unlock 0.07 ± 9% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.__mod_memcg_state 0.07 ± 7% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.release_pages 0.10 ± 17% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.lock_page_lruvec_irqsave 0.06 ± 11% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.mutex_lock 0.11 ± 6% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.memset_erms 0.14 ± 10% -0.0 0.10 ± 19% perf-profile.children.cycles-pp.page_counter_try_charge 0.10 ± 11% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.___might_sleep 0.10 ± 9% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.10 ± 10% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.blk_throtl_bio 0.08 ± 15% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.rcu_idle_exit 0.08 ± 6% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.__blk_queue_split 0.11 ± 9% -0.0 0.08 ± 15% perf-profile.children.cycles-pp.__perf_sw_event 0.11 ± 7% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.ll_back_merge_fn 0.09 ± 7% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.__cond_resched 0.12 ± 7% -0.0 0.09 ± 12% perf-profile.children.cycles-pp.update_rq_clock 0.08 ± 16% -0.0 0.05 ± 47% perf-profile.children.cycles-pp.bio_associate_blkg_from_css 0.07 ± 10% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.___perf_sw_event 0.10 ± 10% -0.0 0.08 ± 4% perf-profile.children.cycles-pp.hrtimer_update_next_event 0.09 ± 7% -0.0 0.07 ± 16% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.07 ± 7% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.update_irq_load_avg 0.09 ± 7% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.08 ± 10% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.update_blocked_averages 1.01 ± 7% +0.2 1.20 ± 6% perf-profile.children.cycles-pp.native_flush_tlb_local 2.16 ± 7% +0.4 2.54 ± 2% perf-profile.children.cycles-pp.default_send_IPI_mask_sequence_phys 1.59 ± 8% +0.6 2.15 ± 3% perf-profile.children.cycles-pp.__default_send_IPI_dest_field 42.06 ± 5% +8.5 50.53 ± 2% perf-profile.children.cycles-pp.shrink_node 42.00 ± 5% +8.5 50.49 ± 2% perf-profile.children.cycles-pp.shrink_lruvec 41.98 ± 5% +8.5 50.48 ± 2% perf-profile.children.cycles-pp.shrink_inactive_list 40.95 ± 5% +8.9 49.83 ± 2% perf-profile.children.cycles-pp.shrink_page_list 5.08 ± 8% +11.0 16.05 ± 3% perf-profile.children.cycles-pp.try_to_unmap_one 5.86 ± 8% +11.0 16.84 ± 3% perf-profile.children.cycles-pp.try_to_unmap 2.15 ± 12% +13.5 15.64 ± 3% perf-profile.children.cycles-pp.page_referenced 1.30 ± 17% +13.7 15.00 ± 3% perf-profile.children.cycles-pp.page_referenced_one 7.62 ± 9% +24.3 31.90 ± 3% perf-profile.children.cycles-pp.rmap_walk_anon 1.58 ± 18% +24.6 26.23 ± 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 2.42 ± 19% +25.7 28.14 ± 3% perf-profile.children.cycles-pp._raw_spin_lock 1.80 ± 27% +27.0 28.78 ± 3% perf-profile.children.cycles-pp.page_vma_mapped_walk 8.26 ± 8% -4.0 4.26 ± 5% perf-profile.self.cycles-pp.smp_call_function_many_cond 3.24 ± 6% -1.1 2.10 ± 9% perf-profile.self.cycles-pp.do_rw_once 2.97 ± 8% -1.0 1.96 ± 10% perf-profile.self.cycles-pp.do_access 2.96 ± 6% -1.0 1.95 ± 6% perf-profile.self.cycles-pp.llist_add_batch 1.73 ± 9% -0.7 0.98 ± 8% perf-profile.self.cycles-pp.try_to_unmap_one 0.79 ± 13% -0.6 0.16 ± 18% perf-profile.self.cycles-pp.page_counter_cancel 1.10 ± 6% -0.6 0.50 ± 5% perf-profile.self.cycles-pp._swap_info_get 0.89 ± 6% -0.6 0.33 ± 8% perf-profile.self.cycles-pp.page_swap_info 0.73 ± 6% -0.5 0.27 ± 5% perf-profile.self.cycles-pp.swap_page_sector 0.98 ± 12% -0.4 0.53 ± 11% perf-profile.self.cycles-pp.clear_page_erms 0.59 ± 6% -0.4 0.17 ± 6% perf-profile.self.cycles-pp.get_swap_page 0.69 ± 13% -0.4 0.31 ± 12% perf-profile.self.cycles-pp.try_charge 2.74 ± 7% -0.3 2.40 ± 7% perf-profile.self.cycles-pp.flush_tlb_func 0.38 ± 8% -0.3 0.09 ± 10% perf-profile.self.cycles-pp.mem_cgroup_id_get_online 0.74 ± 7% -0.3 0.46 ± 4% perf-profile.self.cycles-pp.add_to_swap_cache 0.45 ± 9% -0.3 0.18 ± 10% perf-profile.self.cycles-pp.__delete_from_swap_cache 0.52 ± 10% -0.3 0.26 ± 5% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.42 ± 10% -0.2 0.19 ± 12% perf-profile.self.cycles-pp.mem_cgroup_charge 0.55 ± 16% -0.2 0.33 ± 12% perf-profile.self.cycles-pp._find_next_bit 0.32 ± 4% -0.2 0.11 ± 9% perf-profile.self.cycles-pp.__frontswap_store 0.46 ± 4% -0.2 0.27 ± 8% perf-profile.self.cycles-pp.bio_init 0.32 ± 5% -0.2 0.16 ± 9% perf-profile.self.cycles-pp.native_flush_tlb_multi 0.40 ± 7% -0.2 0.24 ± 10% perf-profile.self.cycles-pp.__list_del_entry_valid 0.31 ± 16% -0.1 0.17 ± 21% perf-profile.self.cycles-pp.page_lock_anon_vma_read 0.19 ± 13% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.scan_swap_map_slots 0.28 ± 9% -0.1 0.17 ± 11% perf-profile.self.cycles-pp.__set_page_dirty_no_writeback 0.36 ± 5% -0.1 0.25 ± 3% perf-profile.self.cycles-pp.shrink_page_list 0.33 ± 8% -0.1 0.22 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.26 ± 8% -0.1 0.16 ± 4% perf-profile.self.cycles-pp.isolate_lru_pages 0.56 ± 6% -0.1 0.47 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state 0.29 ± 12% -0.1 0.21 ± 17% perf-profile.self.cycles-pp.get_page_from_freelist 0.16 ± 6% -0.1 0.08 ± 9% perf-profile.self.cycles-pp.end_page_writeback 0.29 ± 11% -0.1 0.21 ± 12% perf-profile.self.cycles-pp.__mem_cgroup_charge 0.20 ± 5% -0.1 0.13 ± 6% perf-profile.self.cycles-pp.__mod_node_page_state 0.22 ± 3% -0.1 0.15 ± 6% perf-profile.self.cycles-pp.xas_create 0.23 ± 8% -0.1 0.16 ± 7% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state 0.13 ± 9% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.rmqueue_bulk 0.17 ± 9% -0.1 0.10 ± 9% perf-profile.self.cycles-pp.__mod_lruvec_page_state 0.14 ± 8% -0.1 0.08 ± 7% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.15 ± 8% -0.1 0.09 ± 7% perf-profile.self.cycles-pp.__test_set_page_writeback 0.08 ± 9% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.__swap_writepage 0.13 ± 12% -0.1 0.08 ± 9% perf-profile.self.cycles-pp.free_pcppages_bulk 0.11 ± 12% -0.0 0.06 ± 47% perf-profile.self.cycles-pp.page_counter_try_charge 0.15 ± 11% -0.0 0.11 ± 7% perf-profile.self.cycles-pp.page_mapping 0.12 ± 12% -0.0 0.08 ± 19% perf-profile.self.cycles-pp.__pagevec_lru_add 0.08 ± 8% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.pagevec_lru_move_fn 0.12 ± 6% -0.0 0.08 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.10 ± 10% -0.0 0.07 ± 16% perf-profile.self.cycles-pp.pagevec_move_tail_fn 0.10 ± 10% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.mutex_unlock 0.08 ± 19% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.___might_sleep 0.11 ± 8% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.update_sd_lb_stats 0.09 ± 13% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.08 ± 7% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.memset_erms 0.07 ± 11% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.rotate_reclaimable_page 0.06 ± 7% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.update_irq_load_avg 0.08 ± 4% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.ll_back_merge_fn 0.16 ± 5% +0.1 0.21 ± 5% perf-profile.self.cycles-pp.asm_sysvec_call_function 1.01 ± 7% +0.2 1.20 ± 6% perf-profile.self.cycles-pp.native_flush_tlb_local 2.22 ± 6% +0.4 2.59 ± 4% perf-profile.self.cycles-pp.flush_smp_call_function_queue 1.28 ± 5% +0.4 1.72 perf-profile.self.cycles-pp._raw_spin_lock 1.59 ± 8% +0.6 2.15 ± 3% perf-profile.self.cycles-pp.__default_send_IPI_dest_field 0.34 ± 35% +0.6 0.93 ± 8% perf-profile.self.cycles-pp.page_vma_mapped_walk 1.35 ± 14% +20.2 21.54 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 11952 ± 12% -27.8% 8625 ± 13% softirqs.CPU0.RCU 32374 ± 3% +30.1% 42116 ± 2% softirqs.CPU0.SCHED 10010 ± 15% -33.0% 6705 ± 17% softirqs.CPU1.RCU 29521 ± 3% +25.7% 37120 ± 8% softirqs.CPU1.SCHED 9560 ± 18% -35.2% 6197 ± 12% softirqs.CPU10.RCU 9641 ± 15% -42.8% 5512 ± 13% softirqs.CPU100.RCU 27690 ± 2% +31.8% 36488 ± 3% softirqs.CPU100.SCHED 9404 ± 16% -37.4% 5884 ± 14% softirqs.CPU101.RCU 26702 ± 4% +39.4% 37210 ± 4% softirqs.CPU101.SCHED 9636 ± 16% -40.6% 5728 ± 15% softirqs.CPU102.RCU 26856 ± 6% +37.4% 36889 ± 4% softirqs.CPU102.SCHED 9761 ± 16% -42.7% 5591 ± 14% softirqs.CPU103.RCU 26740 ± 3% +32.4% 35414 ± 8% softirqs.CPU103.SCHED 9279 ± 16% -34.8% 6049 ± 14% softirqs.CPU104.RCU 26363 ± 5% +36.1% 35872 ± 9% softirqs.CPU104.SCHED 9410 ± 16% -38.5% 5787 ± 15% softirqs.CPU105.RCU 26533 ± 9% +41.5% 37539 ± 3% softirqs.CPU105.SCHED 9764 ± 26% -39.6% 5894 ± 9% softirqs.CPU106.RCU 26912 ± 4% +38.8% 37354 ± 5% softirqs.CPU106.SCHED 9551 ± 18% -39.0% 5824 ± 12% softirqs.CPU107.RCU 27262 ± 5% +28.8% 35113 ± 4% softirqs.CPU107.SCHED 9607 ± 17% -37.9% 5968 ± 16% softirqs.CPU108.RCU 26904 ± 3% +34.2% 36105 ± 3% softirqs.CPU108.SCHED 9485 ± 18% -43.7% 5341 ± 7% softirqs.CPU109.RCU 26209 ± 6% +38.5% 36295 ± 2% softirqs.CPU109.SCHED 28056 ± 6% +36.9% 38396 ± 5% softirqs.CPU11.SCHED 9807 ± 14% -39.4% 5942 ± 11% softirqs.CPU110.RCU 26349 ± 4% +36.5% 35954 ± 5% softirqs.CPU110.SCHED 9271 ± 15% -40.6% 5506 ± 11% softirqs.CPU111.RCU 26559 ± 8% +36.5% 36243 ± 4% softirqs.CPU111.SCHED 9830 ± 20% -41.0% 5804 ± 14% softirqs.CPU112.RCU 26422 ± 5% +36.5% 36058 ± 7% softirqs.CPU112.SCHED 9658 ± 19% -37.0% 6088 ± 20% softirqs.CPU113.RCU 26912 ± 2% +38.2% 37204 ± 5% softirqs.CPU113.SCHED 9835 ± 17% -41.0% 5798 ± 13% softirqs.CPU114.RCU 27485 ± 2% +32.6% 36438 ± 7% softirqs.CPU114.SCHED 9878 ± 16% -38.1% 6114 ± 16% softirqs.CPU115.RCU 27028 ± 3% +34.2% 36262 ± 4% softirqs.CPU115.SCHED 9542 ± 17% -37.5% 5967 ± 11% softirqs.CPU116.RCU 26216 ± 7% +43.6% 37650 ± 4% softirqs.CPU116.SCHED 9457 ± 18% -39.8% 5691 ± 12% softirqs.CPU117.RCU 27605 ± 5% +34.0% 36982 ± 3% softirqs.CPU117.SCHED 9567 ± 17% -41.5% 5594 ± 15% softirqs.CPU118.RCU 28212 ± 2% +31.3% 37047 ± 5% softirqs.CPU118.SCHED 9858 ± 16% -39.7% 5943 ± 13% softirqs.CPU119.RCU 28244 ± 4% +32.8% 37516 ± 4% softirqs.CPU119.SCHED 10429 ± 16% -31.2% 7175 ± 16% softirqs.CPU12.RCU 27864 ± 3% +33.1% 37090 ± 11% softirqs.CPU12.SCHED 9672 ± 15% -38.5% 5944 ± 13% softirqs.CPU120.RCU 27469 ± 3% +25.6% 34512 ± 9% softirqs.CPU120.SCHED 9419 ± 17% -37.5% 5884 ± 16% softirqs.CPU121.RCU 26369 ± 7% +24.5% 32820 ± 7% softirqs.CPU121.SCHED 9627 ± 17% -41.6% 5626 ± 14% softirqs.CPU122.RCU 27261 ± 4% +37.2% 37408 ± 5% softirqs.CPU122.SCHED 9753 ± 17% -40.5% 5799 ± 12% softirqs.CPU123.RCU 27689 ± 2% +32.5% 36691 ± 7% softirqs.CPU123.SCHED 10024 ± 18% -40.7% 5942 ± 11% softirqs.CPU124.RCU 27738 ± 2% +31.2% 36403 ± 6% softirqs.CPU124.SCHED 9669 ± 15% -38.7% 5930 ± 14% softirqs.CPU125.RCU 27931 ± 2% +29.1% 36046 ± 7% softirqs.CPU125.SCHED 9931 ± 20% -41.1% 5849 ± 13% softirqs.CPU126.RCU 27307 ± 3% +33.4% 36414 ± 7% softirqs.CPU126.SCHED 9518 ± 14% -39.6% 5745 ± 17% softirqs.CPU127.RCU 27125 ± 5% +32.5% 35941 ± 9% softirqs.CPU127.SCHED 9714 ± 16% -39.9% 5837 ± 10% softirqs.CPU128.RCU 27767 ± 3% +32.7% 36842 ± 4% softirqs.CPU128.SCHED 9659 ± 14% -40.8% 5718 ± 10% softirqs.CPU129.RCU 27759 ± 3% +30.9% 36346 ± 4% softirqs.CPU129.SCHED 10789 ± 26% -46.1% 5820 ± 9% softirqs.CPU13.RCU 26247 ± 15% +45.6% 38218 ± 4% softirqs.CPU13.SCHED 9600 ± 15% -38.2% 5930 ± 16% softirqs.CPU130.RCU 27262 ± 4% +34.1% 36562 ± 6% softirqs.CPU130.SCHED 10393 ± 20% -45.2% 5699 ± 14% softirqs.CPU131.RCU 27624 ± 4% +33.4% 36844 ± 3% softirqs.CPU131.SCHED 9593 ± 14% -40.2% 5737 ± 14% softirqs.CPU132.RCU 26882 ± 9% +34.5% 36151 ± 2% softirqs.CPU132.SCHED 9524 ± 17% -39.6% 5749 ± 12% softirqs.CPU133.RCU 27165 ± 5% +31.0% 35586 ± 7% softirqs.CPU133.SCHED 9714 ± 12% -41.1% 5726 ± 14% softirqs.CPU134.RCU 27285 ± 3% +29.3% 35284 ± 5% softirqs.CPU134.SCHED 9561 ± 13% -38.3% 5901 ± 19% softirqs.CPU135.RCU 27567 ± 3% +28.9% 35546 ± 7% softirqs.CPU135.SCHED 9577 ± 14% -40.4% 5711 ± 12% softirqs.CPU136.RCU 28030 ± 3% +31.5% 36846 ± 5% softirqs.CPU136.SCHED 9717 ± 16% -40.6% 5770 ± 15% softirqs.CPU137.RCU 28225 ± 3% +31.7% 37183 ± 6% softirqs.CPU137.SCHED 9676 ± 14% -41.9% 5624 ± 13% softirqs.CPU138.RCU 27095 ± 3% +38.9% 37642 ± 6% softirqs.CPU138.SCHED 9678 ± 17% -40.0% 5805 ± 16% softirqs.CPU139.RCU 27639 ± 2% +29.6% 35811 ± 10% softirqs.CPU139.SCHED 9842 ± 16% -40.1% 5894 ± 11% softirqs.CPU14.RCU 25932 ± 13% +47.9% 38346 ± 3% softirqs.CPU14.SCHED 9769 ± 14% -40.6% 5807 ± 14% softirqs.CPU140.RCU 28046 ± 3% +29.1% 36200 ± 8% softirqs.CPU140.SCHED 9640 ± 15% -39.3% 5847 ± 15% softirqs.CPU141.RCU 9629 ± 13% -38.5% 5924 ± 12% softirqs.CPU142.RCU 27220 ± 3% +34.6% 36638 ± 5% softirqs.CPU142.SCHED 9691 ± 16% -40.7% 5750 ± 13% softirqs.CPU143.RCU 27774 ± 3% +31.3% 36455 ± 5% softirqs.CPU143.SCHED 9922 ± 17% -38.1% 6143 ± 12% softirqs.CPU144.RCU 26996 ± 4% +27.2% 34330 ± 4% softirqs.CPU144.SCHED 9626 ± 17% -40.9% 5693 ± 13% softirqs.CPU145.RCU 9475 ± 15% -41.8% 5519 ± 11% softirqs.CPU146.RCU 26918 ± 9% +35.7% 36521 ± 6% softirqs.CPU146.SCHED 10026 ± 13% -43.5% 5660 ± 14% softirqs.CPU147.RCU 27369 ± 3% +33.8% 36627 ± 5% softirqs.CPU147.SCHED 10079 ± 12% -42.3% 5813 ± 11% softirqs.CPU148.RCU 26659 ± 2% +41.8% 37791 ± 2% softirqs.CPU148.SCHED 9605 ± 14% -39.7% 5796 ± 12% softirqs.CPU149.RCU 27454 ± 2% +36.1% 37356 ± 7% softirqs.CPU149.SCHED 9534 ± 17% -36.3% 6071 ± 10% softirqs.CPU15.RCU 28826 ± 2% +33.7% 38554 ± 4% softirqs.CPU15.SCHED 9821 ± 14% -35.8% 6309 ± 29% softirqs.CPU150.RCU 27391 ± 4% +34.6% 36880 ± 6% softirqs.CPU150.SCHED 9559 ± 15% -41.6% 5585 ± 11% softirqs.CPU151.RCU 26371 ± 2% +37.9% 36354 ± 6% softirqs.CPU151.SCHED 9662 ± 14% -38.1% 5984 ± 16% softirqs.CPU152.RCU 26885 ± 4% +30.6% 35100 ± 10% softirqs.CPU152.SCHED 9520 ± 14% -40.5% 5664 ± 13% softirqs.CPU153.RCU 27614 ± 4% +33.9% 36988 ± 5% softirqs.CPU153.SCHED 9494 ± 14% -40.8% 5622 ± 13% softirqs.CPU154.RCU 27383 +31.7% 36070 ± 7% softirqs.CPU154.SCHED 9623 ± 13% -40.8% 5699 ± 12% softirqs.CPU155.RCU 28108 ± 2% +28.0% 35984 ± 8% softirqs.CPU155.SCHED 9297 ± 15% -38.2% 5747 ± 14% softirqs.CPU156.RCU 28227 ± 4% +28.6% 36312 ± 6% softirqs.CPU156.SCHED 9421 ± 13% -41.2% 5535 ± 15% softirqs.CPU157.RCU 26196 ± 11% +38.9% 36389 ± 6% softirqs.CPU157.SCHED 9572 ± 15% -42.3% 5518 ± 14% softirqs.CPU158.RCU 27901 ± 2% +31.3% 36647 ± 7% softirqs.CPU158.SCHED 9195 ± 17% -38.7% 5634 ± 17% softirqs.CPU159.RCU 27962 ± 2% +30.4% 36469 ± 6% softirqs.CPU159.SCHED 9822 ± 15% -32.6% 6617 ± 18% softirqs.CPU16.RCU 26952 ± 5% +39.9% 37703 ± 8% softirqs.CPU16.SCHED 9209 ± 20% -41.9% 5355 ± 13% softirqs.CPU160.RCU 27793 ± 2% +29.8% 36073 ± 9% softirqs.CPU160.SCHED 9543 ± 16% -40.5% 5678 ± 13% softirqs.CPU161.RCU 28608 ± 2% +28.7% 36806 ± 3% softirqs.CPU161.SCHED 9399 ± 17% -39.8% 5662 ± 12% softirqs.CPU162.RCU 28119 ± 2% +31.9% 37092 ± 8% softirqs.CPU162.SCHED 9334 ± 17% -40.3% 5568 ± 13% softirqs.CPU163.RCU 28083 +32.2% 37136 ± 5% softirqs.CPU163.SCHED 9558 ± 14% -43.1% 5438 ± 11% softirqs.CPU164.RCU 28339 ± 2% +30.8% 37066 ± 7% softirqs.CPU164.SCHED 9522 ± 15% -42.4% 5485 ± 12% softirqs.CPU165.RCU 28064 ± 3% +32.6% 37204 ± 7% softirqs.CPU165.SCHED 9509 ± 17% -41.0% 5609 ± 15% softirqs.CPU166.RCU 27120 ± 8% +33.2% 36135 ± 7% softirqs.CPU166.SCHED 9335 ± 16% -41.2% 5485 ± 11% softirqs.CPU167.RCU 28405 ± 2% +27.8% 36306 ± 3% softirqs.CPU167.SCHED 25148 ± 4% +38.7% 34890 ± 5% softirqs.CPU168.SCHED 8770 ± 17% -40.3% 5237 ± 10% softirqs.CPU169.RCU 10241 ± 25% -36.9% 6461 ± 16% softirqs.CPU17.RCU 26802 ± 6% +34.2% 35979 ± 10% softirqs.CPU17.SCHED 9208 ± 15% -42.2% 5318 ± 11% softirqs.CPU170.RCU 27330 ± 4% +23.0% 33604 ± 10% softirqs.CPU170.SCHED 8924 ± 17% -39.5% 5400 ± 9% softirqs.CPU171.RCU 28100 ± 3% +31.5% 36959 ± 4% softirqs.CPU171.SCHED 8924 ± 17% -38.0% 5529 ± 10% softirqs.CPU172.RCU 28259 ± 3% +31.1% 37040 ± 4% softirqs.CPU172.SCHED 8875 ± 16% -41.0% 5232 ± 13% softirqs.CPU173.RCU 28254 ± 2% +32.8% 37523 ± 4% softirqs.CPU173.SCHED 9176 ± 15% -42.1% 5317 ± 13% softirqs.CPU174.RCU 27499 ± 3% +38.5% 38081 ± 3% softirqs.CPU174.SCHED 8654 ± 15% -38.4% 5333 ± 13% softirqs.CPU175.RCU 27692 ± 2% +35.4% 37507 ± 3% softirqs.CPU175.SCHED 9225 ± 14% -41.6% 5389 ± 15% softirqs.CPU176.RCU 27375 ± 4% +34.9% 36923 ± 8% softirqs.CPU176.SCHED 9289 ± 15% -43.5% 5249 ± 13% softirqs.CPU177.RCU 28263 ± 3% +32.0% 37296 ± 6% softirqs.CPU177.SCHED 9471 ± 14% -41.8% 5510 ± 15% softirqs.CPU178.RCU 28202 ± 3% +29.6% 36560 ± 5% softirqs.CPU178.SCHED 9263 ± 13% -40.8% 5486 ± 15% softirqs.CPU179.RCU 27974 ± 4% +32.7% 37116 ± 4% softirqs.CPU179.SCHED 9755 ± 20% -38.9% 5961 ± 14% softirqs.CPU18.RCU 28714 ± 2% +33.8% 38408 ± 5% softirqs.CPU18.SCHED 9532 ± 17% -42.7% 5461 ± 17% softirqs.CPU180.RCU 28337 ± 2% +29.8% 36778 ± 3% softirqs.CPU180.SCHED 8762 ± 15% -39.3% 5319 ± 14% softirqs.CPU181.RCU 27484 ± 3% +33.9% 36801 ± 2% softirqs.CPU181.SCHED 9105 ± 14% -39.8% 5483 ± 13% softirqs.CPU182.RCU 27679 ± 4% +33.4% 36912 ± 4% softirqs.CPU182.SCHED 9146 ± 16% -41.3% 5367 ± 15% softirqs.CPU183.RCU 9449 ± 14% -38.4% 5822 ± 25% softirqs.CPU184.RCU 28179 ± 2% +34.7% 37964 ± 3% softirqs.CPU184.SCHED 9703 ± 17% -42.4% 5590 ± 21% softirqs.CPU185.RCU 28451 +35.1% 38440 ± 3% softirqs.CPU185.SCHED 8740 ± 17% -40.4% 5210 ± 11% softirqs.CPU186.RCU 27970 ± 3% +33.9% 37441 ± 3% softirqs.CPU186.SCHED 9322 ± 15% -42.1% 5401 ± 14% softirqs.CPU187.RCU 9517 ± 11% -44.2% 5313 ± 14% softirqs.CPU188.RCU 27852 ± 3% +34.4% 37436 ± 5% softirqs.CPU188.SCHED 9375 ± 12% -42.7% 5375 ± 16% softirqs.CPU189.RCU 27896 ± 3% +34.7% 37573 ± 4% softirqs.CPU189.SCHED 9548 ± 17% -38.0% 5921 ± 12% softirqs.CPU19.RCU 28340 ± 2% +34.4% 38104 ± 5% softirqs.CPU19.SCHED 9329 ± 14% -41.7% 5437 ± 16% softirqs.CPU190.RCU 28512 ± 3% +30.6% 37248 ± 3% softirqs.CPU190.SCHED 9956 ± 12% -38.6% 6112 ± 20% softirqs.CPU191.RCU 9920 ± 17% -35.0% 6447 ± 12% softirqs.CPU2.RCU 9792 ± 16% -38.1% 6058 ± 24% softirqs.CPU20.RCU 9552 ± 17% -37.0% 6016 ± 18% softirqs.CPU21.RCU 28148 +36.7% 38489 ± 6% softirqs.CPU21.SCHED 10016 ± 14% -39.9% 6023 ± 17% softirqs.CPU22.RCU 10216 ± 25% -42.6% 5863 ± 11% softirqs.CPU23.RCU 28979 ± 3% +32.4% 38368 ± 8% softirqs.CPU23.SCHED 10358 ± 15% -38.4% 6381 ± 11% softirqs.CPU24.RCU 26220 ± 4% +37.7% 36110 ± 8% softirqs.CPU24.SCHED 9726 ± 15% -35.9% 6235 ± 14% softirqs.CPU25.RCU 10261 ± 14% -41.7% 5986 ± 14% softirqs.CPU26.RCU 27998 ± 6% +36.6% 38241 ± 6% softirqs.CPU26.SCHED 9785 ± 14% -35.1% 6354 ± 13% softirqs.CPU27.RCU 28204 ± 4% +31.8% 37162 ± 7% softirqs.CPU27.SCHED 10151 ± 16% -39.1% 6183 ± 17% softirqs.CPU28.RCU 27983 ± 4% +35.9% 38026 ± 6% softirqs.CPU28.SCHED 10092 ± 14% -40.2% 6036 ± 15% softirqs.CPU29.RCU 28593 ± 3% +34.5% 38465 ± 4% softirqs.CPU29.SCHED 10086 ± 18% -34.8% 6575 ± 26% softirqs.CPU3.RCU 27482 ± 7% +40.4% 38591 ± 3% softirqs.CPU3.SCHED 9883 ± 16% -37.0% 6229 ± 15% softirqs.CPU30.RCU 29738 ± 7% +28.3% 38155 ± 4% softirqs.CPU30.SCHED 9925 ± 17% -38.8% 6077 ± 16% softirqs.CPU31.RCU 10305 ± 14% -39.1% 6275 ± 12% softirqs.CPU32.RCU 28109 ± 3% +34.8% 37891 ± 4% softirqs.CPU32.SCHED 10590 ± 19% -40.9% 6259 ± 17% softirqs.CPU33.RCU 28261 ± 2% +26.6% 35778 ± 7% softirqs.CPU33.SCHED 10001 ± 16% -38.6% 6142 ± 12% softirqs.CPU34.RCU 28560 +35.0% 38550 ± 3% softirqs.CPU34.SCHED 10201 ± 15% -38.8% 6247 ± 12% softirqs.CPU35.RCU 28600 ± 3% +33.1% 38059 ± 3% softirqs.CPU35.SCHED 10116 ± 13% -38.1% 6260 ± 13% softirqs.CPU36.RCU 10122 ± 15% -40.3% 6039 ± 14% softirqs.CPU37.RCU 28116 ± 3% +37.2% 38569 ± 3% softirqs.CPU37.SCHED 10111 ± 13% -39.1% 6158 ± 14% softirqs.CPU38.RCU 28486 +33.3% 37962 ± 4% softirqs.CPU38.SCHED 10005 ± 14% -38.3% 6174 ± 15% softirqs.CPU39.RCU 9741 ± 16% -41.7% 5676 ± 7% softirqs.CPU4.RCU 28352 ± 3% +35.1% 38298 ± 4% softirqs.CPU4.SCHED 10184 ± 12% -40.0% 6115 ± 13% softirqs.CPU40.RCU 28728 ± 2% +33.1% 38233 ± 4% softirqs.CPU40.SCHED 10194 ± 13% -39.6% 6162 ± 13% softirqs.CPU41.RCU 28822 +31.2% 37804 ± 5% softirqs.CPU41.SCHED 10618 ± 18% -41.7% 6195 ± 13% softirqs.CPU42.RCU 28518 +36.7% 38997 ± 3% softirqs.CPU42.SCHED 10139 ± 15% -38.7% 6218 ± 14% softirqs.CPU43.RCU 28442 ± 2% +31.5% 37398 ± 8% softirqs.CPU43.SCHED 10142 ± 13% -39.8% 6108 ± 11% softirqs.CPU44.RCU 10157 ± 14% -35.7% 6529 ± 19% softirqs.CPU45.RCU 28301 ± 2% +31.6% 37252 ± 8% softirqs.CPU45.SCHED 10304 ± 16% -40.1% 6174 ± 18% softirqs.CPU46.RCU 28728 ± 2% +34.4% 38599 ± 4% softirqs.CPU46.SCHED 10163 ± 14% -39.6% 6136 ± 14% softirqs.CPU47.RCU 28510 ± 5% +33.3% 37991 ± 3% softirqs.CPU47.SCHED 10751 ± 15% -39.5% 6504 ± 18% softirqs.CPU48.RCU 24789 ± 10% +37.8% 34152 ± 5% softirqs.CPU48.SCHED 10001 ± 16% -39.8% 6017 ± 12% softirqs.CPU49.RCU 27400 ± 4% +28.0% 35059 ± 10% softirqs.CPU49.SCHED 9899 ± 16% -37.8% 6155 ± 11% softirqs.CPU5.RCU 25043 ± 26% +52.9% 38283 ± 4% softirqs.CPU5.SCHED 9941 ± 15% -40.9% 5880 ± 13% softirqs.CPU50.RCU 27850 ± 5% +33.9% 37300 ± 5% softirqs.CPU50.SCHED 10580 ± 13% -42.0% 6131 ± 13% softirqs.CPU51.RCU 28440 ± 2% +34.7% 38294 ± 4% softirqs.CPU51.SCHED 10532 ± 11% -41.7% 6144 ± 12% softirqs.CPU52.RCU 27522 ± 3% +40.1% 38549 ± 3% softirqs.CPU52.SCHED 10169 ± 14% -39.9% 6116 ± 11% softirqs.CPU53.RCU 28015 ± 2% +38.0% 38675 ± 6% softirqs.CPU53.SCHED 27943 ± 3% +36.6% 38162 ± 5% softirqs.CPU54.SCHED 9711 ± 15% -39.8% 5848 ± 12% softirqs.CPU55.RCU 28057 ± 2% +32.3% 37105 ± 6% softirqs.CPU55.SCHED 9893 ± 14% -39.3% 6004 ± 11% softirqs.CPU56.RCU 28221 ± 2% +34.3% 37914 ± 6% softirqs.CPU56.SCHED 10168 ± 16% -40.7% 6028 ± 12% softirqs.CPU57.RCU 28823 ± 2% +32.2% 38113 ± 6% softirqs.CPU57.SCHED 10014 ± 14% -39.8% 6033 ± 14% softirqs.CPU58.RCU 28179 ± 2% +33.0% 37485 ± 6% softirqs.CPU58.SCHED 10303 ± 13% -41.0% 6083 ± 15% softirqs.CPU59.RCU 29026 +32.1% 38345 ± 4% softirqs.CPU59.SCHED 10178 ± 14% -34.1% 6709 ± 13% softirqs.CPU6.RCU 29174 ± 10% +32.6% 38682 ± 3% softirqs.CPU6.SCHED 9753 ± 15% -37.9% 6057 ± 13% softirqs.CPU60.RCU 28862 ± 2% +32.8% 38320 ± 4% softirqs.CPU60.SCHED 9816 ± 13% -40.9% 5805 ± 14% softirqs.CPU61.RCU 28572 ± 2% +35.2% 38638 ± 3% softirqs.CPU61.SCHED 10000 ± 14% -41.2% 5876 ± 13% softirqs.CPU62.RCU 28515 ± 2% +34.6% 38377 ± 4% softirqs.CPU62.SCHED 9722 ± 16% -35.3% 6293 ± 18% softirqs.CPU63.RCU 28871 +32.9% 38371 ± 3% softirqs.CPU63.SCHED 9335 ± 17% -39.8% 5620 ± 13% softirqs.CPU64.RCU 9671 ± 16% -38.8% 5918 ± 11% softirqs.CPU65.RCU 9635 ± 16% -38.9% 5884 ± 10% softirqs.CPU66.RCU 9620 ± 17% -40.7% 5702 ± 11% softirqs.CPU67.RCU 23386 ± 35% +66.2% 38859 ± 3% softirqs.CPU67.SCHED 9840 ± 16% -42.5% 5661 ± 11% softirqs.CPU68.RCU 28877 ± 3% +31.9% 38083 ± 7% softirqs.CPU68.SCHED 9621 ± 16% -40.2% 5750 ± 10% softirqs.CPU69.RCU 28471 ± 4% +34.2% 38218 ± 4% softirqs.CPU69.SCHED 9500 ± 19% -37.7% 5920 ± 11% softirqs.CPU7.RCU 27971 ± 2% +27.6% 35679 ± 12% softirqs.CPU7.SCHED 9950 ± 19% -42.6% 5713 ± 11% softirqs.CPU70.RCU 28378 ± 4% +36.4% 38698 ± 4% softirqs.CPU70.SCHED 9769 ± 17% -41.5% 5712 ± 11% softirqs.CPU71.RCU 28400 ± 3% +35.0% 38334 ± 2% softirqs.CPU71.SCHED 10186 ± 23% -36.0% 6516 ± 11% softirqs.CPU72.RCU 29528 ± 5% +19.4% 35245 ± 3% softirqs.CPU72.SCHED 9117 ± 16% -37.7% 5682 ± 11% softirqs.CPU73.RCU 8864 ± 16% -37.6% 5527 ± 10% softirqs.CPU74.RCU 9191 ± 15% -40.4% 5474 ± 10% softirqs.CPU75.RCU 28423 ± 4% +28.2% 36424 ± 8% softirqs.CPU75.SCHED 9557 ± 20% -44.1% 5341 ± 10% softirqs.CPU76.RCU 28240 ± 3% +38.6% 39129 ± 3% softirqs.CPU76.SCHED 9108 ± 14% -40.6% 5408 ± 11% softirqs.CPU77.RCU 28923 ± 2% +35.5% 39203 ± 2% softirqs.CPU77.SCHED 9485 ± 16% -41.3% 5571 ± 13% softirqs.CPU78.RCU 28339 ± 2% +38.9% 39369 ± 3% softirqs.CPU78.SCHED 8889 ± 15% -39.2% 5401 ± 12% softirqs.CPU79.RCU 28802 ± 2% +35.8% 39116 ± 3% softirqs.CPU79.SCHED 28293 ± 2% +32.1% 37372 ± 7% softirqs.CPU8.SCHED 9386 ± 13% -40.2% 5612 ± 14% softirqs.CPU80.RCU 28492 ± 3% +36.6% 38911 ± 2% softirqs.CPU80.SCHED 9460 ± 14% -41.5% 5535 ± 13% softirqs.CPU81.RCU 28808 +33.5% 38470 ± 5% softirqs.CPU81.SCHED 9799 ± 14% -41.7% 5717 ± 15% softirqs.CPU82.RCU 28638 ± 3% +33.8% 38305 ± 3% softirqs.CPU82.SCHED 9574 ± 13% -40.1% 5731 ± 14% softirqs.CPU83.RCU 28754 +34.1% 38564 ± 3% softirqs.CPU83.SCHED 9488 ± 18% -41.6% 5541 ± 15% softirqs.CPU84.RCU 28843 +34.3% 38733 ± 4% softirqs.CPU84.SCHED 9072 ± 15% -39.8% 5465 ± 13% softirqs.CPU85.RCU 28412 ± 3% +37.4% 39033 ± 2% softirqs.CPU85.SCHED 9680 ± 17% -41.2% 5696 ± 15% softirqs.CPU86.RCU 28536 ± 2% +35.4% 38644 ± 2% softirqs.CPU86.SCHED 9251 ± 16% -37.9% 5749 ± 13% softirqs.CPU87.RCU 28704 +34.3% 38561 ± 3% softirqs.CPU87.SCHED 9761 ± 14% -40.2% 5842 ± 16% softirqs.CPU88.RCU 28799 ± 2% +36.4% 39277 ± 4% softirqs.CPU88.SCHED 9979 ± 14% -42.2% 5770 ± 14% softirqs.CPU89.RCU 28862 ± 3% +36.8% 39497 ± 3% softirqs.CPU89.SCHED 9789 ± 16% -40.8% 5798 ± 12% softirqs.CPU9.RCU 27851 ± 5% +39.6% 38881 ± 5% softirqs.CPU9.SCHED 8905 ± 14% -39.5% 5388 ± 11% softirqs.CPU90.RCU 28731 ± 2% +36.1% 39093 ± 4% softirqs.CPU90.SCHED 9570 ± 13% -38.1% 5924 ± 18% softirqs.CPU91.RCU 9375 ± 12% -41.8% 5455 ± 14% softirqs.CPU92.RCU 28905 +34.1% 38760 ± 5% softirqs.CPU92.SCHED 9198 ± 15% -35.9% 5893 ± 17% softirqs.CPU93.RCU 9560 ± 13% -37.5% 5976 ± 13% softirqs.CPU94.RCU 29218 ± 2% +28.9% 37660 ± 9% softirqs.CPU94.SCHED 9535 ± 13% -40.6% 5667 ± 14% softirqs.CPU95.RCU 9324 ± 15% -35.0% 6063 ± 21% softirqs.CPU96.RCU 25757 ± 5% +25.0% 32203 ± 9% softirqs.CPU96.SCHED 9674 ± 14% -40.8% 5729 ± 18% softirqs.CPU97.RCU 9371 ± 15% -37.7% 5840 ± 13% softirqs.CPU98.RCU 26068 ± 4% +38.4% 36065 ± 8% softirqs.CPU98.SCHED 9497 ± 15% -41.0% 5602 ± 13% softirqs.CPU99.RCU 26556 ± 6% +36.5% 36239 ± 3% softirqs.CPU99.SCHED 1862214 ± 15% -39.5% 1127214 ± 12% softirqs.RCU 5342380 +32.5% 7080895 ± 3% softirqs.SCHED 42480 ± 3% +28.2% 54463 softirqs.TIMER vm-scalability.time.system_time 1800 +--------------------------------------------------------------------+ 1700 |-+ O O O | | O O O OO O O OO O OO O O O | 1600 |-OO O O O O O O OO O O O OO O | 1500 |-+ O O OO O O O O| | O O O | 1400 |-+ | 1300 |-+ | 1200 |-+ | | | 1100 |-+ | 1000 |-+ | |.++. +.+ .++.++. .+ +. + .++.+ .+ | 900 |-+ + + ++ +.+ ++.+ + + + + | 800 +--------------------------------------------------------------------+ vm-scalability.time.percent_of_cpu_this_job_got 640 +---------------------------------------------------------------------+ | O | 620 |-+ O | 600 |-O O OO OO O O O O O OO O O OO O OO O O | | O O O OO OO OO O OO O | 580 |-+ O O O O O O O | | O | 560 |-+ | | | 540 |-+ + .+. +. | 520 |.+ +.+ .++.+ + + + + ++.+ + | | + : + + + +. :+ +.+ .+ | 500 |-+ +: + + + + | | + | 480 +---------------------------------------------------------------------+ vm-scalability.time.elapsed_time 340 +---------------------------------------------------------------------+ | O O O | 320 |-+ O O O O OO OO OO OO O | | OO O O O OO O O O O O O O O O | 300 |-+ O O O O O O O OO O O | | O O | 280 |-+ | | | 260 |-+ | | | 240 |-+ | | +.++. +. .+ .+ + +. | 220 |.+ ++.++.++.+ ++ + +.+ + +.+.+ ++.+ | | + | 200 +---------------------------------------------------------------------+ vm-scalability.time.elapsed_time.max 340 +---------------------------------------------------------------------+ | O O O | 320 |-+ O O O O OO OO OO OO O | | OO O O O OO O O O O O O O O O | 300 |-+ O O O O O O O OO O O | | O O | 280 |-+ | | | 260 |-+ | | | 240 |-+ | | +.++. +. .+ .+ + +. | 220 |.+ ++.++.++.+ ++ + +.+ + +.+.+ ++.+ | | + | 200 +---------------------------------------------------------------------+ vm-scalability.throughput 3.4e+06 +-----------------------------------------------------------------+ | | 3.2e+06 |-+ + | | .+ + :+ .+ | 3e+06 |+. + ++.++.+ .++. + .+ +: +++. + + | 2.8e+06 |-+++.+ + + + + + | | | 2.6e+06 |-+ | | | 2.4e+06 |-+ | 2.2e+06 |-+ | | O O O O OO O O O O O O O| 2e+06 |O+O O OO O OOO O OO O O O O O OO OO | | O O O OO O O O OO O | 1.8e+06 +-----------------------------------------------------------------+ vm-scalability.median 420000 +------------------------------------------------------------------+ | + | 400000 |-+ : | 380000 |-+ .+ +.+ : : .++.+ | |+. .++ +.++.++ .++. +. : :: +.+++ | 360000 |-+++ + + + + | 340000 |-+ | | | 320000 |-+ | 300000 |-+ | | | 280000 |-+ O O O | 260000 |-+ O OO OO O O O O OO OO O O OO O| |O O OO O O O O O O OO O O OOO O O | 240000 +------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

4
3
0 / 0

[KVM] 39be2e2818: WARNING:at_mm/usercopy.c:#usercopy_warn

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 39be2e28180a2e87af5fbb8d83643812e1a3b371 ("KVM: selftests: Add selftest for KVM statistics data binary interface") https://git.kernel.org/cgit/virt/kvm/kvm.git queue in testcase: kernel-selftests version: kernel-selftests-x86_64-7ae2585b-1_20210607 with following parameters: group: kvm ucode: 0xe2 test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt on test machine: 8 threads Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz with 32G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> [ 386.205633] WARNING: CPU: 5 PID: 6926 at mm/usercopy.c:75 usercopy_warn (kbuild/src/consumer/mm/usercopy.c:75 (discriminator 18)) [ 386.213477] Modules linked in: netconsole btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c intel_rapl_msr intel_rapl_common sd_mod t10_pi sg x86_pkg_temp_thermal i915 intel_powerclamp coretemp crct10dif_pclmul ipmi_devintf crc32_pclmul crc32c_intel ghash_clmulni_intel ipmi_msghandler sdhci_pci intel_wmi_thunderbolt cqhci rapl ahci sdhci vfio_mdev vfio_iommu_type1 libahci intel_cstate mei_me i2c_i801 mmc_core vfio libata mei intel_uncore joydev i2c_smbus mdev intel_pch_thermal ir_rc6_decoder intel_gtt wmi rc_rc6_mce nuvoton_cir rc_core video intel_pmc_core acpi_pad ip_tables [ 386.267149] CPU: 3 PID: 6926 Comm: kvm_binary_stat Tainted: G I 5.13.0-rc1-00239-g39be2e28180a #1 [ 386.277815] Hardware name: /NUC6i7KYB, BIOS KYSKLi70.86A.0041.2016.0817.1130 08/17/2016 [ 386.286169] RIP: 0010:usercopy_warn (kbuild/src/consumer/mm/usercopy.c:75 (discriminator 18)) [ 386.290702] Code: 80 82 41 51 49 89 c0 49 c7 c2 62 a4 7e 82 49 89 f1 48 89 f9 4c 0f 45 d2 48 c7 c7 30 33 80 82 4c 89 da 4c 89 d6 e8 26 30 a9 00 <0f> 0b 48 83 c4 18 c3 48 c7 c6 6b 83 7e 82 49 89 f1 48 89 f0 eb 96 All code ======== 0: 80 82 41 51 49 89 c0 addb $0xc0,-0x76b6aebf(%rdx) 7: 49 c7 c2 62 a4 7e 82 mov $0xffffffff827ea462,%r10 e: 49 89 f1 mov %rsi,%r9 11: 48 89 f9 mov %rdi,%rcx 14: 4c 0f 45 d2 cmovne %rdx,%r10 18: 48 c7 c7 30 33 80 82 mov $0xffffffff82803330,%rdi 1f: 4c 89 da mov %r11,%rdx 22: 4c 89 d6 mov %r10,%rsi 25: e8 26 30 a9 00 callq 0xa93050 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 83 c4 18 add $0x18,%rsp 30: c3 retq 31: 48 c7 c6 6b 83 7e 82 mov $0xffffffff827e836b,%rsi 38: 49 89 f1 mov %rsi,%r9 3b: 48 89 f0 mov %rsi,%rax 3e: eb 96 jmp 0xffffffffffffffd6 Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 48 83 c4 18 add $0x18,%rsp 6: c3 retq 7: 48 c7 c6 6b 83 7e 82 mov $0xffffffff827e836b,%rsi e: 49 89 f1 mov %rsi,%r9 11: 48 89 f0 mov %rsi,%rax 14: eb 96 jmp 0xffffffffffffffac [ 386.310020] RSP: 0018:ffffc90009eefe00 EFLAGS: 00010286 [ 386.315422] RAX: 0000000000000000 RBX: 00000000000000e8 RCX: 0000000000000000 [ 386.322820] RDX: 0000000000000001 RSI: ffffffff8123284f RDI: ffffffff8123284f [ 386.330219] RBP: ffff8888b3318200 R08: 0000000000000001 R09: 0000000000000001 [ 386.337607] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8888b3318118 [ 386.345010] R13: 0000000000000001 R14: 00000000013e4ce0 R15: ffffffff83040260 [ 386.352418] FS: 00007fe185eb6740(0000) GS:ffff8888becc0000(0000) knlGS:0000000000000000 [ 386.360838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 386.366775] CR2: 00007fb475292000 CR3: 00000008b49d6003 CR4: 00000000003726e0 [ 386.374147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 386.381532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 386.388909] Call Trace: [ 386.391469] __check_object_size (kbuild/src/consumer/mm/usercopy.c:240 kbuild/src/consumer/mm/usercopy.c:286 kbuild/src/consumer/mm/usercopy.c:256) [ 386.395869] kvm_stats_read (kbuild/src/consumer/include/linux/uaccess.h:200 kbuild/src/consumer/arch/x86/kvm/../../../virt/kvm/kvm_main.c:3546) [ 386.399875] kvm_vcpu_stats_read (kbuild/src/consumer/arch/x86/kvm/../../../virt/kvm/kvm_main.c:3564) [ 386.404141] vfs_read (kbuild/src/consumer/fs/read_write.c:494) [ 386.407514] ksys_pread64 (kbuild/src/consumer/include/linux/file.h:45 kbuild/src/consumer/fs/read_write.c:687) [ 386.411163] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:47) [ 386.414888] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:112) [ 386.420138] RIP: 0033:0x7fe18608bdf4 [ 386.423892] Code: d8 64 89 02 b8 ff ff ff ff eb c5 66 2e 0f 1f 84 00 00 00 00 00 90 8b 05 5a e6 00 00 49 89 ca 85 c0 75 13 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5c c3 0f 1f 00 41 55 49 89 cd 41 54 49 89 d4 All code ======== 0: d8 64 89 02 fsubs 0x2(%rcx,%rcx,4) 4: b8 ff ff ff ff mov $0xffffffff,%eax 9: eb c5 jmp 0xffffffffffffffd0 b: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1) 12: 00 00 00 15: 90 nop 16: 8b 05 5a e6 00 00 mov 0xe65a(%rip),%eax # 0xe676 1c: 49 89 ca mov %rcx,%r10 1f: 85 c0 test %eax,%eax 21: 75 13 jne 0x36 23: b8 11 00 00 00 mov $0x11,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 5c ja 0x8e 32: c3 retq 33: 0f 1f 00 nopl (%rax) 36: 41 55 push %r13 38: 49 89 cd mov %rcx,%r13 3b: 41 54 push %r12 3d: 49 89 d4 mov %rdx,%r12 Code starting with the faulting instruction =========================================== 0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 5c ja 0x64 8: c3 retq 9: 0f 1f 00 nopl (%rax) c: 41 55 push %r13 e: 49 89 cd mov %rcx,%r13 11: 41 54 push %r12 13: 49 89 d4 mov %rdx,%r12 [ 386.443313] RSP: 002b:00007fff2f95ffe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 386.451151] RAX: ffffffffffffffda RBX: 00000000013e4530 RCX: 00007fe18608bdf4 [ 386.458492] RDX: 00000000000000e8 RSI: 00000000013e4ce0 RDI: 000000000000001d [ 386.465889] RBP: 00000000000000e8 R08: 000000000040e420 R09: 00000000013e4ca0 [ 386.473294] R10: 0000000000000790 R11: 0000000000000246 R12: 0000000000000040 [ 386.480682] R13: 00000000000000e8 R14: 000000000000001d R15: 00000000013e4ce0 [ 386.488056] irq event stamp: 17487 [ 386.491620] hardirqs last enabled at (17505): _raw_spin_unlock_irq (kbuild/src/consumer/arch/x86/include/asm/irqflags.h:45 kbuild/src/consumer/arch/x86/include/asm/irqflags.h:80 kbuild/src/consumer/include/linux/spinlock_api_smp.h:168 kbuild/src/consumer/kernel/locking/spinlock.c:199) [ 386.500893] hardirqs last disabled at (17528): __schedule (kbuild/src/consumer/kernel/sched/core.c:5043 (discriminator 1)) [ 386.509429] softirqs last enabled at (17542): __do_softirq (kbuild/src/consumer/arch/x86/include/asm/preempt.h:27 kbuild/src/consumer/kernel/softirq.c:403 kbuild/src/consumer/kernel/softirq.c:588) [ 386.518160] softirqs last disabled at (17555): irq_exit_rcu (kbuild/src/consumer/kernel/softirq.c:433 kbuild/src/consumer/kernel/softirq.c:637 kbuild/src/consumer/kernel/softirq.c:649) [ 386.526912] ---[ end trace b50de138b37b26c4 ]--- [ 386.574882] ok 42 selftests: kvm: kvm_binary_stats_test [ 386.574891] [ 386.678131] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil... -O /dev/null [ 386.678141] [ 388.157966] kill 1098 vmstat --timestamp -n 10 [ 388.157972] [ 388.170558] kill 1096 dmesg --follow --decode [ 388.170563] [ 388.192089] wait for background processes: 1102 1106 meminfo oom-killer [ 388.192096] [ 396.520977] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil... -O /dev/null [ 396.520991] [ 396.585567] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil... -O /dev/null [ 396.585576] [ 396.638671] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-post-run?job_file=/lkp/job... -O /dev/null [ 396.638679] [ 396.677867] getting new job... [ 396.677873] [ 396.699693] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/gpxelinux.cgi?hostname=lkp-skl... -O /tmp/next-job-lkp [ 396.699702] [ 398.346941] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp//lkp/jobs/scheduled/lkp-skl-nuc2/kerne... -O /tmp/next-job.cgz [ 398.346950] [ 398.355973] 24 blocks [ 398.380914] [ 398.405028] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-wtmp?tbox_name=lkp-skl-nuc... -O /dev/null [ 398.405042] To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

1
0
0 / 0

[x86/fpu] 190757bfda: invalid_opcode:#[##]

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 190757bfda82678e88ce9598fb75a6a6a85615f1 ("x86/fpu: Add PKRU storage outside of task XSAVE buffer") https://git.kernel.org/cgit/linux/kernel/git/tglx/devel.git x86/fpu in testcase: rcuscale version: with following parameters: runtime: 300s scale_type: srcud on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +---------------------------------------------+------------+------------+ | | 65c5ba4c1b | 190757bfda | +---------------------------------------------+------------+------------+ | invalid_opcode:#[##] | 0 | 6 | | RIP:copy_thread | 0 | 6 | +---------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> [ 5.840871] vga=normal [ 5.863284] systemd[1]: RTC configured in localtime, applying delta of 0 minutes to system time. Welcome to Debian GNU/Linux 10 (buster)! [ 5.889702] invalid opcode: 0000 [#1] SMP PTI [ 5.890505] CPU: 1 PID: 1 Comm: systemd Not tainted 5.13.0-rc5-00051-g190757bfda82 #1 [ 5.891832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 5.893260] RIP: 0010:copy_thread (kbuild/src/consumer/arch/x86/include/asm/special_insns.h:83 kbuild/src/consumer/arch/x86/kernel/process.c:170) [ 5.894019] Code: 48 89 85 b0 23 00 00 66 8c c0 66 89 85 a0 23 00 00 66 8c d8 8b 4d 2c 66 89 85 a2 23 00 00 81 e1 00 00 20 00 0f 85 c4 00 00 00 <0f> 01 ee 89 85 14 24 00 00 b9 15 00 00 00 48 89 df 48 c7 43 e8 00 All code ======== 0: 48 89 85 b0 23 00 00 mov %rax,0x23b0(%rbp) 7: 66 8c c0 mov %es,%ax a: 66 89 85 a0 23 00 00 mov %ax,0x23a0(%rbp) 11: 66 8c d8 mov %ds,%ax 14: 8b 4d 2c mov 0x2c(%rbp),%ecx 17: 66 89 85 a2 23 00 00 mov %ax,0x23a2(%rbp) 1e: 81 e1 00 00 20 00 and $0x200000,%ecx 24: 0f 85 c4 00 00 00 jne 0xee 2a:* 0f 01 ee rdpkru <-- trapping instruction 2d: 89 85 14 24 00 00 mov %eax,0x2414(%rbp) 33: b9 15 00 00 00 mov $0x15,%ecx 38: 48 89 df mov %rbx,%rdi 3b: 48 rex.W 3c: c7 .byte 0xc7 3d: 43 rex.XB 3e: e8 .byte 0xe8 ... Code starting with the faulting instruction =========================================== 0: 0f 01 ee rdpkru 3: 89 85 14 24 00 00 mov %eax,0x2414(%rbp) 9: b9 15 00 00 00 mov $0x15,%ecx e: 48 89 df mov %rbx,%rdi 11: 48 rex.W 12: c7 .byte 0xc7 13: 43 rex.XB 14: e8 .byte 0xe8 ... [ 5.896948] RSP: 0018:ffffacebc0013d88 EFLAGS: 00010246 [ 5.897794] RAX: 0000000000000000 RBX: ffffacebc00c7f58 RCX: 0000000000000000 [ 5.898893] RDX: 0000000000000000 RSI: ffff89a98ca9cd00 RDI: 0000000001200000 [ 5.899994] RBP: ffff89acafcca680 R08: 0000000000000000 R09: 0000000000000000 [ 5.901101] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000001200000 [ 5.902208] R13: 0000000000000000 R14: ffffacebc00c4000 R15: 0000000000000000 [ 5.903298] FS: 00007fb1452af940(0000) GS:ffff89acafd00000(0000) knlGS:0000000000000000 [ 5.904594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.905510] CR2: 0000557cb50bdbf0 CR3: 00000002cfb28000 CR4: 00000000000006e0 [ 5.906607] Call Trace: [ 5.907083] copy_process (kbuild/src/consumer/kernel/fork.c:2125) [ 5.907731] ? shmem_init_fs_context (kbuild/src/consumer/mm/shmem.c:3179) [ 5.908489] kernel_clone (kbuild/src/consumer/kernel/fork.c:2503) [ 5.909117] __do_sys_clone (kbuild/src/consumer/kernel/fork.c:2621) [ 5.909750] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:47) [ 5.910375] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:112) [ 5.911205] RIP: 0033:0x7fb14643c7be [ 5.911828] Code: db 0f 85 25 01 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 b6 00 00 00 41 89 c4 85 c0 0f 85 c3 00 00 All code ======== 0: db 0f fisttpl (%rdi) 2: 85 25 01 00 00 64 test %esp,0x64000001(%rip) # 0x64000009 8: 4c 8b 0c 25 10 00 00 mov 0x10,%r9 f: 00 10: 45 31 c0 xor %r8d,%r8d 13: 4d 8d 91 d0 02 00 00 lea 0x2d0(%r9),%r10 1a: 31 d2 xor %edx,%edx 1c: 31 f6 xor %esi,%esi 1e: bf 11 00 20 01 mov $0x1200011,%edi 23: b8 38 00 00 00 mov $0x38,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 0f 87 b6 00 00 00 ja 0xec 36: 41 89 c4 mov %eax,%r12d 39: 85 c0 test %eax,%eax 3b: 0f .byte 0xf 3c: 85 c3 test %eax,%ebx ... Code starting with the faulting instruction =========================================== 0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 0f 87 b6 00 00 00 ja 0xc2 c: 41 89 c4 mov %eax,%r12d f: 85 c0 test %eax,%eax 11: 0f .byte 0xf 12: 85 c3 test %eax,%ebx ... [ 5.914611] RSP: 002b:00007ffff8b392d0 EFLAGS: 00000246 ORIG_RAX: 0000000000000038 [ 5.915833] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb14643c7be [ 5.916940] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 [ 5.918035] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fb1452af940 [ 5.919127] R10: 00007fb1452afc10 R11: 0000000000000246 R12: 0000000000000020 [ 5.920215] R13: 00007fb1462a53f8 R14: 0000000000000000 R15: 0000000000000001 [ 5.921320] Modules linked in: ip_tables [ 5.922005] ---[ end trace d7e147dd4242acd7 ]--- [ 5.922781] RIP: 0010:copy_thread (kbuild/src/consumer/arch/x86/include/asm/special_insns.h:83 kbuild/src/consumer/arch/x86/kernel/process.c:170) [ 5.923509] Code: 48 89 85 b0 23 00 00 66 8c c0 66 89 85 a0 23 00 00 66 8c d8 8b 4d 2c 66 89 85 a2 23 00 00 81 e1 00 00 20 00 0f 85 c4 00 00 00 <0f> 01 ee 89 85 14 24 00 00 b9 15 00 00 00 48 89 df 48 c7 43 e8 00 All code ======== 0: 48 89 85 b0 23 00 00 mov %rax,0x23b0(%rbp) 7: 66 8c c0 mov %es,%ax a: 66 89 85 a0 23 00 00 mov %ax,0x23a0(%rbp) 11: 66 8c d8 mov %ds,%ax 14: 8b 4d 2c mov 0x2c(%rbp),%ecx 17: 66 89 85 a2 23 00 00 mov %ax,0x23a2(%rbp) 1e: 81 e1 00 00 20 00 and $0x200000,%ecx 24: 0f 85 c4 00 00 00 jne 0xee 2a:* 0f 01 ee rdpkru <-- trapping instruction 2d: 89 85 14 24 00 00 mov %eax,0x2414(%rbp) 33: b9 15 00 00 00 mov $0x15,%ecx 38: 48 89 df mov %rbx,%rdi 3b: 48 rex.W 3c: c7 .byte 0xc7 3d: 43 rex.XB 3e: e8 .byte 0xe8 ... Code starting with the faulting instruction =========================================== 0: 0f 01 ee rdpkru 3: 89 85 14 24 00 00 mov %eax,0x2414(%rbp) 9: b9 15 00 00 00 mov $0x15,%ecx e: 48 89 df mov %rbx,%rdi 11: 48 rex.W 12: c7 .byte 0xc7 13: 43 rex.XB 14: e8 .byte 0xe8 ... [ 5.926303] RSP: 0018:ffffacebc0013d88 EFLAGS: 00010246 [ 5.927163] RAX: 0000000000000000 RBX: ffffacebc00c7f58 RCX: 0000000000000000 [ 5.928277] RDX: 0000000000000000 RSI: ffff89a98ca9cd00 RDI: 0000000001200000 [ 5.929405] RBP: ffff89acafcca680 R08: 0000000000000000 R09: 0000000000000000 [ 5.930517] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000001200000 [ 5.931627] R13: 0000000000000000 R14: ffffacebc00c4000 R15: 0000000000000000 [ 5.932751] FS: 00007fb1452af940(0000) GS:ffff89acafd00000(0000) knlGS:0000000000000000 [ 5.934045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.934973] CR2: 0000557cb50bdbf0 CR3: 00000002cfb28000 CR4: 00000000000006e0 [ 5.936084] Kernel panic - not syncing: Fatal exception [ 5.937085] Kernel Offset: 0xde00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) Kboot worker: lkp-worker44 Elapsed time: 60 To reproduce: # build kernel cd linux cp config-5.13.0-rc5-00051-g190757bfda82 .config make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

3
2
0 / 0

[PATCH] mm: relocate 'write_protect_seq' in struct mm_struct

by Feng Tang

0day robot reported a 9.2% regression for will-it-scale mmap1 test case[1], caused by commit 57efa1fe5957 ("mm/gup: prevent gup_fast from racing with COW during fork"). Further debug shows the regression is due to that commit changes the offset of hot fields 'mmap_lock' inside structure 'mm_struct', thus some cache alignment changes. From the perf data, the contention for 'mmap_lock' is very severe and takes around 95% cpu cycles, and it is a rw_semaphore struct rw_semaphore { atomic_long_t count; /* 8 bytes */ atomic_long_t owner; /* 8 bytes */ struct optimistic_spin_queue osq; /* spinner MCS lock */ ... Before commit 57efa1fe5957 adds the 'write_protect_seq', it happens to have a very optimal cache alignment layout, as Linus explained: "and before the addition of the 'write_protect_seq' field, the mmap_sem was at offset 120 in 'struct mm_struct'. Which meant that count and owner were in two different cachelines, and then when you have contention and spend time in rwsem_down_write_slowpath(), this is probably *exactly* the kind of layout you want. Because first the rwsem_write_trylock() will do a cmpxchg on the first cacheline (for the optimistic fast-path), and then in the case of contention, rwsem_down_write_slowpath() will just access the second cacheline. Which is probably just optimal for a load that spends a lot of time contended - new waiters touch that first cacheline, and then they queue themselves up on the second cacheline." After the commit, the rw_semaphore is at offset 128, which means the 'count' and 'owner' fields are now in the same cacheline, and causes more cache bouncing. Currently there are 3 "#ifdef CONFIG_XXX" before 'mmap_lock' which will affect its offset: CONFIG_MMU CONFIG_MEMBARRIER CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES The layout above is on 64 bits system with 0day's default kernel config (similar to RHEL-8.3's config), in which all these 3 options are 'y'. And the layout can vary with different kernel configs. Relayouting a structure is usually a double-edged sword, as sometimes it can helps one case, but hurt other cases. For this case, one solution is, as the newly added 'write_protect_seq' is a 4 bytes long seqcount_t (when CONFIG_DEBUG_LOCK_ALLOC=n), placing it into an existing 4 bytes hole in 'mm_struct' will not change other fields' alignment, while restoring the regression. [1]. https://lore.kernel.org/lkml/20210525031636.GB7744@xsang-OptiPlex-9020/ Reported-by: kernel test robot <oliver.sang(a)intel.com> Signed-off-by: Feng Tang <feng.tang(a)intel.com> Reviewed-by: John Hubbard <jhubbard(a)nvidia.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: Linus Torvalds <torvalds(a)linux-foundation.org> Cc: Peter Xu <peterx(a)redhat.com> --- include/linux/mm_types.h | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5aacc1c..cba6022 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -445,13 +445,6 @@ struct mm_struct { */ atomic_t has_pinned; - /** - * @write_protect_seq: Locked when any thread is write - * protecting pages mapped by this mm to enforce a later COW, - * for instance during page table copying for fork(). - */ - seqcount_t write_protect_seq; - #ifdef CONFIG_MMU atomic_long_t pgtables_bytes; /* PTE page table pages */ #endif @@ -460,6 +453,18 @@ struct mm_struct { spinlock_t page_table_lock; /* Protects page tables and some * counters */ + /* + * With some kernel config, the current mmap_lock's offset + * inside 'mm_struct' is at 0x120, which is very optimal, as + * its two hot fields 'count' and 'owner' sit in 2 different + * cachelines, and when mmap_lock is highly contended, both + * of the 2 fields will be accessed frequently, current layout + * will help to reduce cache bouncing. + * + * So please be careful with adding new fields before + * mmap_lock, which can easily push the 2 fields into one + * cacheline. + */ struct rw_semaphore mmap_lock; struct list_head mmlist; /* List of maybe swapped mm's. These @@ -480,7 +485,15 @@ struct mm_struct { unsigned long stack_vm; /* VM_STACK */ unsigned long def_flags; + /** + * @write_protect_seq: Locked when any thread is write + * protecting pages mapped by this mm to enforce a later COW, + * for instance during page table copying for fork(). + */ + seqcount_t write_protect_seq; + spinlock_t arg_lock; /* protect the below fields */ + unsigned long start_code, end_code, start_data, end_data; unsigned long start_brk, brk, start_stack; unsigned long arg_start, arg_end, env_start, env_end; -- 2.7.4

1 year, 3 months

3
5
0 / 0

Re: follow up

by Jake

1 year, 3 months

1
0
0 / 0

[platform/x86] 51954bff27: BUG:KASAN:null-ptr-deref_in_lwpmudrv_Get_Symbol_Lookup_Func_Addr

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 51954bff279dc62a97c2ce1802c44455d306d971 ("platform/x86: bug fix and re-enable sep driver for Linux 5.12 pull") https://github.com/mchinth/linux sep_socwatch_linux_5_12 in testcase: trinity version: trinity-x86_64-03f10b67-1_20210603 with following parameters: runtime: 300s test-description: Trinity is a linux system call fuzz tester. test-url: http://codemonkey.org.uk/projects/trinity/ on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> [ 732.148425] BUG: KASAN: null-ptr-deref in lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196) [ 732.148425] Read of size 8 at addr 0000000000000026 by task SEPDRV_ABNORMAL/402 [ 732.148425] [ 732.148425] CPU: 0 PID: 402 Comm: SEPDRV_ABNORMAL Not tainted 5.12.0-rc4-00053-g51954bff279d #1 [ 732.211784] sep5_0: [load] [lwpmu_Load@7877]: Graphics support is enabled. [ 732.148425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 732.148425] Call Trace: [ 732.148425] dump_stack (kbuild/src/consumer/lib/dump_stack.c:122) [ 732.148425] ? lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196) [ 732.148425] kasan_report.cold (kbuild/src/consumer/mm/kasan/report.c:403 kbuild/src/consumer/mm/kasan/report.c:416) [ 732.258157] sep5_0: [load] [lwpmu_Load@7880]: NMI will be used for handling PMU interrupts. [ 732.148425] ? kcov_common_handle (kbuild/src/consumer/kernel/kcov.c:1028) [ 732.148425] ? lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196) [ 732.148425] lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196) [ 732.148425] ? lwpmudrv_Samp_Read_PCI_Config (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7180) [ 732.148425] ? rcu_read_lock_any_held (kbuild/src/consumer/kernel/rcu/update.c:346) [ 732.308017] PAX: PMU arbitration service v1.0.2 has been started. [ 732.148425] ? rcu_read_lock_held (kbuild/src/consumer/kernel/rcu/update.c:341) [ 732.148425] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1)) [ 732.148425] ? validate_chain (kbuild/src/consumer/kernel/locking/lockdep.c:3681) [ 732.369358] cros_ec_lpcs: unsupported system. [ 732.351562] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1)) [ 732.351562] ? check_prev_add (kbuild/src/consumer/kernel/locking/lockdep.c:3630) [ 732.429155] random: get_random_u64 called from kmem_cache_open+0x2a/0x460 with crng_init=0 [ 732.429258] random: get_random_u64 called from cache_random_seq_create+0x80/0x180 with crng_init=0 [ 732.442196] surface_gpe: no compatible Microsoft Surface device found, exiting [ 732.351562] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292) [ 732.473751] hv_utils: Registering HyperV Utility Driver [ 732.351562] ? mark_lock (kbuild/src/consumer/kernel/locking/lockdep.c:4456) [ 732.351562] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1)) [ 732.351562] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292) [ 732.515258] hv_vmbus: registering driver hv_utils [ 732.351562] ? sched_clock_cpu (kbuild/src/consumer/kernel/sched/clock.c:382) [ 732.351562] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246) [ 732.564130] hv_vmbus: registering driver hv_balloon [ 732.351562] lwpmudrv_Abnormal_Handler (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7255) [ 732.351562] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246) [ 732.351562] ? lockdep_hardirqs_on (kbuild/src/consumer/kernel/locking/lockdep.c:4215 (discriminator 1)) [ 732.351562] ? _raw_spin_unlock_irqrestore (kbuild/src/consumer/include/linux/spinlock_api_smp.h:160 kbuild/src/consumer/kernel/locking/spinlock.c:191) [ 732.622249] Driver 'dfl-emif' was unable to register with bus_type 'dfl' because the bus was not initialized. [ 732.351562] ? trace_hardirqs_on (kbuild/src/consumer/kernel/trace/trace_preemptirq.c:50 (discriminator 22)) [ 732.351562] ? write_comp_data (kbuild/src/consumer/kernel/kcov.c:218) [ 732.351562] ? __sanitizer_cov_trace_pc (kbuild/src/consumer/kernel/kcov.c:197) [ 732.351562] ? __kthread_parkme (kbuild/src/consumer/kernel/kthread.c:235 (discriminator 4)) [ 732.885996] Board is not the VME system controller [ 732.880567] kthread (kbuild/src/consumer/kernel/kthread.c:292) [ 732.880567] ? kthread_insert_work_sanity_check (kbuild/src/consumer/kernel/kthread.c:245) [ 732.914487] VME geographical address is set to 0 [ 732.914474] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:300) [ 732.914474] ================================================================== [ 732.960810] CR/CSR Offset: 0 [ 732.914474] Disabling lock debugging due to kernel taint [ 732.984459] BUG: kernel NULL pointer dereference, address: 0000000000000026 [ 733.008704] AMD(R) PCI-E Non-Transparent Bridge Driver 1.0 [ 732.988421] #PF: supervisor read access in kernel mode [ 732.988421] #PF: error_code(0x0000) - not-present page [ 732.988421] PGD 0 P4D 0 [ 732.988421] Oops: 0000 [#1] SMP KASAN PTI [ 733.051131] IDT PCI-E Non-Transparent Bridge Driver 2.0 [ 732.988421] CPU: 0 PID: 402 Comm: SEPDRV_ABNORMAL Tainted: G B 5.12.0-rc4-00053-g51954bff279d #1 [ 732.988421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 732.988421] RIP: 0010:lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196) [ 732.988421] Code: b8 86 22 f8 48 85 c0 0f 84 4a 02 00 00 49 89 c4 4d 8d 74 24 28 4d 8d ac 24 30 01 00 00 e8 fa a5 e2 f7 4c 89 f7 e8 72 5b 19 f8 <49> 8b 5c 24 28 48 8d 7b 10 e8 64 5b 19 f8 48 8b 43 10 4c 89 e7 4c All code ======== 0: b8 86 22 f8 48 mov $0x48f82286,%eax 5: 85 c0 test %eax,%eax 7: 0f 84 4a 02 00 00 je 0x257 d: 49 89 c4 mov %rax,%r12 10: 4d 8d 74 24 28 lea 0x28(%r12),%r14 15: 4d 8d ac 24 30 01 00 lea 0x130(%r12),%r13 1c: 00 1d: e8 fa a5 e2 f7 callq 0xfffffffff7e2a61c 22: 4c 89 f7 mov %r14,%rdi 25: e8 72 5b 19 f8 callq 0xfffffffff8195b9c 2a:* 49 8b 5c 24 28 mov 0x28(%r12),%rbx <-- trapping instruction 2f: 48 8d 7b 10 lea 0x10(%rbx),%rdi 33: e8 64 5b 19 f8 callq 0xfffffffff8195b9c 38: 48 8b 43 10 mov 0x10(%rbx),%rax 3c: 4c 89 e7 mov %r12,%rdi 3f: 4c rex.WR Code starting with the faulting instruction =========================================== 0: 49 8b 5c 24 28 mov 0x28(%r12),%rbx 5: 48 8d 7b 10 lea 0x10(%rbx),%rdi 9: e8 64 5b 19 f8 callq 0xfffffffff8195b72 e: 48 8b 43 10 mov 0x10(%rbx),%rax 12: 4c 89 e7 mov %r12,%rdi 15: 4c rex.WR [ 733.094487] Intel(R) PCI-E Non-Transparent Bridge Driver 2.0 [ 732.988421] RSP: 0000:ffffc900081efb48 EFLAGS: 00010286 [ 732.988421] RAX: 0000000000000001 RBX: ffff8881606e9000 RCX: ffffffffa22e19af [ 732.988421] RDX: dffffc0000000000 RSI: ffff8881606e9000 RDI: ffffffffa7dce820 [ 732.988421] RBP: ffff88816070b100 R08: 0000000000000000 R09: 0000000000000000 [ 733.140005] Software Queue-Pair Transport over NTB, version 4 [ 732.988421] R10: ffffffffab65dfc3 R11: fffffbfff56cbbf8 R12: fffffffffffffffe [ 732.988421] R13: 000000000000012e R14: 0000000000000026 R15: 1ffff9200103df6a [ 732.988421] FS: 0000000000000000(0000) GS:ffff88839cc00000(0000) knlGS:0000000000000000 [ 732.988421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 732.988421] CR2: 0000000000000026 CR3: 000000005e8ba000 CR4: 00000000000406f0 [ 732.988421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 733.832632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 733.848575] Call Trace: [ 733.868480] ? lwpmudrv_Samp_Read_PCI_Config (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7180) [ 733.888615] ? rcu_read_lock_any_held (kbuild/src/consumer/kernel/rcu/update.c:346) [ 733.904632] ? rcu_read_lock_held (kbuild/src/consumer/kernel/rcu/update.c:341) [ 733.928610] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1)) [ 733.952605] ? validate_chain (kbuild/src/consumer/kernel/locking/lockdep.c:3681) [ 733.968568] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1)) [ 733.992582] ? check_prev_add (kbuild/src/consumer/kernel/locking/lockdep.c:3630) [ 734.016582] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292) [ 734.036474] ? mark_lock (kbuild/src/consumer/kernel/locking/lockdep.c:4456) [ 734.056609] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1)) [ 734.072599] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292) [ 734.097522] ? sched_clock_cpu (kbuild/src/consumer/kernel/sched/clock.c:382) [ 734.120542] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246) [ 734.136530] lwpmudrv_Abnormal_Handler (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7255) [ 734.160556] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246) [ 734.180494] ? lockdep_hardirqs_on (kbuild/src/consumer/kernel/locking/lockdep.c:4215 (discriminator 1)) [ 734.200593] ? _raw_spin_unlock_irqrestore (kbuild/src/consumer/include/linux/spinlock_api_smp.h:160 kbuild/src/consumer/kernel/locking/spinlock.c:191) [ 734.220551] ? trace_hardirqs_on (kbuild/src/consumer/kernel/trace/trace_preemptirq.c:50 (discriminator 22)) [ 734.240624] ? write_comp_data (kbuild/src/consumer/kernel/kcov.c:218) [ 734.256574] ? __sanitizer_cov_trace_pc (kbuild/src/consumer/kernel/kcov.c:197) [ 734.276481] ? __kthread_parkme (kbuild/src/consumer/kernel/kthread.c:235 (discriminator 4)) [ 734.296593] kthread (kbuild/src/consumer/kernel/kthread.c:292) [ 734.313615] ? kthread_insert_work_sanity_check (kbuild/src/consumer/kernel/kthread.c:245) [ 734.336565] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:300) [ 734.352548] Modules linked in: [ 734.372478] CR2: 0000000000000026 [ 734.384585] _warn_unseeded_randomness: 2 callbacks suppressed [ 734.384585] random: get_random_bytes called from print_oops_end_marker+0x3b/0x60 with crng_init=0 [ 734.384585] ---[ end trace 29029e514c2dd2ac ]--- [ 734.444511] RIP: 0010:lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196) [ 734.456601] Code: b8 86 22 f8 48 85 c0 0f 84 4a 02 00 00 49 89 c4 4d 8d 74 24 28 4d 8d ac 24 30 01 00 00 e8 fa a5 e2 f7 4c 89 f7 e8 72 5b 19 f8 <49> 8b 5c 24 28 48 8d 7b 10 e8 64 5b 19 f8 48 8b 43 10 4c 89 e7 4c All code ======== 0: b8 86 22 f8 48 mov $0x48f82286,%eax 5: 85 c0 test %eax,%eax 7: 0f 84 4a 02 00 00 je 0x257 d: 49 89 c4 mov %rax,%r12 10: 4d 8d 74 24 28 lea 0x28(%r12),%r14 15: 4d 8d ac 24 30 01 00 lea 0x130(%r12),%r13 1c: 00 1d: e8 fa a5 e2 f7 callq 0xfffffffff7e2a61c 22: 4c 89 f7 mov %r14,%rdi 25: e8 72 5b 19 f8 callq 0xfffffffff8195b9c 2a:* 49 8b 5c 24 28 mov 0x28(%r12),%rbx <-- trapping instruction 2f: 48 8d 7b 10 lea 0x10(%rbx),%rdi 33: e8 64 5b 19 f8 callq 0xfffffffff8195b9c 38: 48 8b 43 10 mov 0x10(%rbx),%rax 3c: 4c 89 e7 mov %r12,%rdi 3f: 4c rex.WR Code starting with the faulting instruction =========================================== 0: 49 8b 5c 24 28 mov 0x28(%r12),%rbx 5: 48 8d 7b 10 lea 0x10(%rbx),%rdi 9: e8 64 5b 19 f8 callq 0xfffffffff8195b72 e: 48 8b 43 10 mov 0x10(%rbx),%rax 12: 4c 89 e7 mov %r12,%rdi 15: 4c rex.WR To reproduce: # build kernel cd linux cp config-5.12.0-rc4-00053-g51954bff279d .config make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

1
0
0 / 0

[block/mq] 2aa18c32fd: WARNING:at_block/mq-deadline-main.c:#dd_insert_requests

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 2aa18c32fd0905de7894b1e3a32f84edf77bc840 ("block/mq-deadline: Add cgroup support") https://github.com/bvanassche/linux deadline-scheduler in testcase: mdadm-selftests version: mdadm-selftests-x86_64-5f41845-1_20210508 with following parameters: disk: 1HDD test_prefix: 15 ucode: 0x21 on test machine: 4 threads 1 sockets Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang(a)intel.com> kern :warn : [ 100.174333] WARNING: CPU: 0 PID: 1055 at block/mq-deadline-main.c:714 dd_insert_requests (kbuild/src/consumer/block/mq-deadline-main.c:714 kbuild/src/consumer/block/mq-deadline-main.c:761) kern :warn : [ 100.184543] Modules linked in: multipath loop raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid10 raid1 raid0 netconsole btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c ipmi_devintf ipmi_msghandler sd_mod t10_pi sg intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp i915 kvm_intel kvm intel_gtt drm_kms_helper irqbypass crct10dif_pclmul syscopyarea crc32_pclmul sysfillrect crc32c_intel sysimgblt ghash_clmulni_intel fb_sys_fops rapl wmi_bmof intel_cstate intel_uncore ahci libahci mei_me drm video libata wmi joydev mei ip_tables kern :warn : [ 100.240822] CPU: 0 PID: 1055 Comm: mkfs.ext3 Not tainted 5.13.0-rc3-00092-g2aa18c32fd09 #1 kern :warn : [ 100.250180] Hardware name: Hewlett-Packard HP Pro 3340 MT/17A1, BIOS 8.07 01/24/2013 kern :warn : [ 100.259064] RIP: 0010:dd_insert_requests (kbuild/src/consumer/block/mq-deadline-main.c:714 kbuild/src/consumer/block/mq-deadline-main.c:761) kern :warn : [ 100.265114] Code: c6 07 00 0f 1f 40 00 65 ff 0d a7 fe a0 7e 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 4c 89 f7 e8 d0 21 01 00 e9 f3 fe ff ff <0f> 0b e9 26 ff ff ff 41 8b 47 d0 4c 89 f6 83 e0 01 48 83 c0 01 48 All code ======== 0: c6 07 00 movb $0x0,(%rdi) 3: 0f 1f 40 00 nopl 0x0(%rax) 7: 65 ff 0d a7 fe a0 7e decl %gs:0x7ea0fea7(%rip) # 0x7ea0feb5 e: 48 83 c4 18 add $0x18,%rsp 12: 5b pop %rbx 13: 5d pop %rbp 14: 41 5c pop %r12 16: 41 5d pop %r13 18: 41 5e pop %r14 1a: 41 5f pop %r15 1c: c3 retq 1d: 4c 89 f7 mov %r14,%rdi 20: e8 d0 21 01 00 callq 0x121f5 25: e9 f3 fe ff ff jmpq 0xffffffffffffff1d 2a:* 0f 0b ud2 <-- trapping instruction 2c: e9 26 ff ff ff jmpq 0xffffffffffffff57 31: 41 8b 47 d0 mov -0x30(%r15),%eax 35: 4c 89 f6 mov %r14,%rsi 38: 83 e0 01 and $0x1,%eax 3b: 48 83 c0 01 add $0x1,%rax 3f: 48 rex.W Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: e9 26 ff ff ff jmpq 0xffffffffffffff2d 7: 41 8b 47 d0 mov -0x30(%r15),%eax b: 4c 89 f6 mov %r14,%rsi e: 83 e0 01 and $0x1,%eax 11: 48 83 c0 01 add $0x1,%rax 15: 48 rex.W kern :warn : [ 100.286230] RSP: 0018:ffffc9000101fb40 EFLAGS: 00010246 kern :warn : [ 100.292611] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 kern :warn : [ 100.300920] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881ff9a2f00 kern :warn : [ 100.309253] RBP: ffff8880744c3200 R08: ffffc9000101fbd0 R09: 0000000000000000 kern :warn : [ 100.317578] R10: 0000000000000033 R11: 0000000000000000 R12: ffffc9000101fbd0 kern :warn : [ 100.325885] R13: ffff88821a411908 R14: ffff8881ff981a40 R15: ffff8881ff981a88 kern :warn : [ 100.334200] FS: 00007f21576fe780(0000) GS:ffff88821fa00000(0000) knlGS:0000000000000000 kern :warn : [ 100.343452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kern :warn : [ 100.350333] CR2: 00007f215797f060 CR3: 00000001a07a6002 CR4: 00000000001706f0 kern :warn : [ 100.358636] Call Trace: kern :warn : [ 100.362195] blk_mq_sched_insert_requests (kbuild/src/consumer/block/blk-mq-sched.c:513) kern :warn : [ 100.368316] blk_mq_flush_plug_list (kbuild/src/consumer/include/linux/list.h:282 kbuild/src/consumer/block/blk-mq.c:1960) kern :warn : [ 100.373997] blk_flush_plug_list (kbuild/src/consumer/block/blk-core.c:1734) kern :warn : [ 100.379312] blk_mq_submit_bio (kbuild/src/consumer/arch/x86/include/asm/jump_label.h:19 kbuild/src/consumer/include/linux/jump_label.h:200 kbuild/src/consumer/include/trace/events/block.h:356 kbuild/src/consumer/block/blk-mq.c:2260) kern :warn : [ 100.384521] submit_bio_noacct (kbuild/src/consumer/block/blk-core.c:919 kbuild/src/consumer/block/blk-core.c:968 kbuild/src/consumer/block/blk-core.c:1047) kern :warn : [ 100.389722] submit_bio_wait (kbuild/src/consumer/block/bio.c:1164) kern :warn : [ 100.394597] blkdev_issue_discard (kbuild/src/consumer/block/blk-lib.c:145) kern :warn : [ 100.399964] blk_ioctl_discard (kbuild/src/consumer/block/ioctl.c:141) kern :warn : [ 100.405090] blkdev_common_ioctl (kbuild/src/consumer/block/ioctl.c:468) kern :warn : [ 100.410425] blkdev_ioctl (kbuild/src/consumer/block/ioctl.c:579) kern :warn : [ 100.415175] block_ioctl (kbuild/src/consumer/fs/block_dev.c:1655) kern :warn : [ 100.419642] __x64_sys_ioctl (kbuild/src/consumer/fs/ioctl.c:52 kbuild/src/consumer/fs/ioctl.c:1069 kbuild/src/consumer/fs/ioctl.c:1055 kbuild/src/consumer/fs/ioctl.c:1055) kern :warn : [ 100.424405] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:47) kern :warn : [ 100.429016] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:112) kern :warn : [ 100.435090] RIP: 0033:0x7f2157812427 kern :warn : [ 100.439661] Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48 All code ======== 0: 00 00 add %al,(%rax) 2: 90 nop 3: 48 8b 05 69 aa 0c 00 mov 0xcaa69(%rip),%rax # 0xcaa73 a: 64 c7 00 26 00 00 00 movl $0x26,%fs:(%rax) 11: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax 18: c3 retq 19: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1) 20: 00 00 00 23: b8 10 00 00 00 mov $0x10,%eax 28: 0f 05 syscall 2a:* 48 3d 01 f0 ff ff cmp $0xfffffffffffff001,%rax <-- trapping instruction 30: 73 01 jae 0x33 32: c3 retq 33: 48 8b 0d 39 aa 0c 00 mov 0xcaa39(%rip),%rcx # 0xcaa73 3a: f7 d8 neg %eax 3c: 64 89 01 mov %eax,%fs:(%rcx) 3f: 48 rex.W Code starting with the faulting instruction =========================================== 0: 48 3d 01 f0 ff ff cmp $0xfffffffffffff001,%rax 6: 73 01 jae 0x9 8: c3 retq 9: 48 8b 0d 39 aa 0c 00 mov 0xcaa39(%rip),%rcx # 0xcaa49 10: f7 d8 neg %eax 12: 64 89 01 mov %eax,%fs:(%rcx) 15: 48 rex.W kern :warn : [ 100.460608] RSP: 002b:00007ffe241c15b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 kern :warn : [ 100.469262] RAX: ffffffffffffffda RBX: 00007ffe241c16e0 RCX: 00007f2157812427 kern :warn : [ 100.477447] RDX: 00007ffe241c15c0 RSI: 0000000000001277 RDI: 0000000000000004 kern :warn : [ 100.485637] RBP: 0000557074df39d0 R08: 0000000000000004 R09: 0000000000000000 kern :warn : [ 100.493851] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000002800 kern :warn : [ 100.502067] R13: 0000557074df39d0 R14: 0000000000000000 R15: 0000000000000000 kern :warn : [ 100.510250] ---[ end trace 3c43a36e40e0cc9b ]--- kern :warn : [ 100.515932] ------------[ cut here ]------------ To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang

1 year, 3 months

2
1
0 / 0

Are you struggling with your programming assignments?

by tejparker97＠gmail.com

Use Assignmenthelpshop and get it done now. Everyone is aware of the complexity and toughness of programming assignments. These assignments are not a cup of tea for everyone. You need to be very intelligent and skilled to accomplish programming assignments. Many students are unable to do the programming assignments that lead them to get poor marks. So, it is better to hire a professional expert with sound knowledge and expertise in programming. Assignmenthelpshop provides quality Programming Assignment Help that can solve your all problems related to assignments. Give it a go and boost your grades while being in your comfort zone. https://www.assignmenthelpshop.com/programming-assignment-help/

1 year, 3 months

1
0
0 / 0

Are you struggling with your programming assignments?

by tejparker97＠gmail.com

Use Assignmenthelpshop and get it done now. Everyone is aware of the complexity and toughness of programming assignments. These assignments are not a cup of tea for everyone. You need to be very intelligent and skilled to accomplish programming assignments. Many students are unable to do the programming assignments that lead them to get poor marks. So, it is better to hire a professional expert with sound knowledge and expertise in programming. Assignmenthelpshop provides quality <a href="https://www.assignmenthelpshop.com/programming-assignment-help">Programming assignment help</a> that can solve your all problems related to assignments. Give it a go and boost your grades while being in your comfort zone.

1 year, 3 months

1
0
0 / 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP June 2021