[xfs] 18bb7d74fb: Assertion_failed
by kernel test robot
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 18bb7d74fbd6886c19068d0f6eea3d983ebf4c23 ("xfs: allow queued AG intents to drain before scrubbing")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git scrub-drain-intents
in testcase: xfstests
version: xfstests-x86_64-820b5e3-1_20210609
with following parameters:
disk: 4HDD
fs: xfs
test: xfs-reflink-15
ucode: 0x28
test-description: xfstests is a regression test suite for xfs and other files ystems.
test-url: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
on test machine: 8 threads 1 sockets Intel(R) Core(TM) i7-4790 v3 @ 3.60GHz with 6G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
[ 67.979803] XFS (sdb4): Unmounting Filesystem
[ 68.939007] XFS (sdb4): Mounting V5 Filesystem
[ 68.983028] XFS (sdb4): Ending clean mount
[ 69.017237] xfs filesystem being mounted at /fs/scratch supports timestamps until 2038 (0x7fffffff)
[ 69.257224] XFS (sdb4): Unmounting Filesystem
[ 69.321334] XFS: Assertion failed: atomic_read(&pag->pag_intents) == 0, file: fs/xfs/xfs_mount.c, line: 152
[ 69.331066] ------------[ cut here ]------------
[ 69.335675] kernel BUG at fs/xfs/xfs_message.c:110!
[ 69.340564] invalid opcode: 0000 [#1] SMP PTI
[ 69.344918] CPU: 3 PID: 12251 Comm: umount Not tainted 5.13.0-rc6-00320-g18bb7d74fbd6 #1
[ 69.352995] Hardware name: Dell Inc. OptiPlex 9020/03CPWF, BIOS A11 04/01/2015
[ 69.360198] RIP: 0010:assfail (kbuild/src/consumer/fs/xfs/./xfs_trace.h:164) xfs
[ 69.364634] Code: 67 fc ff ff 0f 0b c3 0f 1f 44 00 00 41 89 c8 48 89 d1 48 89 f2 48 c7 c6 80 48 81 c0 e8 82 f9 ff ff 80 3d 24 df 0c 00 00 74 02 <0f> 0b 0f 0b c3 48 8d 45 10 48 89 e2 4c 89 e6 48 89 1c 24 48 89 44
All code
========
0: 67 fc addr32 cld
2: ff (bad)
3: ff 0f decl (%rdi)
5: 0b c3 or %ebx,%eax
7: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
c: 41 89 c8 mov %ecx,%r8d
f: 48 89 d1 mov %rdx,%rcx
12: 48 89 f2 mov %rsi,%rdx
15: 48 c7 c6 80 48 81 c0 mov $0xffffffffc0814880,%rsi
1c: e8 82 f9 ff ff callq 0xfffffffffffff9a3
21: 80 3d 24 df 0c 00 00 cmpb $0x0,0xcdf24(%rip) # 0xcdf4c
28: 74 02 je 0x2c
2a:* 0f 0b ud2 <-- trapping instruction
2c: 0f 0b ud2
2e: c3 retq
2f: 48 8d 45 10 lea 0x10(%rbp),%rax
33: 48 89 e2 mov %rsp,%rdx
36: 4c 89 e6 mov %r12,%rsi
39: 48 89 1c 24 mov %rbx,(%rsp)
3d: 48 rex.W
3e: 89 .byte 0x89
3f: 44 rex.R
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 0f 0b ud2
4: c3 retq
5: 48 8d 45 10 lea 0x10(%rbp),%rax
9: 48 89 e2 mov %rsp,%rdx
c: 4c 89 e6 mov %r12,%rsi
f: 48 89 1c 24 mov %rbx,(%rsp)
13: 48 rex.W
14: 89 .byte 0x89
15: 44 rex.R
[ 69.383359] RSP: 0018:ffffc9000996fdf0 EFLAGS: 00010202
[ 69.388568] RAX: 0000000000000000 RBX: ffff888191d86000 RCX: 0000000000000000
[ 69.395683] RDX: 00000000ffffffc0 RSI: 000000000000000a RDI: ffffffffc0805241
[ 69.402799] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 69.409921] R10: 000000000000000a R11: f000000000000000 R12: ffff888192010580
[ 69.417045] R13: ffff888192010000 R14: ffff888192010570 R15: ffff888191774f80
[ 69.424161] FS: 00007f9f57447080(0000) GS:ffff88819eac0000(0000) knlGS:0000000000000000
[ 69.432229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 69.437958] CR2: 000055acc59e3ae8 CR3: 0000000191c4e003 CR4: 00000000001706e0
[ 69.445072] Call Trace:
[ 69.447513] xfs_free_perag (kbuild/src/consumer/fs/xfs/xfs_mount.c:152 (discriminator 1)) xfs
[ 69.451917] xfs_unmountfs (kbuild/src/consumer/fs/xfs/xfs_mount.c:1239) xfs
[ 69.456152] xfs_fs_put_super (kbuild/src/consumer/fs/xfs/xfs_super.c:1099) xfs
[ 69.460553] generic_shutdown_super (kbuild/src/consumer/include/linux/list.h:282 kbuild/src/consumer/fs/super.c:467)
[ 69.464987] kill_block_super (kbuild/src/consumer/fs/super.c:1396)
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
Thanks,
Oliver Sang
1 year, 3 months
[mm/mremap] ecf8443e51: vm-scalability.throughput -29.4% regression
by kernel test robot
Greeting,
FYI, we noticed a -29.4% regression of vm-scalability.throughput due to commit:
commit: ecf8443e51a862b261313c2319ab4e4aed9e6b7e ("[PATCH v7 02/11] mm/mremap: Fix race between MOVE_PUD mremap and pageout")
url: https://github.com/0day-ci/linux/commits/Aneesh-Kumar-K-V/Speedup-mremap-...
base: https://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git next
in testcase: vm-scalability
on test machine: 192 threads 4 sockets Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
with following parameters:
runtime: 300
thp_enabled: never
thp_defrag: always
nr_task: 8
nr_ssd: 1
test: swap-w-seq-mt
cpufreq_governor: performance
ucode: 0x4003006
test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_ssd/nr_task/rootfs/runtime/tbox_group/test/testcase/thp_defrag/thp_enabled/ucode:
gcc-9/performance/x86_64-rhel-8.3/1/8/debian-10.4-x86_64-20200603.cgz/300/lkp-csl-2ap1/swap-w-seq-mt/vm-scalability/always/never/0x4003006
commit:
5f80ee2fc0 ("mm/mremap: Fix race between MOVE_PMD mremap and pageout")
ecf8443e51 ("mm/mremap: Fix race between MOVE_PUD mremap and pageout")
5f80ee2fc08b3613 ecf8443e51a862b261313c2319a
---------------- ---------------------------
%stddev %change %stddev
\ | \
7.93 ± 4% +14.6% 9.08 ± 5% vm-scalability.free_time
373210 -29.4% 263582 ± 3% vm-scalability.median
43.45 ± 5% -8.1 35.34 ± 7% vm-scalability.stddev%
2954757 -29.4% 2085715 ± 3% vm-scalability.throughput
220.03 +38.3% 304.37 ± 2% vm-scalability.time.elapsed_time
220.03 +38.3% 304.37 ± 2% vm-scalability.time.elapsed_time.max
32410 ± 2% +29.9% 42117 ± 10% vm-scalability.time.involuntary_context_switches
528.33 +13.1% 597.67 vm-scalability.time.percent_of_cpu_this_job_got
913.19 ± 2% +70.7% 1558 ± 3% vm-scalability.time.system_time
250.22 +4.7% 261.96 vm-scalability.time.user_time
34211 ± 11% -95.9% 1400 ± 38% vm-scalability.time.voluntary_context_switches
4.82 ± 3% +15.8% 5.57 ± 3% iostat.cpu.system
3.77e+08 ± 24% +46.4% 5.52e+08 ± 19% turbostat.IRQ
271.03 +30.2% 352.88 ± 2% uptime.boot
48496 +30.1% 63095 ± 2% uptime.idle
0.66 ± 12% -0.7 0.00 ± 57% mpstat.cpu.all.iowait%
0.06 ± 4% -0.0 0.06 ± 4% mpstat.cpu.all.soft%
3.09 ± 2% +0.9 4.03 ± 3% mpstat.cpu.all.sys%
0.60 -0.1 0.48 ± 4% mpstat.cpu.all.usr%
28255 ± 5% +77.2% 50067 ± 6% meminfo.Active
27866 ± 5% +78.2% 49655 ± 6% meminfo.Active(anon)
28180965 -20.0% 22548765 ± 3% meminfo.MemAvailable
28675444 -19.7% 23016466 ± 3% meminfo.MemFree
38693 ± 5% +56.1% 60412 ± 2% meminfo.Shmem
496106 ± 7% +2180.0% 11311305 ± 12% meminfo.SwapCached
60640 ± 3% -97.9% 1245 ± 46% meminfo.Writeback
87575 ± 3% -43.8% 49246 ± 8% slabinfo.dmaengine-unmap-16.active_objs
2123 ± 3% -44.5% 1179 ± 8% slabinfo.dmaengine-unmap-16.active_slabs
89210 ± 3% -44.5% 49532 ± 8% slabinfo.dmaengine-unmap-16.num_objs
2123 ± 3% -44.5% 1179 ± 8% slabinfo.dmaengine-unmap-16.num_slabs
4866 -20.4% 3874 ± 3% slabinfo.kmalloc-4k.active_objs
623.83 -20.9% 493.50 ± 3% slabinfo.kmalloc-4k.active_slabs
4992 -20.8% 3952 ± 3% slabinfo.kmalloc-4k.num_objs
623.83 -20.9% 493.50 ± 3% slabinfo.kmalloc-4k.num_slabs
374.17 ± 2% -47.8% 195.17 ± 53% vmstat.io.bi
1341570 -23.6% 1025527 ± 2% vmstat.io.bo
28897122 -21.2% 22757749 ± 2% vmstat.memory.free
1.238e+08 +8.1% 1.339e+08 ± 2% vmstat.memory.swpd
1.00 -100.0% 0.00 vmstat.procs.b
8.00 +29.2% 10.33 ± 7% vmstat.procs.r
364.17 ± 2% -48.3% 188.17 ± 55% vmstat.swap.si
1341561 -23.6% 1025520 ± 2% vmstat.swap.so
3328 ± 2% -19.9% 2666 vmstat.system.cs
1511984 +17.7% 1780129 vmstat.system.in
14215 ± 15% -98.5% 209.67 ± 84% numa-meminfo.node0.Writeback
308351 ±116% +1024.4% 3467064 ± 11% numa-meminfo.node1.FilePages
168771 ± 6% -15.1% 143367 ± 12% numa-meminfo.node1.KReclaimable
168771 ± 6% -15.1% 143367 ± 12% numa-meminfo.node1.SReclaimable
127034 ± 32% +2618.6% 3453577 ± 10% numa-meminfo.node1.SwapCached
14382 ± 9% -97.7% 332.00 ± 34% numa-meminfo.node1.Writeback
124583 ± 35% +2602.3% 3366554 ± 14% numa-meminfo.node2.FilePages
77286 ± 26% +55.8% 120430 ± 18% numa-meminfo.node2.KReclaimable
77286 ± 26% +55.8% 120430 ± 18% numa-meminfo.node2.SReclaimable
138348 ± 20% +28.1% 177225 ± 13% numa-meminfo.node2.Slab
113067 ± 23% +2874.7% 3363404 ± 14% numa-meminfo.node2.SwapCached
14263 ± 5% -97.4% 373.17 ± 37% numa-meminfo.node2.Writeback
13835 ± 13% +155.7% 35375 ± 18% numa-meminfo.node3.Active
13835 ± 13% +155.2% 35308 ± 18% numa-meminfo.node3.Active(anon)
261289 ±137% +1236.3% 3491534 ± 7% numa-meminfo.node3.FilePages
16541 ± 12% +130.9% 38192 ± 21% numa-meminfo.node3.Shmem
81973 ± 30% +3903.0% 3281379 ± 9% numa-meminfo.node3.SwapCached
16321 ± 6% -97.7% 372.33 ± 42% numa-meminfo.node3.Writeback
3643 ± 14% -98.5% 55.00 ±102% numa-vmstat.node0.nr_writeback
3619 ± 14% -98.6% 49.67 ±107% numa-vmstat.node0.nr_zone_write_pending
77012 ±116% +1039.9% 877855 ± 10% numa-vmstat.node1.nr_file_pages
42097 ± 6% -14.0% 36208 ± 10% numa-vmstat.node1.nr_slab_reclaimable
31679 ± 32% +2660.4% 874478 ± 10% numa-vmstat.node1.nr_swapcached
11014726 ± 2% +12.6% 12401236 ± 4% numa-vmstat.node1.nr_vmscan_write
3577 ± 10% -97.5% 89.83 ± 41% numa-vmstat.node1.nr_writeback
11011253 ± 2% +12.6% 12401174 ± 4% numa-vmstat.node1.nr_written
3502 ± 10% -97.7% 82.17 ± 49% numa-vmstat.node1.nr_zone_write_pending
31302 ± 35% +2617.2% 850555 ± 13% numa-vmstat.node2.nr_file_pages
19271 ± 26% +57.8% 30407 ± 17% numa-vmstat.node2.nr_slab_reclaimable
28421 ± 24% +2889.9% 849766 ± 12% numa-vmstat.node2.nr_swapcached
11041559 ± 3% +19.1% 13149436 ± 10% numa-vmstat.node2.nr_vmscan_write
3782 ± 4% -97.6% 92.00 ± 34% numa-vmstat.node2.nr_writeback
11037863 ± 3% +19.1% 13149374 ± 10% numa-vmstat.node2.nr_written
3724 ± 4% -97.9% 79.00 ± 39% numa-vmstat.node2.nr_zone_write_pending
3443 ± 14% +160.0% 8954 ± 20% numa-vmstat.node3.nr_active_anon
65304 ±137% +1252.7% 883364 ± 6% numa-vmstat.node3.nr_file_pages
4129 ± 12% +134.5% 9682 ± 23% numa-vmstat.node3.nr_shmem
20479 ± 30% +3956.2% 830690 ± 8% numa-vmstat.node3.nr_swapcached
10829696 ± 5% +14.0% 12346461 ± 4% numa-vmstat.node3.nr_vmscan_write
4124 ± 4% -97.6% 97.00 ± 39% numa-vmstat.node3.nr_writeback
10825671 ± 5% +14.0% 12346386 ± 4% numa-vmstat.node3.nr_written
3444 ± 14% +160.0% 8955 ± 20% numa-vmstat.node3.nr_zone_active_anon
4055 ± 4% -97.9% 83.67 ± 43% numa-vmstat.node3.nr_zone_write_pending
8412 ± 3% +28.0% 10768 ± 13% proc-vmstat.allocstall_movable
5996587 ± 46% +183.4% 16993010 ± 32% proc-vmstat.compact_free_scanned
3451053 ± 29% +75.7% 6064315 ± 15% proc-vmstat.compact_isolated
3259087 ± 27% +89.7% 6184085 ± 15% proc-vmstat.compact_migrate_scanned
6961 ± 5% +78.5% 12425 ± 6% proc-vmstat.nr_active_anon
40993758 -3.1% 39718983 proc-vmstat.nr_anon_pages
698799 -19.8% 560310 ± 3% proc-vmstat.nr_dirty_background_threshold
1399308 -19.8% 1121992 ± 3% proc-vmstat.nr_dirty_threshold
388625 ± 2% +695.9% 3092997 ± 11% proc-vmstat.nr_file_pages
7165507 -19.3% 5779541 ± 3% proc-vmstat.nr_free_pages
41119695 +3.5% 42542747 proc-vmstat.nr_inactive_anon
145036 +3.1% 149568 proc-vmstat.nr_page_table_pages
9675 ± 5% +56.0% 15094 ± 2% proc-vmstat.nr_shmem
123879 ± 8% +2178.6% 2822769 ± 12% proc-vmstat.nr_swapcached
38677277 +7.7% 41656286 ± 3% proc-vmstat.nr_vmscan_write
14819 ± 3% -97.9% 307.33 ± 50% proc-vmstat.nr_writeback
74789620 +5.5% 78882933 ± 2% proc-vmstat.nr_written
6962 ± 5% +78.5% 12426 ± 6% proc-vmstat.nr_zone_active_anon
41119448 +3.5% 42542544 proc-vmstat.nr_zone_inactive_anon
14582 ± 3% -98.1% 274.50 ± 47% proc-vmstat.nr_zone_write_pending
20967960 ± 16% +27.7% 26777776 ± 10% proc-vmstat.numa_pte_updates
16061 ± 13% +34.4% 21592 ± 7% proc-vmstat.pgactivate
1726355 ± 29% +75.7% 3033559 ± 15% proc-vmstat.pgmigrate_success
2.992e+08 +5.5% 3.155e+08 ± 2% proc-vmstat.pgpgout
48003 ± 2% +32.3% 63514 ± 3% proc-vmstat.pgreuse
74393768 -12.6% 65057070 ± 4% proc-vmstat.pgrotated
2.36e+08 +8.4% 2.558e+08 proc-vmstat.pgscan_anon
1.048e+08 ± 2% +24.6% 1.305e+08 ± 6% proc-vmstat.pgscan_direct
74792004 +5.5% 78884465 ± 2% proc-vmstat.pswpout
50067 ± 7% +17.0% 58567 ± 4% proc-vmstat.slabs_scanned
4.724e+09 -18.5% 3.85e+09 perf-stat.i.branch-instructions
3231 ± 2% -19.3% 2608 perf-stat.i.context-switches
1.86 ± 2% +47.3% 2.74 ± 5% perf-stat.i.cpi
3.281e+10 ± 2% +15.9% 3.803e+10 ± 4% perf-stat.i.cpu-cycles
206.24 -1.9% 202.23 perf-stat.i.cpu-migrations
642.02 ± 6% +14.2% 732.98 ± 8% perf-stat.i.cycles-between-cache-misses
4.851e+09 -17.3% 4.014e+09 perf-stat.i.dTLB-loads
3304908 -19.0% 2677570 ± 4% perf-stat.i.dTLB-store-misses
2.027e+09 -19.4% 1.633e+09 perf-stat.i.dTLB-stores
5484115 ± 3% -14.5% 4691022 ± 3% perf-stat.i.iTLB-load-misses
1.857e+10 -17.5% 1.533e+10 perf-stat.i.instructions
5181 ± 3% -16.9% 4306 ± 2% perf-stat.i.instructions-per-iTLB-miss
0.59 ± 3% -24.6% 0.44 ± 3% perf-stat.i.ipc
23.06 ± 6% -49.2% 11.72 ± 54% perf-stat.i.major-faults
0.17 ± 2% +15.9% 0.20 ± 4% perf-stat.i.metric.GHz
60.43 -18.1% 49.46 perf-stat.i.metric.M/sec
585912 -26.9% 428491 ± 2% perf-stat.i.minor-faults
87.62 +3.0 90.65 perf-stat.i.node-load-miss-rate%
1450164 ± 6% -29.1% 1027918 ± 3% perf-stat.i.node-loads
83.34 +4.1 87.48 perf-stat.i.node-store-miss-rate%
1130049 ± 10% -24.5% 853398 ± 11% perf-stat.i.node-stores
585935 -26.9% 428503 ± 2% perf-stat.i.page-faults
1.77 ± 2% +42.5% 2.52 ± 5% perf-stat.overall.cpi
560.05 ± 6% +22.0% 683.28 ± 11% perf-stat.overall.cycles-between-cache-misses
0.56 ± 2% -29.7% 0.40 ± 5% perf-stat.overall.ipc
91.72 +2.0 93.70 perf-stat.overall.node-load-miss-rate%
86.56 +3.1 89.65 perf-stat.overall.node-store-miss-rate%
7112 +13.7% 8086 perf-stat.overall.path-length
4.703e+09 -18.7% 3.821e+09 ± 2% perf-stat.ps.branch-instructions
3220 ± 2% -19.7% 2587 perf-stat.ps.context-switches
3.277e+10 ± 2% +17.3% 3.843e+10 ± 5% perf-stat.ps.cpu-cycles
205.01 -2.7% 199.55 ± 2% perf-stat.ps.cpu-migrations
4.834e+09 -17.5% 3.987e+09 perf-stat.ps.dTLB-loads
3283502 -19.6% 2639256 ± 4% perf-stat.ps.dTLB-store-misses
2.02e+09 -19.9% 1.618e+09 perf-stat.ps.dTLB-stores
5471166 ± 3% -15.0% 4652576 ± 2% perf-stat.ps.iTLB-load-misses
1.85e+10 -17.7% 1.523e+10 perf-stat.ps.instructions
22.79 ± 6% -49.2% 11.57 ± 54% perf-stat.ps.major-faults
581983 -27.5% 422213 ± 2% perf-stat.ps.minor-faults
1442428 ± 6% -29.8% 1013002 ± 3% perf-stat.ps.node-loads
1123088 ± 10% -25.3% 839225 ± 10% perf-stat.ps.node-stores
582006 -27.5% 422225 ± 2% perf-stat.ps.page-faults
4.085e+12 +13.6% 4.642e+12 perf-stat.total.instructions
0.01 ± 23% -49.2% 0.01 ± 13% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.55 ±221% -99.5% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.02 ± 32% -100.0% 0.00 perf-sched.sch_delay.avg.ms.kswapd.kthread.ret_from_fork
0.01 ± 62% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
0.01 ± 17% -47.2% 0.00 ± 10% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.02 ± 21% -49.1% 0.01 ± 22% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.02 ± 37% -60.0% 0.01 ± 34% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
166.76 ±223% -100.0% 0.01 ± 10% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.03 ± 67% -62.5% 0.01 ± 51% perf-sched.sch_delay.max.ms.io_schedule.__lock_page_or_retry.do_swap_page.__handle_mm_fault
0.03 ±130% -77.5% 0.01 ± 61% perf-sched.sch_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
0.02 ± 39% -100.0% 0.00 perf-sched.sch_delay.max.ms.kswapd.kthread.ret_from_fork
0.09 ± 59% -85.5% 0.01 ± 10% perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.05 ± 77% -89.3% 0.01 ± 56% perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
0.02 ± 96% -100.0% 0.00 perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
0.02 ± 36% -62.8% 0.01 ± 21% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
1018 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1018 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
15.36 ± 84% -97.3% 0.42 ± 78% perf-sched.wait_and_delay.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
2042 ± 37% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
8.00 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
8.00 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
93.67 ±133% +261.4% 338.50 ± 12% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__alloc_pages_slowpath.constprop.0
20.17 ±223% +547.1% 130.50 ± 10% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.mempool_alloc.bio_alloc_bioset.__swap_writepage
346.50 ± 74% +194.0% 1018 ± 10% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.shrink_node.do_try_to_free_pages.try_to_free_pages
9.00 ± 19% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
23.00 ± 5% +24.6% 28.67 ± 10% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
5996 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
5996 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
4492 ± 79% -99.9% 3.21 ± 71% perf-sched.wait_and_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
6025 ± 32% -82.0% 1082 ± 4% perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
6024 ± 32% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
1018 ± 14% -99.9% 1.33 ± 15% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1018 ± 14% -99.8% 1.60 ± 57% perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
309.68 ±143% -100.0% 0.01 ± 27% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
224.46 ±171% -99.9% 0.13 ±110% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
378.18 ± 74% -99.8% 0.69 ± 48% perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
15.36 ± 84% -97.3% 0.42 ± 78% perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
105.18 -100.0% 0.00 perf-sched.wait_time.avg.ms.kswapd.kthread.ret_from_fork
57.26 ± 71% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
2.81 ± 22% -68.2% 0.89 ± 45% perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
754.86 ±100% -100.0% 0.00 perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
298.39 ±141% -100.0% 0.00 perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
2042 ± 37% -100.0% 0.37 ± 99% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 ± 10% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kswapd.kthread.ret_from_fork
5996 ± 31% -100.0% 2.66 ± 15% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
5996 ± 31% -100.0% 2.56 ± 22% perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
1179 ±142% -100.0% 0.03 ± 89% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
818.62 ±149% -100.0% 0.13 ±110% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
4236 ± 54% -99.9% 3.55 ± 58% perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
4492 ± 79% -99.9% 3.21 ± 71% perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
107.44 -100.0% 0.00 perf-sched.wait_time.max.ms.kswapd.kthread.ret_from_fork
6025 ± 32% -82.0% 1082 ± 4% perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
2367 ± 71% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
5.00 -56.8% 2.16 ± 93% perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
1888 ±101% -100.0% 0.00 perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1240 ±140% -100.0% 0.00 perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
6024 ± 32% -100.0% 0.74 ±107% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ± 13% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kswapd.kthread.ret_from_fork
443.33 +38.2% 612.67 ± 2% interrupts.9:IO-APIC.9-fasteoi.acpi
2.506e+08 ± 2% +71.0% 4.285e+08 ± 2% interrupts.CAL:Function_call_interrupts
436770 +39.6% 609678 ± 3% interrupts.CPU0.LOC:Local_timer_interrupts
443.33 +38.2% 612.67 ± 2% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
436473 +39.6% 609534 ± 3% interrupts.CPU1.LOC:Local_timer_interrupts
436112 +39.7% 609460 ± 3% interrupts.CPU10.LOC:Local_timer_interrupts
435885 +39.8% 609476 ± 3% interrupts.CPU100.LOC:Local_timer_interrupts
435912 +39.8% 609519 ± 3% interrupts.CPU101.LOC:Local_timer_interrupts
435983 +39.8% 609496 ± 3% interrupts.CPU102.LOC:Local_timer_interrupts
436229 +39.7% 609544 ± 3% interrupts.CPU103.LOC:Local_timer_interrupts
152.33 ± 49% +944.5% 1591 ± 79% interrupts.CPU103.NMI:Non-maskable_interrupts
152.33 ± 49% +944.5% 1591 ± 79% interrupts.CPU103.PMI:Performance_monitoring_interrupts
436056 +39.8% 609461 ± 3% interrupts.CPU104.LOC:Local_timer_interrupts
435861 +39.8% 609478 ± 3% interrupts.CPU105.LOC:Local_timer_interrupts
436021 +39.8% 609379 ± 3% interrupts.CPU106.LOC:Local_timer_interrupts
435843 +39.9% 609554 ± 3% interrupts.CPU107.LOC:Local_timer_interrupts
435999 +39.8% 609466 ± 3% interrupts.CPU108.LOC:Local_timer_interrupts
436372 +39.7% 609463 ± 3% interrupts.CPU109.LOC:Local_timer_interrupts
435946 +39.8% 609652 ± 3% interrupts.CPU11.LOC:Local_timer_interrupts
436096 +39.8% 609463 ± 3% interrupts.CPU110.LOC:Local_timer_interrupts
435856 +39.8% 609436 ± 3% interrupts.CPU111.LOC:Local_timer_interrupts
436180 +39.7% 609470 ± 3% interrupts.CPU112.LOC:Local_timer_interrupts
435902 +39.8% 609466 ± 3% interrupts.CPU113.LOC:Local_timer_interrupts
435823 +39.9% 609521 ± 3% interrupts.CPU114.LOC:Local_timer_interrupts
435831 +39.8% 609416 ± 3% interrupts.CPU115.LOC:Local_timer_interrupts
435716 +39.9% 609492 ± 3% interrupts.CPU116.LOC:Local_timer_interrupts
435934 +39.8% 609518 ± 3% interrupts.CPU117.LOC:Local_timer_interrupts
435946 +39.8% 609459 ± 3% interrupts.CPU118.LOC:Local_timer_interrupts
435735 +39.9% 609492 ± 3% interrupts.CPU119.LOC:Local_timer_interrupts
436018 +39.8% 609544 ± 3% interrupts.CPU12.LOC:Local_timer_interrupts
1730294 ± 54% +231.1% 5729689 ± 73% interrupts.CPU120.CAL:Function_call_interrupts
436513 +38.6% 605176 ± 4% interrupts.CPU120.LOC:Local_timer_interrupts
437125 +38.4% 605185 ± 4% interrupts.CPU121.LOC:Local_timer_interrupts
119.00 ± 49% +2582.6% 3192 ± 78% interrupts.CPU121.NMI:Non-maskable_interrupts
119.00 ± 49% +2582.6% 3192 ± 78% interrupts.CPU121.PMI:Performance_monitoring_interrupts
436783 +38.6% 605196 ± 4% interrupts.CPU122.LOC:Local_timer_interrupts
436134 +38.8% 605202 ± 4% interrupts.CPU123.LOC:Local_timer_interrupts
436402 +38.7% 605173 ± 4% interrupts.CPU124.LOC:Local_timer_interrupts
436347 +38.7% 605194 ± 4% interrupts.CPU125.LOC:Local_timer_interrupts
436510 +38.7% 605240 ± 4% interrupts.CPU126.LOC:Local_timer_interrupts
436805 +38.5% 605169 ± 4% interrupts.CPU127.LOC:Local_timer_interrupts
436659 +38.6% 605187 ± 4% interrupts.CPU128.LOC:Local_timer_interrupts
436231 +38.7% 605179 ± 4% interrupts.CPU129.LOC:Local_timer_interrupts
436500 +39.6% 609536 ± 3% interrupts.CPU13.LOC:Local_timer_interrupts
436424 +38.7% 605184 ± 4% interrupts.CPU130.LOC:Local_timer_interrupts
436297 +38.7% 605194 ± 4% interrupts.CPU131.LOC:Local_timer_interrupts
436562 +38.6% 605189 ± 4% interrupts.CPU132.LOC:Local_timer_interrupts
436922 +38.5% 605198 ± 4% interrupts.CPU133.LOC:Local_timer_interrupts
436581 +38.6% 605204 ± 4% interrupts.CPU134.LOC:Local_timer_interrupts
436103 +38.8% 605178 ± 4% interrupts.CPU135.LOC:Local_timer_interrupts
436818 +38.5% 605171 ± 4% interrupts.CPU136.LOC:Local_timer_interrupts
436327 +38.7% 605173 ± 4% interrupts.CPU137.LOC:Local_timer_interrupts
436190 +38.7% 605185 ± 4% interrupts.CPU138.LOC:Local_timer_interrupts
436439 +38.7% 605194 ± 4% interrupts.CPU139.LOC:Local_timer_interrupts
436084 +39.8% 609534 ± 3% interrupts.CPU14.LOC:Local_timer_interrupts
436238 +38.7% 605188 ± 4% interrupts.CPU140.LOC:Local_timer_interrupts
436692 +38.6% 605188 ± 4% interrupts.CPU141.LOC:Local_timer_interrupts
436428 +38.7% 605187 ± 4% interrupts.CPU142.LOC:Local_timer_interrupts
436618 +38.6% 605181 ± 4% interrupts.CPU143.LOC:Local_timer_interrupts
2395060 ± 21% +210.7% 7440320 ± 40% interrupts.CPU144.CAL:Function_call_interrupts
437076 ± 2% +38.3% 604600 ± 4% interrupts.CPU144.LOC:Local_timer_interrupts
4849080 ± 21% +113.2% 10339751 ± 39% interrupts.CPU144.TLB:TLB_shootdowns
437303 ± 2% +38.3% 604631 ± 4% interrupts.CPU145.LOC:Local_timer_interrupts
437337 ± 2% +38.3% 604650 ± 4% interrupts.CPU146.LOC:Local_timer_interrupts
1628946 ± 35% +144.0% 3974501 ± 26% interrupts.CPU147.CAL:Function_call_interrupts
436987 ± 2% +38.4% 604641 ± 4% interrupts.CPU147.LOC:Local_timer_interrupts
3313139 ± 38% +62.6% 5387495 ± 26% interrupts.CPU147.TLB:TLB_shootdowns
437013 ± 2% +38.4% 604647 ± 4% interrupts.CPU148.LOC:Local_timer_interrupts
437047 ± 2% +38.3% 604641 ± 4% interrupts.CPU149.LOC:Local_timer_interrupts
435764 +39.9% 609486 ± 3% interrupts.CPU15.LOC:Local_timer_interrupts
437104 ± 2% +38.3% 604641 ± 4% interrupts.CPU150.LOC:Local_timer_interrupts
437148 ± 2% +38.3% 604674 ± 4% interrupts.CPU151.LOC:Local_timer_interrupts
437106 ± 2% +38.3% 604644 ± 4% interrupts.CPU152.LOC:Local_timer_interrupts
437065 ± 2% +38.4% 604688 ± 4% interrupts.CPU153.LOC:Local_timer_interrupts
437182 ± 2% +38.3% 604648 ± 4% interrupts.CPU154.LOC:Local_timer_interrupts
437068 ± 2% +38.3% 604649 ± 4% interrupts.CPU155.LOC:Local_timer_interrupts
437126 ± 2% +38.3% 604638 ± 4% interrupts.CPU156.LOC:Local_timer_interrupts
437340 ± 2% +38.3% 604639 ± 4% interrupts.CPU157.LOC:Local_timer_interrupts
437229 ± 2% +38.3% 604631 ± 4% interrupts.CPU158.LOC:Local_timer_interrupts
437031 ± 2% +38.4% 604646 ± 4% interrupts.CPU159.LOC:Local_timer_interrupts
436217 +39.7% 609502 ± 3% interrupts.CPU16.LOC:Local_timer_interrupts
437281 ± 2% +38.3% 604671 ± 4% interrupts.CPU160.LOC:Local_timer_interrupts
437174 ± 2% +38.3% 604664 ± 4% interrupts.CPU161.LOC:Local_timer_interrupts
437093 ± 2% +38.3% 604644 ± 4% interrupts.CPU162.LOC:Local_timer_interrupts
437080 ± 2% +38.3% 604617 ± 4% interrupts.CPU163.LOC:Local_timer_interrupts
437014 ± 2% +38.4% 604623 ± 4% interrupts.CPU164.LOC:Local_timer_interrupts
437064 ± 2% +38.3% 604664 ± 4% interrupts.CPU165.LOC:Local_timer_interrupts
437060 ± 2% +38.3% 604645 ± 4% interrupts.CPU166.LOC:Local_timer_interrupts
437133 ± 2% +38.3% 604647 ± 4% interrupts.CPU167.LOC:Local_timer_interrupts
437691 +38.8% 607710 ± 3% interrupts.CPU168.LOC:Local_timer_interrupts
437778 +38.8% 607719 ± 3% interrupts.CPU169.LOC:Local_timer_interrupts
333.83 ±130% +630.9% 2440 ±113% interrupts.CPU169.NMI:Non-maskable_interrupts
333.83 ±130% +630.9% 2440 ±113% interrupts.CPU169.PMI:Performance_monitoring_interrupts
435893 +39.8% 609457 ± 3% interrupts.CPU17.LOC:Local_timer_interrupts
437845 +38.8% 607745 ± 3% interrupts.CPU170.LOC:Local_timer_interrupts
972.00 ±199% +287.4% 3765 ± 75% interrupts.CPU170.NMI:Non-maskable_interrupts
972.00 ±199% +287.4% 3765 ± 75% interrupts.CPU170.PMI:Performance_monitoring_interrupts
437428 +38.9% 607740 ± 3% interrupts.CPU171.LOC:Local_timer_interrupts
437494 +38.9% 607739 ± 3% interrupts.CPU172.LOC:Local_timer_interrupts
1590512 ± 27% +139.8% 3814104 ± 44% interrupts.CPU173.CAL:Function_call_interrupts
437559 +38.9% 607732 ± 3% interrupts.CPU173.LOC:Local_timer_interrupts
437721 +38.8% 607747 ± 3% interrupts.CPU174.LOC:Local_timer_interrupts
437316 +39.0% 607744 ± 3% interrupts.CPU175.LOC:Local_timer_interrupts
437481 +38.9% 607717 ± 3% interrupts.CPU176.LOC:Local_timer_interrupts
437671 +38.9% 607816 ± 3% interrupts.CPU177.LOC:Local_timer_interrupts
437549 +38.9% 607776 ± 3% interrupts.CPU178.LOC:Local_timer_interrupts
437791 +38.8% 607753 ± 3% interrupts.CPU179.LOC:Local_timer_interrupts
435869 +39.8% 609482 ± 3% interrupts.CPU18.LOC:Local_timer_interrupts
437851 +38.8% 607737 ± 3% interrupts.CPU180.LOC:Local_timer_interrupts
437762 +38.8% 607765 ± 3% interrupts.CPU181.LOC:Local_timer_interrupts
437506 +38.9% 607755 ± 3% interrupts.CPU182.LOC:Local_timer_interrupts
437825 +38.8% 607732 ± 3% interrupts.CPU183.LOC:Local_timer_interrupts
437482 +38.9% 607739 ± 3% interrupts.CPU184.LOC:Local_timer_interrupts
437502 +38.9% 607760 ± 3% interrupts.CPU185.LOC:Local_timer_interrupts
437904 +38.8% 607748 ± 3% interrupts.CPU186.LOC:Local_timer_interrupts
437601 +38.9% 607731 ± 3% interrupts.CPU187.LOC:Local_timer_interrupts
437425 +38.9% 607746 ± 3% interrupts.CPU188.LOC:Local_timer_interrupts
437660 +38.9% 607742 ± 3% interrupts.CPU189.LOC:Local_timer_interrupts
435885 +39.8% 609492 ± 3% interrupts.CPU19.LOC:Local_timer_interrupts
658863 ± 99% +449.3% 3618986 ± 71% interrupts.CPU190.CAL:Function_call_interrupts
437525 +38.9% 607724 ± 3% interrupts.CPU190.LOC:Local_timer_interrupts
437714 +38.9% 607775 ± 3% interrupts.CPU191.LOC:Local_timer_interrupts
436340 +39.7% 609513 ± 3% interrupts.CPU2.LOC:Local_timer_interrupts
435793 +39.9% 609516 ± 3% interrupts.CPU20.LOC:Local_timer_interrupts
435929 +39.8% 609516 ± 3% interrupts.CPU21.LOC:Local_timer_interrupts
435897 +39.8% 609589 ± 3% interrupts.CPU22.LOC:Local_timer_interrupts
436105 +39.7% 609457 ± 3% interrupts.CPU23.LOC:Local_timer_interrupts
436542 +38.6% 605219 ± 4% interrupts.CPU24.LOC:Local_timer_interrupts
437153 +38.4% 605215 ± 4% interrupts.CPU25.LOC:Local_timer_interrupts
436932 +38.5% 605126 ± 4% interrupts.CPU26.LOC:Local_timer_interrupts
436176 +38.8% 605266 ± 4% interrupts.CPU27.LOC:Local_timer_interrupts
436415 +38.7% 605223 ± 4% interrupts.CPU28.LOC:Local_timer_interrupts
436413 +38.7% 605249 ± 4% interrupts.CPU29.LOC:Local_timer_interrupts
435892 +39.8% 609496 ± 3% interrupts.CPU3.LOC:Local_timer_interrupts
1884 ± 93% -90.6% 177.17 ± 79% interrupts.CPU3.NMI:Non-maskable_interrupts
1884 ± 93% -90.6% 177.17 ± 79% interrupts.CPU3.PMI:Performance_monitoring_interrupts
3717304 ± 27% -51.0% 1822336 ± 39% interrupts.CPU3.TLB:TLB_shootdowns
436566 +38.6% 605260 ± 4% interrupts.CPU30.LOC:Local_timer_interrupts
436820 +38.5% 605209 ± 4% interrupts.CPU31.LOC:Local_timer_interrupts
436693 +38.6% 605210 ± 4% interrupts.CPU32.LOC:Local_timer_interrupts
436239 +38.7% 605207 ± 4% interrupts.CPU33.LOC:Local_timer_interrupts
436425 +38.7% 605210 ± 4% interrupts.CPU34.LOC:Local_timer_interrupts
436281 +38.7% 605214 ± 4% interrupts.CPU35.LOC:Local_timer_interrupts
436621 +38.6% 605203 ± 4% interrupts.CPU36.LOC:Local_timer_interrupts
436949 +38.5% 605220 ± 4% interrupts.CPU37.LOC:Local_timer_interrupts
436637 +38.6% 605224 ± 4% interrupts.CPU38.LOC:Local_timer_interrupts
436124 +38.8% 605199 ± 4% interrupts.CPU39.LOC:Local_timer_interrupts
435874 +39.8% 609500 ± 3% interrupts.CPU4.LOC:Local_timer_interrupts
436826 +38.5% 605202 ± 4% interrupts.CPU40.LOC:Local_timer_interrupts
436386 +38.7% 605204 ± 4% interrupts.CPU41.LOC:Local_timer_interrupts
436202 +38.7% 605125 ± 4% interrupts.CPU42.LOC:Local_timer_interrupts
436444 +38.7% 605206 ± 4% interrupts.CPU43.LOC:Local_timer_interrupts
436315 +38.7% 605211 ± 4% interrupts.CPU44.LOC:Local_timer_interrupts
436710 +38.6% 605216 ± 4% interrupts.CPU45.LOC:Local_timer_interrupts
436448 +38.7% 605224 ± 4% interrupts.CPU46.LOC:Local_timer_interrupts
436683 +38.6% 605205 ± 4% interrupts.CPU47.LOC:Local_timer_interrupts
437175 ± 2% +38.3% 604605 ± 4% interrupts.CPU48.LOC:Local_timer_interrupts
428230 ± 65% +243.5% 1470847 ± 61% interrupts.CPU49.CAL:Function_call_interrupts
437324 ± 2% +38.3% 604629 ± 4% interrupts.CPU49.LOC:Local_timer_interrupts
435982 +39.8% 609551 ± 3% interrupts.CPU5.LOC:Local_timer_interrupts
437347 ± 2% +38.3% 604679 ± 4% interrupts.CPU50.LOC:Local_timer_interrupts
436995 ± 2% +38.4% 604678 ± 4% interrupts.CPU51.LOC:Local_timer_interrupts
437070 ± 2% +38.3% 604662 ± 4% interrupts.CPU52.LOC:Local_timer_interrupts
437067 ± 2% +38.3% 604680 ± 4% interrupts.CPU53.LOC:Local_timer_interrupts
437130 ± 2% +38.3% 604671 ± 4% interrupts.CPU54.LOC:Local_timer_interrupts
437155 ± 2% +38.3% 604700 ± 4% interrupts.CPU55.LOC:Local_timer_interrupts
437141 ± 2% +38.3% 604682 ± 4% interrupts.CPU56.LOC:Local_timer_interrupts
437048 ± 2% +38.4% 604671 ± 4% interrupts.CPU57.LOC:Local_timer_interrupts
437189 ± 2% +38.3% 604701 ± 4% interrupts.CPU58.LOC:Local_timer_interrupts
437069 ± 2% +38.4% 604688 ± 4% interrupts.CPU59.LOC:Local_timer_interrupts
436032 +39.8% 609555 ± 3% interrupts.CPU6.LOC:Local_timer_interrupts
437190 ± 2% +38.3% 604670 ± 4% interrupts.CPU60.LOC:Local_timer_interrupts
437388 ± 2% +38.2% 604673 ± 4% interrupts.CPU61.LOC:Local_timer_interrupts
437227 ± 2% +38.3% 604666 ± 4% interrupts.CPU62.LOC:Local_timer_interrupts
437036 ± 2% +38.4% 604662 ± 4% interrupts.CPU63.LOC:Local_timer_interrupts
437304 ± 2% +38.3% 604662 ± 4% interrupts.CPU64.LOC:Local_timer_interrupts
437177 ± 2% +38.3% 604654 ± 4% interrupts.CPU65.LOC:Local_timer_interrupts
437090 ± 2% +38.3% 604659 ± 4% interrupts.CPU66.LOC:Local_timer_interrupts
437100 ± 2% +38.3% 604602 ± 4% interrupts.CPU67.LOC:Local_timer_interrupts
437019 ± 2% +38.4% 604618 ± 4% interrupts.CPU68.LOC:Local_timer_interrupts
437048 ± 2% +38.4% 604675 ± 4% interrupts.CPU69.LOC:Local_timer_interrupts
436254 +39.7% 609555 ± 3% interrupts.CPU7.LOC:Local_timer_interrupts
140.83 ± 25% +500.0% 845.00 ± 93% interrupts.CPU7.NMI:Non-maskable_interrupts
140.83 ± 25% +500.0% 845.00 ± 93% interrupts.CPU7.PMI:Performance_monitoring_interrupts
437047 ± 2% +38.4% 604663 ± 4% interrupts.CPU70.LOC:Local_timer_interrupts
437164 ± 2% +38.3% 604658 ± 4% interrupts.CPU71.LOC:Local_timer_interrupts
437731 +38.8% 607745 ± 3% interrupts.CPU72.LOC:Local_timer_interrupts
437796 +38.8% 607772 ± 3% interrupts.CPU73.LOC:Local_timer_interrupts
437891 +38.8% 607769 ± 3% interrupts.CPU74.LOC:Local_timer_interrupts
488.67 ±180% +653.0% 3679 ± 75% interrupts.CPU74.NMI:Non-maskable_interrupts
488.67 ±180% +653.0% 3679 ± 75% interrupts.CPU74.PMI:Performance_monitoring_interrupts
456635 ± 60% +233.5% 1522817 ± 41% interrupts.CPU75.CAL:Function_call_interrupts
437451 +38.9% 607764 ± 3% interrupts.CPU75.LOC:Local_timer_interrupts
437531 +38.9% 607754 ± 3% interrupts.CPU76.LOC:Local_timer_interrupts
437603 +38.9% 607768 ± 3% interrupts.CPU77.LOC:Local_timer_interrupts
437752 +38.8% 607791 ± 3% interrupts.CPU78.LOC:Local_timer_interrupts
437335 +39.0% 607782 ± 3% interrupts.CPU79.LOC:Local_timer_interrupts
436062 +39.8% 609477 ± 3% interrupts.CPU8.LOC:Local_timer_interrupts
437520 +38.9% 607783 ± 3% interrupts.CPU80.LOC:Local_timer_interrupts
437687 +38.9% 607803 ± 3% interrupts.CPU81.LOC:Local_timer_interrupts
437567 +38.9% 607820 ± 3% interrupts.CPU82.LOC:Local_timer_interrupts
437815 +38.8% 607773 ± 3% interrupts.CPU83.LOC:Local_timer_interrupts
437892 +38.8% 607756 ± 3% interrupts.CPU84.LOC:Local_timer_interrupts
437781 +38.8% 607806 ± 3% interrupts.CPU85.LOC:Local_timer_interrupts
437537 +38.9% 607769 ± 3% interrupts.CPU86.LOC:Local_timer_interrupts
437848 +38.8% 607766 ± 3% interrupts.CPU87.LOC:Local_timer_interrupts
437543 +38.9% 607809 ± 3% interrupts.CPU88.LOC:Local_timer_interrupts
437531 +38.9% 607776 ± 3% interrupts.CPU89.LOC:Local_timer_interrupts
435850 +39.8% 609496 ± 3% interrupts.CPU9.LOC:Local_timer_interrupts
437953 +38.8% 607759 ± 3% interrupts.CPU90.LOC:Local_timer_interrupts
437616 +38.9% 607766 ± 3% interrupts.CPU91.LOC:Local_timer_interrupts
437458 +38.9% 607762 ± 3% interrupts.CPU92.LOC:Local_timer_interrupts
437697 +38.9% 607787 ± 3% interrupts.CPU93.LOC:Local_timer_interrupts
437556 +38.9% 607804 ± 3% interrupts.CPU94.LOC:Local_timer_interrupts
437772 +38.8% 607836 ± 3% interrupts.CPU95.LOC:Local_timer_interrupts
436537 +39.6% 609467 ± 3% interrupts.CPU96.LOC:Local_timer_interrupts
436496 +39.6% 609495 ± 3% interrupts.CPU97.LOC:Local_timer_interrupts
436305 +39.7% 609468 ± 3% interrupts.CPU98.LOC:Local_timer_interrupts
435736 +39.9% 609454 ± 3% interrupts.CPU99.LOC:Local_timer_interrupts
83872531 +38.9% 1.165e+08 ± 4% interrupts.LOC:Local_timer_interrupts
0.00 +1.9e+104% 192.00 interrupts.MCP:Machine_check_polls
122140 ± 6% +15.2% 140671 ± 5% interrupts.NMI:Non-maskable_interrupts
122140 ± 6% +15.2% 140671 ± 5% interrupts.PMI:Performance_monitoring_interrupts
5.166e+08 ± 2% +15.1% 5.948e+08 ± 2% interrupts.TLB:TLB_shootdowns
17.91 ± 6% -7.3 10.57 ± 2% perf-profile.calltrace.cycles-pp.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
17.90 ± 6% -7.3 10.56 ± 2% perf-profile.calltrace.cycles-pp.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec
17.22 ± 6% -7.0 10.22 ± 2% perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list
16.87 ± 6% -6.8 10.07 ± 2% perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list
40.74 ± 7% -4.6 36.13 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
40.30 ± 8% -4.5 35.84 ± 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
40.30 ± 8% -4.5 35.84 ± 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
40.30 ± 8% -4.5 35.83 ± 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
38.05 ± 9% -4.4 33.60 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
37.65 ± 9% -4.4 33.26 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
4.33 ± 9% -3.2 1.13 ± 22% perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
5.57 ± 6% -2.7 2.85 ± 4% perf-profile.calltrace.cycles-pp.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
3.96 ± 5% -2.2 1.76 ± 7% perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
3.72 ± 6% -1.8 1.90 ± 8% perf-profile.calltrace.cycles-pp.__swap_writepage.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec
3.91 ± 6% -1.8 2.13 ± 3% perf-profile.calltrace.cycles-pp.llist_add_batch.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty
1.83 ± 6% -1.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty
3.04 ± 11% -0.9 2.17 ± 11% perf-profile.calltrace.cycles-pp.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
5.24 ± 7% -0.8 4.43 ± 4% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
3.40 ± 6% -0.8 2.62 ± 10% perf-profile.calltrace.cycles-pp.do_rw_once
4.81 ± 7% -0.7 4.06 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
1.94 ± 11% -0.7 1.19 ± 9% perf-profile.calltrace.cycles-pp.__mem_cgroup_charge.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault
3.31 ± 8% -0.5 2.78 ± 5% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
3.29 ± 8% -0.5 2.76 ± 5% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.61 ± 5% -0.3 0.28 ±100% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
1.15 ± 17% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_slowpath.__alloc_pages.alloc_pages_vma.do_anonymous_page
0.73 ± 6% -0.1 0.62 ± 6% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.46 ± 44% +0.4 0.83 ± 17% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.do_access
1.58 ± 8% +0.5 2.07 ± 6% perf-profile.calltrace.cycles-pp.__default_send_IPI_dest_field.default_send_IPI_mask_sequence_phys.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush
0.00 +2.2 2.25 ± 4% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one
0.00 +2.4 2.38 ± 8% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one
0.00 +3.4 3.37 ± 9% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath
0.00 +3.4 3.41 ± 9% perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock
0.00 +3.7 3.72 ± 6% perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk
5.56 ± 8% +10.7 16.22 ± 3% perf-profile.calltrace.cycles-pp.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec
5.84 ± 8% +10.8 16.68 ± 3% perf-profile.calltrace.cycles-pp.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
4.96 ± 8% +10.9 15.85 ± 3% perf-profile.calltrace.cycles-pp.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list
0.10 ±223% +12.3 12.40 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon
0.10 ±223% +12.5 12.62 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon
0.34 ±109% +13.2 13.56 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap
2.00 ± 12% +13.4 15.36 ± 3% perf-profile.calltrace.cycles-pp.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec
2.13 ± 12% +13.4 15.48 ± 3% perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
0.32 ±105% +13.4 13.74 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced
0.50 ± 77% +13.5 14.04 ± 3% perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list
0.51 ± 76% +13.9 14.39 ± 3% perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list
0.87 ± 22% +14.0 14.83 ± 3% perf-profile.calltrace.cycles-pp.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list
18.00 ± 6% -7.4 10.64 ± 2% perf-profile.children.cycles-pp.try_to_unmap_flush_dirty
17.99 ± 6% -7.4 10.64 ± 2% perf-profile.children.cycles-pp.arch_tlbbatch_flush
17.31 ± 6% -7.0 10.29 ± 2% perf-profile.children.cycles-pp.on_each_cpu_cond_mask
17.30 ± 6% -7.0 10.29 ± 3% perf-profile.children.cycles-pp.smp_call_function_many_cond
40.74 ± 7% -4.6 36.13 ± 4% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
40.74 ± 7% -4.6 36.13 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry
40.74 ± 7% -4.6 36.13 ± 4% perf-profile.children.cycles-pp.do_idle
38.48 ± 8% -4.6 33.90 ± 4% perf-profile.children.cycles-pp.cpuidle_enter
38.46 ± 8% -4.6 33.89 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state
40.30 ± 8% -4.5 35.84 ± 4% perf-profile.children.cycles-pp.start_secondary
4.34 ± 9% -3.0 1.33 ± 8% perf-profile.children.cycles-pp.__remove_mapping
5.60 ± 6% -2.7 2.87 ± 4% perf-profile.children.cycles-pp.pageout
3.99 ± 5% -2.1 1.84 ± 3% perf-profile.children.cycles-pp.add_to_swap
4.12 ± 5% -1.8 2.29 ± 6% perf-profile.children.cycles-pp.llist_add_batch
3.74 ± 6% -1.8 1.98 ± 4% perf-profile.children.cycles-pp.__swap_writepage
1.78 ± 12% -1.4 0.38 ± 13% perf-profile.children.cycles-pp.mem_cgroup_swapout
1.64 ± 6% -1.1 0.55 ± 4% perf-profile.children.cycles-pp.get_swap_page
3.25 ± 11% -1.0 2.28 ± 11% perf-profile.children.cycles-pp.mem_cgroup_charge
1.57 ± 6% -0.9 0.66 ± 6% perf-profile.children.cycles-pp.swap_writepage
3.90 ± 6% -0.9 3.01 ± 11% perf-profile.children.cycles-pp.do_rw_once
5.60 ± 7% -0.8 4.78 ± 4% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
1.09 ± 19% -0.8 0.29 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.35 ± 6% -0.8 0.57 ± 5% perf-profile.children.cycles-pp._swap_info_get
1.16 ± 5% -0.8 0.39 ± 8% perf-profile.children.cycles-pp.page_swap_info
5.12 ± 7% -0.7 4.37 ± 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
1.14 ± 5% -0.7 0.39 ± 8% perf-profile.children.cycles-pp.swap_set_page_dirty
2.02 ± 11% -0.7 1.30 ± 9% perf-profile.children.cycles-pp.__mem_cgroup_charge
0.88 ± 13% -0.7 0.19 ± 18% perf-profile.children.cycles-pp.page_counter_uncharge
2.00 ± 10% -0.6 1.36 ± 11% perf-profile.children.cycles-pp.get_page_from_freelist
0.79 ± 13% -0.6 0.16 ± 18% perf-profile.children.cycles-pp.page_counter_cancel
1.26 ± 2% -0.6 0.64 ± 7% perf-profile.children.cycles-pp.handle_irq_event
1.27 ± 2% -0.6 0.65 ± 7% perf-profile.children.cycles-pp.__common_interrupt
1.25 ± 2% -0.6 0.64 ± 7% perf-profile.children.cycles-pp.handle_irq_event_percpu
1.26 ± 2% -0.6 0.65 ± 7% perf-profile.children.cycles-pp.handle_edge_irq
1.25 ± 2% -0.6 0.64 ± 7% perf-profile.children.cycles-pp.__handle_irq_event_percpu
1.24 ± 2% -0.6 0.63 ± 7% perf-profile.children.cycles-pp.nvme_irq
1.28 ± 2% -0.6 0.67 ± 6% perf-profile.children.cycles-pp.common_interrupt
1.28 ± 2% -0.6 0.67 ± 6% perf-profile.children.cycles-pp.asm_common_interrupt
0.93 ± 5% -0.6 0.34 ± 5% perf-profile.children.cycles-pp.swap_page_sector
1.05 ± 7% -0.6 0.48 ± 4% perf-profile.children.cycles-pp.try_to_free_swap
1.02 ± 7% -0.6 0.46 ± 5% perf-profile.children.cycles-pp.page_swapcount
3.59 ± 8% -0.5 3.05 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
3.56 ± 8% -0.5 3.04 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt
1.10 ± 2% -0.5 0.58 ± 8% perf-profile.children.cycles-pp.blk_update_request
1.11 ± 2% -0.5 0.59 ± 7% perf-profile.children.cycles-pp.blk_mq_end_request
0.70 ± 9% -0.5 0.20 ± 10% perf-profile.children.cycles-pp.swap_duplicate
0.70 ± 10% -0.5 0.19 ± 11% perf-profile.children.cycles-pp.__swap_duplicate
1.32 ± 5% -0.5 0.83 ± 2% perf-profile.children.cycles-pp.add_to_swap_cache
0.69 ± 9% -0.5 0.23 ± 5% perf-profile.children.cycles-pp.get_swap_pages
1.21 ± 11% -0.4 0.77 ± 12% perf-profile.children.cycles-pp.prep_new_page
1.13 ± 11% -0.4 0.71 ± 12% perf-profile.children.cycles-pp.clear_page_erms
0.93 ± 12% -0.4 0.52 ± 11% perf-profile.children.cycles-pp.try_charge
0.84 ± 5% -0.4 0.44 ± 4% perf-profile.children.cycles-pp.bio_alloc_bioset
1.12 ± 8% -0.4 0.74 ± 8% perf-profile.children.cycles-pp.submit_bio
0.45 ± 22% -0.4 0.06 ± 11% perf-profile.children.cycles-pp.swap_cgroup_record
1.11 ± 7% -0.4 0.72 ± 8% perf-profile.children.cycles-pp.submit_bio_noacct
0.66 ± 9% -0.4 0.29 ± 8% perf-profile.children.cycles-pp.put_swap_page
0.67 ± 6% -0.4 0.30 ± 7% perf-profile.children.cycles-pp.__delete_from_swap_cache
0.84 ± 8% -0.4 0.47 ± 8% perf-profile.children.cycles-pp.page_lock_anon_vma_read
0.57 ± 8% -0.4 0.20 ± 5% perf-profile.children.cycles-pp.scan_swap_map_slots
0.51 ± 3% -0.3 0.18 ± 11% perf-profile.children.cycles-pp.__frontswap_store
0.65 ± 3% -0.3 0.32 ± 9% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.72 ± 15% -0.3 0.39 ± 11% perf-profile.children.cycles-pp._find_next_bit
0.59 ± 8% -0.3 0.28 ± 7% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.80 ± 4% -0.3 0.48 ± 8% perf-profile.children.cycles-pp.end_swap_bio_write
0.79 ± 4% -0.3 0.48 ± 8% perf-profile.children.cycles-pp.end_page_writeback
0.61 ± 5% -0.3 0.31 ± 7% perf-profile.children.cycles-pp.native_flush_tlb_multi
0.38 ± 8% -0.3 0.09 ± 12% perf-profile.children.cycles-pp.mem_cgroup_id_get_online
0.72 ± 7% -0.3 0.43 ± 5% perf-profile.children.cycles-pp.blk_mq_submit_bio
0.64 ± 11% -0.3 0.35 ± 7% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
0.47 ± 8% -0.3 0.21 ± 7% perf-profile.children.cycles-pp.sysvec_call_function_single
0.44 ± 7% -0.3 0.19 ± 7% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.52 ± 9% -0.2 0.27 ± 4% perf-profile.children.cycles-pp.cgroup_rstat_updated
0.56 ± 4% -0.2 0.33 ± 6% perf-profile.children.cycles-pp.bio_init
0.28 ± 13% -0.2 0.06 ± 13% perf-profile.children.cycles-pp.get_swap_device
0.36 ± 26% -0.2 0.15 ± 26% perf-profile.children.cycles-pp.cpumask_any_but
0.54 ± 5% -0.2 0.36 ± 4% perf-profile.children.cycles-pp.__mod_lruvec_page_state
0.27 ± 11% -0.2 0.09 ± 12% perf-profile.children.cycles-pp.scan_swap_map_try_ssd_cluster
0.44 ± 8% -0.2 0.26 ± 4% perf-profile.children.cycles-pp.isolate_lru_pages
0.38 ± 8% -0.2 0.20 ± 8% perf-profile.children.cycles-pp.__set_page_dirty_no_writeback
0.42 ± 7% -0.2 0.25 ± 10% perf-profile.children.cycles-pp.__list_del_entry_valid
0.27 ± 8% -0.2 0.11 ± 10% perf-profile.children.cycles-pp.mempool_alloc
0.43 ± 6% -0.2 0.26 ± 10% perf-profile.children.cycles-pp.rotate_reclaimable_page
0.30 ± 8% -0.2 0.14 ± 8% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
0.29 ± 8% -0.2 0.13 ± 5% perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
0.39 ± 10% -0.2 0.23 ± 4% perf-profile.children.cycles-pp.__test_set_page_writeback
0.30 ± 8% -0.2 0.14 ± 6% perf-profile.children.cycles-pp.blk_flush_plug_list
0.29 ± 9% -0.2 0.13 ± 6% perf-profile.children.cycles-pp.nvme_queue_rq
0.28 ± 8% -0.2 0.13 ± 5% perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly
0.28 ± 8% -0.2 0.13 ± 6% perf-profile.children.cycles-pp.__blk_mq_try_issue_directly
0.28 ± 9% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.blk_mq_request_issue_directly
0.48 ± 7% -0.1 0.33 ± 4% perf-profile.children.cycles-pp.cpumask_next
0.25 ± 8% -0.1 0.10 ± 9% perf-profile.children.cycles-pp.nvme_map_data
0.38 ± 5% -0.1 0.24 ± 9% perf-profile.children.cycles-pp.pagevec_lru_move_fn
0.40 ± 7% -0.1 0.26 ± 5% perf-profile.children.cycles-pp.rmqueue
0.21 ± 8% -0.1 0.08 ± 20% perf-profile.children.cycles-pp.kmem_cache_alloc
0.34 ± 7% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.free_unref_page_list
0.34 ± 5% -0.1 0.23 ± 7% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.46 ± 11% -0.1 0.35 ± 9% perf-profile.children.cycles-pp.lru_cache_add
0.30 ± 13% -0.1 0.20 ± 10% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.23 ± 6% -0.1 0.12 ± 9% perf-profile.children.cycles-pp.rmqueue_bulk
0.42 ± 10% -0.1 0.32 ± 10% perf-profile.children.cycles-pp.__pagevec_lru_add
0.62 ± 5% -0.1 0.53 ± 7% perf-profile.children.cycles-pp.__softirqentry_text_start
0.12 ± 9% -0.1 0.03 ±100% perf-profile.children.cycles-pp.kmem_cache_free
0.21 ± 5% -0.1 0.13 ± 8% perf-profile.children.cycles-pp.xas_store
0.23 ± 7% -0.1 0.14 ± 14% perf-profile.children.cycles-pp.pagevec_move_tail_fn
0.22 ± 5% -0.1 0.14 ± 3% perf-profile.children.cycles-pp.__mod_node_page_state
0.24 ± 3% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.xas_create
0.23 ± 10% -0.1 0.15 ± 4% perf-profile.children.cycles-pp.free_pcppages_bulk
0.33 ± 8% -0.1 0.25 ± 13% perf-profile.children.cycles-pp.submit_bio_checks
0.24 ± 8% -0.1 0.17 ± 9% perf-profile.children.cycles-pp.blk_attempt_plug_merge
0.21 ± 5% -0.1 0.14 ± 9% perf-profile.children.cycles-pp.test_clear_page_writeback
0.20 ± 6% -0.1 0.14 ± 8% perf-profile.children.cycles-pp.__mod_lruvec_state
0.18 ± 2% -0.1 0.11 ± 4% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.17 ± 8% -0.1 0.11 ± 11% perf-profile.children.cycles-pp.page_add_new_anon_rmap
0.09 ± 14% -0.1 0.03 ±101% perf-profile.children.cycles-pp.__count_memcg_events
0.19 ± 11% -0.1 0.13 ± 6% perf-profile.children.cycles-pp.page_mapping
0.20 ± 6% -0.1 0.14 ± 11% perf-profile.children.cycles-pp.load_balance
0.15 ± 10% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.page_remove_rmap
0.16 ± 2% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.update_sd_lb_stats
0.18 ± 11% -0.1 0.12 ± 10% perf-profile.children.cycles-pp.bio_attempt_back_merge
0.16 ± 4% -0.0 0.11 ± 14% perf-profile.children.cycles-pp.find_busiest_group
0.13 ± 8% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.mutex_unlock
0.07 ± 9% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.__mod_memcg_state
0.07 ± 7% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.release_pages
0.10 ± 17% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.lock_page_lruvec_irqsave
0.06 ± 11% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.mutex_lock
0.11 ± 6% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.memset_erms
0.14 ± 10% -0.0 0.10 ± 19% perf-profile.children.cycles-pp.page_counter_try_charge
0.10 ± 11% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.___might_sleep
0.10 ± 9% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.10 ± 10% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.blk_throtl_bio
0.08 ± 15% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.rcu_idle_exit
0.08 ± 6% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.__blk_queue_split
0.11 ± 9% -0.0 0.08 ± 15% perf-profile.children.cycles-pp.__perf_sw_event
0.11 ± 7% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.ll_back_merge_fn
0.09 ± 7% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.__cond_resched
0.12 ± 7% -0.0 0.09 ± 12% perf-profile.children.cycles-pp.update_rq_clock
0.08 ± 16% -0.0 0.05 ± 47% perf-profile.children.cycles-pp.bio_associate_blkg_from_css
0.07 ± 10% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.___perf_sw_event
0.10 ± 10% -0.0 0.08 ± 4% perf-profile.children.cycles-pp.hrtimer_update_next_event
0.09 ± 7% -0.0 0.07 ± 16% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.07 ± 7% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.update_irq_load_avg
0.09 ± 7% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.08 ± 10% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.update_blocked_averages
1.01 ± 7% +0.2 1.20 ± 6% perf-profile.children.cycles-pp.native_flush_tlb_local
2.16 ± 7% +0.4 2.54 ± 2% perf-profile.children.cycles-pp.default_send_IPI_mask_sequence_phys
1.59 ± 8% +0.6 2.15 ± 3% perf-profile.children.cycles-pp.__default_send_IPI_dest_field
42.06 ± 5% +8.5 50.53 ± 2% perf-profile.children.cycles-pp.shrink_node
42.00 ± 5% +8.5 50.49 ± 2% perf-profile.children.cycles-pp.shrink_lruvec
41.98 ± 5% +8.5 50.48 ± 2% perf-profile.children.cycles-pp.shrink_inactive_list
40.95 ± 5% +8.9 49.83 ± 2% perf-profile.children.cycles-pp.shrink_page_list
5.08 ± 8% +11.0 16.05 ± 3% perf-profile.children.cycles-pp.try_to_unmap_one
5.86 ± 8% +11.0 16.84 ± 3% perf-profile.children.cycles-pp.try_to_unmap
2.15 ± 12% +13.5 15.64 ± 3% perf-profile.children.cycles-pp.page_referenced
1.30 ± 17% +13.7 15.00 ± 3% perf-profile.children.cycles-pp.page_referenced_one
7.62 ± 9% +24.3 31.90 ± 3% perf-profile.children.cycles-pp.rmap_walk_anon
1.58 ± 18% +24.6 26.23 ± 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
2.42 ± 19% +25.7 28.14 ± 3% perf-profile.children.cycles-pp._raw_spin_lock
1.80 ± 27% +27.0 28.78 ± 3% perf-profile.children.cycles-pp.page_vma_mapped_walk
8.26 ± 8% -4.0 4.26 ± 5% perf-profile.self.cycles-pp.smp_call_function_many_cond
3.24 ± 6% -1.1 2.10 ± 9% perf-profile.self.cycles-pp.do_rw_once
2.97 ± 8% -1.0 1.96 ± 10% perf-profile.self.cycles-pp.do_access
2.96 ± 6% -1.0 1.95 ± 6% perf-profile.self.cycles-pp.llist_add_batch
1.73 ± 9% -0.7 0.98 ± 8% perf-profile.self.cycles-pp.try_to_unmap_one
0.79 ± 13% -0.6 0.16 ± 18% perf-profile.self.cycles-pp.page_counter_cancel
1.10 ± 6% -0.6 0.50 ± 5% perf-profile.self.cycles-pp._swap_info_get
0.89 ± 6% -0.6 0.33 ± 8% perf-profile.self.cycles-pp.page_swap_info
0.73 ± 6% -0.5 0.27 ± 5% perf-profile.self.cycles-pp.swap_page_sector
0.98 ± 12% -0.4 0.53 ± 11% perf-profile.self.cycles-pp.clear_page_erms
0.59 ± 6% -0.4 0.17 ± 6% perf-profile.self.cycles-pp.get_swap_page
0.69 ± 13% -0.4 0.31 ± 12% perf-profile.self.cycles-pp.try_charge
2.74 ± 7% -0.3 2.40 ± 7% perf-profile.self.cycles-pp.flush_tlb_func
0.38 ± 8% -0.3 0.09 ± 10% perf-profile.self.cycles-pp.mem_cgroup_id_get_online
0.74 ± 7% -0.3 0.46 ± 4% perf-profile.self.cycles-pp.add_to_swap_cache
0.45 ± 9% -0.3 0.18 ± 10% perf-profile.self.cycles-pp.__delete_from_swap_cache
0.52 ± 10% -0.3 0.26 ± 5% perf-profile.self.cycles-pp.cgroup_rstat_updated
0.42 ± 10% -0.2 0.19 ± 12% perf-profile.self.cycles-pp.mem_cgroup_charge
0.55 ± 16% -0.2 0.33 ± 12% perf-profile.self.cycles-pp._find_next_bit
0.32 ± 4% -0.2 0.11 ± 9% perf-profile.self.cycles-pp.__frontswap_store
0.46 ± 4% -0.2 0.27 ± 8% perf-profile.self.cycles-pp.bio_init
0.32 ± 5% -0.2 0.16 ± 9% perf-profile.self.cycles-pp.native_flush_tlb_multi
0.40 ± 7% -0.2 0.24 ± 10% perf-profile.self.cycles-pp.__list_del_entry_valid
0.31 ± 16% -0.1 0.17 ± 21% perf-profile.self.cycles-pp.page_lock_anon_vma_read
0.19 ± 13% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.scan_swap_map_slots
0.28 ± 9% -0.1 0.17 ± 11% perf-profile.self.cycles-pp.__set_page_dirty_no_writeback
0.36 ± 5% -0.1 0.25 ± 3% perf-profile.self.cycles-pp.shrink_page_list
0.33 ± 8% -0.1 0.22 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.26 ± 8% -0.1 0.16 ± 4% perf-profile.self.cycles-pp.isolate_lru_pages
0.56 ± 6% -0.1 0.47 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state
0.29 ± 12% -0.1 0.21 ± 17% perf-profile.self.cycles-pp.get_page_from_freelist
0.16 ± 6% -0.1 0.08 ± 9% perf-profile.self.cycles-pp.end_page_writeback
0.29 ± 11% -0.1 0.21 ± 12% perf-profile.self.cycles-pp.__mem_cgroup_charge
0.20 ± 5% -0.1 0.13 ± 6% perf-profile.self.cycles-pp.__mod_node_page_state
0.22 ± 3% -0.1 0.15 ± 6% perf-profile.self.cycles-pp.xas_create
0.23 ± 8% -0.1 0.16 ± 7% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.13 ± 9% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.rmqueue_bulk
0.17 ± 9% -0.1 0.10 ± 9% perf-profile.self.cycles-pp.__mod_lruvec_page_state
0.14 ± 8% -0.1 0.08 ± 7% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.15 ± 8% -0.1 0.09 ± 7% perf-profile.self.cycles-pp.__test_set_page_writeback
0.08 ± 9% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.__swap_writepage
0.13 ± 12% -0.1 0.08 ± 9% perf-profile.self.cycles-pp.free_pcppages_bulk
0.11 ± 12% -0.0 0.06 ± 47% perf-profile.self.cycles-pp.page_counter_try_charge
0.15 ± 11% -0.0 0.11 ± 7% perf-profile.self.cycles-pp.page_mapping
0.12 ± 12% -0.0 0.08 ± 19% perf-profile.self.cycles-pp.__pagevec_lru_add
0.08 ± 8% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.pagevec_lru_move_fn
0.12 ± 6% -0.0 0.08 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.10 ± 10% -0.0 0.07 ± 16% perf-profile.self.cycles-pp.pagevec_move_tail_fn
0.10 ± 10% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.mutex_unlock
0.08 ± 19% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.___might_sleep
0.11 ± 8% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.update_sd_lb_stats
0.09 ± 13% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.08 ± 7% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.memset_erms
0.07 ± 11% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.rotate_reclaimable_page
0.06 ± 7% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.update_irq_load_avg
0.08 ± 4% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.ll_back_merge_fn
0.16 ± 5% +0.1 0.21 ± 5% perf-profile.self.cycles-pp.asm_sysvec_call_function
1.01 ± 7% +0.2 1.20 ± 6% perf-profile.self.cycles-pp.native_flush_tlb_local
2.22 ± 6% +0.4 2.59 ± 4% perf-profile.self.cycles-pp.flush_smp_call_function_queue
1.28 ± 5% +0.4 1.72 perf-profile.self.cycles-pp._raw_spin_lock
1.59 ± 8% +0.6 2.15 ± 3% perf-profile.self.cycles-pp.__default_send_IPI_dest_field
0.34 ± 35% +0.6 0.93 ± 8% perf-profile.self.cycles-pp.page_vma_mapped_walk
1.35 ± 14% +20.2 21.54 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
11952 ± 12% -27.8% 8625 ± 13% softirqs.CPU0.RCU
32374 ± 3% +30.1% 42116 ± 2% softirqs.CPU0.SCHED
10010 ± 15% -33.0% 6705 ± 17% softirqs.CPU1.RCU
29521 ± 3% +25.7% 37120 ± 8% softirqs.CPU1.SCHED
9560 ± 18% -35.2% 6197 ± 12% softirqs.CPU10.RCU
9641 ± 15% -42.8% 5512 ± 13% softirqs.CPU100.RCU
27690 ± 2% +31.8% 36488 ± 3% softirqs.CPU100.SCHED
9404 ± 16% -37.4% 5884 ± 14% softirqs.CPU101.RCU
26702 ± 4% +39.4% 37210 ± 4% softirqs.CPU101.SCHED
9636 ± 16% -40.6% 5728 ± 15% softirqs.CPU102.RCU
26856 ± 6% +37.4% 36889 ± 4% softirqs.CPU102.SCHED
9761 ± 16% -42.7% 5591 ± 14% softirqs.CPU103.RCU
26740 ± 3% +32.4% 35414 ± 8% softirqs.CPU103.SCHED
9279 ± 16% -34.8% 6049 ± 14% softirqs.CPU104.RCU
26363 ± 5% +36.1% 35872 ± 9% softirqs.CPU104.SCHED
9410 ± 16% -38.5% 5787 ± 15% softirqs.CPU105.RCU
26533 ± 9% +41.5% 37539 ± 3% softirqs.CPU105.SCHED
9764 ± 26% -39.6% 5894 ± 9% softirqs.CPU106.RCU
26912 ± 4% +38.8% 37354 ± 5% softirqs.CPU106.SCHED
9551 ± 18% -39.0% 5824 ± 12% softirqs.CPU107.RCU
27262 ± 5% +28.8% 35113 ± 4% softirqs.CPU107.SCHED
9607 ± 17% -37.9% 5968 ± 16% softirqs.CPU108.RCU
26904 ± 3% +34.2% 36105 ± 3% softirqs.CPU108.SCHED
9485 ± 18% -43.7% 5341 ± 7% softirqs.CPU109.RCU
26209 ± 6% +38.5% 36295 ± 2% softirqs.CPU109.SCHED
28056 ± 6% +36.9% 38396 ± 5% softirqs.CPU11.SCHED
9807 ± 14% -39.4% 5942 ± 11% softirqs.CPU110.RCU
26349 ± 4% +36.5% 35954 ± 5% softirqs.CPU110.SCHED
9271 ± 15% -40.6% 5506 ± 11% softirqs.CPU111.RCU
26559 ± 8% +36.5% 36243 ± 4% softirqs.CPU111.SCHED
9830 ± 20% -41.0% 5804 ± 14% softirqs.CPU112.RCU
26422 ± 5% +36.5% 36058 ± 7% softirqs.CPU112.SCHED
9658 ± 19% -37.0% 6088 ± 20% softirqs.CPU113.RCU
26912 ± 2% +38.2% 37204 ± 5% softirqs.CPU113.SCHED
9835 ± 17% -41.0% 5798 ± 13% softirqs.CPU114.RCU
27485 ± 2% +32.6% 36438 ± 7% softirqs.CPU114.SCHED
9878 ± 16% -38.1% 6114 ± 16% softirqs.CPU115.RCU
27028 ± 3% +34.2% 36262 ± 4% softirqs.CPU115.SCHED
9542 ± 17% -37.5% 5967 ± 11% softirqs.CPU116.RCU
26216 ± 7% +43.6% 37650 ± 4% softirqs.CPU116.SCHED
9457 ± 18% -39.8% 5691 ± 12% softirqs.CPU117.RCU
27605 ± 5% +34.0% 36982 ± 3% softirqs.CPU117.SCHED
9567 ± 17% -41.5% 5594 ± 15% softirqs.CPU118.RCU
28212 ± 2% +31.3% 37047 ± 5% softirqs.CPU118.SCHED
9858 ± 16% -39.7% 5943 ± 13% softirqs.CPU119.RCU
28244 ± 4% +32.8% 37516 ± 4% softirqs.CPU119.SCHED
10429 ± 16% -31.2% 7175 ± 16% softirqs.CPU12.RCU
27864 ± 3% +33.1% 37090 ± 11% softirqs.CPU12.SCHED
9672 ± 15% -38.5% 5944 ± 13% softirqs.CPU120.RCU
27469 ± 3% +25.6% 34512 ± 9% softirqs.CPU120.SCHED
9419 ± 17% -37.5% 5884 ± 16% softirqs.CPU121.RCU
26369 ± 7% +24.5% 32820 ± 7% softirqs.CPU121.SCHED
9627 ± 17% -41.6% 5626 ± 14% softirqs.CPU122.RCU
27261 ± 4% +37.2% 37408 ± 5% softirqs.CPU122.SCHED
9753 ± 17% -40.5% 5799 ± 12% softirqs.CPU123.RCU
27689 ± 2% +32.5% 36691 ± 7% softirqs.CPU123.SCHED
10024 ± 18% -40.7% 5942 ± 11% softirqs.CPU124.RCU
27738 ± 2% +31.2% 36403 ± 6% softirqs.CPU124.SCHED
9669 ± 15% -38.7% 5930 ± 14% softirqs.CPU125.RCU
27931 ± 2% +29.1% 36046 ± 7% softirqs.CPU125.SCHED
9931 ± 20% -41.1% 5849 ± 13% softirqs.CPU126.RCU
27307 ± 3% +33.4% 36414 ± 7% softirqs.CPU126.SCHED
9518 ± 14% -39.6% 5745 ± 17% softirqs.CPU127.RCU
27125 ± 5% +32.5% 35941 ± 9% softirqs.CPU127.SCHED
9714 ± 16% -39.9% 5837 ± 10% softirqs.CPU128.RCU
27767 ± 3% +32.7% 36842 ± 4% softirqs.CPU128.SCHED
9659 ± 14% -40.8% 5718 ± 10% softirqs.CPU129.RCU
27759 ± 3% +30.9% 36346 ± 4% softirqs.CPU129.SCHED
10789 ± 26% -46.1% 5820 ± 9% softirqs.CPU13.RCU
26247 ± 15% +45.6% 38218 ± 4% softirqs.CPU13.SCHED
9600 ± 15% -38.2% 5930 ± 16% softirqs.CPU130.RCU
27262 ± 4% +34.1% 36562 ± 6% softirqs.CPU130.SCHED
10393 ± 20% -45.2% 5699 ± 14% softirqs.CPU131.RCU
27624 ± 4% +33.4% 36844 ± 3% softirqs.CPU131.SCHED
9593 ± 14% -40.2% 5737 ± 14% softirqs.CPU132.RCU
26882 ± 9% +34.5% 36151 ± 2% softirqs.CPU132.SCHED
9524 ± 17% -39.6% 5749 ± 12% softirqs.CPU133.RCU
27165 ± 5% +31.0% 35586 ± 7% softirqs.CPU133.SCHED
9714 ± 12% -41.1% 5726 ± 14% softirqs.CPU134.RCU
27285 ± 3% +29.3% 35284 ± 5% softirqs.CPU134.SCHED
9561 ± 13% -38.3% 5901 ± 19% softirqs.CPU135.RCU
27567 ± 3% +28.9% 35546 ± 7% softirqs.CPU135.SCHED
9577 ± 14% -40.4% 5711 ± 12% softirqs.CPU136.RCU
28030 ± 3% +31.5% 36846 ± 5% softirqs.CPU136.SCHED
9717 ± 16% -40.6% 5770 ± 15% softirqs.CPU137.RCU
28225 ± 3% +31.7% 37183 ± 6% softirqs.CPU137.SCHED
9676 ± 14% -41.9% 5624 ± 13% softirqs.CPU138.RCU
27095 ± 3% +38.9% 37642 ± 6% softirqs.CPU138.SCHED
9678 ± 17% -40.0% 5805 ± 16% softirqs.CPU139.RCU
27639 ± 2% +29.6% 35811 ± 10% softirqs.CPU139.SCHED
9842 ± 16% -40.1% 5894 ± 11% softirqs.CPU14.RCU
25932 ± 13% +47.9% 38346 ± 3% softirqs.CPU14.SCHED
9769 ± 14% -40.6% 5807 ± 14% softirqs.CPU140.RCU
28046 ± 3% +29.1% 36200 ± 8% softirqs.CPU140.SCHED
9640 ± 15% -39.3% 5847 ± 15% softirqs.CPU141.RCU
9629 ± 13% -38.5% 5924 ± 12% softirqs.CPU142.RCU
27220 ± 3% +34.6% 36638 ± 5% softirqs.CPU142.SCHED
9691 ± 16% -40.7% 5750 ± 13% softirqs.CPU143.RCU
27774 ± 3% +31.3% 36455 ± 5% softirqs.CPU143.SCHED
9922 ± 17% -38.1% 6143 ± 12% softirqs.CPU144.RCU
26996 ± 4% +27.2% 34330 ± 4% softirqs.CPU144.SCHED
9626 ± 17% -40.9% 5693 ± 13% softirqs.CPU145.RCU
9475 ± 15% -41.8% 5519 ± 11% softirqs.CPU146.RCU
26918 ± 9% +35.7% 36521 ± 6% softirqs.CPU146.SCHED
10026 ± 13% -43.5% 5660 ± 14% softirqs.CPU147.RCU
27369 ± 3% +33.8% 36627 ± 5% softirqs.CPU147.SCHED
10079 ± 12% -42.3% 5813 ± 11% softirqs.CPU148.RCU
26659 ± 2% +41.8% 37791 ± 2% softirqs.CPU148.SCHED
9605 ± 14% -39.7% 5796 ± 12% softirqs.CPU149.RCU
27454 ± 2% +36.1% 37356 ± 7% softirqs.CPU149.SCHED
9534 ± 17% -36.3% 6071 ± 10% softirqs.CPU15.RCU
28826 ± 2% +33.7% 38554 ± 4% softirqs.CPU15.SCHED
9821 ± 14% -35.8% 6309 ± 29% softirqs.CPU150.RCU
27391 ± 4% +34.6% 36880 ± 6% softirqs.CPU150.SCHED
9559 ± 15% -41.6% 5585 ± 11% softirqs.CPU151.RCU
26371 ± 2% +37.9% 36354 ± 6% softirqs.CPU151.SCHED
9662 ± 14% -38.1% 5984 ± 16% softirqs.CPU152.RCU
26885 ± 4% +30.6% 35100 ± 10% softirqs.CPU152.SCHED
9520 ± 14% -40.5% 5664 ± 13% softirqs.CPU153.RCU
27614 ± 4% +33.9% 36988 ± 5% softirqs.CPU153.SCHED
9494 ± 14% -40.8% 5622 ± 13% softirqs.CPU154.RCU
27383 +31.7% 36070 ± 7% softirqs.CPU154.SCHED
9623 ± 13% -40.8% 5699 ± 12% softirqs.CPU155.RCU
28108 ± 2% +28.0% 35984 ± 8% softirqs.CPU155.SCHED
9297 ± 15% -38.2% 5747 ± 14% softirqs.CPU156.RCU
28227 ± 4% +28.6% 36312 ± 6% softirqs.CPU156.SCHED
9421 ± 13% -41.2% 5535 ± 15% softirqs.CPU157.RCU
26196 ± 11% +38.9% 36389 ± 6% softirqs.CPU157.SCHED
9572 ± 15% -42.3% 5518 ± 14% softirqs.CPU158.RCU
27901 ± 2% +31.3% 36647 ± 7% softirqs.CPU158.SCHED
9195 ± 17% -38.7% 5634 ± 17% softirqs.CPU159.RCU
27962 ± 2% +30.4% 36469 ± 6% softirqs.CPU159.SCHED
9822 ± 15% -32.6% 6617 ± 18% softirqs.CPU16.RCU
26952 ± 5% +39.9% 37703 ± 8% softirqs.CPU16.SCHED
9209 ± 20% -41.9% 5355 ± 13% softirqs.CPU160.RCU
27793 ± 2% +29.8% 36073 ± 9% softirqs.CPU160.SCHED
9543 ± 16% -40.5% 5678 ± 13% softirqs.CPU161.RCU
28608 ± 2% +28.7% 36806 ± 3% softirqs.CPU161.SCHED
9399 ± 17% -39.8% 5662 ± 12% softirqs.CPU162.RCU
28119 ± 2% +31.9% 37092 ± 8% softirqs.CPU162.SCHED
9334 ± 17% -40.3% 5568 ± 13% softirqs.CPU163.RCU
28083 +32.2% 37136 ± 5% softirqs.CPU163.SCHED
9558 ± 14% -43.1% 5438 ± 11% softirqs.CPU164.RCU
28339 ± 2% +30.8% 37066 ± 7% softirqs.CPU164.SCHED
9522 ± 15% -42.4% 5485 ± 12% softirqs.CPU165.RCU
28064 ± 3% +32.6% 37204 ± 7% softirqs.CPU165.SCHED
9509 ± 17% -41.0% 5609 ± 15% softirqs.CPU166.RCU
27120 ± 8% +33.2% 36135 ± 7% softirqs.CPU166.SCHED
9335 ± 16% -41.2% 5485 ± 11% softirqs.CPU167.RCU
28405 ± 2% +27.8% 36306 ± 3% softirqs.CPU167.SCHED
25148 ± 4% +38.7% 34890 ± 5% softirqs.CPU168.SCHED
8770 ± 17% -40.3% 5237 ± 10% softirqs.CPU169.RCU
10241 ± 25% -36.9% 6461 ± 16% softirqs.CPU17.RCU
26802 ± 6% +34.2% 35979 ± 10% softirqs.CPU17.SCHED
9208 ± 15% -42.2% 5318 ± 11% softirqs.CPU170.RCU
27330 ± 4% +23.0% 33604 ± 10% softirqs.CPU170.SCHED
8924 ± 17% -39.5% 5400 ± 9% softirqs.CPU171.RCU
28100 ± 3% +31.5% 36959 ± 4% softirqs.CPU171.SCHED
8924 ± 17% -38.0% 5529 ± 10% softirqs.CPU172.RCU
28259 ± 3% +31.1% 37040 ± 4% softirqs.CPU172.SCHED
8875 ± 16% -41.0% 5232 ± 13% softirqs.CPU173.RCU
28254 ± 2% +32.8% 37523 ± 4% softirqs.CPU173.SCHED
9176 ± 15% -42.1% 5317 ± 13% softirqs.CPU174.RCU
27499 ± 3% +38.5% 38081 ± 3% softirqs.CPU174.SCHED
8654 ± 15% -38.4% 5333 ± 13% softirqs.CPU175.RCU
27692 ± 2% +35.4% 37507 ± 3% softirqs.CPU175.SCHED
9225 ± 14% -41.6% 5389 ± 15% softirqs.CPU176.RCU
27375 ± 4% +34.9% 36923 ± 8% softirqs.CPU176.SCHED
9289 ± 15% -43.5% 5249 ± 13% softirqs.CPU177.RCU
28263 ± 3% +32.0% 37296 ± 6% softirqs.CPU177.SCHED
9471 ± 14% -41.8% 5510 ± 15% softirqs.CPU178.RCU
28202 ± 3% +29.6% 36560 ± 5% softirqs.CPU178.SCHED
9263 ± 13% -40.8% 5486 ± 15% softirqs.CPU179.RCU
27974 ± 4% +32.7% 37116 ± 4% softirqs.CPU179.SCHED
9755 ± 20% -38.9% 5961 ± 14% softirqs.CPU18.RCU
28714 ± 2% +33.8% 38408 ± 5% softirqs.CPU18.SCHED
9532 ± 17% -42.7% 5461 ± 17% softirqs.CPU180.RCU
28337 ± 2% +29.8% 36778 ± 3% softirqs.CPU180.SCHED
8762 ± 15% -39.3% 5319 ± 14% softirqs.CPU181.RCU
27484 ± 3% +33.9% 36801 ± 2% softirqs.CPU181.SCHED
9105 ± 14% -39.8% 5483 ± 13% softirqs.CPU182.RCU
27679 ± 4% +33.4% 36912 ± 4% softirqs.CPU182.SCHED
9146 ± 16% -41.3% 5367 ± 15% softirqs.CPU183.RCU
9449 ± 14% -38.4% 5822 ± 25% softirqs.CPU184.RCU
28179 ± 2% +34.7% 37964 ± 3% softirqs.CPU184.SCHED
9703 ± 17% -42.4% 5590 ± 21% softirqs.CPU185.RCU
28451 +35.1% 38440 ± 3% softirqs.CPU185.SCHED
8740 ± 17% -40.4% 5210 ± 11% softirqs.CPU186.RCU
27970 ± 3% +33.9% 37441 ± 3% softirqs.CPU186.SCHED
9322 ± 15% -42.1% 5401 ± 14% softirqs.CPU187.RCU
9517 ± 11% -44.2% 5313 ± 14% softirqs.CPU188.RCU
27852 ± 3% +34.4% 37436 ± 5% softirqs.CPU188.SCHED
9375 ± 12% -42.7% 5375 ± 16% softirqs.CPU189.RCU
27896 ± 3% +34.7% 37573 ± 4% softirqs.CPU189.SCHED
9548 ± 17% -38.0% 5921 ± 12% softirqs.CPU19.RCU
28340 ± 2% +34.4% 38104 ± 5% softirqs.CPU19.SCHED
9329 ± 14% -41.7% 5437 ± 16% softirqs.CPU190.RCU
28512 ± 3% +30.6% 37248 ± 3% softirqs.CPU190.SCHED
9956 ± 12% -38.6% 6112 ± 20% softirqs.CPU191.RCU
9920 ± 17% -35.0% 6447 ± 12% softirqs.CPU2.RCU
9792 ± 16% -38.1% 6058 ± 24% softirqs.CPU20.RCU
9552 ± 17% -37.0% 6016 ± 18% softirqs.CPU21.RCU
28148 +36.7% 38489 ± 6% softirqs.CPU21.SCHED
10016 ± 14% -39.9% 6023 ± 17% softirqs.CPU22.RCU
10216 ± 25% -42.6% 5863 ± 11% softirqs.CPU23.RCU
28979 ± 3% +32.4% 38368 ± 8% softirqs.CPU23.SCHED
10358 ± 15% -38.4% 6381 ± 11% softirqs.CPU24.RCU
26220 ± 4% +37.7% 36110 ± 8% softirqs.CPU24.SCHED
9726 ± 15% -35.9% 6235 ± 14% softirqs.CPU25.RCU
10261 ± 14% -41.7% 5986 ± 14% softirqs.CPU26.RCU
27998 ± 6% +36.6% 38241 ± 6% softirqs.CPU26.SCHED
9785 ± 14% -35.1% 6354 ± 13% softirqs.CPU27.RCU
28204 ± 4% +31.8% 37162 ± 7% softirqs.CPU27.SCHED
10151 ± 16% -39.1% 6183 ± 17% softirqs.CPU28.RCU
27983 ± 4% +35.9% 38026 ± 6% softirqs.CPU28.SCHED
10092 ± 14% -40.2% 6036 ± 15% softirqs.CPU29.RCU
28593 ± 3% +34.5% 38465 ± 4% softirqs.CPU29.SCHED
10086 ± 18% -34.8% 6575 ± 26% softirqs.CPU3.RCU
27482 ± 7% +40.4% 38591 ± 3% softirqs.CPU3.SCHED
9883 ± 16% -37.0% 6229 ± 15% softirqs.CPU30.RCU
29738 ± 7% +28.3% 38155 ± 4% softirqs.CPU30.SCHED
9925 ± 17% -38.8% 6077 ± 16% softirqs.CPU31.RCU
10305 ± 14% -39.1% 6275 ± 12% softirqs.CPU32.RCU
28109 ± 3% +34.8% 37891 ± 4% softirqs.CPU32.SCHED
10590 ± 19% -40.9% 6259 ± 17% softirqs.CPU33.RCU
28261 ± 2% +26.6% 35778 ± 7% softirqs.CPU33.SCHED
10001 ± 16% -38.6% 6142 ± 12% softirqs.CPU34.RCU
28560 +35.0% 38550 ± 3% softirqs.CPU34.SCHED
10201 ± 15% -38.8% 6247 ± 12% softirqs.CPU35.RCU
28600 ± 3% +33.1% 38059 ± 3% softirqs.CPU35.SCHED
10116 ± 13% -38.1% 6260 ± 13% softirqs.CPU36.RCU
10122 ± 15% -40.3% 6039 ± 14% softirqs.CPU37.RCU
28116 ± 3% +37.2% 38569 ± 3% softirqs.CPU37.SCHED
10111 ± 13% -39.1% 6158 ± 14% softirqs.CPU38.RCU
28486 +33.3% 37962 ± 4% softirqs.CPU38.SCHED
10005 ± 14% -38.3% 6174 ± 15% softirqs.CPU39.RCU
9741 ± 16% -41.7% 5676 ± 7% softirqs.CPU4.RCU
28352 ± 3% +35.1% 38298 ± 4% softirqs.CPU4.SCHED
10184 ± 12% -40.0% 6115 ± 13% softirqs.CPU40.RCU
28728 ± 2% +33.1% 38233 ± 4% softirqs.CPU40.SCHED
10194 ± 13% -39.6% 6162 ± 13% softirqs.CPU41.RCU
28822 +31.2% 37804 ± 5% softirqs.CPU41.SCHED
10618 ± 18% -41.7% 6195 ± 13% softirqs.CPU42.RCU
28518 +36.7% 38997 ± 3% softirqs.CPU42.SCHED
10139 ± 15% -38.7% 6218 ± 14% softirqs.CPU43.RCU
28442 ± 2% +31.5% 37398 ± 8% softirqs.CPU43.SCHED
10142 ± 13% -39.8% 6108 ± 11% softirqs.CPU44.RCU
10157 ± 14% -35.7% 6529 ± 19% softirqs.CPU45.RCU
28301 ± 2% +31.6% 37252 ± 8% softirqs.CPU45.SCHED
10304 ± 16% -40.1% 6174 ± 18% softirqs.CPU46.RCU
28728 ± 2% +34.4% 38599 ± 4% softirqs.CPU46.SCHED
10163 ± 14% -39.6% 6136 ± 14% softirqs.CPU47.RCU
28510 ± 5% +33.3% 37991 ± 3% softirqs.CPU47.SCHED
10751 ± 15% -39.5% 6504 ± 18% softirqs.CPU48.RCU
24789 ± 10% +37.8% 34152 ± 5% softirqs.CPU48.SCHED
10001 ± 16% -39.8% 6017 ± 12% softirqs.CPU49.RCU
27400 ± 4% +28.0% 35059 ± 10% softirqs.CPU49.SCHED
9899 ± 16% -37.8% 6155 ± 11% softirqs.CPU5.RCU
25043 ± 26% +52.9% 38283 ± 4% softirqs.CPU5.SCHED
9941 ± 15% -40.9% 5880 ± 13% softirqs.CPU50.RCU
27850 ± 5% +33.9% 37300 ± 5% softirqs.CPU50.SCHED
10580 ± 13% -42.0% 6131 ± 13% softirqs.CPU51.RCU
28440 ± 2% +34.7% 38294 ± 4% softirqs.CPU51.SCHED
10532 ± 11% -41.7% 6144 ± 12% softirqs.CPU52.RCU
27522 ± 3% +40.1% 38549 ± 3% softirqs.CPU52.SCHED
10169 ± 14% -39.9% 6116 ± 11% softirqs.CPU53.RCU
28015 ± 2% +38.0% 38675 ± 6% softirqs.CPU53.SCHED
27943 ± 3% +36.6% 38162 ± 5% softirqs.CPU54.SCHED
9711 ± 15% -39.8% 5848 ± 12% softirqs.CPU55.RCU
28057 ± 2% +32.3% 37105 ± 6% softirqs.CPU55.SCHED
9893 ± 14% -39.3% 6004 ± 11% softirqs.CPU56.RCU
28221 ± 2% +34.3% 37914 ± 6% softirqs.CPU56.SCHED
10168 ± 16% -40.7% 6028 ± 12% softirqs.CPU57.RCU
28823 ± 2% +32.2% 38113 ± 6% softirqs.CPU57.SCHED
10014 ± 14% -39.8% 6033 ± 14% softirqs.CPU58.RCU
28179 ± 2% +33.0% 37485 ± 6% softirqs.CPU58.SCHED
10303 ± 13% -41.0% 6083 ± 15% softirqs.CPU59.RCU
29026 +32.1% 38345 ± 4% softirqs.CPU59.SCHED
10178 ± 14% -34.1% 6709 ± 13% softirqs.CPU6.RCU
29174 ± 10% +32.6% 38682 ± 3% softirqs.CPU6.SCHED
9753 ± 15% -37.9% 6057 ± 13% softirqs.CPU60.RCU
28862 ± 2% +32.8% 38320 ± 4% softirqs.CPU60.SCHED
9816 ± 13% -40.9% 5805 ± 14% softirqs.CPU61.RCU
28572 ± 2% +35.2% 38638 ± 3% softirqs.CPU61.SCHED
10000 ± 14% -41.2% 5876 ± 13% softirqs.CPU62.RCU
28515 ± 2% +34.6% 38377 ± 4% softirqs.CPU62.SCHED
9722 ± 16% -35.3% 6293 ± 18% softirqs.CPU63.RCU
28871 +32.9% 38371 ± 3% softirqs.CPU63.SCHED
9335 ± 17% -39.8% 5620 ± 13% softirqs.CPU64.RCU
9671 ± 16% -38.8% 5918 ± 11% softirqs.CPU65.RCU
9635 ± 16% -38.9% 5884 ± 10% softirqs.CPU66.RCU
9620 ± 17% -40.7% 5702 ± 11% softirqs.CPU67.RCU
23386 ± 35% +66.2% 38859 ± 3% softirqs.CPU67.SCHED
9840 ± 16% -42.5% 5661 ± 11% softirqs.CPU68.RCU
28877 ± 3% +31.9% 38083 ± 7% softirqs.CPU68.SCHED
9621 ± 16% -40.2% 5750 ± 10% softirqs.CPU69.RCU
28471 ± 4% +34.2% 38218 ± 4% softirqs.CPU69.SCHED
9500 ± 19% -37.7% 5920 ± 11% softirqs.CPU7.RCU
27971 ± 2% +27.6% 35679 ± 12% softirqs.CPU7.SCHED
9950 ± 19% -42.6% 5713 ± 11% softirqs.CPU70.RCU
28378 ± 4% +36.4% 38698 ± 4% softirqs.CPU70.SCHED
9769 ± 17% -41.5% 5712 ± 11% softirqs.CPU71.RCU
28400 ± 3% +35.0% 38334 ± 2% softirqs.CPU71.SCHED
10186 ± 23% -36.0% 6516 ± 11% softirqs.CPU72.RCU
29528 ± 5% +19.4% 35245 ± 3% softirqs.CPU72.SCHED
9117 ± 16% -37.7% 5682 ± 11% softirqs.CPU73.RCU
8864 ± 16% -37.6% 5527 ± 10% softirqs.CPU74.RCU
9191 ± 15% -40.4% 5474 ± 10% softirqs.CPU75.RCU
28423 ± 4% +28.2% 36424 ± 8% softirqs.CPU75.SCHED
9557 ± 20% -44.1% 5341 ± 10% softirqs.CPU76.RCU
28240 ± 3% +38.6% 39129 ± 3% softirqs.CPU76.SCHED
9108 ± 14% -40.6% 5408 ± 11% softirqs.CPU77.RCU
28923 ± 2% +35.5% 39203 ± 2% softirqs.CPU77.SCHED
9485 ± 16% -41.3% 5571 ± 13% softirqs.CPU78.RCU
28339 ± 2% +38.9% 39369 ± 3% softirqs.CPU78.SCHED
8889 ± 15% -39.2% 5401 ± 12% softirqs.CPU79.RCU
28802 ± 2% +35.8% 39116 ± 3% softirqs.CPU79.SCHED
28293 ± 2% +32.1% 37372 ± 7% softirqs.CPU8.SCHED
9386 ± 13% -40.2% 5612 ± 14% softirqs.CPU80.RCU
28492 ± 3% +36.6% 38911 ± 2% softirqs.CPU80.SCHED
9460 ± 14% -41.5% 5535 ± 13% softirqs.CPU81.RCU
28808 +33.5% 38470 ± 5% softirqs.CPU81.SCHED
9799 ± 14% -41.7% 5717 ± 15% softirqs.CPU82.RCU
28638 ± 3% +33.8% 38305 ± 3% softirqs.CPU82.SCHED
9574 ± 13% -40.1% 5731 ± 14% softirqs.CPU83.RCU
28754 +34.1% 38564 ± 3% softirqs.CPU83.SCHED
9488 ± 18% -41.6% 5541 ± 15% softirqs.CPU84.RCU
28843 +34.3% 38733 ± 4% softirqs.CPU84.SCHED
9072 ± 15% -39.8% 5465 ± 13% softirqs.CPU85.RCU
28412 ± 3% +37.4% 39033 ± 2% softirqs.CPU85.SCHED
9680 ± 17% -41.2% 5696 ± 15% softirqs.CPU86.RCU
28536 ± 2% +35.4% 38644 ± 2% softirqs.CPU86.SCHED
9251 ± 16% -37.9% 5749 ± 13% softirqs.CPU87.RCU
28704 +34.3% 38561 ± 3% softirqs.CPU87.SCHED
9761 ± 14% -40.2% 5842 ± 16% softirqs.CPU88.RCU
28799 ± 2% +36.4% 39277 ± 4% softirqs.CPU88.SCHED
9979 ± 14% -42.2% 5770 ± 14% softirqs.CPU89.RCU
28862 ± 3% +36.8% 39497 ± 3% softirqs.CPU89.SCHED
9789 ± 16% -40.8% 5798 ± 12% softirqs.CPU9.RCU
27851 ± 5% +39.6% 38881 ± 5% softirqs.CPU9.SCHED
8905 ± 14% -39.5% 5388 ± 11% softirqs.CPU90.RCU
28731 ± 2% +36.1% 39093 ± 4% softirqs.CPU90.SCHED
9570 ± 13% -38.1% 5924 ± 18% softirqs.CPU91.RCU
9375 ± 12% -41.8% 5455 ± 14% softirqs.CPU92.RCU
28905 +34.1% 38760 ± 5% softirqs.CPU92.SCHED
9198 ± 15% -35.9% 5893 ± 17% softirqs.CPU93.RCU
9560 ± 13% -37.5% 5976 ± 13% softirqs.CPU94.RCU
29218 ± 2% +28.9% 37660 ± 9% softirqs.CPU94.SCHED
9535 ± 13% -40.6% 5667 ± 14% softirqs.CPU95.RCU
9324 ± 15% -35.0% 6063 ± 21% softirqs.CPU96.RCU
25757 ± 5% +25.0% 32203 ± 9% softirqs.CPU96.SCHED
9674 ± 14% -40.8% 5729 ± 18% softirqs.CPU97.RCU
9371 ± 15% -37.7% 5840 ± 13% softirqs.CPU98.RCU
26068 ± 4% +38.4% 36065 ± 8% softirqs.CPU98.SCHED
9497 ± 15% -41.0% 5602 ± 13% softirqs.CPU99.RCU
26556 ± 6% +36.5% 36239 ± 3% softirqs.CPU99.SCHED
1862214 ± 15% -39.5% 1127214 ± 12% softirqs.RCU
5342380 +32.5% 7080895 ± 3% softirqs.SCHED
42480 ± 3% +28.2% 54463 softirqs.TIMER
vm-scalability.time.system_time
1800 +--------------------------------------------------------------------+
1700 |-+ O O O |
| O O O OO O O OO O OO O O O |
1600 |-OO O O O O O O OO O O O OO O |
1500 |-+ O O OO O O O O|
| O O O |
1400 |-+ |
1300 |-+ |
1200 |-+ |
| |
1100 |-+ |
1000 |-+ |
|.++. +.+ .++.++. .+ +. + .++.+ .+ |
900 |-+ + + ++ +.+ ++.+ + + + + |
800 +--------------------------------------------------------------------+
vm-scalability.time.percent_of_cpu_this_job_got
640 +---------------------------------------------------------------------+
| O |
620 |-+ O |
600 |-O O OO OO O O O O O OO O O OO O OO O O |
| O O O OO OO OO O OO O |
580 |-+ O O O O O O O |
| O |
560 |-+ |
| |
540 |-+ + .+. +. |
520 |.+ +.+ .++.+ + + + + ++.+ + |
| + : + + + +. :+ +.+ .+ |
500 |-+ +: + + + + |
| + |
480 +---------------------------------------------------------------------+
vm-scalability.time.elapsed_time
340 +---------------------------------------------------------------------+
| O O O |
320 |-+ O O O O OO OO OO OO O |
| OO O O O OO O O O O O O O O O |
300 |-+ O O O O O O O OO O O |
| O O |
280 |-+ |
| |
260 |-+ |
| |
240 |-+ |
| +.++. +. .+ .+ + +. |
220 |.+ ++.++.++.+ ++ + +.+ + +.+.+ ++.+ |
| + |
200 +---------------------------------------------------------------------+
vm-scalability.time.elapsed_time.max
340 +---------------------------------------------------------------------+
| O O O |
320 |-+ O O O O OO OO OO OO O |
| OO O O O OO O O O O O O O O O |
300 |-+ O O O O O O O OO O O |
| O O |
280 |-+ |
| |
260 |-+ |
| |
240 |-+ |
| +.++. +. .+ .+ + +. |
220 |.+ ++.++.++.+ ++ + +.+ + +.+.+ ++.+ |
| + |
200 +---------------------------------------------------------------------+
vm-scalability.throughput
3.4e+06 +-----------------------------------------------------------------+
| |
3.2e+06 |-+ + |
| .+ + :+ .+ |
3e+06 |+. + ++.++.+ .++. + .+ +: +++. + + |
2.8e+06 |-+++.+ + + + + + |
| |
2.6e+06 |-+ |
| |
2.4e+06 |-+ |
2.2e+06 |-+ |
| O O O O OO O O O O O O O|
2e+06 |O+O O OO O OOO O OO O O O O O OO OO |
| O O O OO O O O OO O |
1.8e+06 +-----------------------------------------------------------------+
vm-scalability.median
420000 +------------------------------------------------------------------+
| + |
400000 |-+ : |
380000 |-+ .+ +.+ : : .++.+ |
|+. .++ +.++.++ .++. +. : :: +.+++ |
360000 |-+++ + + + + |
340000 |-+ |
| |
320000 |-+ |
300000 |-+ |
| |
280000 |-+ O O O |
260000 |-+ O OO OO O O O O OO OO O O OO O|
|O O OO O O O O O O OO O O OOO O O |
240000 +------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
Thanks,
Oliver Sang
1 year, 3 months
[KVM] 39be2e2818: WARNING:at_mm/usercopy.c:#usercopy_warn
by kernel test robot
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 39be2e28180a2e87af5fbb8d83643812e1a3b371 ("KVM: selftests: Add selftest for KVM statistics data binary interface")
https://git.kernel.org/cgit/virt/kvm/kvm.git queue
in testcase: kernel-selftests
version: kernel-selftests-x86_64-7ae2585b-1_20210607
with following parameters:
group: kvm
ucode: 0xe2
test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel.
test-url: https://www.kernel.org/doc/Documentation/kselftest.txt
on test machine: 8 threads Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz with 32G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
[ 386.205633] WARNING: CPU: 5 PID: 6926 at mm/usercopy.c:75 usercopy_warn (kbuild/src/consumer/mm/usercopy.c:75 (discriminator 18))
[ 386.213477] Modules linked in: netconsole btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c intel_rapl_msr intel_rapl_common sd_mod t10_pi sg x86_pkg_temp_thermal i915 intel_powerclamp coretemp crct10dif_pclmul ipmi_devintf crc32_pclmul crc32c_intel ghash_clmulni_intel ipmi_msghandler sdhci_pci intel_wmi_thunderbolt cqhci rapl ahci sdhci vfio_mdev vfio_iommu_type1 libahci intel_cstate mei_me i2c_i801 mmc_core vfio libata mei intel_uncore joydev i2c_smbus mdev intel_pch_thermal ir_rc6_decoder intel_gtt wmi rc_rc6_mce nuvoton_cir rc_core video intel_pmc_core acpi_pad ip_tables
[ 386.267149] CPU: 3 PID: 6926 Comm: kvm_binary_stat Tainted: G I 5.13.0-rc1-00239-g39be2e28180a #1
[ 386.277815] Hardware name: /NUC6i7KYB, BIOS KYSKLi70.86A.0041.2016.0817.1130 08/17/2016
[ 386.286169] RIP: 0010:usercopy_warn (kbuild/src/consumer/mm/usercopy.c:75 (discriminator 18))
[ 386.290702] Code: 80 82 41 51 49 89 c0 49 c7 c2 62 a4 7e 82 49 89 f1 48 89 f9 4c 0f 45 d2 48 c7 c7 30 33 80 82 4c 89 da 4c 89 d6 e8 26 30 a9 00 <0f> 0b 48 83 c4 18 c3 48 c7 c6 6b 83 7e 82 49 89 f1 48 89 f0 eb 96
All code
========
0: 80 82 41 51 49 89 c0 addb $0xc0,-0x76b6aebf(%rdx)
7: 49 c7 c2 62 a4 7e 82 mov $0xffffffff827ea462,%r10
e: 49 89 f1 mov %rsi,%r9
11: 48 89 f9 mov %rdi,%rcx
14: 4c 0f 45 d2 cmovne %rdx,%r10
18: 48 c7 c7 30 33 80 82 mov $0xffffffff82803330,%rdi
1f: 4c 89 da mov %r11,%rdx
22: 4c 89 d6 mov %r10,%rsi
25: e8 26 30 a9 00 callq 0xa93050
2a:* 0f 0b ud2 <-- trapping instruction
2c: 48 83 c4 18 add $0x18,%rsp
30: c3 retq
31: 48 c7 c6 6b 83 7e 82 mov $0xffffffff827e836b,%rsi
38: 49 89 f1 mov %rsi,%r9
3b: 48 89 f0 mov %rsi,%rax
3e: eb 96 jmp 0xffffffffffffffd6
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 83 c4 18 add $0x18,%rsp
6: c3 retq
7: 48 c7 c6 6b 83 7e 82 mov $0xffffffff827e836b,%rsi
e: 49 89 f1 mov %rsi,%r9
11: 48 89 f0 mov %rsi,%rax
14: eb 96 jmp 0xffffffffffffffac
[ 386.310020] RSP: 0018:ffffc90009eefe00 EFLAGS: 00010286
[ 386.315422] RAX: 0000000000000000 RBX: 00000000000000e8 RCX: 0000000000000000
[ 386.322820] RDX: 0000000000000001 RSI: ffffffff8123284f RDI: ffffffff8123284f
[ 386.330219] RBP: ffff8888b3318200 R08: 0000000000000001 R09: 0000000000000001
[ 386.337607] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8888b3318118
[ 386.345010] R13: 0000000000000001 R14: 00000000013e4ce0 R15: ffffffff83040260
[ 386.352418] FS: 00007fe185eb6740(0000) GS:ffff8888becc0000(0000) knlGS:0000000000000000
[ 386.360838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 386.366775] CR2: 00007fb475292000 CR3: 00000008b49d6003 CR4: 00000000003726e0
[ 386.374147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 386.381532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 386.388909] Call Trace:
[ 386.391469] __check_object_size (kbuild/src/consumer/mm/usercopy.c:240 kbuild/src/consumer/mm/usercopy.c:286 kbuild/src/consumer/mm/usercopy.c:256)
[ 386.395869] kvm_stats_read (kbuild/src/consumer/include/linux/uaccess.h:200 kbuild/src/consumer/arch/x86/kvm/../../../virt/kvm/kvm_main.c:3546)
[ 386.399875] kvm_vcpu_stats_read (kbuild/src/consumer/arch/x86/kvm/../../../virt/kvm/kvm_main.c:3564)
[ 386.404141] vfs_read (kbuild/src/consumer/fs/read_write.c:494)
[ 386.407514] ksys_pread64 (kbuild/src/consumer/include/linux/file.h:45 kbuild/src/consumer/fs/read_write.c:687)
[ 386.411163] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:47)
[ 386.414888] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:112)
[ 386.420138] RIP: 0033:0x7fe18608bdf4
[ 386.423892] Code: d8 64 89 02 b8 ff ff ff ff eb c5 66 2e 0f 1f 84 00 00 00 00 00 90 8b 05 5a e6 00 00 49 89 ca 85 c0 75 13 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5c c3 0f 1f 00 41 55 49 89 cd 41 54 49 89 d4
All code
========
0: d8 64 89 02 fsubs 0x2(%rcx,%rcx,4)
4: b8 ff ff ff ff mov $0xffffffff,%eax
9: eb c5 jmp 0xffffffffffffffd0
b: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
12: 00 00 00
15: 90 nop
16: 8b 05 5a e6 00 00 mov 0xe65a(%rip),%eax # 0xe676
1c: 49 89 ca mov %rcx,%r10
1f: 85 c0 test %eax,%eax
21: 75 13 jne 0x36
23: b8 11 00 00 00 mov $0x11,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 5c ja 0x8e
32: c3 retq
33: 0f 1f 00 nopl (%rax)
36: 41 55 push %r13
38: 49 89 cd mov %rcx,%r13
3b: 41 54 push %r12
3d: 49 89 d4 mov %rdx,%r12
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 5c ja 0x64
8: c3 retq
9: 0f 1f 00 nopl (%rax)
c: 41 55 push %r13
e: 49 89 cd mov %rcx,%r13
11: 41 54 push %r12
13: 49 89 d4 mov %rdx,%r12
[ 386.443313] RSP: 002b:00007fff2f95ffe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000011
[ 386.451151] RAX: ffffffffffffffda RBX: 00000000013e4530 RCX: 00007fe18608bdf4
[ 386.458492] RDX: 00000000000000e8 RSI: 00000000013e4ce0 RDI: 000000000000001d
[ 386.465889] RBP: 00000000000000e8 R08: 000000000040e420 R09: 00000000013e4ca0
[ 386.473294] R10: 0000000000000790 R11: 0000000000000246 R12: 0000000000000040
[ 386.480682] R13: 00000000000000e8 R14: 000000000000001d R15: 00000000013e4ce0
[ 386.488056] irq event stamp: 17487
[ 386.491620] hardirqs last enabled at (17505): _raw_spin_unlock_irq (kbuild/src/consumer/arch/x86/include/asm/irqflags.h:45 kbuild/src/consumer/arch/x86/include/asm/irqflags.h:80 kbuild/src/consumer/include/linux/spinlock_api_smp.h:168 kbuild/src/consumer/kernel/locking/spinlock.c:199)
[ 386.500893] hardirqs last disabled at (17528): __schedule (kbuild/src/consumer/kernel/sched/core.c:5043 (discriminator 1))
[ 386.509429] softirqs last enabled at (17542): __do_softirq (kbuild/src/consumer/arch/x86/include/asm/preempt.h:27 kbuild/src/consumer/kernel/softirq.c:403 kbuild/src/consumer/kernel/softirq.c:588)
[ 386.518160] softirqs last disabled at (17555): irq_exit_rcu (kbuild/src/consumer/kernel/softirq.c:433 kbuild/src/consumer/kernel/softirq.c:637 kbuild/src/consumer/kernel/softirq.c:649)
[ 386.526912] ---[ end trace b50de138b37b26c4 ]---
[ 386.574882] ok 42 selftests: kvm: kvm_binary_stats_test
[ 386.574891]
[ 386.678131] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil... -O /dev/null
[ 386.678141]
[ 388.157966] kill 1098 vmstat --timestamp -n 10
[ 388.157972]
[ 388.170558] kill 1096 dmesg --follow --decode
[ 388.170563]
[ 388.192089] wait for background processes: 1102 1106 meminfo oom-killer
[ 388.192096]
[ 396.520977] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil... -O /dev/null
[ 396.520991]
[ 396.585567] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil... -O /dev/null
[ 396.585576]
[ 396.638671] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-post-run?job_file=/lkp/job... -O /dev/null
[ 396.638679]
[ 396.677867] getting new job...
[ 396.677873]
[ 396.699693] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/gpxelinux.cgi?hostname=lkp-skl... -O /tmp/next-job-lkp
[ 396.699702]
[ 398.346941] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp//lkp/jobs/scheduled/lkp-skl-nuc2/kerne... -O /tmp/next-job.cgz
[ 398.346950]
[ 398.355973] 24 blocks
[ 398.380914]
[ 398.405028] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://internal-lkp-server:80/~lkp/cgi-bin/lkp-wtmp?tbox_name=lkp-skl-nuc... -O /dev/null
[ 398.405042]
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
Thanks,
Oliver Sang
1 year, 3 months
[x86/fpu] 190757bfda: invalid_opcode:#[##]
by kernel test robot
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 190757bfda82678e88ce9598fb75a6a6a85615f1 ("x86/fpu: Add PKRU storage outside of task XSAVE buffer")
https://git.kernel.org/cgit/linux/kernel/git/tglx/devel.git x86/fpu
in testcase: rcuscale
version:
with following parameters:
runtime: 300s
scale_type: srcud
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+---------------------------------------------+------------+------------+
| | 65c5ba4c1b | 190757bfda |
+---------------------------------------------+------------+------------+
| invalid_opcode:#[##] | 0 | 6 |
| RIP:copy_thread | 0 | 6 |
+---------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
[ 5.840871] vga=normal
[ 5.863284] systemd[1]: RTC configured in localtime, applying delta of 0 minutes to system time.
Welcome to Debian GNU/Linux 10 (buster)!
[ 5.889702] invalid opcode: 0000 [#1] SMP PTI
[ 5.890505] CPU: 1 PID: 1 Comm: systemd Not tainted 5.13.0-rc5-00051-g190757bfda82 #1
[ 5.891832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[ 5.893260] RIP: 0010:copy_thread (kbuild/src/consumer/arch/x86/include/asm/special_insns.h:83 kbuild/src/consumer/arch/x86/kernel/process.c:170)
[ 5.894019] Code: 48 89 85 b0 23 00 00 66 8c c0 66 89 85 a0 23 00 00 66 8c d8 8b 4d 2c 66 89 85 a2 23 00 00 81 e1 00 00 20 00 0f 85 c4 00 00 00 <0f> 01 ee 89 85 14 24 00 00 b9 15 00 00 00 48 89 df 48 c7 43 e8 00
All code
========
0: 48 89 85 b0 23 00 00 mov %rax,0x23b0(%rbp)
7: 66 8c c0 mov %es,%ax
a: 66 89 85 a0 23 00 00 mov %ax,0x23a0(%rbp)
11: 66 8c d8 mov %ds,%ax
14: 8b 4d 2c mov 0x2c(%rbp),%ecx
17: 66 89 85 a2 23 00 00 mov %ax,0x23a2(%rbp)
1e: 81 e1 00 00 20 00 and $0x200000,%ecx
24: 0f 85 c4 00 00 00 jne 0xee
2a:* 0f 01 ee rdpkru <-- trapping instruction
2d: 89 85 14 24 00 00 mov %eax,0x2414(%rbp)
33: b9 15 00 00 00 mov $0x15,%ecx
38: 48 89 df mov %rbx,%rdi
3b: 48 rex.W
3c: c7 .byte 0xc7
3d: 43 rex.XB
3e: e8 .byte 0xe8
...
Code starting with the faulting instruction
===========================================
0: 0f 01 ee rdpkru
3: 89 85 14 24 00 00 mov %eax,0x2414(%rbp)
9: b9 15 00 00 00 mov $0x15,%ecx
e: 48 89 df mov %rbx,%rdi
11: 48 rex.W
12: c7 .byte 0xc7
13: 43 rex.XB
14: e8 .byte 0xe8
...
[ 5.896948] RSP: 0018:ffffacebc0013d88 EFLAGS: 00010246
[ 5.897794] RAX: 0000000000000000 RBX: ffffacebc00c7f58 RCX: 0000000000000000
[ 5.898893] RDX: 0000000000000000 RSI: ffff89a98ca9cd00 RDI: 0000000001200000
[ 5.899994] RBP: ffff89acafcca680 R08: 0000000000000000 R09: 0000000000000000
[ 5.901101] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000001200000
[ 5.902208] R13: 0000000000000000 R14: ffffacebc00c4000 R15: 0000000000000000
[ 5.903298] FS: 00007fb1452af940(0000) GS:ffff89acafd00000(0000) knlGS:0000000000000000
[ 5.904594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5.905510] CR2: 0000557cb50bdbf0 CR3: 00000002cfb28000 CR4: 00000000000006e0
[ 5.906607] Call Trace:
[ 5.907083] copy_process (kbuild/src/consumer/kernel/fork.c:2125)
[ 5.907731] ? shmem_init_fs_context (kbuild/src/consumer/mm/shmem.c:3179)
[ 5.908489] kernel_clone (kbuild/src/consumer/kernel/fork.c:2503)
[ 5.909117] __do_sys_clone (kbuild/src/consumer/kernel/fork.c:2621)
[ 5.909750] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:47)
[ 5.910375] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:112)
[ 5.911205] RIP: 0033:0x7fb14643c7be
[ 5.911828] Code: db 0f 85 25 01 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 b6 00 00 00 41 89 c4 85 c0 0f 85 c3 00 00
All code
========
0: db 0f fisttpl (%rdi)
2: 85 25 01 00 00 64 test %esp,0x64000001(%rip) # 0x64000009
8: 4c 8b 0c 25 10 00 00 mov 0x10,%r9
f: 00
10: 45 31 c0 xor %r8d,%r8d
13: 4d 8d 91 d0 02 00 00 lea 0x2d0(%r9),%r10
1a: 31 d2 xor %edx,%edx
1c: 31 f6 xor %esi,%esi
1e: bf 11 00 20 01 mov $0x1200011,%edi
23: b8 38 00 00 00 mov $0x38,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 0f 87 b6 00 00 00 ja 0xec
36: 41 89 c4 mov %eax,%r12d
39: 85 c0 test %eax,%eax
3b: 0f .byte 0xf
3c: 85 c3 test %eax,%ebx
...
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 0f 87 b6 00 00 00 ja 0xc2
c: 41 89 c4 mov %eax,%r12d
f: 85 c0 test %eax,%eax
11: 0f .byte 0xf
12: 85 c3 test %eax,%ebx
...
[ 5.914611] RSP: 002b:00007ffff8b392d0 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
[ 5.915833] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb14643c7be
[ 5.916940] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[ 5.918035] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fb1452af940
[ 5.919127] R10: 00007fb1452afc10 R11: 0000000000000246 R12: 0000000000000020
[ 5.920215] R13: 00007fb1462a53f8 R14: 0000000000000000 R15: 0000000000000001
[ 5.921320] Modules linked in: ip_tables
[ 5.922005] ---[ end trace d7e147dd4242acd7 ]---
[ 5.922781] RIP: 0010:copy_thread (kbuild/src/consumer/arch/x86/include/asm/special_insns.h:83 kbuild/src/consumer/arch/x86/kernel/process.c:170)
[ 5.923509] Code: 48 89 85 b0 23 00 00 66 8c c0 66 89 85 a0 23 00 00 66 8c d8 8b 4d 2c 66 89 85 a2 23 00 00 81 e1 00 00 20 00 0f 85 c4 00 00 00 <0f> 01 ee 89 85 14 24 00 00 b9 15 00 00 00 48 89 df 48 c7 43 e8 00
All code
========
0: 48 89 85 b0 23 00 00 mov %rax,0x23b0(%rbp)
7: 66 8c c0 mov %es,%ax
a: 66 89 85 a0 23 00 00 mov %ax,0x23a0(%rbp)
11: 66 8c d8 mov %ds,%ax
14: 8b 4d 2c mov 0x2c(%rbp),%ecx
17: 66 89 85 a2 23 00 00 mov %ax,0x23a2(%rbp)
1e: 81 e1 00 00 20 00 and $0x200000,%ecx
24: 0f 85 c4 00 00 00 jne 0xee
2a:* 0f 01 ee rdpkru <-- trapping instruction
2d: 89 85 14 24 00 00 mov %eax,0x2414(%rbp)
33: b9 15 00 00 00 mov $0x15,%ecx
38: 48 89 df mov %rbx,%rdi
3b: 48 rex.W
3c: c7 .byte 0xc7
3d: 43 rex.XB
3e: e8 .byte 0xe8
...
Code starting with the faulting instruction
===========================================
0: 0f 01 ee rdpkru
3: 89 85 14 24 00 00 mov %eax,0x2414(%rbp)
9: b9 15 00 00 00 mov $0x15,%ecx
e: 48 89 df mov %rbx,%rdi
11: 48 rex.W
12: c7 .byte 0xc7
13: 43 rex.XB
14: e8 .byte 0xe8
...
[ 5.926303] RSP: 0018:ffffacebc0013d88 EFLAGS: 00010246
[ 5.927163] RAX: 0000000000000000 RBX: ffffacebc00c7f58 RCX: 0000000000000000
[ 5.928277] RDX: 0000000000000000 RSI: ffff89a98ca9cd00 RDI: 0000000001200000
[ 5.929405] RBP: ffff89acafcca680 R08: 0000000000000000 R09: 0000000000000000
[ 5.930517] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000001200000
[ 5.931627] R13: 0000000000000000 R14: ffffacebc00c4000 R15: 0000000000000000
[ 5.932751] FS: 00007fb1452af940(0000) GS:ffff89acafd00000(0000) knlGS:0000000000000000
[ 5.934045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5.934973] CR2: 0000557cb50bdbf0 CR3: 00000002cfb28000 CR4: 00000000000006e0
[ 5.936084] Kernel panic - not syncing: Fatal exception
[ 5.937085] Kernel Offset: 0xde00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Kboot worker: lkp-worker44
Elapsed time: 60
To reproduce:
# build kernel
cd linux
cp config-5.13.0-rc5-00051-g190757bfda82 .config
make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
Thanks,
Oliver Sang
1 year, 3 months
[PATCH] mm: relocate 'write_protect_seq' in struct mm_struct
by Feng Tang
0day robot reported a 9.2% regression for will-it-scale mmap1 test
case[1], caused by commit 57efa1fe5957 ("mm/gup: prevent gup_fast
from racing with COW during fork").
Further debug shows the regression is due to that commit changes
the offset of hot fields 'mmap_lock' inside structure 'mm_struct',
thus some cache alignment changes.
From the perf data, the contention for 'mmap_lock' is very severe
and takes around 95% cpu cycles, and it is a rw_semaphore
struct rw_semaphore {
atomic_long_t count; /* 8 bytes */
atomic_long_t owner; /* 8 bytes */
struct optimistic_spin_queue osq; /* spinner MCS lock */
...
Before commit 57efa1fe5957 adds the 'write_protect_seq', it
happens to have a very optimal cache alignment layout, as
Linus explained:
"and before the addition of the 'write_protect_seq' field, the
mmap_sem was at offset 120 in 'struct mm_struct'.
Which meant that count and owner were in two different cachelines,
and then when you have contention and spend time in
rwsem_down_write_slowpath(), this is probably *exactly* the kind
of layout you want.
Because first the rwsem_write_trylock() will do a cmpxchg on the
first cacheline (for the optimistic fast-path), and then in the
case of contention, rwsem_down_write_slowpath() will just access
the second cacheline.
Which is probably just optimal for a load that spends a lot of
time contended - new waiters touch that first cacheline, and then
they queue themselves up on the second cacheline."
After the commit, the rw_semaphore is at offset 128, which means
the 'count' and 'owner' fields are now in the same cacheline,
and causes more cache bouncing.
Currently there are 3 "#ifdef CONFIG_XXX" before 'mmap_lock' which
will affect its offset:
CONFIG_MMU
CONFIG_MEMBARRIER
CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES
The layout above is on 64 bits system with 0day's default kernel
config (similar to RHEL-8.3's config), in which all these 3 options
are 'y'. And the layout can vary with different kernel configs.
Relayouting a structure is usually a double-edged sword, as sometimes
it can helps one case, but hurt other cases. For this case, one
solution is, as the newly added 'write_protect_seq' is a 4 bytes long
seqcount_t (when CONFIG_DEBUG_LOCK_ALLOC=n), placing it into an
existing 4 bytes hole in 'mm_struct' will not change other fields'
alignment, while restoring the regression.
[1]. https://lore.kernel.org/lkml/20210525031636.GB7744@xsang-OptiPlex-9020/
Reported-by: kernel test robot <oliver.sang(a)intel.com>
Signed-off-by: Feng Tang <feng.tang(a)intel.com>
Reviewed-by: John Hubbard <jhubbard(a)nvidia.com>
Cc: Jason Gunthorpe <jgg(a)nvidia.com>
Cc: Linus Torvalds <torvalds(a)linux-foundation.org>
Cc: Peter Xu <peterx(a)redhat.com>
---
include/linux/mm_types.h | 27 ++++++++++++++++++++-------
1 file changed, 20 insertions(+), 7 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 5aacc1c..cba6022 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -445,13 +445,6 @@ struct mm_struct {
*/
atomic_t has_pinned;
- /**
- * @write_protect_seq: Locked when any thread is write
- * protecting pages mapped by this mm to enforce a later COW,
- * for instance during page table copying for fork().
- */
- seqcount_t write_protect_seq;
-
#ifdef CONFIG_MMU
atomic_long_t pgtables_bytes; /* PTE page table pages */
#endif
@@ -460,6 +453,18 @@ struct mm_struct {
spinlock_t page_table_lock; /* Protects page tables and some
* counters
*/
+ /*
+ * With some kernel config, the current mmap_lock's offset
+ * inside 'mm_struct' is at 0x120, which is very optimal, as
+ * its two hot fields 'count' and 'owner' sit in 2 different
+ * cachelines, and when mmap_lock is highly contended, both
+ * of the 2 fields will be accessed frequently, current layout
+ * will help to reduce cache bouncing.
+ *
+ * So please be careful with adding new fields before
+ * mmap_lock, which can easily push the 2 fields into one
+ * cacheline.
+ */
struct rw_semaphore mmap_lock;
struct list_head mmlist; /* List of maybe swapped mm's. These
@@ -480,7 +485,15 @@ struct mm_struct {
unsigned long stack_vm; /* VM_STACK */
unsigned long def_flags;
+ /**
+ * @write_protect_seq: Locked when any thread is write
+ * protecting pages mapped by this mm to enforce a later COW,
+ * for instance during page table copying for fork().
+ */
+ seqcount_t write_protect_seq;
+
spinlock_t arg_lock; /* protect the below fields */
+
unsigned long start_code, end_code, start_data, end_data;
unsigned long start_brk, brk, start_stack;
unsigned long arg_start, arg_end, env_start, env_end;
--
2.7.4
1 year, 3 months
[platform/x86] 51954bff27: BUG:KASAN:null-ptr-deref_in_lwpmudrv_Get_Symbol_Lookup_Func_Addr
by kernel test robot
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 51954bff279dc62a97c2ce1802c44455d306d971 ("platform/x86: bug fix and re-enable sep driver for Linux 5.12 pull")
https://github.com/mchinth/linux sep_socwatch_linux_5_12
in testcase: trinity
version: trinity-x86_64-03f10b67-1_20210603
with following parameters:
runtime: 300s
test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
[ 732.148425] BUG: KASAN: null-ptr-deref in lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196)
[ 732.148425] Read of size 8 at addr 0000000000000026 by task SEPDRV_ABNORMAL/402
[ 732.148425]
[ 732.148425] CPU: 0 PID: 402 Comm: SEPDRV_ABNORMAL Not tainted 5.12.0-rc4-00053-g51954bff279d #1
[ 732.211784] sep5_0: [load] [lwpmu_Load@7877]: Graphics support is enabled.
[ 732.148425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[ 732.148425] Call Trace:
[ 732.148425] dump_stack (kbuild/src/consumer/lib/dump_stack.c:122)
[ 732.148425] ? lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196)
[ 732.148425] kasan_report.cold (kbuild/src/consumer/mm/kasan/report.c:403 kbuild/src/consumer/mm/kasan/report.c:416)
[ 732.258157] sep5_0: [load] [lwpmu_Load@7880]: NMI will be used for handling PMU interrupts.
[ 732.148425] ? kcov_common_handle (kbuild/src/consumer/kernel/kcov.c:1028)
[ 732.148425] ? lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196)
[ 732.148425] lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196)
[ 732.148425] ? lwpmudrv_Samp_Read_PCI_Config (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7180)
[ 732.148425] ? rcu_read_lock_any_held (kbuild/src/consumer/kernel/rcu/update.c:346)
[ 732.308017] PAX: PMU arbitration service v1.0.2 has been started.
[ 732.148425] ? rcu_read_lock_held (kbuild/src/consumer/kernel/rcu/update.c:341)
[ 732.148425] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1))
[ 732.148425] ? validate_chain (kbuild/src/consumer/kernel/locking/lockdep.c:3681)
[ 732.369358] cros_ec_lpcs: unsupported system.
[ 732.351562] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1))
[ 732.351562] ? check_prev_add (kbuild/src/consumer/kernel/locking/lockdep.c:3630)
[ 732.429155] random: get_random_u64 called from kmem_cache_open+0x2a/0x460 with crng_init=0
[ 732.429258] random: get_random_u64 called from cache_random_seq_create+0x80/0x180 with crng_init=0
[ 732.442196] surface_gpe: no compatible Microsoft Surface device found, exiting
[ 732.351562] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292)
[ 732.473751] hv_utils: Registering HyperV Utility Driver
[ 732.351562] ? mark_lock (kbuild/src/consumer/kernel/locking/lockdep.c:4456)
[ 732.351562] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1))
[ 732.351562] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292)
[ 732.515258] hv_vmbus: registering driver hv_utils
[ 732.351562] ? sched_clock_cpu (kbuild/src/consumer/kernel/sched/clock.c:382)
[ 732.351562] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246)
[ 732.564130] hv_vmbus: registering driver hv_balloon
[ 732.351562] lwpmudrv_Abnormal_Handler (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7255)
[ 732.351562] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246)
[ 732.351562] ? lockdep_hardirqs_on (kbuild/src/consumer/kernel/locking/lockdep.c:4215 (discriminator 1))
[ 732.351562] ? _raw_spin_unlock_irqrestore (kbuild/src/consumer/include/linux/spinlock_api_smp.h:160 kbuild/src/consumer/kernel/locking/spinlock.c:191)
[ 732.622249] Driver 'dfl-emif' was unable to register with bus_type 'dfl' because the bus was not initialized.
[ 732.351562] ? trace_hardirqs_on (kbuild/src/consumer/kernel/trace/trace_preemptirq.c:50 (discriminator 22))
[ 732.351562] ? write_comp_data (kbuild/src/consumer/kernel/kcov.c:218)
[ 732.351562] ? __sanitizer_cov_trace_pc (kbuild/src/consumer/kernel/kcov.c:197)
[ 732.351562] ? __kthread_parkme (kbuild/src/consumer/kernel/kthread.c:235 (discriminator 4))
[ 732.885996] Board is not the VME system controller
[ 732.880567] kthread (kbuild/src/consumer/kernel/kthread.c:292)
[ 732.880567] ? kthread_insert_work_sanity_check (kbuild/src/consumer/kernel/kthread.c:245)
[ 732.914487] VME geographical address is set to 0
[ 732.914474] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:300)
[ 732.914474] ==================================================================
[ 732.960810] CR/CSR Offset: 0
[ 732.914474] Disabling lock debugging due to kernel taint
[ 732.984459] BUG: kernel NULL pointer dereference, address: 0000000000000026
[ 733.008704] AMD(R) PCI-E Non-Transparent Bridge Driver 1.0
[ 732.988421] #PF: supervisor read access in kernel mode
[ 732.988421] #PF: error_code(0x0000) - not-present page
[ 732.988421] PGD 0 P4D 0
[ 732.988421] Oops: 0000 [#1] SMP KASAN PTI
[ 733.051131] IDT PCI-E Non-Transparent Bridge Driver 2.0
[ 732.988421] CPU: 0 PID: 402 Comm: SEPDRV_ABNORMAL Tainted: G B 5.12.0-rc4-00053-g51954bff279d #1
[ 732.988421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[ 732.988421] RIP: 0010:lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196)
[ 732.988421] Code: b8 86 22 f8 48 85 c0 0f 84 4a 02 00 00 49 89 c4 4d 8d 74 24 28 4d 8d ac 24 30 01 00 00 e8 fa a5 e2 f7 4c 89 f7 e8 72 5b 19 f8 <49> 8b 5c 24 28 48 8d 7b 10 e8 64 5b 19 f8 48 8b 43 10 4c 89 e7 4c
All code
========
0: b8 86 22 f8 48 mov $0x48f82286,%eax
5: 85 c0 test %eax,%eax
7: 0f 84 4a 02 00 00 je 0x257
d: 49 89 c4 mov %rax,%r12
10: 4d 8d 74 24 28 lea 0x28(%r12),%r14
15: 4d 8d ac 24 30 01 00 lea 0x130(%r12),%r13
1c: 00
1d: e8 fa a5 e2 f7 callq 0xfffffffff7e2a61c
22: 4c 89 f7 mov %r14,%rdi
25: e8 72 5b 19 f8 callq 0xfffffffff8195b9c
2a:* 49 8b 5c 24 28 mov 0x28(%r12),%rbx <-- trapping instruction
2f: 48 8d 7b 10 lea 0x10(%rbx),%rdi
33: e8 64 5b 19 f8 callq 0xfffffffff8195b9c
38: 48 8b 43 10 mov 0x10(%rbx),%rax
3c: 4c 89 e7 mov %r12,%rdi
3f: 4c rex.WR
Code starting with the faulting instruction
===========================================
0: 49 8b 5c 24 28 mov 0x28(%r12),%rbx
5: 48 8d 7b 10 lea 0x10(%rbx),%rdi
9: e8 64 5b 19 f8 callq 0xfffffffff8195b72
e: 48 8b 43 10 mov 0x10(%rbx),%rax
12: 4c 89 e7 mov %r12,%rdi
15: 4c rex.WR
[ 733.094487] Intel(R) PCI-E Non-Transparent Bridge Driver 2.0
[ 732.988421] RSP: 0000:ffffc900081efb48 EFLAGS: 00010286
[ 732.988421] RAX: 0000000000000001 RBX: ffff8881606e9000 RCX: ffffffffa22e19af
[ 732.988421] RDX: dffffc0000000000 RSI: ffff8881606e9000 RDI: ffffffffa7dce820
[ 732.988421] RBP: ffff88816070b100 R08: 0000000000000000 R09: 0000000000000000
[ 733.140005] Software Queue-Pair Transport over NTB, version 4
[ 732.988421] R10: ffffffffab65dfc3 R11: fffffbfff56cbbf8 R12: fffffffffffffffe
[ 732.988421] R13: 000000000000012e R14: 0000000000000026 R15: 1ffff9200103df6a
[ 732.988421] FS: 0000000000000000(0000) GS:ffff88839cc00000(0000) knlGS:0000000000000000
[ 732.988421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 732.988421] CR2: 0000000000000026 CR3: 000000005e8ba000 CR4: 00000000000406f0
[ 732.988421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 733.832632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 733.848575] Call Trace:
[ 733.868480] ? lwpmudrv_Samp_Read_PCI_Config (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7180)
[ 733.888615] ? rcu_read_lock_any_held (kbuild/src/consumer/kernel/rcu/update.c:346)
[ 733.904632] ? rcu_read_lock_held (kbuild/src/consumer/kernel/rcu/update.c:341)
[ 733.928610] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1))
[ 733.952605] ? validate_chain (kbuild/src/consumer/kernel/locking/lockdep.c:3681)
[ 733.968568] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1))
[ 733.992582] ? check_prev_add (kbuild/src/consumer/kernel/locking/lockdep.c:3630)
[ 734.016582] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292)
[ 734.036474] ? mark_lock (kbuild/src/consumer/kernel/locking/lockdep.c:4456)
[ 734.056609] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/include/asm/pvclock.h:35 (discriminator 1) kbuild/src/consumer/arch/x86/kernel/pvclock.c:78 (discriminator 1))
[ 734.072599] ? sched_clock_local (kbuild/src/consumer/kernel/sched/clock.c:292)
[ 734.097522] ? sched_clock_cpu (kbuild/src/consumer/kernel/sched/clock.c:382)
[ 734.120542] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246)
[ 734.136530] lwpmudrv_Abnormal_Handler (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7255)
[ 734.160556] ? lwpmudrv_Finish_Stop (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7246)
[ 734.180494] ? lockdep_hardirqs_on (kbuild/src/consumer/kernel/locking/lockdep.c:4215 (discriminator 1))
[ 734.200593] ? _raw_spin_unlock_irqrestore (kbuild/src/consumer/include/linux/spinlock_api_smp.h:160 kbuild/src/consumer/kernel/locking/spinlock.c:191)
[ 734.220551] ? trace_hardirqs_on (kbuild/src/consumer/kernel/trace/trace_preemptirq.c:50 (discriminator 22))
[ 734.240624] ? write_comp_data (kbuild/src/consumer/kernel/kcov.c:218)
[ 734.256574] ? __sanitizer_cov_trace_pc (kbuild/src/consumer/kernel/kcov.c:197)
[ 734.276481] ? __kthread_parkme (kbuild/src/consumer/kernel/kthread.c:235 (discriminator 4))
[ 734.296593] kthread (kbuild/src/consumer/kernel/kthread.c:292)
[ 734.313615] ? kthread_insert_work_sanity_check (kbuild/src/consumer/kernel/kthread.c:245)
[ 734.336565] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:300)
[ 734.352548] Modules linked in:
[ 734.372478] CR2: 0000000000000026
[ 734.384585] _warn_unseeded_randomness: 2 callbacks suppressed
[ 734.384585] random: get_random_bytes called from print_oops_end_marker+0x3b/0x60 with crng_init=0
[ 734.384585] ---[ end trace 29029e514c2dd2ac ]---
[ 734.444511] RIP: 0010:lwpmudrv_Get_Symbol_Lookup_Func_Addr (kbuild/src/consumer/drivers/platform/x86/sepdk/sep/lwpmudrv.c:7196)
[ 734.456601] Code: b8 86 22 f8 48 85 c0 0f 84 4a 02 00 00 49 89 c4 4d 8d 74 24 28 4d 8d ac 24 30 01 00 00 e8 fa a5 e2 f7 4c 89 f7 e8 72 5b 19 f8 <49> 8b 5c 24 28 48 8d 7b 10 e8 64 5b 19 f8 48 8b 43 10 4c 89 e7 4c
All code
========
0: b8 86 22 f8 48 mov $0x48f82286,%eax
5: 85 c0 test %eax,%eax
7: 0f 84 4a 02 00 00 je 0x257
d: 49 89 c4 mov %rax,%r12
10: 4d 8d 74 24 28 lea 0x28(%r12),%r14
15: 4d 8d ac 24 30 01 00 lea 0x130(%r12),%r13
1c: 00
1d: e8 fa a5 e2 f7 callq 0xfffffffff7e2a61c
22: 4c 89 f7 mov %r14,%rdi
25: e8 72 5b 19 f8 callq 0xfffffffff8195b9c
2a:* 49 8b 5c 24 28 mov 0x28(%r12),%rbx <-- trapping instruction
2f: 48 8d 7b 10 lea 0x10(%rbx),%rdi
33: e8 64 5b 19 f8 callq 0xfffffffff8195b9c
38: 48 8b 43 10 mov 0x10(%rbx),%rax
3c: 4c 89 e7 mov %r12,%rdi
3f: 4c rex.WR
Code starting with the faulting instruction
===========================================
0: 49 8b 5c 24 28 mov 0x28(%r12),%rbx
5: 48 8d 7b 10 lea 0x10(%rbx),%rdi
9: e8 64 5b 19 f8 callq 0xfffffffff8195b72
e: 48 8b 43 10 mov 0x10(%rbx),%rax
12: 4c 89 e7 mov %r12,%rdi
15: 4c rex.WR
To reproduce:
# build kernel
cd linux
cp config-5.12.0-rc4-00053-g51954bff279d .config
make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
Thanks,
Oliver Sang
1 year, 3 months
[block/mq] 2aa18c32fd: WARNING:at_block/mq-deadline-main.c:#dd_insert_requests
by kernel test robot
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 2aa18c32fd0905de7894b1e3a32f84edf77bc840 ("block/mq-deadline: Add cgroup support")
https://github.com/bvanassche/linux deadline-scheduler
in testcase: mdadm-selftests
version: mdadm-selftests-x86_64-5f41845-1_20210508
with following parameters:
disk: 1HDD
test_prefix: 15
ucode: 0x21
on test machine: 4 threads 1 sockets Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
kern :warn : [ 100.174333] WARNING: CPU: 0 PID: 1055 at block/mq-deadline-main.c:714 dd_insert_requests (kbuild/src/consumer/block/mq-deadline-main.c:714 kbuild/src/consumer/block/mq-deadline-main.c:761)
kern :warn : [ 100.184543] Modules linked in: multipath loop raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid10 raid1 raid0 netconsole btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c ipmi_devintf ipmi_msghandler sd_mod t10_pi sg intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp i915 kvm_intel kvm intel_gtt drm_kms_helper irqbypass crct10dif_pclmul syscopyarea crc32_pclmul sysfillrect crc32c_intel sysimgblt ghash_clmulni_intel fb_sys_fops rapl wmi_bmof intel_cstate intel_uncore ahci libahci mei_me drm video libata wmi joydev mei ip_tables
kern :warn : [ 100.240822] CPU: 0 PID: 1055 Comm: mkfs.ext3 Not tainted 5.13.0-rc3-00092-g2aa18c32fd09 #1
kern :warn : [ 100.250180] Hardware name: Hewlett-Packard HP Pro 3340 MT/17A1, BIOS 8.07 01/24/2013
kern :warn : [ 100.259064] RIP: 0010:dd_insert_requests (kbuild/src/consumer/block/mq-deadline-main.c:714 kbuild/src/consumer/block/mq-deadline-main.c:761)
kern :warn : [ 100.265114] Code: c6 07 00 0f 1f 40 00 65 ff 0d a7 fe a0 7e 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 4c 89 f7 e8 d0 21 01 00 e9 f3 fe ff ff <0f> 0b e9 26 ff ff ff 41 8b 47 d0 4c 89 f6 83 e0 01 48 83 c0 01 48
All code
========
0: c6 07 00 movb $0x0,(%rdi)
3: 0f 1f 40 00 nopl 0x0(%rax)
7: 65 ff 0d a7 fe a0 7e decl %gs:0x7ea0fea7(%rip) # 0x7ea0feb5
e: 48 83 c4 18 add $0x18,%rsp
12: 5b pop %rbx
13: 5d pop %rbp
14: 41 5c pop %r12
16: 41 5d pop %r13
18: 41 5e pop %r14
1a: 41 5f pop %r15
1c: c3 retq
1d: 4c 89 f7 mov %r14,%rdi
20: e8 d0 21 01 00 callq 0x121f5
25: e9 f3 fe ff ff jmpq 0xffffffffffffff1d
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 26 ff ff ff jmpq 0xffffffffffffff57
31: 41 8b 47 d0 mov -0x30(%r15),%eax
35: 4c 89 f6 mov %r14,%rsi
38: 83 e0 01 and $0x1,%eax
3b: 48 83 c0 01 add $0x1,%rax
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 26 ff ff ff jmpq 0xffffffffffffff2d
7: 41 8b 47 d0 mov -0x30(%r15),%eax
b: 4c 89 f6 mov %r14,%rsi
e: 83 e0 01 and $0x1,%eax
11: 48 83 c0 01 add $0x1,%rax
15: 48 rex.W
kern :warn : [ 100.286230] RSP: 0018:ffffc9000101fb40 EFLAGS: 00010246
kern :warn : [ 100.292611] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
kern :warn : [ 100.300920] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881ff9a2f00
kern :warn : [ 100.309253] RBP: ffff8880744c3200 R08: ffffc9000101fbd0 R09: 0000000000000000
kern :warn : [ 100.317578] R10: 0000000000000033 R11: 0000000000000000 R12: ffffc9000101fbd0
kern :warn : [ 100.325885] R13: ffff88821a411908 R14: ffff8881ff981a40 R15: ffff8881ff981a88
kern :warn : [ 100.334200] FS: 00007f21576fe780(0000) GS:ffff88821fa00000(0000) knlGS:0000000000000000
kern :warn : [ 100.343452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kern :warn : [ 100.350333] CR2: 00007f215797f060 CR3: 00000001a07a6002 CR4: 00000000001706f0
kern :warn : [ 100.358636] Call Trace:
kern :warn : [ 100.362195] blk_mq_sched_insert_requests (kbuild/src/consumer/block/blk-mq-sched.c:513)
kern :warn : [ 100.368316] blk_mq_flush_plug_list (kbuild/src/consumer/include/linux/list.h:282 kbuild/src/consumer/block/blk-mq.c:1960)
kern :warn : [ 100.373997] blk_flush_plug_list (kbuild/src/consumer/block/blk-core.c:1734)
kern :warn : [ 100.379312] blk_mq_submit_bio (kbuild/src/consumer/arch/x86/include/asm/jump_label.h:19 kbuild/src/consumer/include/linux/jump_label.h:200 kbuild/src/consumer/include/trace/events/block.h:356 kbuild/src/consumer/block/blk-mq.c:2260)
kern :warn : [ 100.384521] submit_bio_noacct (kbuild/src/consumer/block/blk-core.c:919 kbuild/src/consumer/block/blk-core.c:968 kbuild/src/consumer/block/blk-core.c:1047)
kern :warn : [ 100.389722] submit_bio_wait (kbuild/src/consumer/block/bio.c:1164)
kern :warn : [ 100.394597] blkdev_issue_discard (kbuild/src/consumer/block/blk-lib.c:145)
kern :warn : [ 100.399964] blk_ioctl_discard (kbuild/src/consumer/block/ioctl.c:141)
kern :warn : [ 100.405090] blkdev_common_ioctl (kbuild/src/consumer/block/ioctl.c:468)
kern :warn : [ 100.410425] blkdev_ioctl (kbuild/src/consumer/block/ioctl.c:579)
kern :warn : [ 100.415175] block_ioctl (kbuild/src/consumer/fs/block_dev.c:1655)
kern :warn : [ 100.419642] __x64_sys_ioctl (kbuild/src/consumer/fs/ioctl.c:52 kbuild/src/consumer/fs/ioctl.c:1069 kbuild/src/consumer/fs/ioctl.c:1055 kbuild/src/consumer/fs/ioctl.c:1055)
kern :warn : [ 100.424405] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:47)
kern :warn : [ 100.429016] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:112)
kern :warn : [ 100.435090] RIP: 0033:0x7f2157812427
kern :warn : [ 100.439661] Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48
All code
========
0: 00 00 add %al,(%rax)
2: 90 nop
3: 48 8b 05 69 aa 0c 00 mov 0xcaa69(%rip),%rax # 0xcaa73
a: 64 c7 00 26 00 00 00 movl $0x26,%fs:(%rax)
11: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax
18: c3 retq
19: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
20: 00 00 00
23: b8 10 00 00 00 mov $0x10,%eax
28: 0f 05 syscall
2a:* 48 3d 01 f0 ff ff cmp $0xfffffffffffff001,%rax <-- trapping instruction
30: 73 01 jae 0x33
32: c3 retq
33: 48 8b 0d 39 aa 0c 00 mov 0xcaa39(%rip),%rcx # 0xcaa73
3a: f7 d8 neg %eax
3c: 64 89 01 mov %eax,%fs:(%rcx)
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: 48 3d 01 f0 ff ff cmp $0xfffffffffffff001,%rax
6: 73 01 jae 0x9
8: c3 retq
9: 48 8b 0d 39 aa 0c 00 mov 0xcaa39(%rip),%rcx # 0xcaa49
10: f7 d8 neg %eax
12: 64 89 01 mov %eax,%fs:(%rcx)
15: 48 rex.W
kern :warn : [ 100.460608] RSP: 002b:00007ffe241c15b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kern :warn : [ 100.469262] RAX: ffffffffffffffda RBX: 00007ffe241c16e0 RCX: 00007f2157812427
kern :warn : [ 100.477447] RDX: 00007ffe241c15c0 RSI: 0000000000001277 RDI: 0000000000000004
kern :warn : [ 100.485637] RBP: 0000557074df39d0 R08: 0000000000000004 R09: 0000000000000000
kern :warn : [ 100.493851] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000002800
kern :warn : [ 100.502067] R13: 0000557074df39d0 R14: 0000000000000000 R15: 0000000000000000
kern :warn : [ 100.510250] ---[ end trace 3c43a36e40e0cc9b ]---
kern :warn : [ 100.515932] ------------[ cut here ]------------
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
Thanks,
Oliver Sang
1 year, 3 months
Are you struggling with your programming assignments?
by tejparker97@gmail.com
Use Assignmenthelpshop and get it done now. Everyone is aware of the complexity and toughness of programming assignments. These assignments are not a cup of tea for everyone. You need to be very intelligent and skilled to accomplish programming assignments. Many students are unable to do the programming assignments that lead them to get poor marks. So, it is better to hire a professional expert with sound knowledge and expertise in programming. Assignmenthelpshop provides quality Programming Assignment Help that can solve your all problems related to assignments. Give it a go and boost your grades while being in your comfort zone.
https://www.assignmenthelpshop.com/programming-assignment-help/
1 year, 3 months
Are you struggling with your programming assignments?
by tejparker97@gmail.com
Use Assignmenthelpshop and get it done now. Everyone is aware of the complexity and toughness of programming assignments. These assignments are not a cup of tea for everyone. You need to be very intelligent and skilled to accomplish programming assignments. Many students are unable to do the programming assignments that lead them to get poor marks. So, it is better to hire a professional expert with sound knowledge and expertise in programming. Assignmenthelpshop provides quality <a href="https://www.assignmenthelpshop.com/programming-assignment-help">Programming assignment help</a> that can solve your all problems related to assignments. Give it a go and boost your grades while being in your comfort zone.
1 year, 3 months