Re: [LKP] [another NUMA code dump] -3.9% netperf throughput drop
by Fengguang Wu
Hi Peter,
On Wed, Aug 28, 2013 at 10:02:40PM +0800, Fengguang Wu wrote:
> Hi Peter,
>
> FYI, we find a little netperf performance drop, together with much
> reduced lock contentions. Please let me know if you would like us do
> more tests on it.
>
> 126d9c20bc99ccf1d62581d6e7fd63fd12e222bf is the first bad commit
> commit 126d9c20bc99ccf1d62581d6e7fd63fd12e222bf
> Author: Peter Zijlstra <peterz(a)infradead.org>
> Date: Tue Aug 20 15:36:48 2013 +0200
>
> another NUMA code dump
Now we got more results: there are both ups and downs.
661.10 +2.3% 676.20 TOTAL netperf.Throughput_Mbps
165733.95 -6.7% 154709.48 TOTAL aim7.2000.jobs-per-min
2203.20 -1.7% 2165.50 TOTAL netperf.Throughput_tps
569.20 -68.0% 181.94 TOTAL slabinfo.skbuff_fclone_cache.num_objs
542.84 -66.5% 181.94 TOTAL slabinfo.skbuff_fclone_cache.active_objs
376.92 -69.2% 116.27 TOTAL sched_debug.cpu#2.nr_uninterruptible
441.02 +89.6% 836.36 TOTAL sched_debug.cfs_rq[7]:/.nr_spread_over
62464.94 -36.5% 39676.33 TOTAL interrupts.CPU3.RES
53942.34 -46.8% 28701.16 TOTAL interrupts.CPU1.RES
85.95 +164.3% 227.20 TOTAL sched_debug.cpu#5.nr_uninterruptible
73764.87 -32.5% 49819.94 TOTAL interrupts.CPU2.RES
7698.32 +118.7% 16836.46 TOTAL sched_debug.cfs_rq[7]:/.spread0
89601.90 -31.3% 61544.49 TOTAL interrupts.CPU0.RES
496290.10 -25.8% 368250.18 TOTAL interrupts.RES
11166.54 +82.7% 20397.97 TOTAL sched_debug.cfs_rq[7]:/.min_vruntime
158066.00 -34.2% 103970.26 TOTAL softirqs.CPU0.NET_RX
630988.15 -29.8% 442956.09 TOTAL softirqs.NET_RX
156872.34 -29.2% 110987.49 TOTAL softirqs.CPU2.NET_RX
157622.69 -30.0% 110326.06 TOTAL softirqs.CPU3.NET_RX
7690.77 -15.2% 6523.75 TOTAL sched_debug.cpu#2.ttwu_local
33587.66 -16.5% 28048.84 TOTAL sched_debug.cpu#1.sched_goidle
158427.11 -25.7% 117672.28 TOTAL softirqs.CPU1.NET_RX
8975.45 -21.6% 7038.36 TOTAL sched_debug.cpu#3.ttwu_local
21103.12 -21.6% 16553.45 TOTAL softirqs.CPU3.RCU
21376.43 -20.8% 16934.73 TOTAL softirqs.CPU2.RCU
1663.42 -1.9% 1631.68 TOTAL slabinfo.kmalloc-2048.num_objs
1554.94 +3.8% 1613.65 TOTAL slabinfo.kmalloc-2048.active_objs
3526131.15 +20.5% 4248631.25 TOTAL proc-vmstat.pgalloc_normal
56884.60 +20.3% 68432.74 TOTAL slabinfo.kmalloc-64.num_objs
55951.48 +19.8% 67030.33 TOTAL slabinfo.kmalloc-64.active_objs
16903.28 +18.2% 19976.14 TOTAL softirqs.CPU0.SCHED
24396.71 -16.6% 20337.48 TOTAL interrupts.CPU6.RES
24424.39 -15.4% 20660.17 TOTAL interrupts.CPU5.RES
68269.93 -14.8% 58135.54 TOTAL sched_debug.cpu#6.nr_switches
68301.75 -14.8% 58169.82 TOTAL sched_debug.cpu#6.sched_count
36349.50 -14.3% 31146.73 TOTAL sched_debug.cpu#0.sched_goidle
169088.39 -14.3% 144828.56 TOTAL softirqs.RCU
29181.61 -13.3% 25309.49 TOTAL sched_debug.cpu#5.sched_goidle
29244.19 -14.7% 24941.43 TOTAL sched_debug.cpu#6.sched_goidle
46361.62 +7.7% 49935.54 TOTAL softirqs.SCHED
2254.81 -11.6% 1993.26 TOTAL sched_debug.cfs_rq[2]:/.min_vruntime
84214.17 -12.9% 73322.31 TOTAL sched_debug.cpu#3.nr_switches
21465.75 -15.9% 18061.93 TOTAL softirqs.CPU0.RCU
84259.37 -12.9% 73360.78 TOTAL sched_debug.cpu#3.sched_count
2717.34 +13.4% 3081.63 TOTAL slabinfo.kmalloc-128.num_objs
2717.34 +12.7% 3061.40 TOTAL slabinfo.kmalloc-128.active_objs
44008.82 -13.0% 38280.08 TOTAL sched_debug.cpu#3.ttwu_count
20872.34 -11.8% 18400.51 TOTAL softirqs.CPU4.RCU
1164.51 +13.5% 1321.68 TOTAL buddyinfo.Node.0.zone.Normal.0
1687.56 +12.3% 1895.26 TOTAL slabinfo.filp.active_objs
67219.90 -12.5% 58843.19 TOTAL sched_debug.cpu#5.nr_switches
67253.78 -12.5% 58880.24 TOTAL sched_debug.cpu#5.sched_count
5312453.48 +13.5% 6029326.46 TOTAL proc-vmstat.pgfree
21554.90 -13.8% 18579.77 TOTAL softirqs.CPU6.RCU
36507.99 -14.9% 31078.42 TOTAL sched_debug.cpu#2.sched_goidle
43167.98 -13.2% 37457.64 TOTAL sched_debug.cpu#2.ttwu_count
24019.62 -11.5% 21267.02 TOTAL interrupts.CPU7.RES
82664.49 -15.0% 70263.14 TOTAL sched_debug.cpu#2.sched_count
82620.08 -15.0% 70223.42 TOTAL sched_debug.cpu#2.nr_switches
37659.16 -13.7% 32493.72 TOTAL sched_debug.cpu#6.ttwu_count
20774.12 -10.7% 18544.93 TOTAL softirqs.CPU1.RCU
28506.05 -10.2% 25592.08 TOTAL sched_debug.cpu#7.sched_goidle
35939.20 -11.5% 31822.25 TOTAL sched_debug.cpu#3.sched_goidle
21152.73 -11.4% 18737.90 TOTAL softirqs.CPU5.RCU
861.68 +12.4% 968.22 TOTAL buddyinfo.Node.0.zone.Normal.1
37078.89 -10.4% 33230.12 TOTAL sched_debug.cpu#5.ttwu_count
860788.53 +10.6% 952024.03 TOTAL proc-vmstat.numa_hit
860788.53 +10.6% 952023.96 TOTAL proc-vmstat.numa_local
7802.01 +11.1% 8670.11 TOTAL sched_debug.cpu#5.nr_load_updates
65555.21 -9.3% 59426.89 TOTAL sched_debug.cpu#7.nr_switches
65585.05 -9.3% 59462.10 TOTAL sched_debug.cpu#7.sched_count
8846.95 -14.5% 7563.70 TOTAL sched_debug.cpu#6.ttwu_local
22195.37 -9.0% 20188.74 TOTAL softirqs.CPU3.SCHED
227.63 +8.2% 246.30 TOTAL uptime.idle
23712.01 +9.2% 25893.32 TOTAL interrupts.LOC
4307.40 -99.3% 31.00 TOTAL lock_stat.((&icsk->icsk_retransmit_timer)).acquisitions
7139.90 -96.3% 265.40 TOTAL lock_stat.((&icsk->icsk_delack_timer)).acquisitions
566879.90 -99.9% 657.00 TOTAL lock_stat.&wq->wait.acq-bounces
13827.10 -99.6% 56.80 TOTAL lock_stat.&(&base->lock)->rlock.con-bounces
40546.40 -99.6% 161.40 TOTAL lock_stat.&(&base->lock)->rlock.contentions.lock_timer_base
13827.30 -99.6% 57.00 TOTAL lock_stat.&(&base->lock)->rlock.contentions
74164.23 -99.2% 628.83 TOTAL lock_stat.&(&base->lock)->rlock.waittime-total
4249.11 -100.0% 0.00 TOTAL lock_stat.&(&list->lock)->rlock#2.waittime-total
215936.60 -99.8% 324.60 TOTAL lock_stat.&(&list->lock)->rlock#2.acq-bounces
2975.23 -100.0% 0.00 TOTAL lock_stat.&wq->wait.waittime-total
12327.20 -99.9% 8.30 TOTAL lock_stat.slock-AF_INET.contentions.release_sock
23594.90 -99.9% 13.80 TOTAL lock_stat.slock-AF_INET/1.con-bounces
23594.90 -99.9% 13.80 TOTAL lock_stat.slock-AF_INET/1.contentions
23594.90 -99.9% 13.80 TOTAL lock_stat.slock-AF_INET/1.contentions.tcp_v4_rcv
11909.80 -99.9% 7.40 TOTAL lock_stat.slock-AF_INET/1.contentions.release_sock
11682.50 -99.9% 7.11 TOTAL lock_stat.slock-AF_INET/1.contentions.lock_sock_nested
2891155.00 +793.9% 25844317.50 TOTAL lock_stat.&pipe->mutex/1.acquisitions
584.68 +770.8% 5091.37 TOTAL lock_stat.&pipe->wait.waittime-total
2870693.90 +799.0% 25808748.90 TOTAL lock_stat.&pipe->wait.acquisitions
653854.40 -99.9% 361.90 TOTAL lock_stat.slock-AF_INET/1.acq-bounces
132668.89 -100.0% 58.82 TOTAL lock_stat.slock-AF_INET/1.waittime-total
135481.90 -99.9% 86.50 TOTAL lock_stat.slock-AF_INET.contentions.tcp_v4_rcv
113885.70 -99.9% 75.00 TOTAL lock_stat.slock-AF_INET.contentions.lock_sock_nested
834535.10 -99.9% 695.50 TOTAL lock_stat.slock-AF_INET.acq-bounces
135509.30 -99.9% 86.50 TOTAL lock_stat.slock-AF_INET.con-bounces
115477.01 -96.7% 3808.53 TOTAL lock_stat.((&icsk->icsk_delack_timer)).holdtime-total
135511.30 -99.9% 86.50 TOTAL lock_stat.slock-AF_INET.contentions
916330.86 -93.8% 56991.34 TOTAL lock_stat.&rq->lock.waittime-total
6994048.67 -99.9% 4638.26 TOTAL lock_stat.slock-AF_INET.waittime-total
72918.76 -93.8% 4501.73 TOTAL lock_stat.((&icsk->icsk_retransmit_timer)).holdtime-total
561.79 +366.8% 2622.34 TOTAL lock_stat.&pipe->wait.waittime-max
37855.16 +392.2% 186310.37 TOTAL lock_stat.(&task->u.tk_work).holdtime-max
37856.44 +392.2% 186311.75 TOTAL lock_stat.rpciod-R.holdtime-max
1882726.00 -30.9% 1301055.80 TOTAL lock_stat.&rq->lock.acq-bounces
1192.74 +349.8% 5364.45 TOTAL lock_stat.slock-AF_INET-RPC.waittime-total
1057131.87 -68.5% 332629.78 TOTAL lock_stat.&rq->lock/1.holdtime-total
230850.10 -47.2% 121845.20 TOTAL lock_stat.&rq->lock.contentions.try_to_wake_up
2054554.10 +337.8% 8995698.10 TOTAL lock_stat.&p->pi_lock.acquisitions
958070.10 -86.1% 133493.10 TOTAL lock_stat.&(&base->lock)->rlock.acq-bounces
3001.10 -83.1% 508.26 TOTAL lock_stat.&mm->mmap_sem-R.holdtime-max
41448.45 +277.4% 156411.45 TOTAL lock_stat.zone->wait_table+i.waittime-total
206432.67 +247.1% 716507.42 TOTAL lock_stat.(&task->u.tk_work).holdtime-total
207795.52 +246.0% 718916.47 TOTAL lock_stat.rpciod-R.holdtime-total
42415.30 +13.1% 47964.02 TOTAL lock_stat.&rq->lock/1.waittime-total
1598626.30 +671.8% 12338701.15 TOTAL lock_stat.&pipe->mutex/1.holdtime-total
240154.70 -24.1% 182204.10 TOTAL lock_stat.&rq->lock.contentions
239875.20 -24.1% 182101.50 TOTAL lock_stat.&rq->lock.con-bounces
10334983.54 +260.8% 37290852.98 TOTAL lock_stat.all_cpu_access_lock.holdtime-total
10382840.38 +260.4% 37424524.18 TOTAL lock_stat.trace_event_sem-R.holdtime-total
10682456.18 +257.6% 38195309.13 TOTAL lock_stat.&iter->mutex.holdtime-total
11622.60 +254.2% 41166.90 TOTAL lock_stat.slock-AF_INET-RPC/1.acquisitions
11716.00 +252.2% 41260.90 TOTAL lock_stat._xmit_ETHER#2.acquisitions
23432.00 +252.2% 82521.90 TOTAL lock_stat.&(&list->lock)->rlock.acquisitions
15159239.80 +253.4% 53574107.00 TOTAL lock_stat.buffer->reader_lock_key.acquisitions
13357123.60 +261.4% 48273636.60 TOTAL lock_stat.&rq->lock.acquisitions
3250919.71 +225.8% 10592024.15 TOTAL lock_stat.buffer->reader_lock_key.holdtime-total
15000.00 +219.6% 47943.30 TOTAL lock_stat.slock-AF_INET-RPC.acquisitions
1821865.51 +799.3% 16383840.49 TOTAL lock_stat.&pipe->wait.holdtime-total
6306.35 +216.5% 19962.11 TOTAL lock_stat.&(&list->lock)->rlock.holdtime-total
104745.20 +220.5% 335684.60 TOTAL lock_stat.trace_types_lock.acquisitions
104745.20 +220.5% 335684.00 TOTAL lock_stat.trace_event_sem-R.acquisitions
104745.20 +220.5% 335683.80 TOTAL lock_stat.all_cpu_access_lock.acquisitions
1411.65 +126.4% 3196.43 TOTAL lock_stat.&vfsmount_lock-R.holdtime-max
1669574.80 -60.4% 661408.70 TOTAL lock_stat.&(&base->lock)->rlock.acquisitions
116585.00 +197.0% 346231.10 TOTAL lock_stat.&iter->mutex.acquisitions
39046.00 +188.4% 112601.00 TOTAL lock_stat.trace_types_lock.holdtime-total
2356238.10 -20.5% 1872483.90 TOTAL lock_stat.rcu_read_lock_bh-R.acquisitions
712340.80 -58.5% 295574.20 TOTAL lock_stat.&wq->wait.acquisitions
106482.08 +160.0% 276823.43 TOTAL lock_stat.slock-AF_INET-RPC/1.holdtime-total
161126.72 +183.9% 457435.26 TOTAL lock_stat._xmit_ETHER#2.holdtime-total
134393.55 +183.8% 381435.33 TOTAL lock_stat.sk_lock-AF_INET-RPC.holdtime-total
23948.20 +127.0% 54351.40 TOTAL lock_stat.&sb->s_type->i_lock_key#1.acquisitions
509327.41 -54.9% 229766.53 TOTAL lock_stat.&sig->cred_guard_mutex.holdtime-max
27084.80 +118.9% 59298.00 TOTAL lock_stat.sb_writers#4-R.acquisitions
6221.93 -51.1% 3045.54 TOTAL lock_stat.sk_lock-AF_INET.holdtime-max
4656.87 +181.9% 13126.12 TOTAL lock_stat.&pipe->mutex/1.holdtime-max
10765.33 +123.3% 24037.30 TOTAL lock_stat.(&task->u.tk_work)#2.holdtime-total
43141327.41 -13.1% 37507805.71 TOTAL lock_stat.rcu_read_lock_bh-R.holdtime-total
4100.50 -51.8% 1975.29 TOTAL lock_stat.rcu_read_lock-R.holdtime-max
739556.16 -17.4% 610769.01 TOTAL lock_stat.&mm->mmap_sem-W.holdtime-max
11290.17 +117.2% 24520.74 TOTAL lock_stat.nfsiod-R.holdtime-total
42400.54 -47.1% 22416.67 TOTAL lock_stat.&(&info->si_lock)->rlock.holdtime-total
48336.50 +80.9% 87451.90 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions.ip_defrag
1168351.69 +10.8% 1294640.42 TOTAL lock_stat.&pipe->mutex/1.waittime-total
102707.56 -48.3% 53101.39 TOTAL lock_stat.&sb->s_type->i_mutex_key#2.waittime-total
91318694.78 +26.6% 115632734.76 TOTAL lock_stat.&rq->lock.holdtime-total
50189004.65 +18.5% 59471866.33 TOTAL lock_stat.&p->pi_lock.holdtime-total
133163.30 -44.5% 73942.90 TOTAL lock_stat.&anon_vma->rwsem.acq-bounces
66818.00 -45.6% 36333.20 TOTAL lock_stat.&p->pi_lock.acq-bounces
361874.50 +82.2% 659237.20 TOTAL lock_stat.&(&zone->lock)->rlock.contentions.__free_pages_ok
3957.60 -33.3% 2638.84 TOTAL lock_stat.tasklist_lock-W.waittime-total
244764.50 +72.1% 421131.00 TOTAL lock_stat.&(&zone->lock)->rlock.contentions.get_page_from_freelist
6540563.51 -53.2% 3058553.07 TOTAL lock_stat.&(&base->lock)->rlock.holdtime-total
214843.60 +74.5% 374915.00 TOTAL lock_stat.&(&nf->lru_lock)->rlock.con-bounces
214846.50 +74.5% 374916.00 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions
641418.67 +75.6% 1126321.06 TOTAL lock_stat.&(&nf->lru_lock)->rlock.waittime-total
192855.90 +76.0% 339378.40 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions.inet_frag_find
2322889.38 +111.6% 4915093.78 TOTAL lock_stat.sb_writers#7-R.holdtime-total
63744.63 -42.3% 36809.55 TOTAL lock_stat.&(&n->list_lock)->rlock.holdtime-total
188500.60 +71.4% 323001.70 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions.inet_frag_kill
14078.22 +71.4% 24129.54 TOTAL lock_stat.&(&queue->lock)->rlock.holdtime-total
350330.40 +68.4% 589915.00 TOTAL lock_stat.&(&zone->lock)->rlock.contentions
350276.20 +68.2% 589108.30 TOTAL lock_stat.&(&zone->lock)->rlock.con-bounces
2435.44 +68.8% 4109.92 TOTAL lock_stat.&(&hb->chain_lock)->rlock.waittime-total
11039.58 +38.2% 15255.39 TOTAL lock_stat.(&(&wb->dwork)->work).holdtime-total
45302081.74 -8.2% 41581942.88 TOTAL lock_stat.&wq->wait.holdtime-total
182856.27 +59.9% 292446.19 TOTAL lock_stat.zone->wait_table+i.holdtime-total
2182608.45 +60.5% 3502508.79 TOTAL lock_stat.&(&zone->lock)->rlock.waittime-total
21148.30 +57.5% 33298.65 TOTAL lock_stat.&(&xprt->transport_lock)->rlock.holdtime-total
11841.06 +49.7% 17725.30 TOTAL lock_stat.&sb->s_type->i_lock_key#1.holdtime-total
20350.70 +48.1% 30148.50 TOTAL lock_stat.&pipe->mutex/1.acq-bounces
19903.70 +48.3% 29522.80 TOTAL lock_stat.&pipe->wait.acq-bounces
12025.44 +88.2% 22631.20 TOTAL lock_stat.&type->i_mutex_dir_key#2.holdtime-total
12025.44 +88.2% 22631.20 TOTAL lock_stat.&type->i_mutex_dir_key#2.holdtime-min
12025.44 +88.2% 22631.20 TOTAL lock_stat.&type->i_mutex_dir_key#2.holdtime-max
459953.53 +9.5% 503614.41 TOTAL lock_stat.&mapping->i_mmap_mutex.waittime-max
36732.44 +38.4% 50824.74 TOTAL lock_stat.sb_writers#4-R.holdtime-total
439874.42 -5.3% 416534.11 TOTAL lock_stat.clock-AF_INET-R.holdtime-total
277796.51 +44.6% 401667.68 TOTAL lock_stat.&p->pi_lock.waittime-total
2313283.70 -26.8% 1693267.70 TOTAL lock_stat.&(&list->lock)->rlock#3.acquisitions
74032.10 -28.6% 52864.10 TOTAL lock_stat.&rq->lock/1.acquisitions
931324.30 +34.7% 1254788.40 TOTAL lock_stat.&(&zone->lock)->rlock.acq-bounces
14328.30 +25.0% 17906.30 TOTAL lock_stat.hrtimer_bases.lock#7.holdtime-total
37493.70 -26.3% 27624.80 TOTAL lock_stat.&rq->lock/1.acq-bounces
15085692.76 +36.5% 20597805.11 TOTAL lock_stat.rcu_read_lock-R.holdtime-total
31239.40 +36.1% 42514.40 TOTAL lock_stat.zone->wait_table+i.contentions.__wake_up
159447.37 +32.3% 211019.95 TOTAL lock_stat.rcu_node_0.waittime-total
100236.97 -33.9% 66247.10 TOTAL lock_stat.&pipe->mutex/1.waittime-max
1166309.60 -23.3% 894892.70 TOTAL lock_stat.slock-AF_INET/1.acquisitions
4053.60 +29.4% 5245.03 TOTAL lock_stat.sk_lock-AF_INET6.holdtime-total
5036.72 -23.9% 3834.90 TOTAL lock_stat.&(&tsk->delays->lock)->rlock.holdtime-total
10019.40 +45.8% 14613.28 TOTAL lock_stat.slock-AF_INET-RPC.holdtime-total
1417553.50 +23.9% 1755833.90 TOTAL lock_stat.&(&zone->lock)->rlock.acquisitions
6946024.55 -19.3% 5602189.45 TOTAL lock_stat.&(&list->lock)->rlock#3.holdtime-total
5172.44 -19.2% 4177.08 TOTAL lock_stat.&q->lock.holdtime-total
18369.95 +23.1% 22618.18 TOTAL lock_stat.(&(work)->timer).holdtime-total
834643.80 +22.7% 1024368.10 TOTAL lock_stat.&(&nf->lru_lock)->rlock.acq-bounces
25251.78 +22.4% 30899.04 TOTAL lock_stat.&(&pool->lock)->rlock.holdtime-total
3931866.39 +23.0% 4834661.56 TOTAL lock_stat.((&timer)).holdtime-total
400630.70 -19.6% 322076.80 TOTAL lock_stat.&(&dentry->d_lock)->rlock.contentions.path_get
729459.97 -23.7% 556449.31 TOTAL lock_stat.&mapping->i_mmap_mutex.waittime-total
1205124.95 +17.2% 1412040.97 TOTAL lock_stat.&rsp->gp_wq.holdtime-total
778969.60 -18.0% 639110.20 TOTAL lock_stat.&(&dentry->d_lock)->rlock.contentions.dput
203249.44 -17.2% 168359.45 TOTAL lock_stat.&(&dentry->d_lock)->rlock.waittime-total
787586.90 -16.9% 654513.10 TOTAL lock_stat.&(&dentry->d_lock)->rlock.con-bounces
787737.60 -16.9% 654670.40 TOTAL lock_stat.&(&dentry->d_lock)->rlock.contentions
982029.40 +20.0% 1178821.80 TOTAL lock_stat.&f->lock-R.acq-bounces
11377298.70 -15.7% 9591520.20 TOTAL lock_stat.rcu_read_lock-R.acquisitions
21232.88 +19.0% 25274.64 TOTAL lock_stat.&rnp->nocb_gp_wq[1].holdtime-total
22050.24 +18.9% 26217.07 TOTAL lock_stat.&rnp->nocb_gp_wq[0].holdtime-total
175592.50 +16.0% 203669.97 TOTAL lock_stat.clockevents_lock.waittime-total
182770.97 +18.0% 215662.07 TOTAL lock_stat.rcu_sched_state.onoff_mutex.holdtime-total
169134.80 -14.9% 143897.00 TOTAL lock_stat.zone->wait_table+i.acquisitions
11269453.16 +12.6% 12691439.20 TOTAL lock_stat.&(&zone->lock)->rlock.holdtime-total
13525.95 +17.0% 15830.27 TOTAL lock_stat.(&(work)->work).holdtime-total
8975.59 +16.8% 10485.40 TOTAL lock_stat.proc_subdir_lock.holdtime-total
99222.00 +16.0% 115055.75 TOTAL lock_stat.sb_writers#2-R.holdtime-total
124847.90 -13.7% 107753.10 TOTAL lock_stat.zone->wait_table+i.acq-bounces
1518193.46 +10.4% 1676339.59 TOTAL lock_stat.rcu_node_0.holdtime-total
95007.69 +15.2% 109413.23 TOTAL lock_stat.&sig->wait_chldexit.holdtime-total
256302.50 -13.1% 222725.30 TOTAL lock_stat.&mm->mmap_sem-W.acq-bounces
6934.41 +14.6% 7944.90 TOTAL lock_stat.sb_writers#3-R.holdtime-total
15463.56 +13.7% 17577.66 TOTAL lock_stat.entries_lock-R.holdtime-total
263989.70 -13.1% 229316.00 TOTAL lock_stat.&(&mm->page_table_lock)->rlock.acq-bounces
13700.25 +87.2% 25651.38 TOTAL lock_stat.hrtimer_bases.lock#4.holdtime-total
10082.81 +14.1% 11507.23 TOTAL lock_stat.bdev_lock.holdtime-total
258345.90 -12.6% 225740.10 TOTAL lock_stat.&(&newf->file_lock)->rlock.acq-bounces
28512.94 +12.8% 32154.48 TOTAL lock_stat.events-R.holdtime-total
47912.04 -12.3% 41996.01 TOTAL lock_stat.rename_lock.waittime-total
224803.20 -12.6% 196381.70 TOTAL lock_stat.&(&fs->lock)->rlock.acq-bounces
103370.20 -12.7% 90277.20 TOTAL lock_stat.rename_lock.contentions.SyS_getcwd
13222.56 +18.3% 15637.30 TOTAL lock_stat.hrtimer_bases.lock#5.holdtime-total
702867.20 -12.0% 618778.40 TOTAL lock_stat.&mapping->i_mmap_mutex.contentions.vma_link
238640.40 -11.8% 210514.70 TOTAL lock_stat.&(&sighand->siglock)->rlock.acq-bounces
144491.30 +12.8% 163038.30 TOTAL lock_stat.clockevents_lock.con-bounces
289138.00 +12.8% 326170.80 TOTAL lock_stat.clockevents_lock.contentions.clockevents_notify
144569.00 +12.8% 163085.40 TOTAL lock_stat.clockevents_lock.contentions
585553.00 -11.5% 518161.60 TOTAL lock_stat.&mapping->i_mmap_mutex.con-bounces
585553.00 -11.5% 518161.60 TOTAL lock_stat.&mapping->i_mmap_mutex.contentions
174036.80 +11.7% 194446.50 TOTAL lock_stat.((&timer)).acquisitions
198920.50 +12.5% 223756.40 TOTAL lock_stat.rcu_node_0.acquisitions
159105.97 +12.2% 178524.65 TOTAL lock_stat.&(&p->alloc_lock)->rlock.holdtime-total
12962.55 +11.3% 14431.29 TOTAL lock_stat.&child->perf_event_mutex.holdtime-total
467032.10 -10.9% 416276.60 TOTAL lock_stat.&mapping->i_mmap_mutex.contentions.unlink_file_vma
2639031.80 -10.6% 2360489.00 TOTAL lock_stat.sk_lock-AF_INET.acquisitions
23964.02 +9.5% 26236.37 TOTAL lock_stat.&fsnotify_mark_srcu-R.holdtime-total
706694.52 -9.6% 638876.20 TOTAL lock_stat.jiffies_lock.holdtime-total
14111.25 +36.4% 19245.96 TOTAL lock_stat.hrtimer_bases.lock.holdtime-total
47545.73 +11.3% 52915.50 TOTAL lock_stat.pgd_lock.holdtime-total
319799.80 +10.4% 353118.34 TOTAL lock_stat.tasklist_lock-W.holdtime-total
13131948.50 -10.6% 11739786.10 TOTAL lock_stat.&(&dentry->d_lock)->rlock.acq-bounces
5748518.20 -10.3% 5156548.00 TOTAL lock_stat.&mapping->i_mmap_mutex.acq-bounces
10293967.02 -6.7% 9599300.67 TOTAL lock_stat.&mm->mmap_sem-R.holdtime-total
2486744.60 +590.1% 17161466.60 TOTAL perf-stat.context-switches
91101.74 +19.8% 109116.17 TOTAL vmstat.system.cs
15.96 +17.8% 18.79 TOTAL iostat.cpu.user
19.81 +16.5% 23.07 TOTAL vmstat.cpu.us
1818.70 +29.6% 2356.40 TOTAL time.involuntary_context_switches
313376.90 -20.3% 249842.70 TOTAL perf-stat.cpu-migrations
17606.67 -17.4% 14540.72 TOTAL vmstat.system.in
669432.20 -12.7% 584367.90 TOTAL time.voluntary_context_switches
306.12 +0.1% 306.52 TOTAL iostat.cpu.system
72.62 +7.1% 77.79 TOTAL time.elapsed_time
400.28 +0.0% 400.32 TOTAL vmstat.cpu.sy
188.10 -6.8% 175.40 TOTAL time.percent_of_cpu_this_job_got
1.35 -7.9% 1.24 TOTAL iostat.sda.rkB/s
0.35 -7.9% 0.32 TOTAL vmstat.io.bi
178.19 -4.7% 169.90 TOTAL vmstat.cpu.wa
178.08 -4.6% 169.91 TOTAL iostat.cpu.iowait
311.74 +2.6% 319.92 TOTAL boottime.dhcp
269.22 +2.7% 276.37 TOTAL boottime.boot
568.45 +3.1% 586.18 TOTAL boottime.idle
2699.20 +0.6% 2714.10 TOTAL time.minor_page_faults
82.14 +0.3% 82.40 TOTAL iostat.cpu.idle
82.12 +0.3% 82.39 TOTAL vmstat.cpu.id
Details attached.
Thanks,
Fengguang
7 years, 5 months