Hi Peter,
On Wed, Aug 28, 2013 at 10:02:40PM +0800, Fengguang Wu wrote:
Hi Peter,
FYI, we find a little netperf performance drop, together with much
reduced lock contentions. Please let me know if you would like us do
more tests on it.
126d9c20bc99ccf1d62581d6e7fd63fd12e222bf is the first bad commit
commit 126d9c20bc99ccf1d62581d6e7fd63fd12e222bf
Author: Peter Zijlstra <peterz(a)infradead.org>
Date: Tue Aug 20 15:36:48 2013 +0200
another NUMA code dump
Now we got more results: there are both ups and downs.
661.10 +2.3% 676.20 TOTAL netperf.Throughput_Mbps
165733.95 -6.7% 154709.48 TOTAL aim7.2000.jobs-per-min
2203.20 -1.7% 2165.50 TOTAL netperf.Throughput_tps
569.20 -68.0% 181.94 TOTAL
slabinfo.skbuff_fclone_cache.num_objs
542.84 -66.5% 181.94 TOTAL
slabinfo.skbuff_fclone_cache.active_objs
376.92 -69.2% 116.27 TOTAL
sched_debug.cpu#2.nr_uninterruptible
441.02 +89.6% 836.36 TOTAL
sched_debug.cfs_rq[7]:/.nr_spread_over
62464.94 -36.5% 39676.33 TOTAL interrupts.CPU3.RES
53942.34 -46.8% 28701.16 TOTAL interrupts.CPU1.RES
85.95 +164.3% 227.20 TOTAL
sched_debug.cpu#5.nr_uninterruptible
73764.87 -32.5% 49819.94 TOTAL interrupts.CPU2.RES
7698.32 +118.7% 16836.46 TOTAL sched_debug.cfs_rq[7]:/.spread0
89601.90 -31.3% 61544.49 TOTAL interrupts.CPU0.RES
496290.10 -25.8% 368250.18 TOTAL interrupts.RES
11166.54 +82.7% 20397.97 TOTAL
sched_debug.cfs_rq[7]:/.min_vruntime
158066.00 -34.2% 103970.26 TOTAL softirqs.CPU0.NET_RX
630988.15 -29.8% 442956.09 TOTAL softirqs.NET_RX
156872.34 -29.2% 110987.49 TOTAL softirqs.CPU2.NET_RX
157622.69 -30.0% 110326.06 TOTAL softirqs.CPU3.NET_RX
7690.77 -15.2% 6523.75 TOTAL sched_debug.cpu#2.ttwu_local
33587.66 -16.5% 28048.84 TOTAL sched_debug.cpu#1.sched_goidle
158427.11 -25.7% 117672.28 TOTAL softirqs.CPU1.NET_RX
8975.45 -21.6% 7038.36 TOTAL sched_debug.cpu#3.ttwu_local
21103.12 -21.6% 16553.45 TOTAL softirqs.CPU3.RCU
21376.43 -20.8% 16934.73 TOTAL softirqs.CPU2.RCU
1663.42 -1.9% 1631.68 TOTAL slabinfo.kmalloc-2048.num_objs
1554.94 +3.8% 1613.65 TOTAL
slabinfo.kmalloc-2048.active_objs
3526131.15 +20.5% 4248631.25 TOTAL proc-vmstat.pgalloc_normal
56884.60 +20.3% 68432.74 TOTAL slabinfo.kmalloc-64.num_objs
55951.48 +19.8% 67030.33 TOTAL slabinfo.kmalloc-64.active_objs
16903.28 +18.2% 19976.14 TOTAL softirqs.CPU0.SCHED
24396.71 -16.6% 20337.48 TOTAL interrupts.CPU6.RES
24424.39 -15.4% 20660.17 TOTAL interrupts.CPU5.RES
68269.93 -14.8% 58135.54 TOTAL sched_debug.cpu#6.nr_switches
68301.75 -14.8% 58169.82 TOTAL sched_debug.cpu#6.sched_count
36349.50 -14.3% 31146.73 TOTAL sched_debug.cpu#0.sched_goidle
169088.39 -14.3% 144828.56 TOTAL softirqs.RCU
29181.61 -13.3% 25309.49 TOTAL sched_debug.cpu#5.sched_goidle
29244.19 -14.7% 24941.43 TOTAL sched_debug.cpu#6.sched_goidle
46361.62 +7.7% 49935.54 TOTAL softirqs.SCHED
2254.81 -11.6% 1993.26 TOTAL
sched_debug.cfs_rq[2]:/.min_vruntime
84214.17 -12.9% 73322.31 TOTAL sched_debug.cpu#3.nr_switches
21465.75 -15.9% 18061.93 TOTAL softirqs.CPU0.RCU
84259.37 -12.9% 73360.78 TOTAL sched_debug.cpu#3.sched_count
2717.34 +13.4% 3081.63 TOTAL slabinfo.kmalloc-128.num_objs
2717.34 +12.7% 3061.40 TOTAL
slabinfo.kmalloc-128.active_objs
44008.82 -13.0% 38280.08 TOTAL sched_debug.cpu#3.ttwu_count
20872.34 -11.8% 18400.51 TOTAL softirqs.CPU4.RCU
1164.51 +13.5% 1321.68 TOTAL buddyinfo.Node.0.zone.Normal.0
1687.56 +12.3% 1895.26 TOTAL slabinfo.filp.active_objs
67219.90 -12.5% 58843.19 TOTAL sched_debug.cpu#5.nr_switches
67253.78 -12.5% 58880.24 TOTAL sched_debug.cpu#5.sched_count
5312453.48 +13.5% 6029326.46 TOTAL proc-vmstat.pgfree
21554.90 -13.8% 18579.77 TOTAL softirqs.CPU6.RCU
36507.99 -14.9% 31078.42 TOTAL sched_debug.cpu#2.sched_goidle
43167.98 -13.2% 37457.64 TOTAL sched_debug.cpu#2.ttwu_count
24019.62 -11.5% 21267.02 TOTAL interrupts.CPU7.RES
82664.49 -15.0% 70263.14 TOTAL sched_debug.cpu#2.sched_count
82620.08 -15.0% 70223.42 TOTAL sched_debug.cpu#2.nr_switches
37659.16 -13.7% 32493.72 TOTAL sched_debug.cpu#6.ttwu_count
20774.12 -10.7% 18544.93 TOTAL softirqs.CPU1.RCU
28506.05 -10.2% 25592.08 TOTAL sched_debug.cpu#7.sched_goidle
35939.20 -11.5% 31822.25 TOTAL sched_debug.cpu#3.sched_goidle
21152.73 -11.4% 18737.90 TOTAL softirqs.CPU5.RCU
861.68 +12.4% 968.22 TOTAL buddyinfo.Node.0.zone.Normal.1
37078.89 -10.4% 33230.12 TOTAL sched_debug.cpu#5.ttwu_count
860788.53 +10.6% 952024.03 TOTAL proc-vmstat.numa_hit
860788.53 +10.6% 952023.96 TOTAL proc-vmstat.numa_local
7802.01 +11.1% 8670.11 TOTAL
sched_debug.cpu#5.nr_load_updates
65555.21 -9.3% 59426.89 TOTAL sched_debug.cpu#7.nr_switches
65585.05 -9.3% 59462.10 TOTAL sched_debug.cpu#7.sched_count
8846.95 -14.5% 7563.70 TOTAL sched_debug.cpu#6.ttwu_local
22195.37 -9.0% 20188.74 TOTAL softirqs.CPU3.SCHED
227.63 +8.2% 246.30 TOTAL uptime.idle
23712.01 +9.2% 25893.32 TOTAL interrupts.LOC
4307.40 -99.3% 31.00 TOTAL
lock_stat.((&icsk->icsk_retransmit_timer)).acquisitions
7139.90 -96.3% 265.40 TOTAL
lock_stat.((&icsk->icsk_delack_timer)).acquisitions
566879.90 -99.9% 657.00 TOTAL
lock_stat.&wq->wait.acq-bounces
13827.10 -99.6% 56.80 TOTAL
lock_stat.&(&base->lock)->rlock.con-bounces
40546.40 -99.6% 161.40 TOTAL
lock_stat.&(&base->lock)->rlock.contentions.lock_timer_base
13827.30 -99.6% 57.00 TOTAL
lock_stat.&(&base->lock)->rlock.contentions
74164.23 -99.2% 628.83 TOTAL
lock_stat.&(&base->lock)->rlock.waittime-total
4249.11 -100.0% 0.00 TOTAL
lock_stat.&(&list->lock)->rlock#2.waittime-total
215936.60 -99.8% 324.60 TOTAL
lock_stat.&(&list->lock)->rlock#2.acq-bounces
2975.23 -100.0% 0.00 TOTAL
lock_stat.&wq->wait.waittime-total
12327.20 -99.9% 8.30 TOTAL
lock_stat.slock-AF_INET.contentions.release_sock
23594.90 -99.9% 13.80 TOTAL
lock_stat.slock-AF_INET/1.con-bounces
23594.90 -99.9% 13.80 TOTAL
lock_stat.slock-AF_INET/1.contentions
23594.90 -99.9% 13.80 TOTAL
lock_stat.slock-AF_INET/1.contentions.tcp_v4_rcv
11909.80 -99.9% 7.40 TOTAL
lock_stat.slock-AF_INET/1.contentions.release_sock
11682.50 -99.9% 7.11 TOTAL
lock_stat.slock-AF_INET/1.contentions.lock_sock_nested
2891155.00 +793.9% 25844317.50 TOTAL
lock_stat.&pipe->mutex/1.acquisitions
584.68 +770.8% 5091.37 TOTAL
lock_stat.&pipe->wait.waittime-total
2870693.90 +799.0% 25808748.90 TOTAL
lock_stat.&pipe->wait.acquisitions
653854.40 -99.9% 361.90 TOTAL
lock_stat.slock-AF_INET/1.acq-bounces
132668.89 -100.0% 58.82 TOTAL
lock_stat.slock-AF_INET/1.waittime-total
135481.90 -99.9% 86.50 TOTAL
lock_stat.slock-AF_INET.contentions.tcp_v4_rcv
113885.70 -99.9% 75.00 TOTAL
lock_stat.slock-AF_INET.contentions.lock_sock_nested
834535.10 -99.9% 695.50 TOTAL
lock_stat.slock-AF_INET.acq-bounces
135509.30 -99.9% 86.50 TOTAL
lock_stat.slock-AF_INET.con-bounces
115477.01 -96.7% 3808.53 TOTAL
lock_stat.((&icsk->icsk_delack_timer)).holdtime-total
135511.30 -99.9% 86.50 TOTAL
lock_stat.slock-AF_INET.contentions
916330.86 -93.8% 56991.34 TOTAL
lock_stat.&rq->lock.waittime-total
6994048.67 -99.9% 4638.26 TOTAL
lock_stat.slock-AF_INET.waittime-total
72918.76 -93.8% 4501.73 TOTAL
lock_stat.((&icsk->icsk_retransmit_timer)).holdtime-total
561.79 +366.8% 2622.34 TOTAL
lock_stat.&pipe->wait.waittime-max
37855.16 +392.2% 186310.37 TOTAL
lock_stat.(&task->u.tk_work).holdtime-max
37856.44 +392.2% 186311.75 TOTAL lock_stat.rpciod-R.holdtime-max
1882726.00 -30.9% 1301055.80 TOTAL
lock_stat.&rq->lock.acq-bounces
1192.74 +349.8% 5364.45 TOTAL
lock_stat.slock-AF_INET-RPC.waittime-total
1057131.87 -68.5% 332629.78 TOTAL
lock_stat.&rq->lock/1.holdtime-total
230850.10 -47.2% 121845.20 TOTAL
lock_stat.&rq->lock.contentions.try_to_wake_up
2054554.10 +337.8% 8995698.10 TOTAL
lock_stat.&p->pi_lock.acquisitions
958070.10 -86.1% 133493.10 TOTAL
lock_stat.&(&base->lock)->rlock.acq-bounces
3001.10 -83.1% 508.26 TOTAL
lock_stat.&mm->mmap_sem-R.holdtime-max
41448.45 +277.4% 156411.45 TOTAL
lock_stat.zone->wait_table+i.waittime-total
206432.67 +247.1% 716507.42 TOTAL
lock_stat.(&task->u.tk_work).holdtime-total
207795.52 +246.0% 718916.47 TOTAL
lock_stat.rpciod-R.holdtime-total
42415.30 +13.1% 47964.02 TOTAL
lock_stat.&rq->lock/1.waittime-total
1598626.30 +671.8% 12338701.15 TOTAL
lock_stat.&pipe->mutex/1.holdtime-total
240154.70 -24.1% 182204.10 TOTAL
lock_stat.&rq->lock.contentions
239875.20 -24.1% 182101.50 TOTAL
lock_stat.&rq->lock.con-bounces
10334983.54 +260.8% 37290852.98 TOTAL
lock_stat.all_cpu_access_lock.holdtime-total
10382840.38 +260.4% 37424524.18 TOTAL
lock_stat.trace_event_sem-R.holdtime-total
10682456.18 +257.6% 38195309.13 TOTAL
lock_stat.&iter->mutex.holdtime-total
11622.60 +254.2% 41166.90 TOTAL
lock_stat.slock-AF_INET-RPC/1.acquisitions
11716.00 +252.2% 41260.90 TOTAL
lock_stat._xmit_ETHER#2.acquisitions
23432.00 +252.2% 82521.90 TOTAL
lock_stat.&(&list->lock)->rlock.acquisitions
15159239.80 +253.4% 53574107.00 TOTAL
lock_stat.buffer->reader_lock_key.acquisitions
13357123.60 +261.4% 48273636.60 TOTAL
lock_stat.&rq->lock.acquisitions
3250919.71 +225.8% 10592024.15 TOTAL
lock_stat.buffer->reader_lock_key.holdtime-total
15000.00 +219.6% 47943.30 TOTAL
lock_stat.slock-AF_INET-RPC.acquisitions
1821865.51 +799.3% 16383840.49 TOTAL
lock_stat.&pipe->wait.holdtime-total
6306.35 +216.5% 19962.11 TOTAL
lock_stat.&(&list->lock)->rlock.holdtime-total
104745.20 +220.5% 335684.60 TOTAL
lock_stat.trace_types_lock.acquisitions
104745.20 +220.5% 335684.00 TOTAL
lock_stat.trace_event_sem-R.acquisitions
104745.20 +220.5% 335683.80 TOTAL
lock_stat.all_cpu_access_lock.acquisitions
1411.65 +126.4% 3196.43 TOTAL
lock_stat.&vfsmount_lock-R.holdtime-max
1669574.80 -60.4% 661408.70 TOTAL
lock_stat.&(&base->lock)->rlock.acquisitions
116585.00 +197.0% 346231.10 TOTAL
lock_stat.&iter->mutex.acquisitions
39046.00 +188.4% 112601.00 TOTAL
lock_stat.trace_types_lock.holdtime-total
2356238.10 -20.5% 1872483.90 TOTAL
lock_stat.rcu_read_lock_bh-R.acquisitions
712340.80 -58.5% 295574.20 TOTAL
lock_stat.&wq->wait.acquisitions
106482.08 +160.0% 276823.43 TOTAL
lock_stat.slock-AF_INET-RPC/1.holdtime-total
161126.72 +183.9% 457435.26 TOTAL
lock_stat._xmit_ETHER#2.holdtime-total
134393.55 +183.8% 381435.33 TOTAL
lock_stat.sk_lock-AF_INET-RPC.holdtime-total
23948.20 +127.0% 54351.40 TOTAL
lock_stat.&sb->s_type->i_lock_key#1.acquisitions
509327.41 -54.9% 229766.53 TOTAL
lock_stat.&sig->cred_guard_mutex.holdtime-max
27084.80 +118.9% 59298.00 TOTAL
lock_stat.sb_writers#4-R.acquisitions
6221.93 -51.1% 3045.54 TOTAL
lock_stat.sk_lock-AF_INET.holdtime-max
4656.87 +181.9% 13126.12 TOTAL
lock_stat.&pipe->mutex/1.holdtime-max
10765.33 +123.3% 24037.30 TOTAL
lock_stat.(&task->u.tk_work)#2.holdtime-total
43141327.41 -13.1% 37507805.71 TOTAL
lock_stat.rcu_read_lock_bh-R.holdtime-total
4100.50 -51.8% 1975.29 TOTAL
lock_stat.rcu_read_lock-R.holdtime-max
739556.16 -17.4% 610769.01 TOTAL
lock_stat.&mm->mmap_sem-W.holdtime-max
11290.17 +117.2% 24520.74 TOTAL
lock_stat.nfsiod-R.holdtime-total
42400.54 -47.1% 22416.67 TOTAL
lock_stat.&(&info->si_lock)->rlock.holdtime-total
48336.50 +80.9% 87451.90 TOTAL
lock_stat.&(&nf->lru_lock)->rlock.contentions.ip_defrag
1168351.69 +10.8% 1294640.42 TOTAL
lock_stat.&pipe->mutex/1.waittime-total
102707.56 -48.3% 53101.39 TOTAL
lock_stat.&sb->s_type->i_mutex_key#2.waittime-total
91318694.78 +26.6% 115632734.76 TOTAL
lock_stat.&rq->lock.holdtime-total
50189004.65 +18.5% 59471866.33 TOTAL
lock_stat.&p->pi_lock.holdtime-total
133163.30 -44.5% 73942.90 TOTAL
lock_stat.&anon_vma->rwsem.acq-bounces
66818.00 -45.6% 36333.20 TOTAL
lock_stat.&p->pi_lock.acq-bounces
361874.50 +82.2% 659237.20 TOTAL
lock_stat.&(&zone->lock)->rlock.contentions.__free_pages_ok
3957.60 -33.3% 2638.84 TOTAL
lock_stat.tasklist_lock-W.waittime-total
244764.50 +72.1% 421131.00 TOTAL
lock_stat.&(&zone->lock)->rlock.contentions.get_page_from_freelist
6540563.51 -53.2% 3058553.07 TOTAL
lock_stat.&(&base->lock)->rlock.holdtime-total
214843.60 +74.5% 374915.00 TOTAL
lock_stat.&(&nf->lru_lock)->rlock.con-bounces
214846.50 +74.5% 374916.00 TOTAL
lock_stat.&(&nf->lru_lock)->rlock.contentions
641418.67 +75.6% 1126321.06 TOTAL
lock_stat.&(&nf->lru_lock)->rlock.waittime-total
192855.90 +76.0% 339378.40 TOTAL
lock_stat.&(&nf->lru_lock)->rlock.contentions.inet_frag_find
2322889.38 +111.6% 4915093.78 TOTAL
lock_stat.sb_writers#7-R.holdtime-total
63744.63 -42.3% 36809.55 TOTAL
lock_stat.&(&n->list_lock)->rlock.holdtime-total
188500.60 +71.4% 323001.70 TOTAL
lock_stat.&(&nf->lru_lock)->rlock.contentions.inet_frag_kill
14078.22 +71.4% 24129.54 TOTAL
lock_stat.&(&queue->lock)->rlock.holdtime-total
350330.40 +68.4% 589915.00 TOTAL
lock_stat.&(&zone->lock)->rlock.contentions
350276.20 +68.2% 589108.30 TOTAL
lock_stat.&(&zone->lock)->rlock.con-bounces
2435.44 +68.8% 4109.92 TOTAL
lock_stat.&(&hb->chain_lock)->rlock.waittime-total
11039.58 +38.2% 15255.39 TOTAL
lock_stat.(&(&wb->dwork)->work).holdtime-total
45302081.74 -8.2% 41581942.88 TOTAL
lock_stat.&wq->wait.holdtime-total
182856.27 +59.9% 292446.19 TOTAL
lock_stat.zone->wait_table+i.holdtime-total
2182608.45 +60.5% 3502508.79 TOTAL
lock_stat.&(&zone->lock)->rlock.waittime-total
21148.30 +57.5% 33298.65 TOTAL
lock_stat.&(&xprt->transport_lock)->rlock.holdtime-total
11841.06 +49.7% 17725.30 TOTAL
lock_stat.&sb->s_type->i_lock_key#1.holdtime-total
20350.70 +48.1% 30148.50 TOTAL
lock_stat.&pipe->mutex/1.acq-bounces
19903.70 +48.3% 29522.80 TOTAL
lock_stat.&pipe->wait.acq-bounces
12025.44 +88.2% 22631.20 TOTAL
lock_stat.&type->i_mutex_dir_key#2.holdtime-total
12025.44 +88.2% 22631.20 TOTAL
lock_stat.&type->i_mutex_dir_key#2.holdtime-min
12025.44 +88.2% 22631.20 TOTAL
lock_stat.&type->i_mutex_dir_key#2.holdtime-max
459953.53 +9.5% 503614.41 TOTAL
lock_stat.&mapping->i_mmap_mutex.waittime-max
36732.44 +38.4% 50824.74 TOTAL
lock_stat.sb_writers#4-R.holdtime-total
439874.42 -5.3% 416534.11 TOTAL
lock_stat.clock-AF_INET-R.holdtime-total
277796.51 +44.6% 401667.68 TOTAL
lock_stat.&p->pi_lock.waittime-total
2313283.70 -26.8% 1693267.70 TOTAL
lock_stat.&(&list->lock)->rlock#3.acquisitions
74032.10 -28.6% 52864.10 TOTAL
lock_stat.&rq->lock/1.acquisitions
931324.30 +34.7% 1254788.40 TOTAL
lock_stat.&(&zone->lock)->rlock.acq-bounces
14328.30 +25.0% 17906.30 TOTAL
lock_stat.hrtimer_bases.lock#7.holdtime-total
37493.70 -26.3% 27624.80 TOTAL
lock_stat.&rq->lock/1.acq-bounces
15085692.76 +36.5% 20597805.11 TOTAL
lock_stat.rcu_read_lock-R.holdtime-total
31239.40 +36.1% 42514.40 TOTAL
lock_stat.zone->wait_table+i.contentions.__wake_up
159447.37 +32.3% 211019.95 TOTAL
lock_stat.rcu_node_0.waittime-total
100236.97 -33.9% 66247.10 TOTAL
lock_stat.&pipe->mutex/1.waittime-max
1166309.60 -23.3% 894892.70 TOTAL
lock_stat.slock-AF_INET/1.acquisitions
4053.60 +29.4% 5245.03 TOTAL
lock_stat.sk_lock-AF_INET6.holdtime-total
5036.72 -23.9% 3834.90 TOTAL
lock_stat.&(&tsk->delays->lock)->rlock.holdtime-total
10019.40 +45.8% 14613.28 TOTAL
lock_stat.slock-AF_INET-RPC.holdtime-total
1417553.50 +23.9% 1755833.90 TOTAL
lock_stat.&(&zone->lock)->rlock.acquisitions
6946024.55 -19.3% 5602189.45 TOTAL
lock_stat.&(&list->lock)->rlock#3.holdtime-total
5172.44 -19.2% 4177.08 TOTAL
lock_stat.&q->lock.holdtime-total
18369.95 +23.1% 22618.18 TOTAL
lock_stat.(&(work)->timer).holdtime-total
834643.80 +22.7% 1024368.10 TOTAL
lock_stat.&(&nf->lru_lock)->rlock.acq-bounces
25251.78 +22.4% 30899.04 TOTAL
lock_stat.&(&pool->lock)->rlock.holdtime-total
3931866.39 +23.0% 4834661.56 TOTAL
lock_stat.((&timer)).holdtime-total
400630.70 -19.6% 322076.80 TOTAL
lock_stat.&(&dentry->d_lock)->rlock.contentions.path_get
729459.97 -23.7% 556449.31 TOTAL
lock_stat.&mapping->i_mmap_mutex.waittime-total
1205124.95 +17.2% 1412040.97 TOTAL
lock_stat.&rsp->gp_wq.holdtime-total
778969.60 -18.0% 639110.20 TOTAL
lock_stat.&(&dentry->d_lock)->rlock.contentions.dput
203249.44 -17.2% 168359.45 TOTAL
lock_stat.&(&dentry->d_lock)->rlock.waittime-total
787586.90 -16.9% 654513.10 TOTAL
lock_stat.&(&dentry->d_lock)->rlock.con-bounces
787737.60 -16.9% 654670.40 TOTAL
lock_stat.&(&dentry->d_lock)->rlock.contentions
982029.40 +20.0% 1178821.80 TOTAL
lock_stat.&f->lock-R.acq-bounces
11377298.70 -15.7% 9591520.20 TOTAL
lock_stat.rcu_read_lock-R.acquisitions
21232.88 +19.0% 25274.64 TOTAL
lock_stat.&rnp->nocb_gp_wq[1].holdtime-total
22050.24 +18.9% 26217.07 TOTAL
lock_stat.&rnp->nocb_gp_wq[0].holdtime-total
175592.50 +16.0% 203669.97 TOTAL
lock_stat.clockevents_lock.waittime-total
182770.97 +18.0% 215662.07 TOTAL
lock_stat.rcu_sched_state.onoff_mutex.holdtime-total
169134.80 -14.9% 143897.00 TOTAL
lock_stat.zone->wait_table+i.acquisitions
11269453.16 +12.6% 12691439.20 TOTAL
lock_stat.&(&zone->lock)->rlock.holdtime-total
13525.95 +17.0% 15830.27 TOTAL
lock_stat.(&(work)->work).holdtime-total
8975.59 +16.8% 10485.40 TOTAL
lock_stat.proc_subdir_lock.holdtime-total
99222.00 +16.0% 115055.75 TOTAL
lock_stat.sb_writers#2-R.holdtime-total
124847.90 -13.7% 107753.10 TOTAL
lock_stat.zone->wait_table+i.acq-bounces
1518193.46 +10.4% 1676339.59 TOTAL
lock_stat.rcu_node_0.holdtime-total
95007.69 +15.2% 109413.23 TOTAL
lock_stat.&sig->wait_chldexit.holdtime-total
256302.50 -13.1% 222725.30 TOTAL
lock_stat.&mm->mmap_sem-W.acq-bounces
6934.41 +14.6% 7944.90 TOTAL
lock_stat.sb_writers#3-R.holdtime-total
15463.56 +13.7% 17577.66 TOTAL
lock_stat.entries_lock-R.holdtime-total
263989.70 -13.1% 229316.00 TOTAL
lock_stat.&(&mm->page_table_lock)->rlock.acq-bounces
13700.25 +87.2% 25651.38 TOTAL
lock_stat.hrtimer_bases.lock#4.holdtime-total
10082.81 +14.1% 11507.23 TOTAL
lock_stat.bdev_lock.holdtime-total
258345.90 -12.6% 225740.10 TOTAL
lock_stat.&(&newf->file_lock)->rlock.acq-bounces
28512.94 +12.8% 32154.48 TOTAL
lock_stat.events-R.holdtime-total
47912.04 -12.3% 41996.01 TOTAL
lock_stat.rename_lock.waittime-total
224803.20 -12.6% 196381.70 TOTAL
lock_stat.&(&fs->lock)->rlock.acq-bounces
103370.20 -12.7% 90277.20 TOTAL
lock_stat.rename_lock.contentions.SyS_getcwd
13222.56 +18.3% 15637.30 TOTAL
lock_stat.hrtimer_bases.lock#5.holdtime-total
702867.20 -12.0% 618778.40 TOTAL
lock_stat.&mapping->i_mmap_mutex.contentions.vma_link
238640.40 -11.8% 210514.70 TOTAL
lock_stat.&(&sighand->siglock)->rlock.acq-bounces
144491.30 +12.8% 163038.30 TOTAL
lock_stat.clockevents_lock.con-bounces
289138.00 +12.8% 326170.80 TOTAL
lock_stat.clockevents_lock.contentions.clockevents_notify
144569.00 +12.8% 163085.40 TOTAL
lock_stat.clockevents_lock.contentions
585553.00 -11.5% 518161.60 TOTAL
lock_stat.&mapping->i_mmap_mutex.con-bounces
585553.00 -11.5% 518161.60 TOTAL
lock_stat.&mapping->i_mmap_mutex.contentions
174036.80 +11.7% 194446.50 TOTAL
lock_stat.((&timer)).acquisitions
198920.50 +12.5% 223756.40 TOTAL
lock_stat.rcu_node_0.acquisitions
159105.97 +12.2% 178524.65 TOTAL
lock_stat.&(&p->alloc_lock)->rlock.holdtime-total
12962.55 +11.3% 14431.29 TOTAL
lock_stat.&child->perf_event_mutex.holdtime-total
467032.10 -10.9% 416276.60 TOTAL
lock_stat.&mapping->i_mmap_mutex.contentions.unlink_file_vma
2639031.80 -10.6% 2360489.00 TOTAL
lock_stat.sk_lock-AF_INET.acquisitions
23964.02 +9.5% 26236.37 TOTAL
lock_stat.&fsnotify_mark_srcu-R.holdtime-total
706694.52 -9.6% 638876.20 TOTAL
lock_stat.jiffies_lock.holdtime-total
14111.25 +36.4% 19245.96 TOTAL
lock_stat.hrtimer_bases.lock.holdtime-total
47545.73 +11.3% 52915.50 TOTAL
lock_stat.pgd_lock.holdtime-total
319799.80 +10.4% 353118.34 TOTAL
lock_stat.tasklist_lock-W.holdtime-total
13131948.50 -10.6% 11739786.10 TOTAL
lock_stat.&(&dentry->d_lock)->rlock.acq-bounces
5748518.20 -10.3% 5156548.00 TOTAL
lock_stat.&mapping->i_mmap_mutex.acq-bounces
10293967.02 -6.7% 9599300.67 TOTAL
lock_stat.&mm->mmap_sem-R.holdtime-total
2486744.60 +590.1% 17161466.60 TOTAL perf-stat.context-switches
91101.74 +19.8% 109116.17 TOTAL vmstat.system.cs
15.96 +17.8% 18.79 TOTAL iostat.cpu.user
19.81 +16.5% 23.07 TOTAL vmstat.cpu.us
1818.70 +29.6% 2356.40 TOTAL
time.involuntary_context_switches
313376.90 -20.3% 249842.70 TOTAL perf-stat.cpu-migrations
17606.67 -17.4% 14540.72 TOTAL vmstat.system.in
669432.20 -12.7% 584367.90 TOTAL time.voluntary_context_switches
306.12 +0.1% 306.52 TOTAL iostat.cpu.system
72.62 +7.1% 77.79 TOTAL time.elapsed_time
400.28 +0.0% 400.32 TOTAL vmstat.cpu.sy
188.10 -6.8% 175.40 TOTAL
time.percent_of_cpu_this_job_got
1.35 -7.9% 1.24 TOTAL iostat.sda.rkB/s
0.35 -7.9% 0.32 TOTAL vmstat.io.bi
178.19 -4.7% 169.90 TOTAL vmstat.cpu.wa
178.08 -4.6% 169.91 TOTAL iostat.cpu.iowait
311.74 +2.6% 319.92 TOTAL boottime.dhcp
269.22 +2.7% 276.37 TOTAL boottime.boot
568.45 +3.1% 586.18 TOTAL boottime.idle
2699.20 +0.6% 2714.10 TOTAL time.minor_page_faults
82.14 +0.3% 82.40 TOTAL iostat.cpu.idle
82.12 +0.3% 82.39 TOTAL vmstat.cpu.id
Details attached.
Thanks,
Fengguang