Hi, One vm related test case run by LKP on a Haswell EP with 128GiB memory showed that compaction code would cause performance drop about 30%. To illustrate the problem, I've simplified the test with a program called usemem(see attached). The test goes like this: 1 Boot up the server; 2 modprobe scsi_debug(a module that could use memory as SCSI device), dev_size set to 4/5 free memory, i.e. about 100GiB. Use it as swap. 3 run the usemem test, which use mmap to map a MAP_PRIVATE | MAP_ANON region with the size set to 3/4 of (remaining_free_memory + swap), and then write to that region sequentially to trigger page fault and swap out. The above test runs with two configs regarding the below two sysfs files: /sys/kernel/mm/transparent_hugepage/enabled /sys/kernel/mm/transparent_hugepage/defrag 1 transparent hugepage and defrag are both set to always, let's call it always-always case; 2 transparent hugepage is set to always while defrag is set to never, let's call it always-never case. The output from the always-always case is: Setting up swapspace version 1, size = 104627196 KiB no label, UUID=aafa53ae-af9e-46c9-acb9-8b3d4f57f610 cmdline: /lkp/aaron/src/bin/usemem 99994672128 99994672128 transferred in 95 seconds, throughput: 1003 MB/s And the output from the always-never case is: etting up swapspace version 1, size = 104629244 KiB no label, UUID=60563c82-d1c6-4d86-b9fa-b52f208097e9 cmdline: /lkp/aaron/src/bin/usemem 99995965440 99995965440 transferred in 67 seconds, throughput: 1423 MB/s

The vmstat and perf-profile are also attached, please let me know if you need any more information, thanks.

On 11/19/2015 09:29 PM, Vlastimil Babka wrote: > +CC Andrea, David, Joonsoo > > On 11/19/2015 10:29 AM, Aaron Lu wrote: >> The vmstat and perf-profile are also attached, please let me know if you >> need any more information, thanks. > > Output from vmstat (the tool) isn't much useful here, a periodic "cat > /proc/vmstat" would be much better. No problem. > The perf profiles are somewhat weirdly sorted by children cost (?), but > I noticed a very high cost (46%) in pageblock_pfn_to_page(). This could > be due to a very large but sparsely populated zone. Could you provide > /proc/zoneinfo? Is a one time /proc/zoneinfo enough or also a periodic one?

On 11/20/2015 10:33 AM, Aaron Lu wrote: >On 11/20/2015 04:55 PM, Aaron Lu wrote: >>On 11/19/2015 09:29 PM, Vlastimil Babka wrote: >>>+CC Andrea, David, Joonsoo >>> >>>On 11/19/2015 10:29 AM, Aaron Lu wrote: >>>>The vmstat and perf-profile are also attached, please let me know if you >>>>need any more information, thanks. >>> >>>Output from vmstat (the tool) isn't much useful here, a periodic "cat >>>/proc/vmstat" would be much better. >> >>No problem. >> >>>The perf profiles are somewhat weirdly sorted by children cost (?), but >>>I noticed a very high cost (46%) in pageblock_pfn_to_page(). This could >>>be due to a very large but sparsely populated zone. Could you provide >>>/proc/zoneinfo? >> >>Is a one time /proc/zoneinfo enough or also a periodic one? > >Please see attached, note that this is a new run so the perf profile is >a little different. > >Thanks, >Aaron Thanks. DMA32 is a bit sparse: Node 0, zone DMA32 pages free 62829 min 327 low 408 high 490 scanned 0 spanned 1044480 present 495951 managed 479559 Since the other zones are much larger, probably this is not the culprit. But tracepoints should tell us more. I have a theory that updating free scanner's cached pfn doesn't happen if it aborts due to need_resched() during isolate_freepages(), before hitting a valid pageblock, if the zone has a large hole in it. But zoneinfo doesn't tell us if the large difference between "spanned" and "present"/"managed" is due to a large hole, or many smaller holes... compact_migrate_scanned 1982396 compact_free_scanned 40576943 compact_isolated 2096602 compact_stall 9070 compact_fail 6025 compact_success 3045 So it's struggling to find free pages, no wonder about that. I'm

On 11/23/2015 04:16 PM, Joonsoo Kim wrote: > Numbers looks fine to me. I guess this performance degradation is > caused by COMPACT_CLUSTER_MAX change (from 32 to 256). THP allocation > is async so should be aborted quickly. But, after isolating 256 > migratable pages, it can't be aborted and will finish 256 pages > migration (at least, current implementation).

> Aaron, please test again with setting COMPACT_CLUSTER_MAX to 32 > (in swap.h)? This is what I found in include/linux/swap.h: #define SWAP_CLUSTER_MAX 32UL #define COMPACT_CLUSTER_MAX SWAP_CLUSTER_MAX Looks like it is already 32, or am I looking at the wrong place? BTW, I'm using v4.3 for all these tests, and I just checked v4.4-rc2, the above definition doesn't change.

> > And, please attach always-always's vmstat numbers, too. Sure, attached the vmstat tool output, taken every second.

On 11/23/2015 05:24 PM, Joonsoo Kim wrote: > 2015-11-23 17:33 GMT+09:00 Aaron Lu <aaron.lu(a)intel.com&gt;: >> On 11/23/2015 04:16 PM, Joonsoo Kim wrote: >>> >>> And, please attach always-always's vmstat numbers, too. >> >> Sure, attached the vmstat tool output, taken every second. > > Oops... I'd like to see '1 sec interval cat /proc/vmstat' for always-never. Here it is, the proc-vmstat for always-never.

BTW, I'm still learning how to do proper ftrace for this case and it may take a while.

On 11/24/2015 12:55 PM, Joonsoo Kim wrote: > On Tue, Nov 24, 2015 at 11:40:28AM +0800, Aaron Lu wrote: >> BTW, I'm still learning how to do proper ftrace for this case and it may >> take a while. > > You can do it simply with trace-cmd. > > sudo trace-cmd record -e compaction & > run test program > fg > Ctrl + c > > sudo trace-cmd report Thanks for the tip, I just recorded it like this: trace-cmd record -e compaction ./usemem xxx Due to the big size of trace.out(6MB after compress), I've uploaed it: https://drive.google.com/open?id=0B49uX3igf4K4UkJBOGt3cHhOU00 The pagetypeinfo, perf and proc-vmstat is also there.

On 11/24/2015 09:29 AM, Joonsoo Kim wrote: > On Tue, Nov 24, 2015 at 03:27:43PM +0800, Aaron Lu wrote: > > Thanks. > > Okay. Output proves the theory. pagetypeinfo shows that there are > too many unmovable pageblocks. isolate_freepages() should skip these > so it's not easy to meet proper pageblock until need_resched(). Hence, > updating cached pfn doesn't happen. (You can see unchanged free_pfn > with 'grep compaction_begin tracepoint-output') Hm to me it seems that the scanners meet a lot, so they restart at zone boundaries and that's fine. There's nothing to cache. > But, I don't think that updating cached pfn is enough to solve your problem. > More complex change would be needed, I guess. One factor is probably that THP only use async compaction and those don't result in deferred compaction, which should help here. It also means that pageblock_skip bits are not being reset except by kswapd... Oh and pageblock_pfn_to_page is done before checking the pageblock skip bits, so that's why it's prominent in the profiles. Although it was less prominent (9% vs 46% before) in the last data... was perf collected while tracing, thus generating extra noise?