We've been having a devil of a time figuring out what's been going on with what appears to be a memory leak in the Lustre page cache. Some specifics: we are running Lustre 2.1.2, on RHEL 6.3, with kernel 2.6.32-279.11.1.el6. The symptoms are we end up with a system with all of the memory consumed by pages on the inactive free list. Here's what the relevant lines from /proc/meminfo look like: MemTotal: 32914040 kB MemFree: 9382100 kB [...] Active(file): 33348 kB Inactive(file): 19620564 kB What happens is a couple of users seem to be triggering this behavior; exactly what causes it is still unclear, but one user triggers it just taring up and removing some files. Eventually the inactive list keeps growing and the OOM killer will come along, kill off various things in a futile attempt to get rid of memory, and then the system will need to be rebooted. I can't claim to be an expert on the Linux VM system, but I decided to try my best at digging into this some more. I figured out where the inactive page list was in memory and started looking at the pages using "crash". There were over 5 million entries on the inactive page list, so obviously I didn't look at them all; I picked a few at random and looked at them. What I found was: - They were all pages owned by Lustre (their backing store pointer was pointing to Lustre). - They weren't dirty; they were all marked up-to-date. From my reading of the Linux VM code, pages marked with private data (which is the case here) get their releasepage op called, which in this case is a pointer to ll_releasepage. I took at look at ll_releasepage, but it wasn't easy to determine exactly why the page wouldn't be released in this case. I'm glad to look at anything people could suggest; I was going to instrument ll_releasepage to see if a) it not releasing pages is the cause of this problem and b) if it is the problem, to see exactly why it isn't releasing the pages. I took a look at what (I think) was the contents of the pages are, but it was just binary data, so it wasn't anything helpful to me. I was thinking that maybe it was directory data (we have one user here who has directory files that have sizes in the hundreds of megabytes), but it wasn't obvious to me that it was directory data. --Ken _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss