2:31 p.m.
On Sat, May 09, 2015 at 10:45:10AM +0200, Ingo Molnar wrote:
If we 'think big', we can create something very exciting
IMHO, that
also gets rid of most of the complications with DIO, DAX, etc:
"Directly mapped pmem integrated into the page cache":
------------------------------------------------------
- The pmem filesystem is mapped directly in all cases, it has device
side struct page arrays, and its struct pages are directly in the
page cache, write-through cached. (See further below about how we
can do this.)
Note that this is radically different from the current approach
that tries to use DIO and DAX to provide specialized "direct
access" APIs.
With the 'directly mapped' approach we have numerous advantages:
- no double buffering to main RAM: the device pages represent
file content.
- no bdflush, no VM pressure, no writeback pressure, no
swapping: this is a very simple VM model where the device is
RAM and we don't have much dirty state. The primary kernel
cache is the dcache and the directly mapped page cache, which
is not a writeback cache in this case but essentially a
logical->physical index cache of filesystem indexing
metadata.
- every binary mmap()ed would be XIP mapped in essence
- every read() would be equivalent a DIO read, without the
complexity of DIO.
- every read() or write() done into a data mmap() area would
allow device-to-device zero copy DMA.
- main RAM caching would still be avilable and would work in
many cases by default: as most apps use file processing
buffers in anonymous memory into which they read() data.
I admire your big vision, but I think there are problems that it doesn't
solve.
1. The difference in lifetimes between filesystem blocks and page cache
pages that represent them. Existing filesystems have their own block
allocators which have their own notions of when blocks are available for
reallocation which may differ from when a page in the page cache can be
reused for caching another block.
Concrete example: A mapped page of a file is used as the source or target
of a direct I/O. That file is simultaneously truncated, which in our
current paths calls the filesystem to free the block, while leaving the
page cache page in place in order to be the source or destination of
the I/O. Once the I/O completes, the page's reference count drops to
zero and the page can be freed.
If we do not modify the filesystem, that page/block may end up referring
to a block in a different file, with the usual security & integrity
problems.
2. Some of the media which currently exist (not exactly supported
well by the current DAX framework either) have great read properties,
but abysmal write properties. For example, they may have only a small
number of write cycles, or they may take milliseconds to absorb a write.
These media might work well for mapping some read-mostly files directly,
but be poor choices for putting things like struct page in, which contains
cachelines which are frquently modified.
We can achieve this by statically allocating all page structs on the
device, in the following way:
- For every 128MB of pmem data we allocate 2MB of struct-page
descriptors, 64 bytes each, that describes that 128MB data range
in a 4K granular way. We never have to allocate page structs as
they are always there.
- Filesystems don't directly see the preallocated page arrays, they
still get a 'logical block space' presented that to them looks
like a continuous block device (which is 1.5% smaller than the
true size of the device): this allows arbitrary filesystems to be
put into such pmem devices, fsck will just work, etc.
I.e. no special pmem filesystem: the full range of existing block
device based Linux filesystems can be used.
I think the goal of "use any Linux filesystem" is laudable, but
impractical. Since we're modifying filesystems anyway, is there an
advantage to doing this in the block device instead of just allocating the
struct pages in a special file in the filesystem (like modern filesystems
do for various structures)?