Re: [RFC 0/2] New MAP_PMEM_AWARE mmap flag

On 02/21/2016 09:03 AM, Boaz Harrosh wrote: > > Hi all > > Recent DAX code fixed the cl_flushing ie durability of mmap access > of direct persistent-memory from applications. It uses the radix-tree > per inode to track the indexes of a file that where page-faulted for > write. Then at m/fsync time it would cl_flush these pages and clean > the radix-tree, for the next round. > > Sigh, that is life, for legacy applications this is the price we must > pay. But for NV aware applications like nvml library, we pay extra extra > price, even if we do not actually call m/fsync eventually. For these > applications these extra resources and especially the extra radix locking > per page-fault, costs a lot, like x3 a lot. > > What we propose here is a way for those applications to enjoy the > boost and still not sacrifice any correctness of legacy applications. > Any concurrent access from legacy apps vs nv-aware apps even to the same > file / same page, will work correctly. > > We do that by defining a new MMAP flag that is set by the nv-aware > app. this flag is carried by the VMA. In the dax code we bypass any > radix handling of the page if this flag is set. Those pages accessed > *without* > this flag will be added to the radix-tree, those with will not. > At m/fsync time if the radix tree is then empty nothing will happen. > I'm a little late to the party, but let me offer a variant that might be considerably safer: Add a flag MAP_DAX_WRITETHROUGH (name could be debated -- MAP_DAX_FASTFLUSH might be more architecture-neutral, but I'm only familiar with the x86 semantics). MAP_DAX_WRITETHROUGH does whatever is needed to ensure that writing through the mapping and then calling fsync is both safe and fast. On x86, it would (surprise, surprise!) map the pages writethrough and skip adding them to the radix tree. fsync makes sure to do sfence before pcommit. This is totally safe. You *can't* abuse this to cause fsync to leave non-persistent dirty cached data anywhere. It makes sufficiently DAX-aware applications very fast. Reads are unaffected, and non-temporal writes should be the same speed as they are under any other circumstances. It makes applications that set it blindly very slow. Applications that use standard writes (i.e. plain stores that are neither fast string operations nor explicit non-temporal writes) will suffer. But they'll still work correctly. Applications that want a WB mapping with manually-managed persistence can still do it, but fsync will be slow. Adding an fmetadatasync() for their benefit might be a decent idea, but it would just be icing on the cake. Unlike with MAP_DAX_AWARE, there's no issue with malicious users who map the thing with the wrong flag, write, call fsync, and snicker because now the other applications might read data and be surprised that the data they just read isn't persistent even if they subsequently call fsync. There would be details to be hashed out in case a page is mapped normally and with MAP_DAX_WRITETHROUGH in separate mappings.