On 14/03/16 03:57 PM, Jason Gunthorpe wrote:
Someone should probably explain in more detail what this is even
for, DAX on PCI-E bar memory seems goofy in the general case. I was
under the impression the main use case involved the CPU never touching
these memories and just using them to route-through to another IO
device (eg network). So all these discussions about CPU coherency seem
a bit strange.
Yes, the primary purpose is to enable P2P transactions that don't
involve the CPU at all. To enable this, we do mmap the BAR region into
user space which is then technically able to read/write to it using the
CPU. However, you're right, it is silly to write to the mmap'd PCI BAR
for anything but debug/testing purposes -- this type of access also has
horrible performance. Really, the mmaping is just a convenient way to
pass around the addresses with existing interfaces that expect system
RAM (RDMA, O_DIRECT).
Putting DAX on the PCI-E bar is a actually more of a curiosity at the
moment than anything. The current plan for NVMe with CMB would not
involve DAX. CMB buffers would be allocated perhaps by mapping the nvmeX
char device which could then be used with O_DIRECT access on a file on
the NVME device and also be passed to RDMA devices. In this way data
could flow from the NVMe device to an RDMA network without using system
memory to buffer it.