Re: [PATCH RFC 1/1] Add support for ZONE_DEVICE IO memory with struct pages.
On Mon, Mar 14, 2016 at 12:14:37PM -0600, Stephen Bates wrote:
3. Coherency Issues. When IOMEM is written from both the CPU and a
PCIe
peer there is potential for coherency issues and for writes to occur out
of order. This is something that users of this feature need to be
cognizant of and may necessitate the use of CONFIG_EXPERT. Though really,
this isn't much different than the existing situation with RDMA: if
userspace sets up an MR for remote use, they need to be careful about
using that memory region themselves.
There's more to the coherency problem than this. As I understand it, on
x86, memory in a PCI BAR does not participate in the coherency protocol.
So you can get a situation where CPU A stores 4 bytes to offset 8 in a
cacheline, then CPU B stores 4 bytes to offset 16 in the same cacheline,
and CPU A's write mysteriously goes missing.
I may have misunderstood the exact details when this was explained to me a
few years ago, but the details were horrible enough to run away screaming.
Pretending PCI BARs are real memory? Just Say No.