Hi Furkan,
thank you for your question! Let me try to clarify a couple of points before
addressing your question directly.
OPAE is targeted at supporting a wide range of programmable accelerator
platforms, and its fundamental concepts are kept intentionally broad to account
for differences in platform implementation. As such, it may include features
that are not available on all supported platforms or may differ in realization.
I believe you're referring to the FPGA platform that's part of the current Intel
Hardware Accelerator Research Program (that was the target platform of the
presentation at FPL2017). That platform does not include an IOMMU, so any
addresses returned by fpgaGetIOAddress() will indeed be physical addresses.
(Trivia: fpgaGetIOAddress() was originally called fpgaGetIOVA(), but we changed
it precisely because of the reasons you're outlining.) It appears that not all
variable names carrying this IO address have been renamed to account for this,
though; I'm sorry for any confusion this may have caused.
If you look at other potential platforms that OPAE could support (or even at
actual platforms OPAE supports right now, such as
http://altera.com/pac), you
will find examples of configurations that do have an IOMMU (the PAC, for
example, is a PCIe card, so it lives behind any IOMMU of the platform it's
plugged into).
An IOMMU does not necessarily provide the same virtual-to-phyiscal translations
that are created for software applications by the MMU -- IO devices in
virtualized systems may have their own IO virtual address space (hence the
name "IO Virtual Address"). In many virtualized systems, the IOMMU will only
provide a single mapping for each virtual machine. To allow AFUs to use true
"process virtual addresses" instead of "IO addresses" (which may be
virtual or
physical depending on IOMMU availability) you'll need additional logic in the
FPGA to do the translation in sync with the OS page tables.
As it happens, the "intel-fpga-bbb" repository
https://github.com/OPAE/intel-fpga-bbb
includes an IP block (we call then "basic building blocks" or BBBs) for this
named MPF -- you may recall a section of the FPL 2017 workshop that explained
it. MPF includes a feature for virtual-to-physical address translations (VTP)
for AFUs. If you instantiate this block in your AFU design, your AFU can use the
virtual addresses of its owning process (you can share pointers) -- limited to
explicitly shared buffers, of course --, and there is no need to use
fpgaGetIOAddress() at all.
On Fri, Feb 16, 2018 at 04:12:22PM +0100, Furkan Turan wrote:
Hello Dears,
I would like to consult on you about the memory view of AFUs.
In OPAE, fpgaGetIOAddress is declared to get IO address, which is returned
as updating the ioaddr parameter passed to it. A variable for this parameter
in one example code is name as buf_pa, as it will return the IO address for
the buffer and that will be the physical address "pa". However, in some
other examples, e.g. hello_fpga.c, the same variable is names as iova which
basically stands for IO virtual address.
As far as I remember the presentation in FPL2017, it was mentioned that
there is no IOMMU right now; therefore, iova and physical address should be
same things. However, the opae.github.io explains the functions referring to
IOMMU, so maybe Intel has already introduced it. In that case, AFUs should
not work with physical addresses, but indeed have virtual addresses which
will be translated to physical by the IOMMU.
I would like to consult on you about this:
* Is IOMMU available or not?
It depends on the target platform. The integrated Xeon+FPGA system deployed as
part of HARP does not; a deployment of the PAC card may very well do (as may
other platforms).
* If there is IOMMU, then how the page tables are maintained: by OS,
or by
OPAE drivers?
The IOMMU translations are managed by the operating system. The OPAE driver uses
the Linux kernel DMA subsystem, which is aware of (and manages) these mappings.
* Can this provide the same virtual addresses on both SW and HW side
in the
future, making the fpgaGetIOAddress function obsolete?
Yes, using MPF (see above). Other (future?) platforms may also provide other
solutions to this -- it has been done before for other accelerator technologies.
* May the IOMMU allow getting rid of the contagious memory space
allocation
requirement for the buffers in the future?
That's an interesting question. I believe you could program an IOMMU to mirror
the page mappings of an MMU and thus allow page-level translations. I don't know
how that would be implemented using the Linux kernel, though (I believe it
currently only creates mappings for explicitly requested DMA buffers, which have
to be contiguous (duh)). You may also run into issues where the IOMMU is already
used for translating between guest physical and host physical addresses for
virtual machines -- you'll need something like a second-level address
translation mechanism for that (similar to what MMUs do with EPT/RVI).
I hope this answers your questions.
Thank you
- Enno