On 12/2/2021 7:12 PM, niketkandya(a)google.com wrote:
From my reading of the code and docs this is what I understand:
1. Huge pages are needed by SPDK/DPDK so that user pages are always present and pinned.
2. In systems with IOMMU and VFIO support, there should be no need for huge pages.
spdk_mem_register on an externally allocated buffer should do the same pinning.
3. This is also what happens to DPDK's internal heaps in the memory callbacks
(memory_iter_cb).
When I tried to use --no-huge with SPDK it failed with error in alignment/size
restrictions for DPDK's internal heap. I was able to hack DPDK code to return 2MB
aligned addresses for its internal heap and this worked.
So, to confirm my understanding here: for SPDK to work without hugepages, just a
guarantee that (2MB)aligned addresses are returned by DPDK's internal heap should be
enough?
I can't think of any other requirements. We store virtual to physical
mappings in a radix tree with 2MB granularity, so as long as you can
assign a physical/io address to every virtual address in 2MB units, I
think it should be fine. I'd be curious to see how you changed DPDK
because I'd love to be able to officially support SPDK with no huge pages.
I *think* the step of programming the IOMMU to associate the virtual
address to the io address pins the pages. I'm about 99% sure that's the
case. If for some reason it's not the case, you'd also need to mlock all
the memory.
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>