On Fri, Nov 25, 2016 at 09:40:10PM +0100, Christian König wrote:
We call this "userptr" and it's just a combination of
command submission and making sure the returned list of pages stays valid
using a MMU notifier.
Doesn't that still pin the page?
The "big" problem with this approach is that it is horrible
slow. I mean
seriously horrible slow so that we actually can't use it for some of the
purposes we wanted to use it.
>The code moving the page will move it and the next GPU command that
>needs it will refault it in the usual way, just like the CPU would.
And here comes the problem. CPU do this on a page by page basis, so they
fault only what needed and everything else gets filled in on demand. This
results that faulting a page is relatively light weight operation.
But for GPU command submission we don't know which pages might be accessed
beforehand, so what we do is walking all possible pages and make sure all of
them are present.
Little confused why this is slow? So you fault the entire user MM into
your page tables at start of day and keep track of it with mmu
>This might be much more efficient since it optimizes for the
>case of unchanging translation tables.
Yeah, completely agree. It works perfectly fine as long as you don't have
two drivers trying to mess with the same page.
Well, the idea would be to not have the GPU block the other driver
beyond hinting that the page shouldn't be swapped out.
>This assumes the commands are fairly short lived of course, the
>expectation of the mmu notifiers is that a flush is reasonably prompt
Correct, this is another problem. GFX command submissions usually don't take
longer than a few milliseconds, but compute command submission can easily
take multiple hours.
So, that won't work - you have the same issue as RDMA with work loads
If you can't somehow fence the hardware then pinning is the only
solution. Felix has the right kind of suggestion for what is needed -
globally stop the GPU, fence the DMA, fix the page tables, and start
it up again. :\
I can easily imagine what would happen when kswapd is blocked by a
command submission for an hour or so while the system is under memory
Right. The advantage of pinning is it tells the other stuff not to
touch the page and doesn't block it, MMU notifiers have to be able to
I'm thinking on this problem for about a year now and going in
quite a while. So if you have ideas on this even if they sound totally
crazy, feel free to come up.
Well, it isn't a software problem. From what I've seen in this thread
the GPU application requires coherent page table mirroring, so the
only full & complete solution is going to be to actually implement
that somehow in GPU hardware.
Everything else is going to be deeply flawed somehow. Linux just
doesn't have the support for this kind of stuff - and I'm honestly not
sure something better is even possible considering the hardware
This doesn't have to be faulting, but really anything that lets you
pause the GPU DMA and reload the page tables.
You might look at trying to use the IOMMU and/or PCI ATS in very new
hardware. IIRC the physical IOMMU hardware can do the fault and fence
and block stuff, but I'm not sure about software support for using the
IOMMU to create coherent user page table mirrors - that is something
Linux doesn't do today. But there is demand for this kind of capability..