6:44 a.m.
On Wed, Aug 21, 2019 at 04:48:10PM -0300, Jason Gunthorpe wrote:
On Wed, Aug 21, 2019 at 11:57:03AM -0700, Ira Weiny wrote:
> > Oh, I didn't think we were talking about that. Hanging the close of
> > the datafile fd contingent on some other FD's closure is a recipe for
> > deadlock..
>
> The discussion between Jan and Dave was concerning what happens when a user
> calls
>
> fd = open()
> fnctl(...getlease...)
> addr = mmap(fd...)
> ib_reg_mr() <pin>
> munmap(addr...)
> close(fd)
I don't see how blocking close(fd) could work.
Well Dave was saying this _could_ work. FWIW I'm not 100% sure it will but I
can't prove it won't.. Maybe we are all just touching a different part of this
elephant[1] but the above scenario or one without munmap is very reasonably
something a user would do. So we can either allow the close to complete (my
current patches) or try to make it block like Dave is suggesting.
I don't disagree with Dave with the semantics being nice and clean for the
filesystem. But the fact that RDMA, and potentially others, can "pass the
pins" to other processes is something I spent a lot of time trying to work out.
Write it like this:
fd = open()
uverbs = open(/dev/uverbs)
fnctl(...getlease...)
addr = mmap(fd...)
ib_reg_mr() <pin>
munmap(addr...)
<sigkill>
The order FD's are closed during sigkill is not deterministic, so when
all the fputs happen during a kill'd exit we could end up blocking in
close(fd) as close(uverbs) will come after in the close
list. close(uverbs) is the thing that does the dereg_mr and releases
the pin.
Of course, that is a different scenario which needs to be fixed in my patch
set. Now that my servers are back up I can hopefully make progress. (Power
was down for them yesterday).
We don't need complexity with dup to create problems.
No but that complexity _will_ come unless we "zombie" layout leases.
Ira
[1] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant
Jason