10:26 p.m.
In order to provide the ability to debug enclaves independently of their
programming model, while supporting live attach (meaning the debugger
might not have seen enclave creation), a debugger needs to following
capabilities for a debugged process:
1) Identify enclaves
a) Given a page, identify it as an enclave page
b) Given an enclave page, find that enclave's base address
c) Given a particular enclave, find the TCSes
d) Given a particular enclave, get the attributes/miscselect
2) Read/write enclave memory
3) Find debugging symbols for enclaves
I think (1)(a) and (1)(b) can be done using /proc/[pid]/maps, but I'd
like confirmation. I think the following is true: If a page is in a
memory range that is listed as mapped from the sgx device, it is an
enclave page. From a memory range that is listed as mapped from the sgx
device, you can compute the enclave base address as (start-offset). (2)
is supported already by the driver using the ptrace interface.
(1)(c) is necessary to find the SSAs and (1)(d) is necessary to
determine SSAFRAMESIZE and make sense of the data in the SSAs. What
would be the best way to expose this information? Best I can think of
currently is a dedicated file in /proc/[pid] where the driver exposes
this information which was originally provided at enclave creation time.
I don't think (3) can universally be done without help from the user
program. Yes, it might be possible to learn something from ELF headers
but Linux does not generally have a requirement that user processes must
have valid ELF information in their memory mappings. Most current
userspace debug symbol loading is based on filenames. SGX does not
provide any record of where enclave memory came from. Perhaps the create
ioctl could include some user data that is later exposed again in the
same proc file mentioned above. Frameworks and toolchains could
establish a convention regarding this user data to find symbols.
--
Jethro Beekman | Fortanix
10:36 a.m.
On Tue, 2017-03-21 at 22:26 -0700, Jethro Beekman wrote:
In order to provide the ability to debug enclaves independently of
their
programming model, while supporting live attach (meaning the debugger
might not have seen enclave creation), a debugger needs to following
capabilities for a debugged process:
1) Identify enclaves
a) Given a page, identify it as an enclave page
b) Given an enclave page, find that enclave's base address
c) Given a particular enclave, find the TCSes
d) Given a particular enclave, get the attributes/miscselect
2) Read/write enclave memory
3) Find debugging symbols for enclaves
I think (1)(a) and (1)(b) can be done using /proc/[pid]/maps, but I'd
like confirmation. I think the following is true: If a page is in a
memory range that is listed as mapped from the sgx device, it is an
enclave page.
This will generally hold true for applications built with the Intel
SDK, but
there is nothing that prevents an application from holding onto memory mapped to
/dev/sgx without an associated enclave. Even with the Intel SDK there will be
(very small) windows where this isn't true, e.g. between mmap and ioctl to
create the enclave.
There might even be legitimate use cases where an application maps more memory
to /dev/sgx than it currently needs, e.g. an application that launches multiple
instances of an enclave on-demand might keep multiple unused mappings available.
From a memory range that is listed as mapped from the sgx
device, you can compute the enclave base address as (start-offset).
(1)(b) could be
especially problematic for applications that create multiple
enclaves, e.g. if several enclaves are contiguous in virtual memory.
(2) is supported already by the driver using the ptrace interface.
(1)(c) is necessary to find the SSAs and (1)(d) is necessary to
determine SSAFRAMESIZE and make sense of the data in the SSAs. What
would be the best way to expose this information? Best I can think of
currently is a dedicated file in /proc/[pid] where the driver exposes
this information which was originally provided at enclave creation time.
I like the idea of exposing a small amount of debug information in /proc/[pid],
even if some of the information could theoretically be derived from other
sources. This would also be a convenient location to dump statistics such as
EPC usage, a la /proc/[pid]/smaps.
I don't think (3) can universally be done without help from the user
program. Yes, it might be possible to learn something from ELF headers
but Linux does not generally have a requirement that user processes must
have valid ELF information in their memory mappings. Most current
userspace debug symbol loading is based on filenames. SGX does not
provide any record of where enclave memory came from. Perhaps the create
ioctl could include some user data that is later exposed again in the
same proc file mentioned above. Frameworks and toolchains could
establish a convention regarding this user data to find symbols.
Hmm, assuming we can't come up with a more clever method of providing additional
debug information, I would prefer adding a dedicated ioctl command over
expanding any of the existing commands. I don't think we want to pollute the
functional commands with debug information, especially since any information
would be completely optional (and unverifiable). This would also give the user
program some amount of flexibility over when the debug information is provided.
Because any debug information would be purely optional, the debug command struct
could also be designed in such a way that future versions could provide backward
compatibility, e.g. by defining reserved fields in the initial version.
--
Jethro Beekman | Fortanix
_______________________________________________
intel-sgx-kernel-dev mailing list
intel-sgx-kernel-dev(a)lists.01.org
https://lists.01.org/mailman/listinfo/intel-sgx-kernel-dev
11:35 a.m.
On 2017-03-22 10:36, Sean Christopherson wrote:
On Tue, 2017-03-21 at 22:26 -0700, Jethro Beekman wrote:
> In order to provide the ability to debug enclaves independently of their
> programming model, while supporting live attach (meaning the debugger
> might not have seen enclave creation), a debugger needs to following
> capabilities for a debugged process:
>
> 1) Identify enclaves
> a) Given a page, identify it as an enclave page
> b) Given an enclave page, find that enclave's base address
> c) Given a particular enclave, find the TCSes
> d) Given a particular enclave, get the attributes/miscselect
> 2) Read/write enclave memory
> 3) Find debugging symbols for enclaves
>
> I think (1)(a) and (1)(b) can be done using /proc/[pid]/maps, but I'd
> like confirmation. I think the following is true: If a page is in a
> memory range that is listed as mapped from the sgx device, it is an
> enclave page.
This will generally hold true for applications built with the Intel SDK, but
there is nothing that prevents an application from holding onto memory mapped to
/dev/sgx without an associated enclave. Even with the Intel SDK there will be
(very small) windows where this isn't true, e.g. between mmap and ioctl to
create the enclave.
There might even be legitimate use cases where an application maps more memory
to /dev/sgx than it currently needs, e.g. an application that launches multiple
instances of an enclave on-demand might keep multiple unused mappings available.
> From a memory range that is listed as mapped from the sgx
> device, you can compute the enclave base address as (start-offset).
(1)(b) could be especially problematic for applications that create multiple
enclaves, e.g. if several enclaves are contiguous in virtual memory.
Good points. The mmap interface, while clearly being the best interface
for creating memory mappings in userspace, is not a 100% fit for SGX.
It's not similar to other devices where there is a fixed range that a
process can map into as they see fit. Normally if you have one or more
fds to a device and map the same offset multiple times at different
addresses, you're viewing the same physical memory multiple times. This
is not the case with SGX, and it also wouldn't make sense for SGX.
It would be good to think about what the best way is to deal with this
dichotomy. The current situation is a little weird. On ECREATE, the vma
needs to be at the right start address and it needs to have the right
size. The VMA offset is ignored. Later, you can split the VMA through
e.g. calls of mprotect. This will adjust start and end addresses and
offsets logically. I suppose it's technically possible with the current
setup to map a single range and map multiple enclaves in it, it requires
a precarious sequence of mmap and mprotect calls to pull it off.
I think the easiest way to "fix" this is to add another restriction to
ECREATE: the VMA offset must be 0. Combined with the other restrictions
this will mean it's easy to identify individual enclaves in the memory
map. Alternatively we could put almost no restrictions on VMA addresses
and offsets and export all the relevant information through the proc
file. The best abstraction in the sense that it best maps what's going
on in terms of enclaves is to have a different "device" per enclave.
This way it's clear what userspace means when it maps memory at an
offset, etc. There's still the issue of the fixed linear addressing.
Another question is what needs to happen with
mmap/munmap/mremap/mprotect calls that would normally lead to VMAs being
consolidated, as well as how to handle all this with SGX2.
Jethro Beekman | Fortanix
8:02 a.m.
On Tue, Mar 21, 2017 at 10:26:30PM -0700, Jethro Beekman wrote:
In order to provide the ability to debug enclaves independently of
their
programming model, while supporting live attach (meaning the debugger might
not have seen enclave creation), a debugger needs to following capabilities
for a debugged process:
I have to read this with thought later on. That's why I don't want to
say anything *right now*.
The only concern I have at this point s any of the features such that it
couldn't be postponed after upstreaming? I'm targetting patches
hopefully for 4.13 so I rather fix regressions than add new
functionality as you probably understand.
Anyway with quick skim at least some of your proposals made sense. Would
you be interested to contribute some of this post upstreaming?
/Jarkko
10:16 a.m.
The only concern I have at this point s any of the features such
that
it couldn't be postponed after upstreaming? I'm targetting patches
hopefully for 4.13 so I rather fix regressions than add new
functionality as you probably understand.
As long as we don't inadvertently create user ABI that needs to be
changed I think that's fine. I think before anything will be accepted
upstream we need to at least figure out the mmap story as described in
my second email yesterday.
Anyway with quick skim at least some of your proposals made sense.
Would you be interested to contribute some of this post upstreaming?
Yes, I can work on implementing these changes.
Jethro Beekman | Fortanix
On 2017-03-23 08:02, Jarkko Sakkinen wrote:
On Tue, Mar 21, 2017 at 10:26:30PM -0700, Jethro Beekman wrote:
> In order to provide the ability to debug enclaves independently of their
> programming model, while supporting live attach (meaning the debugger might
> not have seen enclave creation), a debugger needs to following capabilities
> for a debugged process:
I have to read this with thought later on. That's why I don't want to
say anything *right now*.
The only concern I have at this point s any of the features such that it
couldn't be postponed after upstreaming? I'm targetting patches
hopefully for 4.13 so I rather fix regressions than add new
functionality as you probably understand.
Anyway with quick skim at least some of your proposals made sense. Would
you be interested to contribute some of this post upstreaming?
/Jarkko
11:14 a.m.
On Thu, Mar 23, 2017 at 10:16:00AM -0700, Jethro Beekman wrote:
> The only concern I have at this point s any of the features such
that
> it couldn't be postponed after upstreaming? I'm targetting patches
> hopefully for 4.13 so I rather fix regressions than add new
> functionality as you probably understand.
As long as we don't inadvertently create user ABI that needs to be changed I
think that's fine. I think before anything will be accepted upstream we need
to at least figure out the mmap story as described in my second email
yesterday.
Great, I'll try to find time to study them through. I'm soon done
with the LE. After I've finished that and before doing accompanying
kernel changes I'll check this through.
> Anyway with quick skim at least some of your proposals made
sense.
> Would you be interested to contribute some of this post upstreaming?
Yes, I can work on implementing these changes.
Good to hear!
Jethro Beekman | Fortanix
/Jarkko
8:54 a.m.
On 2017-03-23 11:14, Jarkko Sakkinen wrote:
On Thu, Mar 23, 2017 at 10:16:00AM -0700, Jethro Beekman wrote:
> As long as we don't inadvertently create user ABI that needs to be changed I
> think that's fine. I think before anything will be accepted upstream we need
> to at least figure out the mmap story as described in my second email
> yesterday.
Great, I'll try to find time to study them through. I'm soon done
with the LE. After I've finished that and before doing accompanying
kernel changes I'll check this through.
I don't know what your envisioned timeline for upstreaming is, but I
think it's important that we upstream an interface that's going to be
forward compatible with our plans for debugging. We don't need to
implement anything right now, but I do want to make sure the user
interface we settle on isn't forever inconvenient. As stated, in
particular the points mention in my email dated 2017-03-22 11:35 PDT
should be considered.
Jethro Beekman | Fortanix
2:13 p.m.
On Thu, Jun 15, 2017 at 08:54:57AM -0700, Jethro Beekman wrote:
On 2017-03-23 11:14, Jarkko Sakkinen wrote:
> On Thu, Mar 23, 2017 at 10:16:00AM -0700, Jethro Beekman wrote:
> > As long as we don't inadvertently create user ABI that needs to be changed
I
> > think that's fine. I think before anything will be accepted upstream we
need
> > to at least figure out the mmap story as described in my second email
> > yesterday.
>
> Great, I'll try to find time to study them through. I'm soon done
> with the LE. After I've finished that and before doing accompanying
> kernel changes I'll check this through.
I don't know what your envisioned timeline for upstreaming is, but I think
it's important that we upstream an interface that's going to be forward
I was planning to send patches already for 4.13 but got bumped in the
problems with my signing tool that have been since fixed [1].
The LE is working together with a host program and has zero SDK
dependecies. I'm now working on the kernel changes and try to target
them to 4.14.
[1] Probably would make sense to validate the permission bits in secinfo
when TCS page is supplied and return -EINVAL if any of the RWX bits are
set. The CPU will silently just reset them. This caused the mismatch in
the hash and is not trivial to deduce if you don't know already that the
hardware behaves this way.
compatible with our plans for debugging. We don't need to
implement anything
right now, but I do want to make sure the user interface we settle on isn't
forever inconvenient. As stated, in particular the points mention in my
email dated 2017-03-22 11:35 PDT should be considered.
Jethro Beekman | Fortanix
It is your job to check the patches once I send them that they are not
incompatible with your requirements and report if they are. It is my
job to fix the issues to make them align with your requirements.
/Jarkko
11:15 a.m.
On 2017-06-15 14:13, Jarkko Sakkinen wrote:
It is your job to check the patches once I send them that they are
not
incompatible with your requirements and report if they are. It is my
job to fix the issues to make them align with your requirements.
I can't do these things:
On 2017-03-21 22:26, Jethro Beekman wrote:
1) Identify enclaves
a) Given a page, identify it as an enclave page
b) Given an enclave page, find that enclave's base address
c) Given a particular enclave, find the TCSes
d) Given a particular enclave, get the attributes/miscselect
...
3) Find debugging symbols for enclaves
More details in the rest of the email chain.
/Jarkko
Jethro Beekman | Fortanix
1812
days inactive
1926
days old
intel-sgx-kernel-dev@lists.01.org
8 comments
3 participants
participants (3)
-
Jarkko Sakkinen -
Jethro Beekman -
Sean Christopherson