Thanks Nathan for your interests. Just take a glance at Nested Kernel, it seems that we can work together complementarily… not sure if I’m understanding correctly or not, do you assumed that CR0.WP bit won’t be disabled/clear by untrusted code in outer kernel? -Bing From: Intel-KGT [mailto:intel-kgt-bounces@lists.01.org] On Behalf Of Dautenhahn, Nathan Daniel Sent: Thursday, June 11, 2015 10:40 AM To: intel-kgt(a)lists.01.org Cc: Edouard Bugnion; Adve, Vikram Sadanand Subject: [iKGT] Nested Kernel and Intel-KGT Hi All- First I want to say, really cool work! I am a PhD student working at the University of Illinois at Urbana Champaign and recently observed your IKGT work. This looks extremely interesting. The first thing I would like to mention is that IKGT greatly overlaps with my recent efforts to create a new operating system architecture called the Nested Kernel. In fact it appears as almost the same thing—the example policies listed on the web page are all enforced by the Nested Kernel. The two approaches seem to provide equivalent protection capabilities, but we would like to understand the exact capabilities of IKGT better. One key difference is that the Nested Kernel x86-64 design provides a new technique that virtualizes ring 0 using the WP-bit as the privilege switch mechanism in contrast to the VT-x based isolation. I am wondering if you have a paper on it so that I can get a few more details? I am working through the source code as well. Also, I would be interested if any of the creators would be able to review the nested kernel paper at nestedkernel.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__n...;. I am interested to ascertain how much overlap exists between the features. Some technical questions: — Are you using VMFUNC for fast context switching? The code appears to do this, but I am just wondering. — Do you have any performance evaluation of the system? Thanks, - Nathan Dautenhahn

Hi Nathan, Thank you for sharing the pointers with us. We (ikgt team) are looking at the paper and related collateral and will provide a response soon. Regards -harsh From: Intel-KGT [mailto:intel-kgt-bounces@lists.01.org] On Behalf Of Dautenhahn, Nathan Daniel Sent: Thursday, June 11, 2015 12:30 PM To: Zhu, Bing Cc: Edouard Bugnion; Adve, Vikram Sadanand; intel-kgt(a)lists.01.org Subject: Re: [iKGT] Nested Kernel and Intel-KGT On Jun 11, 2015, at 1:31 AM, Zhu, Bing <bing.zhu@intel.com<mailto:bing.zhu@intel.com>> wrote: Thanks Nathan for your interests. Just take a glance at Nested Kernel, it seems that we can work together complementarily… They may be complementary, I think the KGT effort might be considered an “instance of” the nested kernel design. We split our system into two parts: 1) the Nested Kernel architecture using on the MMU, and 2) the x86-64 design and implementation realizing that architecture. not sure if I’m understanding correctly or not, do you assumed that CR0.WP bit won’t be disabled/clear by untrusted code in outer kernel? We do not assume CR0.WP won’t be disabled; we enforce that it cannot be disabled by the outer kernel code by employing a combination of kernel code static deprivileging (paravirtualize to use the trusted component for operations like modifying CR0), load time verification, and runtime integrity to ensure that CR0.WP can only be disabled by code in the nested kernel where we assert the invariant that CR0.WP is always set when the outer kernel is in operation. Thanks, ::nathan:: -Bing From: Intel-KGT [mailto:intel-kgt-bounces@lists.01.org] On Behalf Of Dautenhahn, Nathan Daniel Sent: Thursday, June 11, 2015 10:40 AM To: intel-kgt@lists.01.org<mailto:intel-kgt@lists.01.org> Cc: Edouard Bugnion; Adve, Vikram Sadanand Subject: [iKGT] Nested Kernel and Intel-KGT Hi All- First I want to say, really cool work! I am a PhD student working at the University of Illinois at Urbana Champaign and recently observed your IKGT work. This looks extremely interesting. The first thing I would like to mention is that IKGT greatly overlaps with my recent efforts to create a new operating system architecture called the Nested Kernel. In fact it appears as almost the same thing—the example policies listed on the web page are all enforced by the Nested Kernel. The two approaches seem to provide equivalent protection capabilities, but we would like to understand the exact capabilities of IKGT better. One key difference is that the Nested Kernel x86-64 design provides a new technique that virtualizes ring 0 using the WP-bit as the privilege switch mechanism in contrast to the VT-x based isolation. I am wondering if you have a paper on it so that I can get a few more details? I am working through the source code as well. Also, I would be interested if any of the creators would be able to review the nested kernel paper at nestedkernel.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__n...;. I am interested to ascertain how much overlap exists between the features. Some technical questions: — Are you using VMFUNC for fast context switching? The code appears to do this, but I am just wondering. — Do you have any performance evaluation of the system? Thanks, - Nathan Dautenhahn

On Jun 22, 2015, at 7:07 PM, Vipat, Harshawardhan <harshawardhan.vipat@intel.com<mailto:harshawardhan.vipat@intel.com>> wrote: Hi Nathan, First, my apologies for the delay in responding to this. We reviewed the paper and presentations and have following questions/comments. BTW, we liked the presentations – short and log versions, zooming in/out to the big picture were helpful.. Nicely done! Thank you. Regarding your questions: VMFUNC: The code that was released is not using vmfunc or #ve. The current release only has support for hardening CPU assets (CRs, MSRs) and the policy actions are enforced from vmxroot. What exactly is #ve? Performance: Our solution will have the standard overhead introduced by turning on VTx and EPT. The simple hardening policies that we are currently considering we don’t believe would add any significant overhead, if any, for the normal operation. We can discuss this more later. Our questions: The nested kernel architecture can also realize several system security properties because it controls all virtual memory mappings in the system. One example is lifetime kernel code integrity (as Section 3.5 explains). This single use case effectively thwarts an entire class of kernel malware (namely code injection attacks). In addition to code integrity, PerspicuOS also marks memory pages as non-executable by default and enables superuser mode execution prevention of user-mode code and data. Even if commodity kernels use these hardware features, they cannot prevent malware from disabling them. PerspicuOS, in contrast, enables these protections and prevents malicious code from disabling them. PerspicuOS can also be used for any type of security monitor that inserts explicit calls into source code to ensure that the monitor both executes and is isolated from the untrusted code. Q1:Could you walk us through what would happen in the attack scenario where a malicious driver is dynamically loaded and has code that turns off WP and SMEP? How would this be prevented? What is addressed by code scanning, and what is addressed by non-extensibility ? All mappings in the system are configured as non executable by default. Therefore, if a driver is dynamically loaded as code it must be explicitly requested by the outer kernel if the mapping is to be changed. The inner kernel performs a validation scan to verify that privileged MMU operations---instructions that modify special control registers--- are not in the loaded code. If any such operations exist the nested kernel can either deny loading or include a binary rewriting mechanism to fix the instruction at load time. We did not implement the load time scanner. I'm not quite sure what you mean by non-extensibility, can you clarify? dynamic lifetime outer kernel code integrity by configuring the processor and pMMU so that 1) by default all kernel pages are mapped as non-executable (enforced by the no execute bit (NX-bit) in the EFER MSR), 2) validated kernel code pages are mapped with read-only permissions, and 3) user-space code and data are mapped as non-executable in supervisor-mode by employing supervisor-mode execution prevention (SMEP in CR4) [2], thereby preventing the outer kernel from executing any protected instructions contained within user-mode pages. Q2:This is RX (not RO), right ? Yes, sorry for the confusion. Q3:What is the load sequence ? The bootloader must be trusted to initialize the system so that read only permissions and code integrity properties are set prior to outer kernel execution---at least this is how the paper presents it. It turns out that this constraint may not be necessary. In the paper we assume some type of trusted boot so that the nested kernel can ensure no other modifications of the page tables, but the page tables can be found via a page walk and then a scan can be performed to dynamically verify the loaded code. After the loader finishes the nested kernel has full control of the mappings. How does the nested kernel get its memory? It seems that there is a circular dependency. What parts are measure by secure boot? The nested kernel currently statically pre-allocates memory for itself, however it is feasible to have the nested kernel obtain pages from an untrusted allocator and ensure that no other mappings to the particular page exists, sanitize the page, and insert it for nested kernel use. As per secure boot measurement, it is not something we investigated fully. However, as I mentioned above, with respect to verifying the various invariants, I think the nested kernel can learn via a page walk all of the valid page table mappings, configure them as read only, and then even perform code validation. I need to investigate this further to verify it is correct. Q4: How is above implemented? Could you provide more details on this ? Is there a page table walk involved? Yes there is a page table walk. We did not implement the measurement or runtime verification. We implement a static offline scanner to show that it is possible to generate a kernel without the privileged instructions. Q5: When and how physical page descriptor list is built ? During the page table walk in an initialization function that executes after the initial page tables are setup: the DMAP. Q6: How are the violations handled? Do you simply fail nk_* call and let outer kernel deal with it ? For direct writes to page table pages we don't have an implementation, but it can just fail. For writes though the nk_ interface we make sure the proper mappings are set. Q7: You mention a separate nested kernel stack. Is it a per thread kernel stack or per core kernel stack? This is a per core kernel stack, which is only active for the lifetime of each nested kernel call invocation. Q8: How is the heap memory managed? We do not manage heap memory other than mapping it as NX. Q9: Would it be desirable to move to a model where the inner kernel is implemented without changes to the kernel (e.g. using just a driver with the assistance of something like iKGT) Yes, I think this is a great way of implementing it, although you have to play the trap and emulate game, which may be more costly in terms of performance. On the other hand that approach would increase the security of the MMU isolation. Best, ::Nathan:: Thanks. Regards -harsh From: Vipat, Harshawardhan Sent: Thursday, June 11, 2015 2:15 PM To: Dautenhahn, Nathan Daniel; Zhu, Bing Cc: Edouard Bugnion; Adve, Vikram Sadanand; intel-kgt@lists.01.org<mailto:intel-kgt@lists.01.org> Subject: RE: Nested Kernel and Intel-KGT Hi Nathan, Thank you for sharing the pointers with us. We (ikgt team) are looking at the paper and related collateral and will provide a response soon. Regards -harsh From: Intel-KGT [mailto:intel-kgt-bounces@lists.01.org] On Behalf Of Dautenhahn, Nathan Daniel Sent: Thursday, June 11, 2015 12:30 PM To: Zhu, Bing Cc: Edouard Bugnion; Adve, Vikram Sadanand; intel-kgt@lists.01.org<mailto:intel-kgt@lists.01.org> Subject: Re: [iKGT] Nested Kernel and Intel-KGT On Jun 11, 2015, at 1:31 AM, Zhu, Bing <bing.zhu@intel.com<mailto:bing.zhu@intel.com>> wrote: Thanks Nathan for your interests. Just take a glance at Nested Kernel, it seems that we can work together complementarily… They may be complementary, I think the KGT effort might be considered an “instance of” the nested kernel design. We split our system into two parts: 1) the Nested Kernel architecture using on the MMU, and 2) the x86-64 design and implementation realizing that architecture. not sure if I’m understanding correctly or not, do you assumed that CR0.WP bit won’t be disabled/clear by untrusted code in outer kernel? We do not assume CR0.WP won’t be disabled; we enforce that it cannot be disabled by the outer kernel code by employing a combination of kernel code static deprivileging (paravirtualize to use the trusted component for operations like modifying CR0), load time verification, and runtime integrity to ensure that CR0.WP can only be disabled by code in the nested kernel where we assert the invariant that CR0.WP is always set when the outer kernel is in operation. Thanks, ::nathan:: -Bing From: Intel-KGT [mailto:intel-kgt-bounces@lists.01.org] On Behalf Of Dautenhahn, Nathan Daniel Sent: Thursday, June 11, 2015 10:40 AM To: intel-kgt@lists.01.org<mailto:intel-kgt@lists.01.org> Cc: Edouard Bugnion; Adve, Vikram Sadanand Subject: [iKGT] Nested Kernel and Intel-KGT Hi All- First I want to say, really cool work! I am a PhD student working at the University of Illinois at Urbana Champaign and recently observed your IKGT work. This looks extremely interesting. The first thing I would like to mention is that IKGT greatly overlaps with my recent efforts to create a new operating system architecture called the Nested Kernel. In fact it appears as almost the same thing—the example policies listed on the web page are all enforced by the Nested Kernel. The two approaches seem to provide equivalent protection capabilities, but we would like to understand the exact capabilities of IKGT better. One key difference is that the Nested Kernel x86-64 design provides a new technique that virtualizes ring 0 using the WP-bit as the privilege switch mechanism in contrast to the VT-x based isolation. I am wondering if you have a paper on it so that I can get a few more details? I am working through the source code as well. Also, I would be interested if any of the creators would be able to review the nested kernel paper at nestedkernel.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__n...;. I am interested to ascertain how much overlap exists between the features. Some technical questions: — Are you using VMFUNC for fast context switching? The code appears to do this, but I am just wondering. — Do you have any performance evaluation of the system? Thanks, - Nathan Dautenhahn