Re: Sys call list handled by LWK (MOS)
by Rolf Riesen
Hi Brian,
I'll cc this to the mOS mailing list (mostly just the mOS team plus
a few external, interested parties.) We might get a more interesting
discussion that way.
Glancing through your Mascot paper I find it interesting and intend to
fully read it. However, I'd like to make a couple of comments about
system calls, particularly in mOS.
Traditional lightweight kernels (LWK) had limited abilities to execute
system calls and most calls were shipped to a full-featured OS on
another node, or with multi-kernels, an OS in another partition on
the same node. For traditional HPC workloads the overhead of shipping
does not matter. These applications do not make system calls in the
performance critical sections of their codes. That is beginning
to change with AI, ML, DL workloads, but that is a fairly recent
phenomena in high-end HPC.
Today, mOS ships 99% of all available system calls to the Linux
partition, but we will change that in the next release. Because the
mOS LWK is embedded in the Linux kernel, we can easily make these
calls locally on the core where they were requested.
In the early days of mOS we thought local system calls might
contribute to noise, and there was no need to make them more efficient
(traditional HPC.) As we have gained experience with mOS and are
looking to run a more diverse set of applications, we have decided
that we will make all system calls local. We will retain the ability
to ship them, and will do so if we see a need for that in the feature.
But, the next release of mOS will have all calls local.
That said, you can see the table of calls we currently handle locally
at the end of the include/linux/syscalls.h file in the mOS source. The
32 calls listed in __mos_do_on_original_cpu() are handled locally and
the rest is shipped.
On Fri Dec 6, 2019 15:55:46, Brian Richard Tauro wrote:
> Hi,
>
> I am a phd student studying at Illinois Institute of Technology Chicago,
> working under my advisor Dr Kyle Hale <http://cs.iit.edu/~khale/> at IIT
> Chicago, I want to benchmark multi kernel environments with various compute
> and system intensive applications for extension of my previous work on
> modelling
> speedup in multi kernels
> <http://cs.iit.edu/~khale/docs/multios-mascots19.pdf> for an
> upcoming journal paper.
>
> We would like to run some experiments on moS for our research.
>
> Currently i want to know where can I find the system calls list which tells
> me which system calls are forwarded to linux and the ones which are
> handled by lwk.
>
> Please can you help me with this!!
>
> Thanks,
> Brian
Thanks,
Rolf
+++-+--+----+-------+------------+--------------------+------------------------
Rolf Riesen, Ph.D. Email: rolf.riesen(a)intel.com
Software Architect Phone: +1 (503) 613-5514
Extreme-scale Software System Pathfinding Mobile: +1 (505) 363-6871
Outlook users: Turn off "extra line break removal" in File > Options > Mail > Message Format
1 year, 5 months
Understanding some conflicting yod options resulting in deadlock
by Avinash Maurya (RIT Student)
Dear team,
I would like to sincerely thank you for your time, efforts and speedy
responses to help us understand mOS better for our research.
As we were doing some experiments with mOS and Spark, we realized that
using the move-syscalls-disable and the lwkmem-prot-none-delegation-enable
options simultaneously resulted in something like a futex deadlock. We use
the lwkmem-prot-none-delegation-enable flag because Java won't run without
this and since we are performing benchmarks for move-syscalls-disable
options to check for performance improvements in jobs, we need this flag as
well.
With my limited knowledge, I presume this is because the
lwkmem-prot-none-delegation-enable option is somehow using the Linux
resources to defer backing of the PROT_NONE memory regions. And on the
other hand, the move-syscalls-disable calls restrict yod from invoking
those Linux resources resulting in a futex deadlock. I am not sure if my
understanding of this is correct but our experiments ran successfully
without the move-syscalls-disable option.
Strace file with futex deadlock: http://dpaste.com/1HNM6HX
Also, we look forward to understanding the mOS codebase and being
relatively new to this field, we would love to know about some starting
points in the mOS codebase and some references for the basic understanding
of HPC kernels.
Thank you,
Avinash Maurya.
1 year, 5 months
yod command getting stuck
by Avinash Maurya (RIT Student)
Dear team,
I have been trying to perform a couple of benchmarks using Apache Spark on
mos.
The commands used to work fine initially, but I have started experiencing a
very long delay in the execution of yod commands after allocating the
resources to the LWK using lwkctl -c <config>.
This might be an issue on our cloud servers but I just wanted your opinion
to make sure it's nothing wrong on the configuration end. We tried
creating a new instance with mos and this same issue persists on the new
machines as well.
Please find the execution time of a simple echo command and kernel logs
from /var/log/messages below:
$ sudo lwkctl -c 'lwkcpus=auto lwkmem=auto'
$ tail -100f /var/log/messages/ # Log generated on running sudo
lwkctl -c 'lwkcpus=auto lwkmem=auto'
http://dpaste.com/3AN4ZAK
$ lwkctl -s
mOS version : 0.7
Linux CPU(s): 0,10,20,30 [ 4 CPU(s) ]
LWK CPU(s): 1-9,11-19,21-29,31-39 [ 36 CPU(s) ]
Utility CPU(s): 0,10,20,30 [ 4 CPU(s) ]
LWK Memory(KB): 73400320 73400320 [ 2 NUMA nodes ]
CPU specification was automatically generated.
Memory specification was automatically generated.
$ time yod echo hello
hello
real 1m20.255s
user 0m0.002s
sys 0m0.007s
$ tail -100f /var/log/messages # Log generated on running the echo
hello command.
http://dpaste.com/2VW88FC
<http://dpaste.com/3AN4ZAK>
The above file shows some logs like *EDAC sbridge: Couldn't find mci
handler. *However, I think it's a warning and probably shouldn't be
stalling the execution. Thoughts?
Please let me know if you need more debug info.
Thank you,
Avinash Maurya.
1 year, 5 months