Configuration of mOS environment
by Avinash Maurya (RIT Student)
Hello guys,
Thanks a lot for your help in installing mOS on our server.
I was successful in running a couple of shell scripts and commands to
verify the working of the LWK.
However, when I tried running commands like java -version and spark-shell,
the LWK either crashed or it ran on the Linux CPUs instead.
*1. Java:** (LWK crashes)*
*> yod /bin/java -version *
OpenJDK 64-Bit Server VM warning: INFO:
os::commit_memory(0x0000300040000000, 2555904, 1) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2555904 bytes for
committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid30868.log (http://dpaste.com/14G9SVJ)
*> lwkctl -s*
Linux CPU(s): 0,10 [ 2 CPU(s) ]
LWK CPU(s): 1-9,11-19 [ 18 CPU(s) ]
Utility CPU(s): 0,10 [ 2 CPU(s) ]
LWK Memory(KB): 48234496 [ 1 NUMA nodes ]
CPU specification was automatically generated.
Memory specification was automatically generated.
*Tried solutions:*
1. I tried supplying yod arguments for memory and # CPUs
2. Tried checking the Java heap size:
*> java -XX:+PrintFlagsFinal -version | grep -iE
'HeapSize|PermSize|ThreadStackSize'*
intx CompilerThreadStackSize = 0
{pd product}
uintx ErgoHeapSizeLimit = 0
{product}
uintx HeapSizePerGCThread = 87241520
{product}
uintx InitialHeapSize := 1054867456
{product}
uintx LargePageHeapSizeThreshold = 134217728
{product}
uintx MaxHeapSize := 16856907776
{product}
intx ThreadStackSize = 1024
{pd product}
intx VMThreadStackSize = 1024
{pd product}
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
3. I tried running 'java -version' inside a shell script and invoked that
shell script through the 'yod' command. Surprisingly *it worked*, but still
only the Linux cores showed slight consumption, whereas the LWK CPUs were
still inactive.
4. I tried running 'yod /bin/java -version' inside the shell script instead
of 'java --version' and then ran the shell script through 'yod /bin/sh
/path/to/shell-script' which threw the error as `[yod:32001] ERROR: No LWK
CPUs are available. (rc=-16)`. According to my understanding, all the LWK
CPUs are already locked by the 'yod /bin/sh /path/to/shell-script' command
and hence when the execution reaches 'yod /bin/java -version' command, it
can't find any LWK CPUs. So we cannot span nested yod calls, it would throw
an error.
Please let me know if my understanding is correct here.
*2. Apache Spark:** (Runs on Linux instead)*
*> yod /bin/sh /opt/spark/bin/spark-shell*
19/10/01 20:10:14 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Spark context Web UI available at http://hp099.utah.cloudlab.us:4040
Spark context available as 'sc' (master = local[*], app id =
local-1569982222983).
Spark session available as 'spark'.
*The Problem:*
When I try to run this command (spark-shell) on a regular (without mOS)
centos server, the workload gets distributed uniformly on all the 20 CPUs,
whereas when I run it through `yod`, it just consumes the 2 Linux CPUs
instead of running on the 18 LWK cores.
*Probable Explanation:*
My guess would be spark-shell uses some commands/routines that fall in the
Linux domain and are not available in the LWK, so even with `yod`, the
processing is directed on the Linux cores. And in order to make it work for
LWK, I might have to tweak config or calls in spark.
Please let me know if I'm correct in the above understanding or I'm missing
something.
Thanks,
Avinash Maurya.
1 year, 6 months