Some type of crash
by Harms, Kevin
I tried to run the following:
mpirun -f ${COBALT_NODEFILE} -n 16 -ppn 8 -genv OMP_NUM_THREADS=8 yod -R 1/8 src/lmp_knl -in rhodo/1node/lmp.in -sf omp -var NSTEPS 4 -var REPX 2 -var REPY 2 -var REPZ 2
The head node for launch was knl10 and we see this in the dmesg log. Any ideas? There were three yod processes and the mpirun process on knl10 which just seemed to be sitting and waiting.
thanks,
kevin
[568091.206081] pmi_proxy[143190]: segfault at 50d2770 ip 00007fb552cdbaa5 sp 00007ffdc69412b0 error 6 in libc-2.17.so[7fb552c5e000+1b6000]
[568352.376321] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaaaf600000, 0x2aaaaf601000, ...)=-12
[568352.376358] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaaf5dd000 -> fffffffffffffff4
[568352.379951] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaaafc00000, 0x2aaaafc01000, ...)=-12
[568352.379986] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaafbdd000 -> fffffffffffffff4
[568352.384012] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab0200000, 0x2aaab0201000, ...)=-12
[568352.384048] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab01dd000 -> fffffffffffffff4
[568352.391709] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab0600000, 0x2aaab0601000, ...)=-12
[568352.391746] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab05e9000 -> fffffffffffffff4
[568352.395453] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab0a00000, 0x2aaab0a01000, ...)=-12
[568352.395488] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab09e9000 -> fffffffffffffff4
[568352.398778] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab1200000, 0x2aaab1201000, ...)=-12
[568352.398813] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab11d1000 -> fffffffffffffff4
[568352.403703] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab1a00000, 0x2aaab1a01000, ...)=-12
[568352.403739] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab19d1000 -> fffffffffffffff4
[568352.408628] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab2200000, 0x2aaab2201000, ...)=-12
[568352.408663] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab21d1000 -> fffffffffffffff4
[568352.413573] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab2a00000, 0x2aaab2a01000, ...)=-12
[568352.413610] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab29d1000 -> fffffffffffffff4
[568352.418714] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab3c00000, 0x2aaab3c01000, ...)=-12
[568352.418751] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab3b95000 -> fffffffffffffff4
[568352.428164] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab4e00000, 0x2aaab4e01000, ...)=-12
[568352.428198] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab4d95000 -> fffffffffffffff4
[568352.437631] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab6000000, 0x2aaab6001000, ...)=-12
[568352.437668] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab5f95000 -> fffffffffffffff4
[568352.447198] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab7200000, 0x2aaab7201000, ...)=-12
[568352.447236] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab7195000 -> fffffffffffffff4
[568352.456742] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaab8400000, 0x2aaab8401000, ...)=-12
[568352.456777] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaab8395000 -> fffffffffffffff4
[568353.421758] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaabf800000, 0x2aaabf8c2000, ...)=-12
[568353.421793] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaabf7c1000 -> fffffffffffffff4
[568353.539554] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaac4600000, 0x2aaac46aa000, ...)=-12
[568353.539590] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaac45e1000 -> fffffffffffffff4
[568353.559821] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaac1200000, 0x2aaac1277000, ...)=-12
[568353.559858] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaac11e1000 -> fffffffffffffff4
[568353.566209] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaac2c00000, 0x2aaac2c58000, ...)=-12
[568353.566245] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaac2be1000 -> fffffffffffffff4
[568353.626980] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaac6e00000, 0x2aaac6e8b000, ...)=-12
[568353.627016] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaac6d01000 -> fffffffffffffff4
[568353.666456] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaac8a00000, 0x2aaac8a21000, ...)=-12
[568353.666493] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaac8901000 -> fffffffffffffff4
[568353.686839] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaaca600000, 0x2aaaca621000, ...)=-12
[568353.686875] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaca501000 -> fffffffffffffff4
[568353.693451] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaacc200000, 0x2aaacc202000, ...)=-12
[568353.693486] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacc101000 -> fffffffffffffff4
[568353.858711] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaace000000, 0x2aaace040000, ...)=-12
[568353.858748] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacdf41000 -> fffffffffffffff4
[568353.889876] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaacfe00000, 0x2aaacfe21000, ...)=-12
[568353.889912] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacfd41000 -> fffffffffffffff4
[568353.912539] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaad1c00000, 0x2aaad1c42000, ...)=-12
[568353.912576] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaad1b41000 -> fffffffffffffff4
[568353.951239] mOS-mem: build_lwkvma: find_vma_links(ffff881813a27c00, 0x2aaad4800000, 0x2aaad4802000, ...)=-12
[568353.951273] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaad47e1000 -> fffffffffffffff4
[568497.474574] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaac8600000, 0x2aaac8601000, ...)=-12
[568497.474612] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaac85dd000 -> fffffffffffffff4
[568497.478473] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaac8c00000, 0x2aaac8c01000, ...)=-12
[568497.478510] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaac8bdd000 -> fffffffffffffff4
[568497.482539] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaca400000, 0x2aaaca401000, ...)=-12
[568497.482574] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaca371000 -> fffffffffffffff4
[568497.493623] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaacac00000, 0x2aaacac01000, ...)=-12
[568497.493659] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacabd1000 -> fffffffffffffff4
[568497.503412] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaad9800000, 0x2aaad9801000, ...)=-12
[568497.503446] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaad9765000 -> fffffffffffffff4
[568497.516642] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaacb000000, 0x2aaacb001000, ...)=-12
[568497.516676] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacafe9000 -> fffffffffffffff4
[568497.519762] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaacb400000, 0x2aaacb401000, ...)=-12
[568497.519799] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacb3e9000 -> fffffffffffffff4
[568497.523069] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaacbc00000, 0x2aaacbc01000, ...)=-12
[568497.523106] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacbbd1000 -> fffffffffffffff4
[568497.528012] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaacc400000, 0x2aaacc401000, ...)=-12
[568497.528049] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacc3d1000 -> fffffffffffffff4
[568497.532961] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaccc00000, 0x2aaaccc01000, ...)=-12
[568497.532997] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaccbd1000 -> fffffffffffffff4
[568497.537878] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaacd400000, 0x2aaacd401000, ...)=-12
[568497.537913] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacd3d1000 -> fffffffffffffff4
[568497.543005] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaace600000, 0x2aaace601000, ...)=-12
[568497.543042] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaace595000 -> fffffffffffffff4
[568497.552534] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaacf800000, 0x2aaacf801000, ...)=-12
[568497.552570] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaacf795000 -> fffffffffffffff4
[568497.562032] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaad0a00000, 0x2aaad0a01000, ...)=-12
[568497.562069] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaad0995000 -> fffffffffffffff4
[568497.571449] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaad1c00000, 0x2aaad1c01000, ...)=-12
[568497.571484] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaad1b95000 -> fffffffffffffff4
[568497.580872] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaad2e00000, 0x2aaad2e01000, ...)=-12
[568497.580907] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaad2d95000 -> fffffffffffffff4
[568498.176840] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaadb200000, 0x2aaadb24a000, ...)=-12
[568498.176874] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaadb181000 -> fffffffffffffff4
[568498.196175] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaadce00000, 0x2aaadcf21000, ...)=-12
[568498.196213] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaadcda1000 -> fffffffffffffff4
[568498.271477] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaadea00000, 0x2aaadea40000, ...)=-12
[568498.271515] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaade901000 -> fffffffffffffff4
[568498.291725] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaae0800000, 0x2aaae0862000, ...)=-12
[568498.291761] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaae0741000 -> fffffffffffffff4
[568498.375974] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaae2600000, 0x2aaae2601000, ...)=-12
[568498.376009] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaae2481000 -> fffffffffffffff4
[568498.592045] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaae6200000, 0x2aaae62c1000, ...)=-12
[568498.592081] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaae61a1000 -> fffffffffffffff4
[568498.612481] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaae7a00000, 0x2aaae7a29000, ...)=-12
[568498.612517] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaae79a1000 -> fffffffffffffff4
[568498.679053] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaae4a00000, 0x2aaae4a0a000, ...)=-12
[568498.679091] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaae49c1000 -> fffffffffffffff4
[568498.750655] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaed200000, 0x2aaaed201000, ...)=-12
[568498.750691] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaed081000 -> fffffffffffffff4
[568498.811247] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaeec00000, 0x2aaaeec22000, ...)=-12
[568498.811284] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaeebe1000 -> fffffffffffffff4
[568498.817150] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaae9400000, 0x2aaae9421000, ...)=-12
[568498.817184] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaae93e1000 -> fffffffffffffff4
[568498.846813] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaeae00000, 0x2aaaeaec1000, ...)=-12
[568498.846849] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaeade1000 -> fffffffffffffff4
[568498.888275] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaf3a00000, 0x2aaaf3a23000, ...)=-12
[568498.888313] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaf3901000 -> fffffffffffffff4
[568498.942028] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaf5600000, 0x2aaaf5621000, ...)=-12
[568498.942066] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaf5501000 -> fffffffffffffff4
[568498.950061] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaf7200000, 0x2aaaf7202000, ...)=-12
[568498.950097] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaf7101000 -> fffffffffffffff4
[568499.002231] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaf1200000, 0x2aaaf1236000, ...)=-12
[568499.002268] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaf1181000 -> fffffffffffffff4
[568499.030192] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aaaf9c00000, 0x2aaaf9c21000, ...)=-12
[568499.030228] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aaaf9aa1000 -> fffffffffffffff4
[568499.111812] mOS-mem: build_lwkvma: find_vma_links(ffff8818fb2cf000, 0x2aab01e00000, 0x2aab01e44000, ...)=-12
[568499.111848] mOS-mmap: lwk_sys_mremap() : unexpected remap 2aab01d01000 -> fffffffffffffff...
3 years, 10 months
Test
by Rolf Riesen
Just making sure our mailing list works...
Thanks,
Rolf
+++-+--+----+-------+------------+--------------------+------------------------
Rolf Riesen, Ph.D. Email: rolf.riesen(a)intel.com
Software Architect Phone: +1 (503) 613-5514
Extreme-scale Software System Pathfinding Mobile: +1 (505) 363-6871
Outlook users: Turn off "extra line break removal" in File > Options > Mail > Message Format
3 years, 10 months
