[SPDK] [CASS SPAM]Re: [CASS SPAM]Re: spdk exits with failure if hugemem is very large

wuzhouhui wuzhouhui14 at mails.ucas.ac.cn
Tue Oct 16 02:18:47 PDT 2018


> -----Original Messages-----
> From: wuzhouhui <wuzhouhui14 at mails.ucas.ac.cn>
> Sent Time: 2018-10-16 09:32:09 (Tuesday)
> To: "storage performance development kit" <spdk at lists.01.org>
> Cc: 
> Subject: [CASS SPAM]Re: [SPDK] [CASS SPAM]Re: spdk exits with failure if hugemem is very large
> 
> > -----原始邮件-----
> > 发件人: "Stojaczyk, Dariusz" <dariusz.stojaczyk at intel.com>
> > 发送时间: 2018-10-15 21:18:40 (星期一)
> > 收件人: "Storage Performance Development Kit" <spdk at lists.01.org>
> > 抄送: 
> > 主题: [CASS SPAM]Re: [SPDK] spdk exits with failure if hugemem is very large
> > 
> > I could guess you're hitting a max virtual memory limit. 
The manual of mmap() says:

ENOMEM  The  process's  maximum number of mappings would have been exceeded. 
        This error can also occur for munmap(), when unmapping a region in 
        the middle of an existing mapping, since this results in two smaller  
        mappings  on either side of the region being unmapped.

I also write a demo to mmap() 69632 files, each file is 8-bytes long. The demo
report error when mmap() 65515th file. So it is the limit of number of mappings
that kernel enforced, although I haven't check kernel code.

> > 
> > Do you mind sharing the output of `ulimit -a`?
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 1542344
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 1542344
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> > 
> > This is a limitation of the legacy memory management in DPDK. There's already a new, dynamic memory management in DPDK which overcomes this problem, but we're still getting spdk ready for it. Once we get all patches merged your case should start working automatically, without tweaking any ulimits.
> > 
> > D.
> > 
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of wuzhouhui
> > > Sent: Monday, October 15, 2018 12:14 PM
> > > To: spdk at lists.01.org
> > > Subject: [SPDK] spdk exits with failure if hugemem is very large
> > > 
> > > The dpdk will map all hugepage and unmap unneeded even though the
> > > vhost-target
> > > just needs 1024MB hugepage. Map all hugepage will cost a lot of time and
> > > may
> > > cause mmap() in create_shared_memory() failed with ENOMEM. In my
> > > system, the
> > > /proc/sys/vm/nr_hugepages is 69632, and vhost-target will always failed:
> > > 
> > > Starting SPDK v18.10-pre / DPDK 18.05.0 initialization...
> > > [ DPDK EAL parameters: vhost --no-shconf -c 0x1 -m 1024 --legacy-mem --
> > > file-prefix=spdk_pid162034 ]
> > > EAL: Detected 64 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: Multi-process socket /var/run/dpdk/spdk_pid162034/mp_socket
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: VFIO support initialized
> > > EAL: Failed to create shared memory!
> > > EAL: FATAL: Cannot init memory
> > > 
> > > EAL: Cannot init memory
> > > 
> > > Failed to initialize DPDK
> > > Unable to initialize SPDK env
> > > 
> > > And I found that the option -g of vhost-target can help app bypass this issue.
> > > 
> > > Can we resolve this issue by optimizing dpdk?
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK at lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK at lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK at lists.01.org
> https://lists.01.org/mailman/listinfo/spdk


More information about the SPDK mailing list