February 2020 - SPDK - Ml01.01.Org

Does BlobFS Asynchronous API support multi thread writing?

by chen.zhenghua＠zte.com.cn

Hi everyone, I simply tested the BlobFS Asynchronous API by using SPDK events framework to execute multi tasks, each task writes one file. But it doesn't work, the spdk_file_write_async() reported an error when resizing the file size. The call stack looks like this: spdk_file_write_async() -> __readwrite() -> spdk_file_truncate_async() -> spdk_blob_resize() The resize operation must be done in the metadata thread which invoked the spdk_fs_load(), so only the task dispatched to the metadata CPU core works. That's to say only one thread can be used to write files. It's hard to use, and performance issues may arise. Does anyone knows further more about this? thanks very much

5 months, 2 weeks

2
1
0 / 0

Reporting intermittent test failures

by Harris, James R

Hi all, I’ve seen a lot of cases recently where -1 votes from the test pool have been removed from a patch due to a failure unrelated to the patch, but then nothing was filed in GitHub for that failure. The filing in GitHub could be a new issue, or a comment on an existing issue. Please make those GitHub updates a priority. It’s the only way the project can understand the frequency of those intermittent failures and gather to get them fixed. If you’re not sure if a failure has been seen before, search GitHub issues with the “Intermittent Failure” label, or ask on Slack if anyone else has seen the issue. There is no harm in filing a new issue that may be a duplicate – we can always clean these up later during the next bug scrub meeting. The important thing is that we get the failure tracked. Thanks, -Jim

1 year, 10 months

3
2
0 / 0

SPDK socket abstraction layer

by Zawadzki, Tomasz

Hello everyone, Summary: With this message I wanted to update SPDK community on state of VPP socket abstraction as of SPDK 19.07 release. At this time there does not seem to be a clear efficiency improvements with VPP. There is no further work planned on SPDK and VPP integration. Details: As some of you may remember, SPDK 18.04 release introduced support for alternative socket types. Along with that release, Vector Packet Processing (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by expanding socket abstraction to use VPP Communications Library (VCL). TCP/IP stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early stages back then and has seen improvements throughout the last year. To better use VPP capabilities, following fruitful collaboration with VPP team, in SPDK 19.07, this implementation was changed from VCL to VPP Session API from VPP 19.04.2. VPP socket abstraction has met some challenges due to inherent design of both projects, in particular related to running separate processes and memory copies. Seeing improvements from original implementation was encouraging, yet measuring against posix socket abstraction (taking into consideration entire system, i.e. both processes), results are comparable. In other words, at this time there does not seem to be a clear benefit of either socket abstraction from standpoint of CPU efficiency or IOPS. With this message I just wanted to update SPDK community on state of socket abstraction layers as of SPDK 19.07 release. Each SPDK release always brings improvements to the abstraction and its implementations, with exciting work on more efficient use of kernel TCP stack - changes in SPDK 19.10 and SPDK 20.01. However there is no active involvement at this point around VPP implementation of socket abstraction in SPDK. Contributions in this area are always welcome. In case you're interested in implementing further enhancements of VPP and SPDK integration feel free to reply, or to use one of the many SPDK community communications channels<https://spdk.io/community/>. Thanks, Tomek

2 years, 4 months

6
12
0 / 0

EAL: set_mempolicy failed: Invalid argument

by Andrey Kuzmin

Hi guys, I'm running 20.01 (default configuration) on a CentOS 7 Epyc-based system, and getting a lot of subj warnings like in the below example. Does that point to a system configuration issue that needs to be fixed? Thanks, Andrey Starting SPDK v20.01 git sha1 5ffffe9 / DPDK 19.11.0 initialization... [ DPDK EAL parameters: hotplug --no-shconf -c 0x1 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid109760 ] EAL: No available hugepages reported in hugepages-1048576kB EAL: VFIO support initialized EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Initializing NVMe Controllers EAL: set_mempolicy failed: Invalid argument EAL: using IOMMU type 1 (Type 1) EAL: set_mempolicy failed: Invalid argument Attaching to 0000:45:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:46:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4b:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4c:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4d:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4e:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:45:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:46:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4b:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4c:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4d:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4e:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Initialization complete. Starting I/O...

2 years, 5 months

3
6
0 / 0

Running SPDK without app framework

by Stojaczyk, Dariusz

Tomek, I'm looking through your patch example/fio: add option to load json_config [1] and have some concerns to the overall design of applications utilizing SPDK without the app framework. First of all, we seem to use spdk_app_json_config_load() to load a JSON config in fio_plugin. The spdk_app_ prefix in that function suggests it's the app framework API, which we don't use in fio_plugin. I'm guessing it should be at least renamed? spdk_json_config_load() maybe? spdk_app_json_config_load() is defined in spdk_internal/event.h. Why is this internal anyway? This practically forces external SPDK users to stick with either legacy json config files, or spdk_app_start(). The question is where should we move it? lib/event.h is currently specific to the app framework. Do you have any plans for SPDK app design without the app framework? I'm guessing we do want to support it (see fio_plugin), but currently it's messy. D. [1] https://review.gerrithub.io/c/spdk/spdk/+/463979/

2 years, 6 months

1
1
0 / 0

Re: ask for help about SPDK CI

by Yang, Ziye

Hi Richael, I think that you are not in the default list, so your patch needs to be approved first, then the patch will be tested. Or some one needs to add you into the default list. Let me drop a message to Karol or Seth for help. Thanks. Best Regards Ziye Yang -----Original Message----- From: Richael Zhuang <Richael.Zhuang(a)arm.com> Sent: Friday, February 14, 2020 9:47 AM To: spdk(a)lists.01.org Subject: [SPDK] ask for help about SPDK CI Hi, I got one problem about the new CI system. The CI is not triggered after I submit a patch to review.spdk.io. So I'm wondering whether I miss something about the process to submit a patch? Best Regards, Richael IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

2 years, 6 months

3
2
0 / 0

To be Rescheduled: 2020 SPDK, PMDK and VTune™ Profiler US Summit

by Plichko, Vladimir

With the continued concern over the spread of the coronavirus and out of an abundance of caution for our partners & employees we will be rescheduling the SPDK, PMDK and VTune™ Profiler Summit for later in 2020, targeting the second half 2020. If you already registered, or made hotel reservations, we will take care of cancelling those (we will send an update on that later next week). Thank you for your understanding and let us know if you have any questions. Best regards, Vladimir Plichko

2 years, 7 months

1
0
0 / 0

Re: About "core_mask" for DPDK EAL

by Harris, James R

Hi Niu, The core mask is used to specify which cores will be used for polling threads managed by the SPDK/DPDK application framework. There really isn't a way to disable the core mask, but there are some ways to minimize its effects. I'm curious - in your environment, when do you create your DAOS threads? And you create them from the same thread that is calling spdk_env_init()? Note that the thread calling spdk_env_init() will be pinned to that single core you've specified in your core mask. And threads created after spdk_env_init() will inherit that affinity. You can call spdk_unaffinitize_thread() immediately after spdk_env_init() returns to eliminate that affinity. Could you try that and see if it helps when specifying core_mask 0x1 in your environment? -Jim On 2/27/20, 2:02 AM, "Niu, Yawei" <yawei.niu(a)intel.com> wrote: Hi, We (DAOS) used to call spdk_env_init() with default ‘core_mask’ “0x1” to initialize DPDK EAL, during recent tests, we realized that performance can be boosted a lot by changing the ‘core_mask’ to some other value like ‘0x8’, so I’m wondering if it because some sort of DPDK service thread was stealing CPU cycles from DAOS service thread, and when the “core_mask” is changed, the DPDK thread is happen to be created on some unused core? But I didn’t observe any suspicious thread by ‘ps’ command (with core_mask ‘0x1’ or ‘0x8’). So my question is what this ‘core_mask’ is used for? I suppose it’s not useful for our server (we have our own polling thread and user level scheduler), is there any way to disable it? What’s the best practice of this ‘core_mask’ configuration for SPDK apps? Thanks -Niu _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

2 years, 7 months

2
3
0 / 0

About "core_mask" for DPDK EAL

by Niu, Yawei

Hi, We (DAOS) used to call spdk_env_init() with default ‘core_mask’ “0x1” to initialize DPDK EAL, during recent tests, we realized that performance can be boosted a lot by changing the ‘core_mask’ to some other value like ‘0x8’, so I’m wondering if it because some sort of DPDK service thread was stealing CPU cycles from DAOS service thread, and when the “core_mask” is changed, the DPDK thread is happen to be created on some unused core? But I didn’t observe any suspicious thread by ‘ps’ command (with core_mask ‘0x1’ or ‘0x8’). So my question is what this ‘core_mask’ is used for? I suppose it’s not useful for our server (we have our own polling thread and user level scheduler), is there any way to disable it? What’s the best practice of this ‘core_mask’ configuration for SPDK apps? Thanks -Niu

2 years, 7 months

1
0
0 / 0

Performance of SPDK in BlobFS

by gaiaismus＠outlook.com

Hi All, I recently tested random 4K read based on BlobFS by single thread, but OPS is only 210K. What is the normal OPS under BlobFS？ If anyone has the answer, please share with me. Thanks!

2 years, 7 months

2
2
0 / 0

2022

2021

2020

2019

2018

2017

2016

SPDK February 2020