11:36 p.m.
Hi,
Basically all runtime operations should be executed via spdk_thread_poll()
During runtime, all one-time operations are requested by
spdk_thread_send_msg() and executed via spdk_thread_poll().
All periodic operations are executed as pollers. Each poller is registered
via spdk_poller_register(), and spdk_poller_register() is called in a
message handler, and the message is requested by spdk_thread_send_msg().
You read that at the first time call spdk_thread_send_msg in
spdk_app_start, the local thread is not a spdk thread.
This means the first message call is to execute the start function of the
first SPDK thread.
We don't specify any start function or main function when creating a SPDK
thread.
After the first SPDK thread processes the first sent message, it creates
another SPDK thread or a poller.
Hence the first SPDK thread can be executed via message or poller in
spdk_thread_poll().
If your g_master_thread does not use a reactor, you have to call
spdk_set_thread() by yourself before calling spdk_thread_send_msg().
Please take a look at example/nvmf/nvmf.c
Thanks,
Shuhei
On Thu, Sep 17, 2020 at 11:07 AM Sunshihao (Euler OS) <sunshihao(a)huawei.com>
wrote:
Dear maintainer Jim,
We have modified spdk for some requirement, our io workflow is :
Frist we do init wok:
Spdk Init : spdk_env_init-> spdk_reactors_init->
spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we
have modify code to make the g_masterThread never reactor run, just return
to do io submit job, so we execte every msg send to g_masterThread)
And then, construct io resource:
Io resource construct: get a spdk_thread point thread_1->
spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to
thread_n to get io channel)
After finish that, submit io to thread_1
Submit io: spdk_thread_send_msg(thread_n, LaunchIo, io);
LaunchIo will call bdev submit io function given by spdk;
Io is a struct have all args needed by bdev submit fuction.
We do init work one time and do io resource construct and submit work many
times for io read/write.
I have read the bdevperf code, and at the frist time call
spdk_thread_send_msg in spdk_app_start, the local thread is not a
spdk_thread, we get msg form pool; after that, in the reactor_run to call
spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache.
In our submit work, we only get the spdk_thread thread_1 to send msg; I
set the local thread to thread_1 and create two or more thread to do submit
work; so, it cause the problem when get msg.
Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n,
LaunchIo, io); the problem has been solved.
Above is what i used spdk to submit io, maybe I used wrong way somewhere,
please give me some advice.
Thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris@intel.com]
发送时间: 2020年9月17日 1:33
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: 答复: Re: thread->msg_cache is not safe when multiple threads
send messages to same spdk_thread
Hi,
Can you describe more about your application and when and where it submits
IO? Applications using the SPDK application framework (i.e.
spdk_app_start()) typically do not need to create their own spdk_threads -
they submit IO from the spdk_threads created by the reactors.
Thanks,
Jim
On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com>
wrote:
Dear maintainer Jim, thank you for your reply.
After reading your email, I know I have used the wrong way to
submiting io.
When I create a thread to submit io, I don't create spdk_thread
structure for the thread, this structure only be created in
spdk_reactors_start api for every lcore;
Must I create spdk_thread structure for every submit thread?
Please give me some advice, thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris@intel.com]
发送时间: 2020年9月16日 3:32
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads
send messages to same spdk_thread
Hi,
local_thread gets its value from _get_thread(). _get_thread() returns
the thread-local spdk_thread structure for the calling thread. So if
multiple threads call spdk_thread_send_msg() in parallel, they will each
use their own thread-local spdk_thread structure and no lock or
synchronization is needed.
-Jim
On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com>
wrote:
hello，maintainer：
when i use spdk api spdk_thread_send_msg to send io read/write
msg，i found a bug：
multiple threads send msg to one spdk_thread，they may get msg form
local_thread->msg_cache，there is not a lock or cas to
ensure the safety of the linked list。
msg = NULL;
if (local_thread != NULL) {
if (local_thread->msg_cache_count > 0) {
msg =
SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */
assert(msg != NULL);
SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
local_thread->msg_cache_count--;
}
}
i don't konw whether is my wrong way to use spdk_thread_send_msg
or the list is not safe，so please give me some advice。
Thank you！
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org