4:13 a.m.
Hi All,
Is there any way to directly use the memory allocated by the malloc()
function? The spdk_malloc() allocates a pinned memory buffer in huge
page, and the spdk_nvme_ns_cmd_write also uses the pinned memory for
write. However, if we allocate a memory buffer by malloc(), how could we
directly write data to NVMe device by SPDK without copying data to the
buffer allocated by spdk_malloc()?
Thanks,
Yue
4:46 a.m.
On Dec 12, 2016, at 9:13 PM, yzhu <yzhu(a)cs.fsu.edu> wrote:
Hi All,
Is there any way to directly use the memory allocated by the malloc() function? The
spdk_malloc() allocates a pinned memory buffer in huge page, and the
spdk_nvme_ns_cmd_write also uses the pinned memory for write. However, if we allocate a
memory buffer by malloc(), how could we directly write data to NVMe device by SPDK without
copying data to the buffer allocated by spdk_malloc()?
Hi Yue,
There is no safe way to use a malloc() buffer for DMA, since the buffer could be swapped
out at any time. mlock() can be used to ensure the buffer stays present in RAM, but it
does not guarantee the buffer will not get migrated to a different physical location.
mlock() also has a lot of software overhead and is not something you would want to call
for every I/O.
-Jim
3:49 p.m.
Hi Jim,
Thanks for your quick reply. So I was wondering if spdk_malloc() is the
only safe and correct way to allocate buffer for writing to NVMe device
via spdk_nvme_ns_cmd_write.
Previously, I tried to lock the buffer allocated by malloc() in RAM by
mlock(), and immediately write the buffer to NVMe device by
spdk_nvme_ns_cmd_write. However, the write failed because of the failure
of converting virtual address to physical address. The error was "could
not find 2MB vfn 0x11 in DPDK mem config". Based on this, I was also
wondering how to correctly write to NVMe device via spdk if we still
have to use the buffer allocated by malloc()?
Thank you,
Yue
On 2016-12-12 23:46, Harris, James R wrote:
> On Dec 12, 2016, at 9:13 PM, yzhu <yzhu(a)cs.fsu.edu> wrote:
>
> Hi All,
>
> Is there any way to directly use the memory allocated by the malloc()
> function? The spdk_malloc() allocates a pinned memory buffer in huge
> page, and the spdk_nvme_ns_cmd_write also uses the pinned memory for
> write. However, if we allocate a memory buffer by malloc(), how could
> we directly write data to NVMe device by SPDK without copying data to
> the buffer allocated by spdk_malloc()?
>
Hi Yue,
There is no safe way to use a malloc() buffer for DMA, since the
buffer could be swapped out at any time. mlock() can be used to
ensure the buffer stays present in RAM, but it does not guarantee the
buffer will not get migrated to a different physical location.
mlock() also has a lot of software overhead and is not something you
would want to call for every I/O.
-Jim
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
4:05 p.m.
On Dec 13, 2016, at 8:49 AM, yzhu <yzhu(a)cs.fsu.edu> wrote:
Hi Jim,
Thanks for your quick reply. So I was wondering if spdk_malloc() is the only safe and
correct way to allocate buffer for writing to NVMe device via spdk_nvme_ns_cmd_write.
spdk_malloc() is the best way to allocate the buffer for use with the SPDK NVMe API. One
alternative is to pass your own 2MB-aligned pinned memory buffer, consisting of >= 2MB
huge pages, to the spdk_vtophys_register() function - this will insert the correct vtophys
mappings into the SPDK userspace page table. But spdk_vtophys_register() will not work
with buffers allocated from standard malloc() which will not be 2MB aligned, pinned, nor
guaranteed to be allocated from huge pages.
Previously, I tried to lock the buffer allocated by malloc() in RAM by mlock(), and
immediately write the buffer to NVMe device by spdk_nvme_ns_cmd_write. However, the write
failed because of the failure of converting virtual address to physical address. The error
was "could not find 2MB vfn 0x11 in DPDK mem config". Based on this, I was also
wondering how to correctly write to NVMe device via spdk if we still have to use the
buffer allocated by malloc()?
SPDK is not designed to handle buffers allocated by standard malloc(). Would using glibc
malloc hooks or overriding malloc() such that malloc()/free() calls get forwarded to
spdk_malloc()/spdk_free() help in your situation?
Thank you,
Yue
On 2016-12-12 23:46, Harris, James R wrote:
>> On Dec 12, 2016, at 9:13 PM, yzhu <yzhu(a)cs.fsu.edu> wrote:
>> Hi All,
>> Is there any way to directly use the memory allocated by the malloc() function?
The spdk_malloc() allocates a pinned memory buffer in huge page, and the
spdk_nvme_ns_cmd_write also uses the pinned memory for write. However, if we allocate a
memory buffer by malloc(), how could we directly write data to NVMe device by SPDK without
copying data to the buffer allocated by spdk_malloc()?
> Hi Yue,
> There is no safe way to use a malloc() buffer for DMA, since the
> buffer could be swapped out at any time. mlock() can be used to
> ensure the buffer stays present in RAM, but it does not guarantee the
> buffer will not get migrated to a different physical location.
> mlock() also has a lot of software overhead and is not something you
> would want to call for every I/O.
> -Jim
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
2022
days inactive
2022
days old
3 comments
2 participants
participants (2)
-
Harris, James R -
yzhu