On Tue, Apr 28, 2015 at 11:24 AM, Dan Williams <dan.j.williams(a)intel.com&gt; wrote: I'd kind of like to make it more clear what's going on here. It doesn't help that the spec chose poor names. How about "NVDIMM physical aperture" for E820_PMEM and "legacy persistent RAM" for E820_PRAM? Otherwise this looks generaly sensible, although I don't really understand why e820_type_to_string and e820_print_type are different. --Andy

On Tue, Apr 28, 2015 at 1:57 PM, Dan Williams <dan.j.williams(a)intel.com&gt; wrote: Oh, I missed that. Yuck. What happens when you repartition one of these things? (Can you even do that?) Can we change that? -- Andy Lutomirski AMA Capital Management, LLC

On Fri, May 15, 2015 at 8:43 AM, Jeff Moyer <jmoyer(a)redhat.com&gt; wrote: Had we not polluted the waters with E820_PRAM I would have chosen "Persistent Memory", but as we now need to support both I'd assume they need different names. Thanks Jeff.

1/ Autodetect an NFIT table for the ACPI namespace device with _HID of "ACPI0012" 2/ libnd bus registration The NFIT provided by ACPI is one possible method by which platforms will discover NVDIMM resources. However, the intent of the nd_bus_descriptor abstraction is to abstract "provider" specific details, leaving libnd to be independent of the specific NVDIMM resource discovery mechanism. This flexibility is later exploited later to implement custom-defined nd buses. Cc: <linux-acpi(a)vger.kernel.org&gt; Cc: Robert Moore <robert.moore(a)intel.com&gt; Cc: Rafael J. Wysocki <rafael.j.wysocki(a)intel.com&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/Kconfig | 2 drivers/block/Makefile | 1 drivers/block/nd/Kconfig | 40 +++ drivers/block/nd/Makefile | 6 + drivers/block/nd/acpi.c | 475 +++++++++++++++++++++++++++++++++++++++++ drivers/block/nd/acpi_nfit.h | 254 ++++++++++++++++++++++ drivers/block/nd/core.c | 67 ++++++ drivers/block/nd/libnd.h | 33 +++ drivers/block/nd/nd-private.h | 23 ++ 9 files changed, 901 insertions(+) create mode 100644 drivers/block/nd/Kconfig create mode 100644 drivers/block/nd/Makefile create mode 100644 drivers/block/nd/acpi.c create mode 100644 drivers/block/nd/acpi_nfit.h create mode 100644 drivers/block/nd/core.c create mode 100644 drivers/block/nd/libnd.h create mode 100644 drivers/block/nd/nd-private.h diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig index eb1fed5bd516..dfe40e5ca9bd 100644 --- a/drivers/block/Kconfig +++ b/drivers/block/Kconfig @@ -321,6 +321,8 @@ config BLK_DEV_NVME To compile this driver as a module, choose M here: the module will be called nvme. +source "drivers/block/nd/Kconfig" + config BLK_DEV_SKD tristate "STEC S1120 Block Driver" depends on PCI diff --git a/drivers/block/Makefile b/drivers/block/Makefile index 9cc6c18a1c7e..07a6acecf4d8 100644 --- a/drivers/block/Makefile +++ b/drivers/block/Makefile @@ -24,6 +24,7 @@ obj-$(CONFIG_CDROM_PKTCDVD) += pktcdvd.o obj-$(CONFIG_MG_DISK) += mg_disk.o obj-$(CONFIG_SUNVDC) += sunvdc.o obj-$(CONFIG_BLK_DEV_NVME) += nvme.o +obj-$(CONFIG_ND_DEVICES) += nd/ obj-$(CONFIG_BLK_DEV_SKD) += skd.o obj-$(CONFIG_BLK_DEV_OSD) += osdblk.o diff --git a/drivers/block/nd/Kconfig b/drivers/block/nd/Kconfig new file mode 100644 index 000000000000..6d5d6b732f82 --- /dev/null +++ b/drivers/block/nd/Kconfig @@ -0,0 +1,40 @@ +menuconfig ND_DEVICES + bool "NVDIMM Support" + depends on PHYS_ADDR_T_64BIT + help + Generic support for non-volatile memory devices including + ACPI-6-NFIT defined resources. On platforms that define an + NFIT, or otherwise can discover NVDIMM resources, a libnd + bus is registered to advertise PMEM (persistent memory) + namespaces (/dev/pmemX) and BLK (sliding mmio window(s)) + namespaces (/dev/ndX). A PMEM namespace refers to a memory + resource that may span multiple DIMMs and support DAX (see + CONFIG_DAX). A BLK namespace refers to an NVDIMM control + region which exposes an mmio register set for windowed + access mode to non-volatile memory. + +if ND_DEVICES + +config LIBND + tristate "LIBND: libnd device driver support" + help + Platform agnostic device model for a libnd bus. Publishes + resources for a PMEM (persistent-memory) driver and/or BLK + (sliding mmio window(s)) driver to attach. Exposes a device + topology under a "ndX" bus device, a "/dev/ndctlX" bus-ioctl + message passing interface, and a "/dev/nmemX" dimm-ioctl + message interface for each memory device registered on the + bus. instance. A userspace library "ndctl" provides an API + to enumerate/manage this subsystem. + +config ND_ACPI + tristate "ACPI: NFIT to libnd bus support" + select LIBND + depends on ACPI + help + Infrastructure to probe ACPI 6 compliant platforms for + NVDIMMs (NFIT) and register a libnd device tree. In + addition to storage devices this also enables libnd craft + ACPI._DSM messages for platform/dimm configuration. + +endif diff --git a/drivers/block/nd/Makefile b/drivers/block/nd/Makefile new file mode 100644 index 000000000000..944b5947c0cb --- /dev/null +++ b/drivers/block/nd/Makefile @@ -0,0 +1,6 @@ +obj-$(CONFIG_LIBND) += libnd.o +obj-$(CONFIG_ND_ACPI) += nd_acpi.o + +nd_acpi-y := acpi.o + +libnd-y := core.o diff --git a/drivers/block/nd/acpi.c b/drivers/block/nd/acpi.c new file mode 100644 index 000000000000..9f0b24390d1b --- /dev/null +++ b/drivers/block/nd/acpi.c @@ -0,0 +1,475 @@ +/* + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#include <linux/list_sort.h> +#include <linux/module.h> +#include <linux/list.h> +#include <linux/acpi.h> +#include "acpi_nfit.h" +#include "libnd.h" + +static bool warn_checksum; +module_param(warn_checksum, bool, S_IRUGO|S_IWUSR); +MODULE_PARM_DESC(warn_checksum, "Turn checksum errors into warnings"); + +enum { + NFIT_ACPI_NOTIFY_TABLE = 0x80, +}; + +static int nd_acpi_ctl(struct nd_bus_descriptor *nd_desc, + struct nd_dimm *nd_dimm, unsigned int cmd, void *buf, + unsigned int buf_len) +{ + return -ENOTTY; +} + +static const char *spa_type_name(u16 type) +{ + switch (type) { + case NFIT_SPA_VOLATILE: return "volatile"; + case NFIT_SPA_PM: return "pmem"; + case NFIT_SPA_DCR: return "dimm-control-region"; + case NFIT_SPA_BDW: return "block-data-window"; + default: return "unknown"; + } +} + +static int nfit_spa_type(struct acpi_nfit_spa *spa) +{ + if (memcmp(&nfit_spa_uuid_volatile, spa->type_uuid, 16) == 0) + return NFIT_SPA_VOLATILE; + + if (memcmp(&nfit_spa_uuid_pm, spa->type_uuid, 16) == 0) + return NFIT_SPA_PM; + + if (memcmp(&nfit_spa_uuid_dcr, spa->type_uuid, 16) == 0) + return NFIT_SPA_DCR; + + if (memcmp(&nfit_spa_uuid_bdw, spa->type_uuid, 16) == 0) + return NFIT_SPA_BDW; + + if (memcmp(&nfit_spa_uuid_vdisk, spa->type_uuid, 16) == 0) + return NFIT_SPA_VDISK; + + if (memcmp(&nfit_spa_uuid_vcd, spa->type_uuid, 16) == 0) + return NFIT_SPA_VCD; + + if (memcmp(&nfit_spa_uuid_pdisk, spa->type_uuid, 16) == 0) + return NFIT_SPA_PDISK; + + if (memcmp(&nfit_spa_uuid_pcd, spa->type_uuid, 16) == 0) + return NFIT_SPA_PCD; + + return -1; +} + +struct nfit_table_header { + __le16 type; + __le16 length; +}; + +static void *add_table(struct acpi_nfit_desc *acpi_desc, void *table, const void *end) +{ + struct device *dev = acpi_desc->dev; + struct nfit_table_header *hdr; + void *err = ERR_PTR(-ENOMEM); + + if (table >= end) + return NULL; + + hdr = (struct nfit_table_header *) table; + switch (hdr->type) { + case NFIT_TABLE_SPA: { + struct nfit_spa *nfit_spa = devm_kzalloc(dev, sizeof(*nfit_spa), + GFP_KERNEL); + struct acpi_nfit_spa *spa = table; + + if (!nfit_spa) + return err; + INIT_LIST_HEAD(&nfit_spa->list); + nfit_spa->spa = spa; + list_add_tail(&nfit_spa->list, &acpi_desc->spas); + dev_dbg(dev, "%s: spa index: %d type: %s\n", __func__, + spa->spa_index, + spa_type_name(nfit_spa_type(spa))); + break; + } + case NFIT_TABLE_MEM: { + struct nfit_memdev *nfit_memdev = devm_kzalloc(dev, + sizeof(*nfit_memdev), GFP_KERNEL); + struct acpi_nfit_memdev *memdev = table; + + if (!nfit_memdev) + return err; + INIT_LIST_HEAD(&nfit_memdev->list); + nfit_memdev->memdev = memdev; + list_add_tail(&nfit_memdev->list, &acpi_desc->memdevs); + dev_dbg(dev, "%s: memdev handle: %#x spa: %d dcr: %d\n", + __func__, memdev->nfit_handle, memdev->spa_index, + memdev->dcr_index); + break; + } + case NFIT_TABLE_DCR: { + struct nfit_dcr *nfit_dcr = devm_kzalloc(dev, sizeof(*nfit_dcr), + GFP_KERNEL); + struct acpi_nfit_dcr *dcr = table; + + if (!nfit_dcr) + return err; + INIT_LIST_HEAD(&nfit_dcr->list); + nfit_dcr->dcr = dcr; + list_add_tail(&nfit_dcr->list, &acpi_desc->dcrs); + dev_dbg(dev, "%s: dcr index: %d num_bcw: %d\n", __func__, + dcr->dcr_index, dcr->num_bcw); + break; + } + case NFIT_TABLE_BDW: { + struct nfit_bdw *nfit_bdw = devm_kzalloc(dev, sizeof(*nfit_bdw), + GFP_KERNEL); + struct acpi_nfit_bdw *bdw = table; + + if (!nfit_bdw) + return err; + INIT_LIST_HEAD(&nfit_bdw->list); + nfit_bdw->bdw = bdw; + list_add_tail(&nfit_bdw->list, &acpi_desc->bdws); + dev_dbg(dev, "%s: bdw dcr: %d num_bdw: %d\n", __func__, + bdw->dcr_index, bdw->num_bdw); + break; + } + /* TODO */ + case NFIT_TABLE_IDT: + dev_dbg(dev, "%s: idt\n", __func__); + break; + case NFIT_TABLE_FLUSH: + dev_dbg(dev, "%s: flush\n", __func__); + break; + case NFIT_TABLE_SMBIOS: + dev_dbg(dev, "%s: smbios\n", __func__); + break; + default: + dev_err(dev, "unknown table '%d' parsing nfit\n", hdr->type); + return ERR_PTR(-ENXIO); + } + + return table + hdr->length; +} + +static void nfit_mem_find_spa_bdw(struct acpi_nfit_desc *acpi_desc, + struct nfit_mem *nfit_mem) +{ + u32 nfit_handle = __to_nfit_memdev(nfit_mem)->nfit_handle; + u16 dcr_index = nfit_mem->dcr->dcr_index; + struct nfit_spa *nfit_spa; + + list_for_each_entry(nfit_spa, &acpi_desc->spas, list) { + u16 spa_index = nfit_spa->spa->spa_index; + int type = nfit_spa_type(nfit_spa->spa); + struct nfit_memdev *nfit_memdev; + + if (type != NFIT_SPA_BDW) + continue; + + list_for_each_entry(nfit_memdev, &acpi_desc->memdevs, list) { + if (nfit_memdev->memdev->spa_index != spa_index) + continue; + if (nfit_memdev->memdev->nfit_handle != nfit_handle) + continue; + if (nfit_memdev->memdev->dcr_index != dcr_index) + continue; + + nfit_mem->spa_bdw = nfit_spa->spa; + return; + } + } + + dev_dbg(acpi_desc->dev, "SPA-BDW not found for SPA-DCR %d\n", + nfit_mem->spa_dcr->spa_index); + nfit_mem->bdw = NULL; +} + +static int nfit_mem_add(struct acpi_nfit_desc *acpi_desc, + struct nfit_mem *nfit_mem, struct acpi_nfit_spa *spa) +{ + u16 dcr_index = __to_nfit_memdev(nfit_mem)->dcr_index; + struct nfit_dcr *nfit_dcr; + struct nfit_bdw *nfit_bdw; + + list_for_each_entry(nfit_dcr, &acpi_desc->dcrs, list) { + if (nfit_dcr->dcr->dcr_index != dcr_index) + continue; + nfit_mem->dcr = nfit_dcr->dcr; + break; + } + + if (!nfit_mem->dcr) { + dev_dbg(acpi_desc->dev, "SPA %d missing:%s%s\n", spa->spa_index, + __to_nfit_memdev(nfit_mem) ? "" : " MEMDEV", + nfit_mem->dcr ? "" : " DCR"); + return -ENODEV; + } + + /* + * We've found enough to create an nd_dimm, optionally + * find an associated BDW + */ + list_add(&nfit_mem->list, &acpi_desc->dimms); + + list_for_each_entry(nfit_bdw, &acpi_desc->bdws, list) { + if (nfit_bdw->bdw->dcr_index != dcr_index) + continue; + nfit_mem->bdw = nfit_bdw->bdw; + break; + } + + if (!nfit_mem->bdw) + return 0; + + nfit_mem_find_spa_bdw(acpi_desc, nfit_mem); + return 0; +} + +static int nfit_mem_dcr_init(struct acpi_nfit_desc *acpi_desc, + struct acpi_nfit_spa *spa) +{ + struct nfit_mem *nfit_mem, *found; + struct nfit_memdev *nfit_memdev; + int type = nfit_spa_type(spa); + u16 dcr_index; + + switch (type) { + case NFIT_SPA_DCR: + case NFIT_SPA_PM: + break; + default: + return 0; + } + + list_for_each_entry(nfit_memdev, &acpi_desc->memdevs, list) { + int rc; + + if (nfit_memdev->memdev->spa_index != spa->spa_index) + continue; + found = NULL; + dcr_index = nfit_memdev->memdev->dcr_index; + list_for_each_entry(nfit_mem, &acpi_desc->dimms, list) + if (__to_nfit_memdev(nfit_mem)->dcr_index == dcr_index) { + found = nfit_mem; + break; + } + + if (found) + nfit_mem = found; + else { + nfit_mem = devm_kzalloc(acpi_desc->dev, + sizeof(*nfit_mem), GFP_KERNEL); + if (!nfit_mem) + return -ENOMEM; + INIT_LIST_HEAD(&nfit_mem->list); + } + + if (type == NFIT_SPA_DCR) { + /* multiple dimms may share a SPA when interleaved */ + nfit_mem->spa_dcr = spa; + nfit_mem->memdev_dcr = nfit_memdev->memdev; + } else { + /* + * A single dimm may belong to multiple SPA-PM + * ranges, record at least one in addition to + * any SPA-DCR range. + */ + nfit_mem->memdev_pmem = nfit_memdev->memdev; + } + + if (found) + continue; + + rc = nfit_mem_add(acpi_desc, nfit_mem, spa); + if (rc) + return rc; + } + + return 0; +} + +static int nfit_mem_cmp(void *priv, struct list_head *__a, struct list_head *__b) +{ + struct nfit_mem *a = container_of(__a, typeof(*a), list); + struct nfit_mem *b = container_of(__b, typeof(*b), list); + u32 handleA, handleB; + + handleA = __to_nfit_memdev(a)->nfit_handle; + handleB = __to_nfit_memdev(b)->nfit_handle; + if (handleA < handleB) + return -1; + else if (handleA > handleB) + return 1; + return 0; +} + +static int nfit_mem_init(struct acpi_nfit_desc *acpi_desc) +{ + struct nfit_spa *nfit_spa; + + /* + * For each SPA-DCR or SPA-PMEM address range find its + * corresponding MEMDEV(s). From each MEMDEV find the + * corresponding DCR. Then, if we're operating on a SPA-DCR, + * try to find a SPA-BDW and a corresponding BDW that references + * the DCR. Throw it all into an nfit_mem object. Note, that + * BDWs are optional. + */ + list_for_each_entry(nfit_spa, &acpi_desc->spas, list) { + int rc; + + rc = nfit_mem_dcr_init(acpi_desc, nfit_spa->spa); + if (rc) + return rc; + } + + list_sort(NULL, &acpi_desc->dimms, nfit_mem_cmp); + + return 0; +} + +static int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) +{ + struct device *dev = acpi_desc->dev; + const void *end; + u8 *data, sum; + acpi_size i; + + INIT_LIST_HEAD(&acpi_desc->spas); + INIT_LIST_HEAD(&acpi_desc->dcrs); + INIT_LIST_HEAD(&acpi_desc->bdws); + INIT_LIST_HEAD(&acpi_desc->memdevs); + INIT_LIST_HEAD(&acpi_desc->dimms); + + data = (u8 *) acpi_desc->nfit; + for (i = 0, sum = 0; i < sz; i++) + sum += readb(data + i); + if (sum != 0 && !warn_checksum) { + dev_dbg(dev, "%s: nfit checksum failure\n", __func__); + return -ENXIO; + } + WARN_TAINT_ONCE(sum != 0, TAINT_FIRMWARE_WORKAROUND, + "nfit checksum failure, continuing...\n"); + + end = data + sz; + data += sizeof(struct acpi_nfit); + while (!IS_ERR_OR_NULL(data)) + data = add_table(acpi_desc, data, end); + + if (IS_ERR(data)) { + dev_dbg(dev, "%s: nfit table parsing error: %ld\n", __func__, + PTR_ERR(data)); + return PTR_ERR(data); + } + + if (nfit_mem_init(acpi_desc) != 0) + return -ENOMEM; + + return 0; +} + +static int nd_acpi_add(struct acpi_device *adev) +{ + struct nd_bus_descriptor *nd_desc; + struct acpi_nfit_desc *acpi_desc; + struct device *dev = &adev->dev; + struct acpi_table_header *tbl; + acpi_status status = AE_OK; + acpi_size sz; + int rc; + + status = acpi_get_table_with_size("NFIT", 0, &tbl, &sz); + if (ACPI_FAILURE(status)) { + dev_err(dev, "failed to find NFIT\n"); + return -ENXIO; + } + + acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL); + if (!acpi_desc) + return -ENOMEM; + + dev_set_drvdata(dev, acpi_desc); + acpi_desc->dev = dev; + acpi_desc->nfit = (struct acpi_nfit *) tbl; + nd_desc = &acpi_desc->nd_desc; + nd_desc->provider_name = "ACPI.NFIT"; + nd_desc->ndctl = nd_acpi_ctl; + + acpi_desc->nd_bus = nd_bus_register(dev, nd_desc); + if (!acpi_desc->nd_bus) + return -ENXIO; + + rc = nd_acpi_nfit_init(acpi_desc, sz); + if (rc) { + nd_bus_unregister(acpi_desc->nd_bus); + return rc; + } + return 0; +} + +static int nd_acpi_remove(struct acpi_device *adev) +{ + struct acpi_nfit_desc *acpi_desc = dev_get_drvdata(&adev->dev); + + nd_bus_unregister(acpi_desc->nd_bus); + return 0; +} + +static void nd_acpi_notify(struct acpi_device *adev, u32 event) +{ + /* TODO: handle ACPI_NOTIFY_BUS_CHECK notification */ + dev_dbg(&adev->dev, "%s: event: %d\n", __func__, event); +} + +static const struct acpi_device_id nd_acpi_ids[] = { + { "ACPI0012", 0 }, + { "", 0 }, +}; +MODULE_DEVICE_TABLE(acpi, nd_acpi_ids); + +static struct acpi_driver nd_acpi_driver = { + .name = KBUILD_MODNAME, + .ids = nd_acpi_ids, + .flags = ACPI_DRIVER_ALL_NOTIFY_EVENTS, + .ops = { + .add = nd_acpi_add, + .remove = nd_acpi_remove, + .notify = nd_acpi_notify + }, +}; + +static __init int nd_acpi_init(void) +{ + BUILD_BUG_ON(sizeof(struct acpi_nfit) != 40); + BUILD_BUG_ON(sizeof(struct acpi_nfit_spa) != 56); + BUILD_BUG_ON(sizeof(struct acpi_nfit_memdev) != 48); + BUILD_BUG_ON(sizeof(struct acpi_nfit_idt) != 16); + BUILD_BUG_ON(sizeof(struct acpi_nfit_smbios) != 8); + BUILD_BUG_ON(sizeof(struct acpi_nfit_dcr) != 80); + BUILD_BUG_ON(sizeof(struct acpi_nfit_bdw) != 40); + + return acpi_bus_register_driver(&nd_acpi_driver); +} + +static __exit void nd_acpi_exit(void) +{ + acpi_bus_unregister_driver(&nd_acpi_driver); +} + +module_init(nd_acpi_init); +module_exit(nd_acpi_exit); +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Intel Corporation"); diff --git a/drivers/block/nd/acpi_nfit.h b/drivers/block/nd/acpi_nfit.h new file mode 100644 index 000000000000..e0b0f12736bf --- /dev/null +++ b/drivers/block/nd/acpi_nfit.h @@ -0,0 +1,254 @@ +/* + * NVDIMM Firmware Interface Table - NFIT + * + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#ifndef __NFIT_H__ +#define __NFIT_H__ +#include <linux/types.h> +#include <linux/uuid.h> +#include <linux/acpi.h> +#include "libnd.h" + +static const uuid_le nfit_spa_uuid_volatile __maybe_unused = UUID_LE(0x7305944f, + 0xfdda, 0x44e3, 0xb1, 0x6c, 0x3f, 0x22, 0xd2, 0x52, 0xe5, 0xd0); + +static const uuid_le nfit_spa_uuid_pm __maybe_unused = UUID_LE(0x66f0d379, + 0xb4f3, 0x4074, 0xac, 0x43, 0x0d, 0x33, 0x18, 0xb7, 0x8c, 0xdb); + +static const uuid_le nfit_spa_uuid_dcr __maybe_unused = UUID_LE(0x92f701f6, + 0x13b4, 0x405d, 0x91, 0x0b, 0x29, 0x93, 0x67, 0xe8, 0x23, 0x4c); + +static const uuid_le nfit_spa_uuid_bdw __maybe_unused = UUID_LE(0x91af0530, + 0x5d86, 0x470e, 0xa6, 0xb0, 0x0a, 0x2d, 0xb9, 0x40, 0x82, 0x49); + +static const uuid_le nfit_spa_uuid_vdisk __maybe_unused = UUID_LE(0x77ab535a, + 0x45fc, 0x624b, 0x55, 0x60, 0xf7, 0xb2, 0x81, 0xd1, 0xf9, 0x6e); + +static const uuid_le nfit_spa_uuid_vcd __maybe_unused = UUID_LE(0x3d5abd30, + 0x4175, 0x87ce, 0x6d, 0x64, 0xd2, 0xad, 0xe5, 0x23, 0xc4, 0xbb); + +static const uuid_le nfit_spa_uuid_pdisk __maybe_unused = UUID_LE(0x5cea02c9, + 0x4d07, 0x69d3, 0x26, 0x9f, 0x44, 0x96, 0xfb, 0xe0, 0x96, 0xf9); + +static const uuid_le nfit_spa_uuid_pcd __maybe_unused = UUID_LE(0x08018188, + 0x42cd, 0xbb48, 0x10, 0x0f, 0x53, 0x87, 0xd5, 0x3d, 0xed, 0x3d); + +enum { + NFIT_TABLE_SPA = 0, + NFIT_TABLE_MEM = 1, + NFIT_TABLE_IDT = 2, + NFIT_TABLE_SMBIOS = 3, + NFIT_TABLE_DCR = 4, + NFIT_TABLE_BDW = 5, + NFIT_TABLE_FLUSH = 6, + NFIT_SPA_VOLATILE = 0, + NFIT_SPA_PM = 1, + NFIT_SPA_DCR = 2, + NFIT_SPA_BDW = 3, + NFIT_SPA_VDISK = 4, + NFIT_SPA_VCD = 5, + NFIT_SPA_PDISK = 6, + NFIT_SPA_PCD = 7, + NFIT_SPAF_DCR_HOT_ADD = 1 << 0, + NFIT_SPAF_PDVALID = 1 << 1, + NFIT_MEMF_SAVE_FAIL = 1 << 0, + NFIT_MEMF_RESTORE_FAIL = 1 << 1, + NFIT_MEMF_FLUSH_FAIL = 1 << 2, + NFIT_MEMF_UNARMED = 1 << 3, + NFIT_MEMF_NOTIFY_SMART = 1 << 4, + NFIT_MEMF_SMART_READY = 1 << 5, + NFIT_DCRF_BUFFERED = 1 << 0, +}; + +/** + * struct acpi_nfit - Nvdimm Firmware Interface Table + * @signature: "NFIT" + * @length: sum of size of this table plus all appended subtables + */ +struct acpi_nfit { + u8 signature[4]; + u32 length; + u8 revision; + u8 checksum; + u8 oemid[6]; + u64 oem_tbl_id; + u32 oem_revision; + u32 creator_id; + u32 creator_revision; + u32 reserved; +}; + +/** + * struct acpi_nfit_spa - System Physical Address Range Descriptor Table + */ +struct acpi_nfit_spa { + u16 type; + u16 length; + u16 spa_index; + u16 flags; + u32 reserved; + u32 proximity_domain; + u8 type_uuid[16]; + u64 spa_base; + u64 spa_length; + u64 mem_attr; +}; + +/** + * struct acpi_nfit_mem - Memory Device to SPA Mapping Table + */ +struct acpi_nfit_memdev { + u16 type; + u16 length; + u32 nfit_handle; + u16 phys_id; + u16 region_id; + u16 spa_index; + u16 dcr_index; + u64 region_len; + u64 region_spa_offset; + u64 region_dpa; + u16 idt_index; + u16 interleave_ways; + u16 flags; + u16 reserved; +}; + +/** + * struct acpi_nfit_idt - Interleave description Table + */ +struct acpi_nfit_idt { + u16 type; + u16 length; + u16 idt_index; + u16 reserved; + u32 num_lines; + u32 line_size; + u32 line_offset[0]; +}; + +/** + * struct acpi_nfit_smbios - SMBIOS Management Information Table + */ +struct acpi_nfit_smbios { + u16 type; + u16 length; + u32 reserved; + u8 data[0]; +}; + +/** + * struct acpi_nfit_dcr - NVDIMM Control Region Table + * @fic: Format Interface Code + * @cmd_offset: command registers relative to block control window + * @status_offset: status registers relative to block control window + */ +struct acpi_nfit_dcr { + u16 type; + u16 length; + u16 dcr_index; + u16 vendor_id; + u16 device_id; + u16 revision_id; + u16 sub_vendor_id; + u16 sub_device_id; + u16 sub_revision_id; + u8 reserved[6]; + u32 serial_number; + u16 fic; + u16 num_bcw; + u64 bcw_size; + u64 cmd_offset; + u64 cmd_size; + u64 status_offset; + u64 status_size; + u16 flags; + u8 reserved2[6]; +}; + +/** + * struct acpi_nfit_bdw - NVDIMM Block Data Window Region Table + */ +struct acpi_nfit_bdw { + u16 type; + u16 length; + u16 dcr_index; + u16 num_bdw; + u64 bdw_offset; + u64 bdw_size; + u64 blk_capacity; + u64 blk_offset; +}; + +/** + * struct acpi_nfit_flush - Flush Hint Address Structure + */ +struct acpi_nfit_flush { + u16 type; + u16 length; + u32 nfit_handle; + u16 num_hints; + u8 reserved[6]; + u64 hint_addr[0]; +}; + +struct nfit_spa { + struct acpi_nfit_spa *spa; + struct list_head list; +}; + +struct nfit_dcr { + struct acpi_nfit_dcr *dcr; + struct list_head list; +}; + +struct nfit_bdw { + struct acpi_nfit_bdw *bdw; + struct list_head list; +}; + +struct nfit_memdev { + struct acpi_nfit_memdev *memdev; + struct list_head list; +}; + +/* assembled tables for a given dimm/memory-device */ +struct nfit_mem { + struct acpi_nfit_memdev *memdev_dcr; + struct acpi_nfit_memdev *memdev_pmem; + struct acpi_nfit_dcr *dcr; + struct acpi_nfit_bdw *bdw; + struct acpi_nfit_spa *spa_dcr; + struct acpi_nfit_spa *spa_bdw; + struct list_head list; +}; + +struct acpi_nfit_desc { + struct nd_bus_descriptor nd_desc; + struct acpi_nfit *nfit; + struct list_head memdevs; + struct list_head dimms; + struct list_head spas; + struct list_head dcrs; + struct list_head bdws; + struct nd_bus *nd_bus; + struct device *dev; +}; + +static inline struct acpi_nfit_memdev *__to_nfit_memdev(struct nfit_mem *nfit_mem) +{ + if (nfit_mem->memdev_dcr) + return nfit_mem->memdev_dcr; + return nfit_mem->memdev_pmem; +} +#endif /* __NFIT_H__ */ diff --git a/drivers/block/nd/core.c b/drivers/block/nd/core.c new file mode 100644 index 000000000000..3cccdbc0f3b7 --- /dev/null +++ b/drivers/block/nd/core.c @@ -0,0 +1,67 @@ +/* + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#include <linux/export.h> +#include <linux/module.h> +#include <linux/device.h> +#include <linux/slab.h> +#include "nd-private.h" +#include "libnd.h" + +static DEFINE_IDA(nd_ida); + +static void nd_bus_release(struct device *dev) +{ + struct nd_bus *nd_bus = container_of(dev, struct nd_bus, dev); + + ida_simple_remove(&nd_ida, nd_bus->id); + kfree(nd_bus); +} + +struct nd_bus *nd_bus_register(struct device *parent, + struct nd_bus_descriptor *nd_desc) +{ + struct nd_bus *nd_bus = kzalloc(sizeof(*nd_bus), GFP_KERNEL); + int rc; + + if (!nd_bus) + return NULL; + nd_bus->id = ida_simple_get(&nd_ida, 0, 0, GFP_KERNEL); + if (nd_bus->id < 0) { + kfree(nd_bus); + return NULL; + } + nd_bus->nd_desc = nd_desc; + nd_bus->dev.parent = parent; + nd_bus->dev.release = nd_bus_release; + dev_set_name(&nd_bus->dev, "ndbus%d", nd_bus->id); + rc = device_register(&nd_bus->dev); + if (rc) { + dev_dbg(&nd_bus->dev, "device registration failed: %d\n", rc); + put_device(&nd_bus->dev); + return NULL; + } + + return nd_bus; +} +EXPORT_SYMBOL_GPL(nd_bus_register); + +void nd_bus_unregister(struct nd_bus *nd_bus) +{ + if (!nd_bus) + return; + device_unregister(&nd_bus->dev); +} +EXPORT_SYMBOL_GPL(nd_bus_unregister); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Intel Corporation"); diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h new file mode 100644 index 000000000000..163832937e9c --- /dev/null +++ b/drivers/block/nd/libnd.h @@ -0,0 +1,33 @@ +/* + * libnd - Non-volatile-memory Devices Subsystem + * + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#ifndef __LIBND_H__ +#define __LIBND_H__ +struct nd_dimm; +struct nd_bus_descriptor; +typedef int (*ndctl_fn)(struct nd_bus_descriptor *nd_desc, + struct nd_dimm *nd_dimm, unsigned int cmd, void *buf, + unsigned int buf_len); + +struct nd_bus_descriptor { + unsigned long dsm_mask; + char *provider_name; + ndctl_fn ndctl; +}; + +struct nd_bus; +struct nd_bus *nd_bus_register(struct device *parent, + struct nd_bus_descriptor *nfit_desc); +void nd_bus_unregister(struct nd_bus *nd_bus); +#endif /* __LIBND_H__ */ diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h new file mode 100644 index 000000000000..3dbab29fa0f9 --- /dev/null +++ b/drivers/block/nd/nd-private.h @@ -0,0 +1,23 @@ +/* + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#ifndef __ND_PRIVATE_H__ +#define __ND_PRIVATE_H__ +#include <linux/device.h> +#include "libnd.h" + +struct nd_bus { + struct nd_bus_descriptor *nd_desc; + struct device dev; + int id; +}; +#endif /* __ND_PRIVATE_H__ */

On Thu, Apr 30, 2015 at 4:23 PM, Rafael J. Wysocki <rjw(a)rjwysocki.net&gt; wrote: Later on in the series we introduce ND_E820 which supports creating a libnd-bus from e820-type-12 memory ranges on pre-NFIT systems. I'm also considering a configfs defined libnd-bus because e820 types are not nearly enough information to safely define nvdimm resources outside of NFIT. Um, no, both CONFIG_ND_ACPI and CONFIG_LIBND can be =m. Yes, I should have deferred this to the patch that fills in nd_acpi_ctl later in the series. Having this stub here in this patch just trips up reviewers, sorry. Yes. Yes, for future-proofing we should just ignore tables that we don't understand. Will fix. No, you misunderstood, this will certainly be modular and loaded on-demand. Yes, as soon as definitions those are available we will drop this header and rebase on the ACPICA implementation. [..]

On Thu, Apr 30, 2015 at 6:21 PM, Rafael J. Wysocki <rjw(a)rjwysocki.net&gt; wrote: [..] You can have ND_E820 support and ND_ACPI support in the same system. Likely an NFIT enabled system will never have e820-type-12 ranges, but if a user messes up and uses the new memmap=ss!nn command line to overlap NFIT-defined memory then the request_mem_region() calls in the driver will collide. First to load wins in that scenario. I agree that if the user selects ND_DEVICES then ND_ACPI should probably default on, but otherwise turning it off is a useful option. If you know your system is pre-ACPI-6 then why bother including support? The .notify handler will eventually be filled in to handle hot-add of NFIT structures, but yes I'll drop it for now.

On Fri, May 15, 2015 at 12:44 PM, Jeff Moyer <jmoyer(a)redhat.com&gt; wrote: Yes, but I've deleted it in the next rev of the code since the ACPI core will have already done the checksum. The driver need not implement it's own checksum. Yes. I went ahead and deleted all the ACPI notification infrastructure until we're ready to implement hot-add. Thanks Jeff.

This is the position (device topology) independent method to find all the libnd buses in the system. The expectation is that there will only ever be one "nd" bus discovered via /sys/class/nd/ndctl0. However, we allow for the possibility of multiple buses and they will listed in discovery order as ndctl0...ndctlN. This character device hosts the ioctl for passing control messages (inspired by the ACPI-NFIT DSM interface commands). Note, nd_ioctl() and the backing ->ndctl() implementation are defined in a subsequent patch. Cc: Neil Brown <neilb(a)suse.de&gt; Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; Cc: <linux-acpi(a)vger.kernel.org&gt; Cc: Robert Moore <robert.moore(a)intel.com&gt; Cc: Rafael J. Wysocki <rafael.j.wysocki(a)intel.com&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/Makefile | 1 drivers/block/nd/acpi.c | 29 ++++++++++++++ drivers/block/nd/acpi_nfit.h | 5 ++ drivers/block/nd/bus.c | 83 +++++++++++++++++++++++++++++++++++++++ drivers/block/nd/core.c | 87 ++++++++++++++++++++++++++++++++++++++++- drivers/block/nd/libnd.h | 5 ++ drivers/block/nd/nd-private.h | 6 +++ drivers/block/nd/test/nfit.c | 3 + 8 files changed, 217 insertions(+), 2 deletions(-) create mode 100644 drivers/block/nd/bus.c diff --git a/drivers/block/nd/Makefile b/drivers/block/nd/Makefile index cf064db92589..7defe18ed009 100644 --- a/drivers/block/nd/Makefile +++ b/drivers/block/nd/Makefile @@ -19,3 +19,4 @@ obj-$(CONFIG_NFIT_TEST) += test/ nd_acpi-y := acpi.o libnd-y := core.o +libnd-y += bus.o diff --git a/drivers/block/nd/acpi.c b/drivers/block/nd/acpi.c index 54344ef9c837..dd8505f766ed 100644 --- a/drivers/block/nd/acpi.c +++ b/drivers/block/nd/acpi.c @@ -341,6 +341,34 @@ static int nfit_mem_init(struct acpi_nfit_desc *acpi_desc) return 0; } +static ssize_t revision_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_bus *nd_bus = to_nd_bus(dev); + struct nd_bus_descriptor *nd_desc = to_nd_desc(nd_bus); + struct acpi_nfit_desc *acpi_desc = to_acpi_desc(nd_desc); + + return sprintf(buf, "%d\n", acpi_desc->nfit->revision); +} +static DEVICE_ATTR_RO(revision); + +static struct attribute *nd_acpi_attributes[] = { + &dev_attr_revision.attr, + NULL, +}; + +static struct attribute_group nd_acpi_attribute_group = { + .name = "nfit", + .attrs = nd_acpi_attributes, +}; + +const struct attribute_group *nd_acpi_attribute_groups[] = { + &nd_bus_attribute_group, + &nd_acpi_attribute_group, + NULL, +}; +EXPORT_SYMBOL_GPL(nd_acpi_attribute_groups); + int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) { struct device *dev = acpi_desc->dev; @@ -408,6 +436,7 @@ static int nd_acpi_add(struct acpi_device *adev) nd_desc = &acpi_desc->nd_desc; nd_desc->provider_name = "ACPI.NFIT"; nd_desc->ndctl = nd_acpi_ctl; + nd_desc->attr_groups = nd_acpi_attribute_groups; acpi_desc->nd_bus = nd_bus_register(dev, nd_desc); if (!acpi_desc->nd_bus) diff --git a/drivers/block/nd/acpi_nfit.h b/drivers/block/nd/acpi_nfit.h index a26f69e32244..b65745ca3cbc 100644 --- a/drivers/block/nd/acpi_nfit.h +++ b/drivers/block/nd/acpi_nfit.h @@ -261,5 +261,10 @@ static inline struct acpi_nfit_memdev *__to_nfit_memdev(struct nfit_mem *nfit_me return nfit_mem->memdev_pmem; } +static inline struct acpi_nfit_desc *to_acpi_desc(struct nd_bus_descriptor *nd_desc) +{ + return container_of(nd_desc, struct acpi_nfit_desc, nd_desc); +} + int nd_acpi_nfit_init(struct acpi_nfit_desc *nfit, acpi_size sz); #endif /* __NFIT_H__ */ diff --git a/drivers/block/nd/bus.c b/drivers/block/nd/bus.c new file mode 100644 index 000000000000..635f2e926426 --- /dev/null +++ b/drivers/block/nd/bus.c @@ -0,0 +1,83 @@ +/* + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include <linux/uaccess.h> +#include <linux/fcntl.h> +#include <linux/slab.h> +#include <linux/fs.h> +#include <linux/io.h> +#include "nd-private.h" + +static int nd_bus_major; +static struct class *nd_class; + +int nd_bus_create_ndctl(struct nd_bus *nd_bus) +{ + dev_t devt = MKDEV(nd_bus_major, nd_bus->id); + struct device *dev; + + dev = device_create(nd_class, &nd_bus->dev, devt, nd_bus, "ndctl%d", + nd_bus->id); + + if (IS_ERR(dev)) { + dev_dbg(&nd_bus->dev, "failed to register ndctl%d: %ld\n", + nd_bus->id, PTR_ERR(dev)); + return PTR_ERR(dev); + } + return 0; +} + +void nd_bus_destroy_ndctl(struct nd_bus *nd_bus) +{ + device_destroy(nd_class, MKDEV(nd_bus_major, nd_bus->id)); +} + +static long nd_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + return -ENXIO; +} + +static const struct file_operations nd_bus_fops = { + .owner = THIS_MODULE, + .open = nonseekable_open, + .unlocked_ioctl = nd_ioctl, + .compat_ioctl = nd_ioctl, + .llseek = noop_llseek, +}; + +int __init nd_bus_init(void) +{ + int rc; + + rc = register_chrdev(0, "ndctl", &nd_bus_fops); + if (rc < 0) + return rc; + nd_bus_major = rc; + + nd_class = class_create(THIS_MODULE, "nd"); + if (IS_ERR(nd_class)) + goto err_class; + + return 0; + + err_class: + unregister_chrdev(nd_bus_major, "ndctl"); + + return rc; +} + +void __exit nd_bus_exit(void) +{ + class_destroy(nd_class); + unregister_chrdev(nd_bus_major, "ndctl"); +} diff --git a/drivers/block/nd/core.c b/drivers/block/nd/core.c index 3cccdbc0f3b7..55603ff264ff 100644 --- a/drivers/block/nd/core.c +++ b/drivers/block/nd/core.c @@ -13,10 +13,13 @@ #include <linux/export.h> #include <linux/module.h> #include <linux/device.h> +#include <linux/mutex.h> #include <linux/slab.h> #include "nd-private.h" #include "libnd.h" +LIST_HEAD(nd_bus_list); +DEFINE_MUTEX(nd_bus_list_mutex); static DEFINE_IDA(nd_ida); static void nd_bus_release(struct device *dev) @@ -27,6 +30,54 @@ static void nd_bus_release(struct device *dev) kfree(nd_bus); } +struct nd_bus *to_nd_bus(struct device *dev) +{ + struct nd_bus *nd_bus = container_of(dev, struct nd_bus, dev); + + WARN_ON(nd_bus->dev.release != nd_bus_release); + return nd_bus; +} +EXPORT_SYMBOL_GPL(to_nd_bus); + +struct nd_bus_descriptor *to_nd_desc(struct nd_bus *nd_bus) +{ + /* struct nd_bus definition is private to libnd */ + return nd_bus->nd_desc; +} +EXPORT_SYMBOL_GPL(to_nd_desc); + +static const char *nd_bus_provider(struct nd_bus *nd_bus) +{ + struct nd_bus_descriptor *nd_desc = nd_bus->nd_desc; + struct device *parent = nd_bus->dev.parent; + + if (nd_desc->provider_name) + return nd_desc->provider_name; + else if (parent) + return dev_name(parent); + else + return "unknown"; +} + +static ssize_t provider_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_bus *nd_bus = to_nd_bus(dev); + + return sprintf(buf, "%s\n", nd_bus_provider(nd_bus)); +} +static DEVICE_ATTR_RO(provider); + +static struct attribute *nd_bus_attributes[] = { + &dev_attr_provider.attr, + NULL, +}; + +struct attribute_group nd_bus_attribute_group = { + .attrs = nd_bus_attributes, +}; +EXPORT_SYMBOL_GPL(nd_bus_attribute_group); + struct nd_bus *nd_bus_register(struct device *parent, struct nd_bus_descriptor *nd_desc) { @@ -35,6 +86,7 @@ struct nd_bus *nd_bus_register(struct device *parent, if (!nd_bus) return NULL; + INIT_LIST_HEAD(&nd_bus->list); nd_bus->id = ida_simple_get(&nd_ida, 0, 0, GFP_KERNEL); if (nd_bus->id < 0) { kfree(nd_bus); @@ -43,15 +95,26 @@ struct nd_bus *nd_bus_register(struct device *parent, nd_bus->nd_desc = nd_desc; nd_bus->dev.parent = parent; nd_bus->dev.release = nd_bus_release; + nd_bus->dev.groups = nd_desc->attr_groups; dev_set_name(&nd_bus->dev, "ndbus%d", nd_bus->id); rc = device_register(&nd_bus->dev); if (rc) { dev_dbg(&nd_bus->dev, "device registration failed: %d\n", rc); - put_device(&nd_bus->dev); - return NULL; + goto err; } + rc = nd_bus_create_ndctl(nd_bus); + if (rc) + goto err; + + mutex_lock(&nd_bus_list_mutex); + list_add_tail(&nd_bus->list, &nd_bus_list); + mutex_unlock(&nd_bus_list_mutex); + return nd_bus; + err: + put_device(&nd_bus->dev); + return NULL; } EXPORT_SYMBOL_GPL(nd_bus_register); @@ -59,9 +122,29 @@ void nd_bus_unregister(struct nd_bus *nd_bus) { if (!nd_bus) return; + + mutex_lock(&nd_bus_list_mutex); + list_del_init(&nd_bus->list); + mutex_unlock(&nd_bus_list_mutex); + + nd_bus_destroy_ndctl(nd_bus); + device_unregister(&nd_bus->dev); } EXPORT_SYMBOL_GPL(nd_bus_unregister); +static __init int libnd_init(void) +{ + return nd_bus_init(); +} + +static __exit void libnd_exit(void) +{ + WARN_ON(!list_empty(&nd_bus_list)); + nd_bus_exit(); +} + MODULE_LICENSE("GPL v2"); MODULE_AUTHOR("Intel Corporation"); +module_init(libnd_init); +module_exit(libnd_exit); diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h index 163832937e9c..86cf3e0573b0 100644 --- a/drivers/block/nd/libnd.h +++ b/drivers/block/nd/libnd.h @@ -14,6 +14,8 @@ */ #ifndef __LIBND_H__ #define __LIBND_H__ +extern struct attribute_group nd_bus_attribute_group; + struct nd_dimm; struct nd_bus_descriptor; typedef int (*ndctl_fn)(struct nd_bus_descriptor *nd_desc, @@ -21,6 +23,7 @@ typedef int (*ndctl_fn)(struct nd_bus_descriptor *nd_desc, unsigned int buf_len); struct nd_bus_descriptor { + const struct attribute_group **attr_groups; unsigned long dsm_mask; char *provider_name; ndctl_fn ndctl; @@ -30,4 +33,6 @@ struct nd_bus; struct nd_bus *nd_bus_register(struct device *parent, struct nd_bus_descriptor *nfit_desc); void nd_bus_unregister(struct nd_bus *nd_bus); +struct nd_bus *to_nd_bus(struct device *dev); +struct nd_bus_descriptor *to_nd_desc(struct nd_bus *nd_bus); #endif /* __LIBND_H__ */ diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index 3dbab29fa0f9..960dd2f29cdd 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -17,7 +17,13 @@ struct nd_bus { struct nd_bus_descriptor *nd_desc; + struct list_head list; struct device dev; int id; }; + +int __init nd_bus_init(void); +void __exit nd_bus_exit(void); +int nd_bus_create_ndctl(struct nd_bus *nd_bus); +void nd_bus_destroy_ndctl(struct nd_bus *nd_bus); #endif /* __ND_PRIVATE_H__ */ diff --git a/drivers/block/nd/test/nfit.c b/drivers/block/nd/test/nfit.c index 8691a903515b..1c79f32376fc 100644 --- a/drivers/block/nd/test/nfit.c +++ b/drivers/block/nd/test/nfit.c @@ -833,6 +833,8 @@ static void nfit_test1_setup(struct nfit_test *t) nfit->checksum = nfit_checksum(nfit_buf, size); } +extern const struct attribute_group *nd_acpi_attribute_groups[]; + static int nfit_test_probe(struct platform_device *pdev) { struct nd_bus_descriptor *nd_desc; @@ -882,6 +884,7 @@ static int nfit_test_probe(struct platform_device *pdev) acpi_desc->dev = &pdev->dev; acpi_desc->nfit = nfit_test->nfit_buf; nd_desc = &acpi_desc->nd_desc; + nd_desc->attr_groups = nd_acpi_attribute_groups; acpi_desc->nd_bus = nd_bus_register(&pdev->dev, nd_desc); if (!acpi_desc->nd_bus) return -ENXIO;

Register the memory devices described in the nfit as libnd 'dimm' devices on an nd bus. The kernel assigned device id for dimms is dynamic. If userspace needs a more static identifier it should consult a provider-specific attribute. In the case where NFIT is the provider, the 'nmemX/nfit/handle' or 'nmemX/nfit/serial' attributes may be used for this purpose. Cc: Neil Brown <neilb(a)suse.de&gt; Cc: <linux-acpi(a)vger.kernel.org&gt; Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; Cc: Robert Moore <robert.moore(a)intel.com&gt; Cc: Rafael J. Wysocki <rafael.j.wysocki(a)intel.com&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/Makefile | 1 drivers/block/nd/acpi.c | 160 +++++++++++++++++++++++++++++++++++++++++ drivers/block/nd/acpi_nfit.h | 1 drivers/block/nd/bus.c | 14 +++- drivers/block/nd/core.c | 29 +++++++ drivers/block/nd/dimm_devs.c | 92 ++++++++++++++++++++++++ drivers/block/nd/libnd.h | 11 +++ drivers/block/nd/nd-private.h | 12 +++ 8 files changed, 318 insertions(+), 2 deletions(-) create mode 100644 drivers/block/nd/dimm_devs.c diff --git a/drivers/block/nd/Makefile b/drivers/block/nd/Makefile index 7defe18ed009..35e4c1a7a8ff 100644 --- a/drivers/block/nd/Makefile +++ b/drivers/block/nd/Makefile @@ -20,3 +20,4 @@ nd_acpi-y := acpi.o libnd-y := core.o libnd-y += bus.o +libnd-y += dimm_devs.o diff --git a/drivers/block/nd/acpi.c b/drivers/block/nd/acpi.c index dd8505f766ed..af6684341c9b 100644 --- a/drivers/block/nd/acpi.c +++ b/drivers/block/nd/acpi.c @@ -369,6 +369,164 @@ const struct attribute_group *nd_acpi_attribute_groups[] = { }; EXPORT_SYMBOL_GPL(nd_acpi_attribute_groups); +static struct acpi_nfit_memdev *to_nfit_memdev(struct device *dev) +{ + struct nd_dimm *nd_dimm = to_nd_dimm(dev); + struct nfit_mem *nfit_mem = nd_dimm_provider_data(nd_dimm); + + return __to_nfit_memdev(nfit_mem); +} + +static struct acpi_nfit_dcr *to_nfit_dcr(struct device *dev) +{ + struct nd_dimm *nd_dimm = to_nd_dimm(dev); + struct nfit_mem *nfit_mem = nd_dimm_provider_data(nd_dimm); + + return nfit_mem->dcr; +} + +static ssize_t handle_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct acpi_nfit_memdev *memdev = to_nfit_memdev(dev); + + return sprintf(buf, "%#x\n", memdev->nfit_handle); +} +static DEVICE_ATTR_RO(handle); + +static ssize_t phys_id_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct acpi_nfit_memdev *memdev = to_nfit_memdev(dev); + + return sprintf(buf, "%#x\n", memdev->phys_id); +} +static DEVICE_ATTR_RO(phys_id); + +static ssize_t vendor_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct acpi_nfit_dcr *dcr = to_nfit_dcr(dev); + + return sprintf(buf, "%#x\n", dcr->vendor_id); +} +static DEVICE_ATTR_RO(vendor); + +static ssize_t rev_id_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct acpi_nfit_dcr *dcr = to_nfit_dcr(dev); + + return sprintf(buf, "%#x\n", dcr->revision_id); +} +static DEVICE_ATTR_RO(rev_id); + +static ssize_t device_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct acpi_nfit_dcr *dcr = to_nfit_dcr(dev); + + return sprintf(buf, "%#x\n", dcr->device_id); +} +static DEVICE_ATTR_RO(device); + +static ssize_t format_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct acpi_nfit_dcr *dcr = to_nfit_dcr(dev); + + return sprintf(buf, "%#x\n", dcr->fic); +} +static DEVICE_ATTR_RO(format); + +static ssize_t serial_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct acpi_nfit_dcr *dcr = to_nfit_dcr(dev); + + return sprintf(buf, "%#x\n", dcr->serial_number); +} +static DEVICE_ATTR_RO(serial); + +static struct attribute *nd_acpi_dimm_attributes[] = { + &dev_attr_handle.attr, + &dev_attr_phys_id.attr, + &dev_attr_vendor.attr, + &dev_attr_device.attr, + &dev_attr_format.attr, + &dev_attr_serial.attr, + &dev_attr_rev_id.attr, + NULL, +}; + +static umode_t nd_acpi_dimm_attr_visible(struct kobject *kobj, struct attribute *a, int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + + if (to_nfit_dcr(dev)) + return a->mode; + else + return 0; +} + +static struct attribute_group nd_acpi_dimm_attribute_group = { + .name = "nfit", + .attrs = nd_acpi_dimm_attributes, + .is_visible = nd_acpi_dimm_attr_visible, +}; + +static const struct attribute_group *nd_acpi_dimm_attribute_groups[] = { + &nd_acpi_dimm_attribute_group, + NULL, +}; + +static struct nd_dimm *nd_acpi_dimm_by_handle(struct acpi_nfit_desc *acpi_desc, + u32 nfit_handle) +{ + struct nfit_mem *nfit_mem; + + list_for_each_entry(nfit_mem, &acpi_desc->dimms, list) + if (__to_nfit_memdev(nfit_mem)->nfit_handle == nfit_handle) + return nfit_mem->nd_dimm; + + return NULL; +} + +static int nd_acpi_register_dimms(struct acpi_nfit_desc *acpi_desc) +{ + struct nfit_mem *nfit_mem; + + list_for_each_entry(nfit_mem, &acpi_desc->dimms, list) { + struct nd_dimm *nd_dimm; + unsigned long flags = 0; + u32 nfit_handle; + + nfit_handle = __to_nfit_memdev(nfit_mem)->nfit_handle; + nd_dimm = nd_acpi_dimm_by_handle(acpi_desc, nfit_handle); + if (nd_dimm) { + /* + * If for some reason we find multiple DCRs the + * first one wins + */ + dev_err(acpi_desc->dev, "duplicate DCR detected: %s\n", + nd_dimm_name(nd_dimm)); + continue; + } + + if (nfit_mem->bdw && nfit_mem->memdev_pmem) + flags |= NDD_ALIASING; + + nd_dimm = nd_dimm_create(acpi_desc->nd_bus, nfit_mem, + nd_acpi_dimm_attribute_groups, flags); + if (!nd_dimm) + return -ENOMEM; + + nfit_mem->nd_dimm = nd_dimm; + } + + return 0; +} + int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) { struct device *dev = acpi_desc->dev; @@ -406,7 +564,7 @@ int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) if (nfit_mem_init(acpi_desc) != 0) return -ENOMEM; - return 0; + return nd_acpi_register_dimms(acpi_desc); } EXPORT_SYMBOL_GPL(nd_acpi_nfit_init); diff --git a/drivers/block/nd/acpi_nfit.h b/drivers/block/nd/acpi_nfit.h index b65745ca3cbc..00aaaee5953b 100644 --- a/drivers/block/nd/acpi_nfit.h +++ b/drivers/block/nd/acpi_nfit.h @@ -233,6 +233,7 @@ struct nfit_memdev { /* assembled tables for a given dimm/memory-device */ struct nfit_mem { + struct nd_dimm *nd_dimm; struct acpi_nfit_memdev *memdev_dcr; struct acpi_nfit_memdev *memdev_pmem; struct acpi_nfit_dcr *dcr; diff --git a/drivers/block/nd/bus.c b/drivers/block/nd/bus.c index 635f2e926426..ee56aa1ab2ad 100644 --- a/drivers/block/nd/bus.c +++ b/drivers/block/nd/bus.c @@ -13,6 +13,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include <linux/uaccess.h> #include <linux/fcntl.h> +#include <linux/async.h> #include <linux/slab.h> #include <linux/fs.h> #include <linux/io.h> @@ -21,6 +22,10 @@ static int nd_bus_major; static struct class *nd_class; +struct bus_type nd_bus_type = { + .name = "nd", +}; + int nd_bus_create_ndctl(struct nd_bus *nd_bus) { dev_t devt = MKDEV(nd_bus_major, nd_bus->id); @@ -59,9 +64,13 @@ int __init nd_bus_init(void) { int rc; + rc = bus_register(&nd_bus_type); + if (rc) + return rc; + rc = register_chrdev(0, "ndctl", &nd_bus_fops); if (rc < 0) - return rc; + goto err_chrdev; nd_bus_major = rc; nd_class = class_create(THIS_MODULE, "nd"); @@ -72,6 +81,8 @@ int __init nd_bus_init(void) err_class: unregister_chrdev(nd_bus_major, "ndctl"); + err_chrdev: + bus_unregister(&nd_bus_type); return rc; } @@ -80,4 +91,5 @@ void __exit nd_bus_exit(void) { class_destroy(nd_class); unregister_chrdev(nd_bus_major, "ndctl"); + bus_unregister(&nd_bus_type); } diff --git a/drivers/block/nd/core.c b/drivers/block/nd/core.c index 55603ff264ff..24046db897c1 100644 --- a/drivers/block/nd/core.c +++ b/drivers/block/nd/core.c @@ -46,6 +46,19 @@ struct nd_bus_descriptor *to_nd_desc(struct nd_bus *nd_bus) } EXPORT_SYMBOL_GPL(to_nd_desc); +struct nd_bus *walk_to_nd_bus(struct device *nd_dev) +{ + struct device *dev; + + for (dev = nd_dev; dev; dev = dev->parent) + if (dev->release == nd_bus_release) + break; + dev_WARN_ONCE(nd_dev, !dev, "invalid dev, not on nd bus\n"); + if (dev) + return to_nd_bus(dev); + return NULL; +} + static const char *nd_bus_provider(struct nd_bus *nd_bus) { struct nd_bus_descriptor *nd_desc = nd_bus->nd_desc; @@ -118,6 +131,21 @@ struct nd_bus *nd_bus_register(struct device *parent, } EXPORT_SYMBOL_GPL(nd_bus_register); +static int child_unregister(struct device *dev, void *data) +{ + /* + * the singular ndctl class device per bus needs to be + * "device_destroy"ed, so skip it here + * + * i.e. remove classless children + */ + if (dev->class) + /* pass */; + else + device_unregister(dev); + return 0; +} + void nd_bus_unregister(struct nd_bus *nd_bus) { if (!nd_bus) @@ -127,6 +155,7 @@ void nd_bus_unregister(struct nd_bus *nd_bus) list_del_init(&nd_bus->list); mutex_unlock(&nd_bus_list_mutex); + device_for_each_child(&nd_bus->dev, NULL, child_unregister); nd_bus_destroy_ndctl(nd_bus); device_unregister(&nd_bus->dev); diff --git a/drivers/block/nd/dimm_devs.c b/drivers/block/nd/dimm_devs.c new file mode 100644 index 000000000000..19b081392f2f --- /dev/null +++ b/drivers/block/nd/dimm_devs.c @@ -0,0 +1,92 @@ +/* + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include <linux/device.h> +#include <linux/slab.h> +#include <linux/io.h> +#include <linux/fs.h> +#include <linux/mm.h> +#include "nd-private.h" + +static DEFINE_IDA(dimm_ida); + +static void nd_dimm_release(struct device *dev) +{ + struct nd_dimm *nd_dimm = to_nd_dimm(dev); + + ida_simple_remove(&dimm_ida, nd_dimm->id); + kfree(nd_dimm); +} + +static struct device_type nd_dimm_device_type = { + .name = "nd_dimm", + .release = nd_dimm_release, +}; + +static bool is_nd_dimm(struct device *dev) +{ + return dev->type == &nd_dimm_device_type; +} + +struct nd_dimm *to_nd_dimm(struct device *dev) +{ + struct nd_dimm *nd_dimm = container_of(dev, struct nd_dimm, dev); + + WARN_ON(!is_nd_dimm(dev)); + return nd_dimm; +} +EXPORT_SYMBOL_GPL(to_nd_dimm); + +const char *nd_dimm_name(struct nd_dimm *nd_dimm) +{ + return dev_name(&nd_dimm->dev); +} +EXPORT_SYMBOL_GPL(nd_dimm_name); + +void *nd_dimm_provider_data(struct nd_dimm *nd_dimm) +{ + return nd_dimm->provider_data; +} +EXPORT_SYMBOL_GPL(nd_dimm_provider_data); + +struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, + const struct attribute_group **groups, unsigned long flags) +{ + struct nd_dimm *nd_dimm = kzalloc(sizeof(*nd_dimm), GFP_KERNEL); + struct device *dev; + + if (!nd_dimm) + return NULL; + + nd_dimm->id = ida_simple_get(&dimm_ida, 0, 0, GFP_KERNEL); + if (nd_dimm->id < 0) { + kfree(nd_dimm); + return NULL; + } + nd_dimm->provider_data = provider_data; + nd_dimm->flags = flags; + + dev = &nd_dimm->dev; + dev_set_name(dev, "nmem%d", nd_dimm->id); + dev->parent = &nd_bus->dev; + dev->type = &nd_dimm_device_type; + dev->bus = &nd_bus_type; + dev->groups = groups; + if (device_register(dev) != 0) { + put_device(dev); + return NULL; + } + + return nd_dimm; +} +EXPORT_SYMBOL_GPL(nd_dimm_create); diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h index 86cf3e0573b0..4f8f2ebbcf7b 100644 --- a/drivers/block/nd/libnd.h +++ b/drivers/block/nd/libnd.h @@ -14,6 +14,12 @@ */ #ifndef __LIBND_H__ #define __LIBND_H__ + +enum { + /* when a dimm supports both PMEM and BLK access a label is required */ + NDD_ALIASING = 1 << 0, +}; + extern struct attribute_group nd_bus_attribute_group; struct nd_dimm; @@ -34,5 +40,10 @@ struct nd_bus *nd_bus_register(struct device *parent, struct nd_bus_descriptor *nfit_desc); void nd_bus_unregister(struct nd_bus *nd_bus); struct nd_bus *to_nd_bus(struct device *dev); +struct nd_dimm *to_nd_dimm(struct device *dev); struct nd_bus_descriptor *to_nd_desc(struct nd_bus *nd_bus); +const char *nd_dimm_name(struct nd_dimm *nd_dimm); +void *nd_dimm_provider_data(struct nd_dimm *nd_dimm); +struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, + const struct attribute_group **groups, unsigned long flags); #endif /* __LIBND_H__ */ diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index 960dd2f29cdd..cfb5a7241ed1 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -15,6 +15,10 @@ #include <linux/device.h> #include "libnd.h" +extern struct list_head nd_bus_list; +extern struct mutex nd_bus_list_mutex; +extern struct bus_type nd_bus_type; + struct nd_bus { struct nd_bus_descriptor *nd_desc; struct list_head list; @@ -22,6 +26,14 @@ struct nd_bus { int id; }; +struct nd_dimm { + unsigned long flags; + void *provider_data; + struct device dev; + int id; +}; + +struct nd_bus *walk_to_nd_bus(struct device *nd_dev); int __init nd_bus_init(void); void __exit nd_bus_exit(void); int nd_bus_create_ndctl(struct nd_bus *nd_bus);

Most configuration of the nd-subsystem is done via nd-sysfs attributes. However, some nd buses, particularly the ACPI.NFIT bus, define a small set of messages that can be passed to the platform. For convenience we derivce the initial nd-ioctl-command formats directly from the NFIT DSM formats. ND_CMD_SMART: media health and diagnostics ND_CMD_GET_CONFIG_SIZE: size of the label space ND_CMD_GET_CONFIG_DATA: read label space ND_CMD_SET_CONFIG_DATA: write label space ND_CMD_VENDOR: vendor-specific command passthrough ND_CMD_ARS_CAP: report address-range-scrubbing capabilities ND_CMD_START_ARS: initiate scrubbing ND_CMD_QUERY_ARS: report on scrubbing state ND_CMD_SMART_THRESHOLD: configure alarm thresholds for smart events If a platform later defines different commands than this set it is straightforward to extend support to those formats. Most of the commands target a specific dimm. However, the address-range-scrubbing commands target the bus. The 'commands' attribute in sysfs of an nd-bus, or an nd-dimm enumerate the supported commands for that object. Cc: <linux-acpi(a)vger.kernel.org&gt; Cc: Robert Moore <robert.moore(a)intel.com&gt; Cc: Rafael J. Wysocki <rafael.j.wysocki(a)intel.com&gt; Reported-by: Nicholas Moulin <nicholas.w.moulin(a)linux.intel.com&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/Kconfig | 12 ++ drivers/block/nd/acpi.c | 237 ++++++++++++++++++++++++++++++ drivers/block/nd/acpi_nfit.h | 3 drivers/block/nd/bus.c | 324 ++++++++++++++++++++++++++++++++++++++++- drivers/block/nd/core.c | 16 ++ drivers/block/nd/dimm_devs.c | 38 ++++- drivers/block/nd/libnd.h | 25 +++ drivers/block/nd/nd-private.h | 3 drivers/block/nd/test/nfit.c | 78 ++++++++++ include/uapi/linux/Kbuild | 1 include/uapi/linux/ndctl.h | 178 +++++++++++++++++++++++ 11 files changed, 903 insertions(+), 12 deletions(-) create mode 100644 include/uapi/linux/ndctl.h diff --git a/drivers/block/nd/Kconfig b/drivers/block/nd/Kconfig index 09f0135147ca..d2d84451e82c 100644 --- a/drivers/block/nd/Kconfig +++ b/drivers/block/nd/Kconfig @@ -37,6 +37,18 @@ config ND_ACPI addition to storage devices this also enables libnd craft ACPI._DSM messages for platform/dimm configuration. +config ND_ACPI_DEBUG + bool "ACPI: Extra nd_acpi debugging" + depends on ND_ACPI + depends on DYNAMIC_DEBUG + default n + help + Enabling this option causes the nd_acpi driver to dump the + input and output buffers of _DSM operations on the ACPI0012 + device and its children. This can be very verbose, so leave + it disabled unless you are debugging a hardware / firmware + issue. + config NFIT_TEST tristate "NFIT TEST: Manufactured NFIT for interface testing" depends on DMA_CMA diff --git a/drivers/block/nd/acpi.c b/drivers/block/nd/acpi.c index af6684341c9b..c46e166695f7 100644 --- a/drivers/block/nd/acpi.c +++ b/drivers/block/nd/acpi.c @@ -12,6 +12,7 @@ */ #include <linux/list_sort.h> #include <linux/module.h> +#include <linux/ndctl.h> #include <linux/list.h> #include <linux/acpi.h> #include "acpi_nfit.h" @@ -25,11 +26,160 @@ enum { NFIT_ACPI_NOTIFY_TABLE = 0x80, }; +static u8 nd_acpi_uuids[2][16]; /* initialized at nd_acpi_init */ + +static u8 *nd_acpi_bus_uuid(void) +{ + return nd_acpi_uuids[0]; +} + +static u8 *nd_acpi_dimm_uuid(void) +{ + return nd_acpi_uuids[1]; +} + +static struct acpi_nfit_desc *to_acpi_nfit_desc(struct nd_bus_descriptor *nd_desc) +{ + return container_of(nd_desc, struct acpi_nfit_desc, nd_desc); +} + +static struct acpi_device *to_acpi_dev(struct acpi_nfit_desc *acpi_desc) +{ + struct nd_bus_descriptor *nd_desc = &acpi_desc->nd_desc; + + /* + * If provider == 'ACPI.NFIT' we can assume 'dev' is a struct + * acpi_device. + */ + if (!nd_desc->provider_name + || strcmp(nd_desc->provider_name, "ACPI.NFIT") != 0) + return NULL; + + return to_acpi_device(acpi_desc->dev); +} + static int nd_acpi_ctl(struct nd_bus_descriptor *nd_desc, struct nd_dimm *nd_dimm, unsigned int cmd, void *buf, unsigned int buf_len) { - return -ENOTTY; + struct acpi_nfit_desc *acpi_desc = to_acpi_nfit_desc(nd_desc); + const struct nd_cmd_desc const *desc = NULL; + union acpi_object in_obj, in_buf, *out_obj; + struct device *dev = acpi_desc->dev; + const char *cmd_name, *dimm_name; + unsigned long dsm_mask; + acpi_handle handle; + u32 offset; + int rc, i; + u8 *uuid; + + if (nd_dimm) { + struct nfit_mem *nfit_mem = nd_dimm_provider_data(nd_dimm); + struct acpi_device *adev = nfit_mem->adev; + + if (!adev) + return -ENOTTY; + dimm_name = dev_name(&adev->dev); + cmd_name = nd_dimm_cmd_name(cmd); + dsm_mask = nfit_mem->dsm_mask; + desc = nd_cmd_dimm_desc(cmd); + uuid = nd_acpi_dimm_uuid(); + handle = adev->handle; + } else { + struct acpi_device *adev = to_acpi_dev(acpi_desc); + + cmd_name = nd_bus_cmd_name(cmd); + dsm_mask = nd_desc->dsm_mask; + desc = nd_cmd_bus_desc(cmd); + uuid = nd_acpi_bus_uuid(); + handle = adev->handle; + dimm_name = "bus"; + } + + if (!desc || (cmd && (desc->out_num + desc->in_num == 0))) + return -ENOTTY; + + if (!test_bit(cmd, &dsm_mask)) + return -ENOTTY; + + in_obj.type = ACPI_TYPE_PACKAGE; + in_obj.package.count = 1; + in_obj.package.elements = &in_buf; + in_buf.type = ACPI_TYPE_BUFFER; + in_buf.buffer.pointer = buf; + in_buf.buffer.length = 0; + + /* libnd has already validated the input envelope */ + for (i = 0; i < desc->in_num; i++) + in_buf.buffer.length += nd_cmd_in_size(nd_dimm, cmd, desc, i, buf); + + dev_dbg(dev, "%s:%s cmd: %s input length: %d\n", __func__, dimm_name, + cmd_name, in_buf.buffer.length); + if (IS_ENABLED(CONFIG_ND_ACPI_DEBUG)) + print_hex_dump_debug(cmd_name, DUMP_PREFIX_OFFSET, 4, + 4, in_buf.buffer.pointer, min_t(u32, 128, + in_buf.buffer.length), true); + + out_obj = acpi_evaluate_dsm(handle, uuid, 1, cmd, &in_obj); + if (!out_obj) { + dev_dbg(dev, "%s:%s _DSM failed cmd: %s\n", __func__, dimm_name, + cmd_name); + return -EINVAL; + } + + if (out_obj->package.type != ACPI_TYPE_BUFFER) { + dev_dbg(dev, "%s:%s unexpected output object type cmd: %s type: %d\n", + __func__, dimm_name, cmd_name, out_obj->type); + rc = -EINVAL; + goto out; + } + + dev_dbg(dev, "%s:%s cmd: %s output length: %d\n", __func__, dimm_name, + cmd_name, out_obj->buffer.length); + if (IS_ENABLED(CONFIG_ND_ACPI_DEBUG)) + print_hex_dump_debug(cmd_name, DUMP_PREFIX_OFFSET, 4, + 4, out_obj->buffer.pointer, min_t(u32, 128, + out_obj->buffer.length), true); + + for (i = 0, offset = 0; i < desc->out_num; i++) { + u32 out_size = nd_cmd_out_size(nd_dimm, cmd, desc, i, buf, + (u32 *) out_obj->buffer.pointer); + + if (offset + out_size > out_obj->buffer.length) { + dev_dbg(dev, "%s:%s output object underflow cmd: %s field: %d\n", + __func__, dimm_name, cmd_name, i); + break; + } + + if (in_buf.buffer.length + offset + out_size > buf_len) { + dev_dbg(dev, "%s:%s output overrun cmd: %s field: %d\n", + __func__, dimm_name, cmd_name, i); + rc = -ENXIO; + goto out; + } + memcpy(buf + in_buf.buffer.length + offset, + out_obj->buffer.pointer + offset, out_size); + offset += out_size; + } + if (offset + in_buf.buffer.length < buf_len) { + if (i >= 1) { + /* + * status valid, return the number of bytes left + * unfilled in the output buffer + */ + rc = buf_len - offset - in_buf.buffer.length; + } else { + dev_err(dev, "%s:%s underrun cmd: %s buf_len: %d out_len: %d\n", + __func__, dimm_name, cmd_name, buf_len, offset); + rc = -ENXIO; + } + } else + rc = 0; + + out: + ACPI_FREE(out_obj); + + return rc; } static const char *spa_type_name(u16 type) @@ -476,6 +626,7 @@ static struct attribute_group nd_acpi_dimm_attribute_group = { }; static const struct attribute_group *nd_acpi_dimm_attribute_groups[] = { + &nd_dimm_attribute_group, &nd_acpi_dimm_attribute_group, NULL, }; @@ -492,6 +643,50 @@ static struct nd_dimm *nd_acpi_dimm_by_handle(struct acpi_nfit_desc *acpi_desc, return NULL; } +static int nd_acpi_add_dimm(struct acpi_nfit_desc *acpi_desc, + struct nfit_mem *nfit_mem, u32 nfit_handle) +{ + struct acpi_device *adev, *adev_dimm; + struct device *dev = acpi_desc->dev; + u8 *uuid = nd_acpi_dimm_uuid(); + unsigned long long sta; + int i, rc = -ENODEV; + acpi_status status; + + nfit_mem->dsm_mask = acpi_desc->dimm_dsm_force_en; + adev = to_acpi_dev(acpi_desc); + if (!adev) + return 0; + + adev_dimm = acpi_find_child_device(adev, nfit_handle, false); + nfit_mem->adev = adev_dimm; + if (!adev_dimm) { + dev_err(dev, "no ACPI.NFIT device with _ADR %#x, disabling...\n", + nfit_handle); + return -ENODEV; + } + + status = acpi_evaluate_integer(adev_dimm->handle, "_STA", NULL, &sta); + if (status == AE_NOT_FOUND) { + dev_dbg(dev, "%s missing _STA, assuming enabled...\n", + dev_name(&adev_dimm->dev)); + rc = 0; + } else if (ACPI_FAILURE(status)) + dev_err(dev, "%s failed to retrieve_STA, disabling...\n", + dev_name(&adev_dimm->dev)); + else if ((sta & ACPI_STA_DEVICE_ENABLED) == 0) + dev_info(dev, "%s disabled by firmware\n", + dev_name(&adev_dimm->dev)); + else + rc = 0; + + for (i = ND_CMD_SMART; i <= ND_CMD_VENDOR; i++) + if (acpi_check_dsm(adev_dimm->handle, uuid, 1, 1ULL << i)) + set_bit(i, &nfit_mem->dsm_mask); + + return rc; +} + static int nd_acpi_register_dimms(struct acpi_nfit_desc *acpi_desc) { struct nfit_mem *nfit_mem; @@ -500,6 +695,7 @@ static int nd_acpi_register_dimms(struct acpi_nfit_desc *acpi_desc) struct nd_dimm *nd_dimm; unsigned long flags = 0; u32 nfit_handle; + int rc; nfit_handle = __to_nfit_memdev(nfit_mem)->nfit_handle; nd_dimm = nd_acpi_dimm_by_handle(acpi_desc, nfit_handle); @@ -516,8 +712,13 @@ static int nd_acpi_register_dimms(struct acpi_nfit_desc *acpi_desc) if (nfit_mem->bdw && nfit_mem->memdev_pmem) flags |= NDD_ALIASING; + rc = nd_acpi_add_dimm(acpi_desc, nfit_mem, nfit_handle); + if (rc) + continue; + nd_dimm = nd_dimm_create(acpi_desc->nd_bus, nfit_mem, - nd_acpi_dimm_attribute_groups, flags); + nd_acpi_dimm_attribute_groups, + flags, &nfit_mem->dsm_mask); if (!nd_dimm) return -ENOMEM; @@ -527,6 +728,22 @@ static int nd_acpi_register_dimms(struct acpi_nfit_desc *acpi_desc) return 0; } +static void nd_acpi_init_dsms(struct acpi_nfit_desc *acpi_desc) +{ + struct nd_bus_descriptor *nd_desc = &acpi_desc->nd_desc; + u8 *uuid = nd_acpi_bus_uuid(); + struct acpi_device *adev; + int i; + + adev = to_acpi_dev(acpi_desc); + if (!adev) + return; + + for (i = ND_CMD_ARS_CAP; i <= ND_CMD_ARS_QUERY; i++) + if (acpi_check_dsm(adev->handle, uuid, 1, 1ULL << i)) + set_bit(i, &nd_desc->dsm_mask); +} + int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) { struct device *dev = acpi_desc->dev; @@ -564,6 +781,8 @@ int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) if (nfit_mem_init(acpi_desc) != 0) return -ENOMEM; + nd_acpi_init_dsms(acpi_desc); + return nd_acpi_register_dimms(acpi_desc); } EXPORT_SYMBOL_GPL(nd_acpi_nfit_init); @@ -641,6 +860,14 @@ static struct acpi_driver nd_acpi_driver = { static __init int nd_acpi_init(void) { + char *uuids[] = { + /* bus interface */ + "2f10e7a4-9e91-11e4-89d3-123b93f75cba", + /* per-dimm interface */ + "4309ac30-0d11-11e4-9191-0800200c9a66", + }; + int i; + BUILD_BUG_ON(sizeof(struct acpi_nfit) != 40); BUILD_BUG_ON(sizeof(struct acpi_nfit_spa) != 56); BUILD_BUG_ON(sizeof(struct acpi_nfit_memdev) != 48); @@ -649,6 +876,12 @@ static __init int nd_acpi_init(void) BUILD_BUG_ON(sizeof(struct acpi_nfit_dcr) != 80); BUILD_BUG_ON(sizeof(struct acpi_nfit_bdw) != 40); + for (i = 0; i < ARRAY_SIZE(uuids); i++) + if (acpi_str_to_uuid(uuids[i], nd_acpi_uuids[i]) != AE_OK) { + WARN_ON_ONCE(1); + return -ENXIO; + } + return acpi_bus_register_driver(&nd_acpi_driver); } diff --git a/drivers/block/nd/acpi_nfit.h b/drivers/block/nd/acpi_nfit.h index 00aaaee5953b..2faac336c07d 100644 --- a/drivers/block/nd/acpi_nfit.h +++ b/drivers/block/nd/acpi_nfit.h @@ -241,6 +241,8 @@ struct nfit_mem { struct acpi_nfit_spa *spa_dcr; struct acpi_nfit_spa *spa_bdw; struct list_head list; + struct acpi_device *adev; + unsigned long dsm_mask; }; struct acpi_nfit_desc { @@ -253,6 +255,7 @@ struct acpi_nfit_desc { struct list_head bdws; struct nd_bus *nd_bus; struct device *dev; + unsigned long dimm_dsm_force_en; }; static inline struct acpi_nfit_memdev *__to_nfit_memdev(struct nfit_mem *nfit_mem) diff --git a/drivers/block/nd/bus.c b/drivers/block/nd/bus.c index ee56aa1ab2ad..a271e01af4a9 100644 --- a/drivers/block/nd/bus.c +++ b/drivers/block/nd/bus.c @@ -11,14 +11,18 @@ * General Public License for more details. */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include <linux/vmalloc.h> #include <linux/uaccess.h> #include <linux/fcntl.h> #include <linux/async.h> +#include <linux/ndctl.h> #include <linux/slab.h> #include <linux/fs.h> #include <linux/io.h> +#include <linux/mm.h> #include "nd-private.h" +int nd_dimm_major; static int nd_bus_major; static struct class *nd_class; @@ -47,19 +51,323 @@ void nd_bus_destroy_ndctl(struct nd_bus *nd_bus) device_destroy(nd_class, MKDEV(nd_bus_major, nd_bus->id)); } +static const struct nd_cmd_desc const __nd_cmd_dimm_descs[] = { + [ND_CMD_IMPLEMENTED] = { }, + [ND_CMD_SMART] = { + .out_num = 2, + .out_sizes = { 4, 8, }, + }, + [ND_CMD_SMART_THRESHOLD] = { + .out_num = 2, + .out_sizes = { 4, 8, }, + }, + [ND_CMD_DIMM_FLAGS] = { + .out_num = 2, + .out_sizes = { 4, 4 }, + }, + [ND_CMD_GET_CONFIG_SIZE] = { + .out_num = 3, + .out_sizes = { 4, 4, 4, }, + }, + [ND_CMD_GET_CONFIG_DATA] = { + .in_num = 2, + .in_sizes = { 4, 4, }, + .out_num = 2, + .out_sizes = { 4, UINT_MAX, }, + }, + [ND_CMD_SET_CONFIG_DATA] = { + .in_num = 3, + .in_sizes = { 4, 4, UINT_MAX, }, + .out_num = 1, + .out_sizes = { 4, }, + }, + [ND_CMD_VENDOR] = { + .in_num = 3, + .in_sizes = { 4, 4, UINT_MAX, }, + .out_num = 3, + .out_sizes = { 4, 4, UINT_MAX, }, + }, +}; + +struct nd_cmd_desc const *nd_cmd_dimm_desc(int cmd) +{ + if (cmd < ARRAY_SIZE(__nd_cmd_dimm_descs)) + return &__nd_cmd_dimm_descs[cmd]; + return NULL; +} +EXPORT_SYMBOL_GPL(nd_cmd_dimm_desc); + +static const struct nd_cmd_desc const __nd_cmd_bus_descs[] = { + [ND_CMD_IMPLEMENTED] = { }, + [ND_CMD_ARS_CAP] = { + .in_num = 2, + .in_sizes = { 8, 8, }, + .out_num = 2, + .out_sizes = { 4, 4, }, + }, + [ND_CMD_ARS_START] = { + .in_num = 4, + .in_sizes = { 8, 8, 2, 6, }, + .out_num = 1, + .out_sizes = { 4, }, + }, + [ND_CMD_ARS_QUERY] = { + .out_num = 2, + .out_sizes = { 4, UINT_MAX, }, + }, +}; + +const struct nd_cmd_desc *nd_cmd_bus_desc(int cmd) +{ + if (cmd < ARRAY_SIZE(__nd_cmd_bus_descs)) + return &__nd_cmd_bus_descs[cmd]; + return NULL; +} +EXPORT_SYMBOL_GPL(nd_cmd_bus_desc); + +u32 nd_cmd_in_size(struct nd_dimm *nd_dimm, int cmd, + const struct nd_cmd_desc *desc, int idx, void *buf) +{ + if (idx >= desc->in_num) + return UINT_MAX; + + if (desc->in_sizes[idx] < UINT_MAX) + return desc->in_sizes[idx]; + + if (nd_dimm && cmd == ND_CMD_SET_CONFIG_DATA && idx == 2) { + struct nd_cmd_set_config_hdr *hdr = buf; + + return hdr->in_length; + } else if (nd_dimm && cmd == ND_CMD_VENDOR && idx == 2) { + struct nd_cmd_vendor_hdr *hdr = buf; + + return hdr->in_length; + } + + return UINT_MAX; +} +EXPORT_SYMBOL_GPL(nd_cmd_in_size); + +u32 nd_cmd_out_size(struct nd_dimm *nd_dimm, int cmd, + const struct nd_cmd_desc *desc, int idx, const u32 *in_field, + const u32 *out_field) +{ + if (idx >= desc->out_num) + return UINT_MAX; + + if (desc->out_sizes[idx] < UINT_MAX) + return desc->out_sizes[idx]; + + if (nd_dimm && cmd == ND_CMD_GET_CONFIG_DATA && idx == 1) + return in_field[1]; + else if (nd_dimm && cmd == ND_CMD_VENDOR && idx == 2) + return out_field[1]; + else if (!nd_dimm && cmd == ND_CMD_ARS_QUERY && idx == 1) + return ND_CMD_ARS_QUERY_MAX; + + return UINT_MAX; +} +EXPORT_SYMBOL_GPL(nd_cmd_out_size); + +static int __nd_ioctl(struct nd_bus *nd_bus, struct nd_dimm *nd_dimm, + int read_only, unsigned int ioctl_cmd, unsigned long arg) +{ + struct nd_bus_descriptor *nd_desc = nd_bus->nd_desc; + size_t buf_len = 0, in_len = 0, out_len = 0; + static char out_env[ND_CMD_MAX_ENVELOPE]; + static char in_env[ND_CMD_MAX_ENVELOPE]; + const struct nd_cmd_desc *desc = NULL; + unsigned int cmd = _IOC_NR(ioctl_cmd); + void __user *p = (void __user *) arg; + struct device *dev = &nd_bus->dev; + const char *cmd_name, *dimm_name; + unsigned long dsm_mask; + void *buf; + int rc, i; + + if (nd_dimm) { + desc = nd_cmd_dimm_desc(cmd); + cmd_name = nd_dimm_cmd_name(cmd); + dsm_mask = nd_dimm->dsm_mask ? *(nd_dimm->dsm_mask) : 0; + dimm_name = dev_name(&nd_dimm->dev); + } else { + desc = nd_cmd_bus_desc(cmd); + cmd_name = nd_bus_cmd_name(cmd); + dsm_mask = nd_desc->dsm_mask; + dimm_name = "bus"; + } + + if (!desc || (desc->out_num + desc->in_num == 0) || + !test_bit(cmd, &dsm_mask)) + return -ENOTTY; + + /* fail write commands (when read-only) */ + if (read_only) + switch (ioctl_cmd) { + case ND_IOCTL_VENDOR: + case ND_IOCTL_SET_CONFIG_DATA: + case ND_IOCTL_ARS_START: + dev_dbg(&nd_bus->dev, "'%s' command while read-only.\n", + nd_dimm ? nd_dimm_cmd_name(cmd) + : nd_bus_cmd_name(cmd)); + return -EPERM; + default: + break; + } + + /* process an input envelope */ + for (i = 0; i < desc->in_num; i++) { + u32 in_size, copy; + + in_size = nd_cmd_in_size(nd_dimm, cmd, desc, i, in_env); + if (in_size == UINT_MAX) { + dev_err(dev, "%s:%s unknown input size cmd: %s field: %d\n", + __func__, dimm_name, cmd_name, i); + return -ENXIO; + } + if (!access_ok(VERIFY_READ, p + in_len, in_size)) + return -EFAULT; + if (in_len < sizeof(in_env)) + copy = min_t(u32, sizeof(in_env) - in_len, in_size); + else + copy = 0; + if (copy && copy_from_user(&in_env[in_len], p + in_len, copy)) + return -EFAULT; + in_len += in_size; + } + + /* process an output envelope */ + for (i = 0; i < desc->out_num; i++) { + u32 out_size = nd_cmd_out_size(nd_dimm, cmd, desc, i, + (u32 *) in_env, (u32 *) out_env); + u32 copy; + + if (out_size == UINT_MAX) { + dev_dbg(dev, "%s:%s unknown output size cmd: %s field: %d\n", + __func__, dimm_name, cmd_name, i); + return -EFAULT; + } + if (!access_ok(VERIFY_WRITE, p + in_len + out_len, out_size)) + return -EFAULT; + if (out_len < sizeof(out_env)) + copy = min_t(u32, sizeof(out_env) - out_len, out_size); + else + copy = 0; + if (copy && copy_from_user(&out_env[out_len], p + in_len + out_len, + copy)) + return -EFAULT; + out_len += out_size; + } + + buf_len = out_len + in_len; + if (!access_ok(VERIFY_WRITE, p, sizeof(buf_len))) + return -EFAULT; + + if (buf_len > ND_IOCTL_MAX_BUFLEN) { + dev_dbg(dev, "%s:%s cmd: %s buf_len: %zd > %d\n", __func__, + dimm_name, cmd_name, buf_len, + ND_IOCTL_MAX_BUFLEN); + return -EINVAL; + } + + buf = vmalloc(buf_len); + if (!buf) + return -ENOMEM; + + if (copy_from_user(buf, p, buf_len)) { + rc = -EFAULT; + goto out; + } + + rc = nd_desc->ndctl(nd_desc, nd_dimm, cmd, buf, buf_len); + if (rc < 0) + goto out; + if (copy_to_user(p, buf, buf_len)) + rc = -EFAULT; + out: + vfree(buf); + return rc; +} + static long nd_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { - return -ENXIO; + long id = (long) file->private_data; + int rc = -ENXIO, read_only; + struct nd_bus *nd_bus; + + read_only = (O_RDWR != (file->f_flags & O_ACCMODE)); + mutex_lock(&nd_bus_list_mutex); + list_for_each_entry(nd_bus, &nd_bus_list, list) { + if (nd_bus->id == id) { + rc = __nd_ioctl(nd_bus, NULL, read_only, cmd, arg); + break; + } + } + mutex_unlock(&nd_bus_list_mutex); + + return rc; +} + +static int match_dimm(struct device *dev, void *data) +{ + long id = (long) data; + + if (is_nd_dimm(dev)) { + struct nd_dimm *nd_dimm = to_nd_dimm(dev); + + return nd_dimm->id == id; + } + + return 0; +} + +static long nd_dimm_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + int rc = -ENXIO, read_only; + struct nd_bus *nd_bus; + + read_only = (O_RDWR != (file->f_flags & O_ACCMODE)); + mutex_lock(&nd_bus_list_mutex); + list_for_each_entry(nd_bus, &nd_bus_list, list) { + struct device *dev = device_find_child(&nd_bus->dev, + file->private_data, match_dimm); + + if (!dev) + continue; + + rc = __nd_ioctl(nd_bus, to_nd_dimm(dev), read_only, cmd, arg); + put_device(dev); + break; + } + mutex_unlock(&nd_bus_list_mutex); + + return rc; +} + +static int nd_open(struct inode *inode, struct file *file) +{ + long minor = iminor(inode); + + file->private_data = (void *) minor; + return 0; } static const struct file_operations nd_bus_fops = { .owner = THIS_MODULE, - .open = nonseekable_open, + .open = nd_open, .unlocked_ioctl = nd_ioctl, .compat_ioctl = nd_ioctl, .llseek = noop_llseek, }; +static const struct file_operations nd_dimm_fops = { + .owner = THIS_MODULE, + .open = nd_open, + .unlocked_ioctl = nd_dimm_ioctl, + .compat_ioctl = nd_dimm_ioctl, + .llseek = noop_llseek, +}; + int __init nd_bus_init(void) { int rc; @@ -70,9 +378,14 @@ int __init nd_bus_init(void) rc = register_chrdev(0, "ndctl", &nd_bus_fops); if (rc < 0) - goto err_chrdev; + goto err_bus_chrdev; nd_bus_major = rc; + rc = register_chrdev(0, "dimmctl", &nd_dimm_fops); + if (rc < 0) + goto err_dimm_chrdev; + nd_dimm_major = rc; + nd_class = class_create(THIS_MODULE, "nd"); if (IS_ERR(nd_class)) goto err_class; @@ -80,8 +393,10 @@ int __init nd_bus_init(void) return 0; err_class: + unregister_chrdev(nd_dimm_major, "dimmctl"); + err_dimm_chrdev: unregister_chrdev(nd_bus_major, "ndctl"); - err_chrdev: + err_bus_chrdev: bus_unregister(&nd_bus_type); return rc; @@ -91,5 +406,6 @@ void __exit nd_bus_exit(void) { class_destroy(nd_class); unregister_chrdev(nd_bus_major, "ndctl"); + unregister_chrdev(nd_dimm_major, "dimmctl"); bus_unregister(&nd_bus_type); } diff --git a/drivers/block/nd/core.c b/drivers/block/nd/core.c index 24046db897c1..b14bf47dc292 100644 --- a/drivers/block/nd/core.c +++ b/drivers/block/nd/core.c @@ -13,6 +13,7 @@ #include <linux/export.h> #include <linux/module.h> #include <linux/device.h> +#include <linux/ndctl.h> #include <linux/mutex.h> #include <linux/slab.h> #include "nd-private.h" @@ -59,6 +60,20 @@ struct nd_bus *walk_to_nd_bus(struct device *nd_dev) return NULL; } +static ssize_t commands_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + int cmd, len = 0; + struct nd_bus *nd_bus = to_nd_bus(dev); + struct nd_bus_descriptor *nd_desc = nd_bus->nd_desc; + + for_each_set_bit(cmd, &nd_desc->dsm_mask, BITS_PER_LONG) + len += sprintf(buf + len, "%s ", nd_bus_cmd_name(cmd)); + len += sprintf(buf + len, "\n"); + return len; +} +static DEVICE_ATTR_RO(commands); + static const char *nd_bus_provider(struct nd_bus *nd_bus) { struct nd_bus_descriptor *nd_desc = nd_bus->nd_desc; @@ -82,6 +97,7 @@ static ssize_t provider_show(struct device *dev, static DEVICE_ATTR_RO(provider); static struct attribute *nd_bus_attributes[] = { + &dev_attr_commands.attr, &dev_attr_provider.attr, NULL, }; diff --git a/drivers/block/nd/dimm_devs.c b/drivers/block/nd/dimm_devs.c index 19b081392f2f..3fa26f61c3db 100644 --- a/drivers/block/nd/dimm_devs.c +++ b/drivers/block/nd/dimm_devs.c @@ -12,6 +12,7 @@ */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include <linux/device.h> +#include <linux/ndctl.h> #include <linux/slab.h> #include <linux/io.h> #include <linux/fs.h> @@ -33,7 +34,7 @@ static struct device_type nd_dimm_device_type = { .release = nd_dimm_release, }; -static bool is_nd_dimm(struct device *dev) +bool is_nd_dimm(struct device *dev) { return dev->type == &nd_dimm_device_type; } @@ -55,12 +56,41 @@ EXPORT_SYMBOL_GPL(nd_dimm_name); void *nd_dimm_provider_data(struct nd_dimm *nd_dimm) { - return nd_dimm->provider_data; + if (nd_dimm) + return nd_dimm->provider_data; + return NULL; } EXPORT_SYMBOL_GPL(nd_dimm_provider_data); +static ssize_t commands_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_dimm *nd_dimm = to_nd_dimm(dev); + int cmd, len = 0; + + if (!nd_dimm->dsm_mask) + return sprintf(buf, "\n"); + + for_each_set_bit(cmd, nd_dimm->dsm_mask, BITS_PER_LONG) + len += sprintf(buf + len, "%s ", nd_dimm_cmd_name(cmd)); + len += sprintf(buf + len, "\n"); + return len; +} +static DEVICE_ATTR_RO(commands); + +static struct attribute *nd_dimm_attributes[] = { + &dev_attr_commands.attr, + NULL, +}; + +struct attribute_group nd_dimm_attribute_group = { + .attrs = nd_dimm_attributes, +}; +EXPORT_SYMBOL_GPL(nd_dimm_attribute_group); + struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, - const struct attribute_group **groups, unsigned long flags) + const struct attribute_group **groups, unsigned long flags, + unsigned long *dsm_mask) { struct nd_dimm *nd_dimm = kzalloc(sizeof(*nd_dimm), GFP_KERNEL); struct device *dev; @@ -75,12 +105,14 @@ struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, } nd_dimm->provider_data = provider_data; nd_dimm->flags = flags; + nd_dimm->dsm_mask = dsm_mask; dev = &nd_dimm->dev; dev_set_name(dev, "nmem%d", nd_dimm->id); dev->parent = &nd_bus->dev; dev->type = &nd_dimm_device_type; dev->bus = &nd_bus_type; + dev->devt = MKDEV(nd_dimm_major, nd_dimm->id); dev->groups = groups; if (device_register(dev) != 0) { put_device(dev); diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h index 4f8f2ebbcf7b..23dae8c1e65b 100644 --- a/drivers/block/nd/libnd.h +++ b/drivers/block/nd/libnd.h @@ -14,13 +14,21 @@ */ #ifndef __LIBND_H__ #define __LIBND_H__ +#include <linux/sizes.h> enum { /* when a dimm supports both PMEM and BLK access a label is required */ NDD_ALIASING = 1 << 0, + + /* need to set a limit somewhere, but yes, this is likely overkill */ + ND_IOCTL_MAX_BUFLEN = SZ_4M, + ND_CMD_MAX_ELEM = 4, + ND_CMD_MAX_ENVELOPE = 16, + ND_CMD_ARS_QUERY_MAX = SZ_4K, }; extern struct attribute_group nd_bus_attribute_group; +extern struct attribute_group nd_dimm_attribute_group; struct nd_dimm; struct nd_bus_descriptor; @@ -35,6 +43,13 @@ struct nd_bus_descriptor { ndctl_fn ndctl; }; +struct nd_cmd_desc { + int in_num; + int out_num; + u32 in_sizes[ND_CMD_MAX_ELEM]; + int out_sizes[ND_CMD_MAX_ELEM]; +}; + struct nd_bus; struct nd_bus *nd_bus_register(struct device *parent, struct nd_bus_descriptor *nfit_desc); @@ -45,5 +60,13 @@ struct nd_bus_descriptor *to_nd_desc(struct nd_bus *nd_bus); const char *nd_dimm_name(struct nd_dimm *nd_dimm); void *nd_dimm_provider_data(struct nd_dimm *nd_dimm); struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, - const struct attribute_group **groups, unsigned long flags); + const struct attribute_group **groups, unsigned long flags, + unsigned long *dsm_mask); +const struct nd_cmd_desc const *nd_cmd_dimm_desc(int cmd); +const struct nd_cmd_desc const *nd_cmd_bus_desc(int cmd); +u32 nd_cmd_in_size(struct nd_dimm *nd_dimm, int cmd, + const struct nd_cmd_desc *desc, int idx, void *buf); +u32 nd_cmd_out_size(struct nd_dimm *nd_dimm, int cmd, + const struct nd_cmd_desc *desc, int idx, const u32 *in_field, + const u32 *out_field); #endif /* __LIBND_H__ */ diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index cfb5a7241ed1..968e5089f72c 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -18,6 +18,7 @@ extern struct list_head nd_bus_list; extern struct mutex nd_bus_list_mutex; extern struct bus_type nd_bus_type; +extern int nd_dimm_major; struct nd_bus { struct nd_bus_descriptor *nd_desc; @@ -29,10 +30,12 @@ struct nd_bus { struct nd_dimm { unsigned long flags; void *provider_data; + unsigned long *dsm_mask; struct device dev; int id; }; +bool is_nd_dimm(struct device *dev); struct nd_bus *walk_to_nd_bus(struct device *nd_dev); int __init nd_bus_init(void); void __exit nd_bus_exit(void); diff --git a/drivers/block/nd/test/nfit.c b/drivers/block/nd/test/nfit.c index 1c79f32376fc..50916f0ca901 100644 --- a/drivers/block/nd/test/nfit.c +++ b/drivers/block/nd/test/nfit.c @@ -14,6 +14,7 @@ #include <linux/platform_device.h> #include <linux/dma-mapping.h> #include <linux/module.h> +#include <linux/ndctl.h> #include <linux/sizes.h> #include <linux/slab.h> #include "nfit_test.h" @@ -142,7 +143,74 @@ static int nfit_test_ctl(struct nd_bus_descriptor *nd_desc, struct nd_dimm *nd_dimm, unsigned int cmd, void *buf, unsigned int buf_len) { - return -ENOTTY; + struct acpi_nfit_desc *acpi_desc = to_acpi_desc(nd_desc); + struct nfit_test *t = container_of(acpi_desc, typeof(*t), acpi_desc); + struct nfit_mem *nfit_mem = nd_dimm_provider_data(nd_dimm); + int i, rc; + + if (!nfit_mem || !test_bit(cmd, &nfit_mem->dsm_mask)) + return -ENXIO; + + /* lookup label space for the given dimm */ + for (i = 0; i < ARRAY_SIZE(handle); i++) + if (__to_nfit_memdev(nfit_mem)->nfit_handle == handle[i]) + break; + if (i >= ARRAY_SIZE(handle)) + return -ENXIO; + + switch (cmd) { + case ND_CMD_GET_CONFIG_SIZE: { + struct nd_cmd_get_config_size *nd_cmd = buf; + + if (buf_len < sizeof(*nd_cmd)) + return -EINVAL; + nd_cmd->status = 0; + nd_cmd->config_size = LABEL_SIZE; + nd_cmd->max_xfer = SZ_4K; + rc = 0; + break; + } + case ND_CMD_GET_CONFIG_DATA: { + struct nd_cmd_get_config_data_hdr *nd_cmd = buf; + unsigned int len, offset = nd_cmd->in_offset; + + if (buf_len < sizeof(*nd_cmd)) + return -EINVAL; + if (offset >= LABEL_SIZE) + return -EINVAL; + if (nd_cmd->in_length + sizeof(*nd_cmd) > buf_len) + return -EINVAL; + + nd_cmd->status = 0; + len = min(nd_cmd->in_length, LABEL_SIZE - offset); + memcpy(nd_cmd->out_buf, t->label[i] + offset, len); + rc = buf_len - sizeof(*nd_cmd) - len; + break; + } + case ND_CMD_SET_CONFIG_DATA: { + struct nd_cmd_set_config_hdr *nd_cmd = buf; + unsigned int len, offset = nd_cmd->in_offset; + u32 *status; + + if (buf_len < sizeof(*nd_cmd)) + return -EINVAL; + if (offset >= LABEL_SIZE) + return -EINVAL; + if (nd_cmd->in_length + sizeof(*nd_cmd) + 4 > buf_len) + return -EINVAL; + + status = buf + nd_cmd->in_length + sizeof(*nd_cmd); + *status = 0; + len = min(nd_cmd->in_length, LABEL_SIZE - offset); + memcpy(t->label[i] + offset, nd_cmd->in_buf, len); + rc = buf_len - sizeof(*nd_cmd) - (len + 4); + break; + } + default: + return -ENOTTY; + } + + return rc; } static DEFINE_SPINLOCK(nfit_test_lock); @@ -280,6 +348,7 @@ static int nfit_test0_alloc(struct nfit_test *t) t->label[i] = test_alloc(t, LABEL_SIZE, &t->label_dma[i]); if (!t->label[i]) return -ENOMEM; + sprintf(t->label[i], "label%d", i); } for (i = 0; i < NUM_DCR; i++) { @@ -322,6 +391,7 @@ static int nfit_test1_alloc(struct nfit_test *t) static void nfit_test0_setup(struct nfit_test *t) { struct nd_bus_descriptor *nd_desc; + struct acpi_nfit_desc *acpi_desc; struct acpi_nfit_memdev *memdev; void *nfit_buf = t->nfit_buf; size_t size = t->nfit_size; @@ -763,7 +833,11 @@ static void nfit_test0_setup(struct nfit_test *t) nfit->checksum = nfit_checksum(nfit_buf, size); - nd_desc = &t->acpi_desc.nd_desc; + acpi_desc = &t->acpi_desc; + set_bit(ND_CMD_GET_CONFIG_SIZE, &acpi_desc->dimm_dsm_force_en); + set_bit(ND_CMD_GET_CONFIG_DATA, &acpi_desc->dimm_dsm_force_en); + set_bit(ND_CMD_SET_CONFIG_DATA, &acpi_desc->dimm_dsm_force_en); + nd_desc = &acpi_desc->nd_desc; nd_desc->ndctl = nfit_test_ctl; } diff --git a/include/uapi/linux/Kbuild b/include/uapi/linux/Kbuild index 68ceb97c458c..384e8d212b04 100644 --- a/include/uapi/linux/Kbuild +++ b/include/uapi/linux/Kbuild @@ -270,6 +270,7 @@ header-y += ncp_fs.h header-y += ncp.h header-y += ncp_mount.h header-y += ncp_no.h +header-y += ndctl.h header-y += neighbour.h header-y += netconf.h header-y += netdevice.h diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h new file mode 100644 index 000000000000..62c01bf76198 --- /dev/null +++ b/include/uapi/linux/ndctl.h @@ -0,0 +1,178 @@ +/* + * Copyright (c) 2014-2015, Intel Corporation. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU Lesser General Public License, + * version 2.1, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT ANY + * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS + * FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for + * more details. + */ +#ifndef __NDCTL_H__ +#define __NDCTL_H__ + +#include <linux/types.h> + +struct nd_cmd_smart { + __u32 status; + __u8 data[128]; +} __packed; + +struct nd_cmd_smart_threshold { + __u32 status; + __u8 data[8]; +} __packed; + +struct nd_cmd_dimm_flags { + __u32 status; + __u32 flags; +} __packed; + +struct nd_cmd_get_config_size { + __u32 status; + __u32 config_size; + __u32 max_xfer; +} __packed; + +struct nd_cmd_get_config_data_hdr { + __u32 in_offset; + __u32 in_length; + __u32 status; + __u8 out_buf[0]; +} __packed; + +struct nd_cmd_set_config_hdr { + __u32 in_offset; + __u32 in_length; + __u8 in_buf[0]; +} __packed; + +struct nd_cmd_vendor_hdr { + __u32 opcode; + __u32 in_length; + __u8 in_buf[0]; +} __packed; + +struct nd_cmd_vendor_tail { + __u32 status; + __u32 out_length; + __u8 out_buf[0]; +} __packed; + +struct nd_cmd_ars_cap { + __u64 address; + __u64 length; + __u32 status; + __u32 max_ars_out; +} __packed; + +struct nd_cmd_ars_start { + __u64 address; + __u64 length; + __u16 type; + __u8 reserved[6]; + __u32 status; +} __packed; + +struct nd_cmd_ars_query { + __u32 status; + __u32 out_length; + __u64 address; + __u64 length; + __u16 type; + __u32 num_records; + struct nd_ars_record { + __u32 handle; + __u32 flags; + __u64 err_address; + __u64 mask; + } __packed records[0]; +} __packed; + +enum { + ND_CMD_IMPLEMENTED = 0, + + /* bus commands */ + ND_CMD_ARS_CAP = 1, + ND_CMD_ARS_START = 2, + ND_CMD_ARS_QUERY = 3, + + /* per-dimm commands */ + ND_CMD_SMART = 1, + ND_CMD_SMART_THRESHOLD = 2, + ND_CMD_DIMM_FLAGS = 3, + ND_CMD_GET_CONFIG_SIZE = 4, + ND_CMD_GET_CONFIG_DATA = 5, + ND_CMD_SET_CONFIG_DATA = 6, + ND_CMD_VENDOR_EFFECT_LOG_SIZE = 7, + ND_CMD_VENDOR_EFFECT_LOG = 8, + ND_CMD_VENDOR = 9, +}; + +static inline const char *nd_bus_cmd_name(unsigned cmd) +{ + static const char * const names[] = { + [ND_CMD_ARS_CAP] = "ars_cap", + [ND_CMD_ARS_START] = "ars_start", + [ND_CMD_ARS_QUERY] = "ars_query", + }; + + if (cmd < ARRAY_SIZE(names) && names[cmd]) + return names[cmd]; + return "unknown"; +} + +static inline const char *nd_dimm_cmd_name(unsigned cmd) +{ + static const char * const names[] = { + [ND_CMD_SMART] = "smart", + [ND_CMD_SMART_THRESHOLD] = "smart_thresh", + [ND_CMD_DIMM_FLAGS] = "flags", + [ND_CMD_GET_CONFIG_SIZE] = "get_size", + [ND_CMD_GET_CONFIG_DATA] = "get_data", + [ND_CMD_SET_CONFIG_DATA] = "set_data", + [ND_CMD_VENDOR_EFFECT_LOG_SIZE] = "effect_size", + [ND_CMD_VENDOR_EFFECT_LOG] = "effect_log", + [ND_CMD_VENDOR] = "vendor", + }; + + if (cmd < ARRAY_SIZE(names) && names[cmd]) + return names[cmd]; + return "unknown"; +} + +#define ND_IOCTL 'N' + +#define ND_IOCTL_SMART _IOWR(ND_IOCTL, ND_CMD_SMART,\ + struct nd_cmd_smart) + +#define ND_IOCTL_SMART_THRESHOLD _IOWR(ND_IOCTL, ND_CMD_SMART_THRESHOLD,\ + struct nd_cmd_smart_threshold) + +#define ND_IOCTL_DIMM_FLAGS _IOWR(ND_IOCTL, ND_CMD_DIMM_FLAGS,\ + struct nd_cmd_dimm_flags) + +#define ND_IOCTL_GET_CONFIG_SIZE _IOWR(ND_IOCTL, ND_CMD_GET_CONFIG_SIZE,\ + struct nd_cmd_get_config_size) + +#define ND_IOCTL_GET_CONFIG_DATA _IOWR(ND_IOCTL, ND_CMD_GET_CONFIG_DATA,\ + struct nd_cmd_get_config_data_hdr) + +#define ND_IOCTL_SET_CONFIG_DATA _IOWR(ND_IOCTL, ND_CMD_SET_CONFIG_DATA,\ + struct nd_cmd_set_config_hdr) + +#define ND_IOCTL_VENDOR _IOWR(ND_IOCTL, ND_CMD_VENDOR,\ + struct nd_cmd_vendor_hdr) + +#define ND_IOCTL_ARS_CAP _IOWR(ND_IOCTL, ND_CMD_ARS_CAP,\ + struct nd_cmd_ars_cap) + +#define ND_IOCTL_ARS_START _IOWR(ND_IOCTL, ND_CMD_ARS_START,\ + struct nd_cmd_ars_start) + +#define ND_IOCTL_ARS_QUERY _IOWR(ND_IOCTL, ND_CMD_ARS_QUERY,\ + struct nd_cmd_ars_query) + +#endif /* __NDCTL_H__ */

... It would reduce the slight window for stale pointer usage if you add: dev_set_drvdata(dev, NULL); before free_data(ndd); in both nd_dimm_remove and the err exit for nd_dimm_probe. The same comment applies to all the drivers that store pointers with dev_set_drvdata - btt, pmem, etc.

A "region" device represents the maximum capacity of a BLK range (mmio block-data-window(s)), or a PMEM range (DAX-capable persistent memory or volatile memory), without regard for aliasing. Aliasing, in the dimm-local address space (DPA), is resolved by metadata on a dimm to designate which exclusive interface will access the aliased DPA ranges. Support for the per-dimm metadata/label arrvies is in a subsequent patch. The name format of "region" devices is "regionN" where, like dimms, N is a global ida index assigned at discovery time. This id is not reliable across reboots nor in the presence of hotplug. Look to attributes of the region or static id-data of the sub-namespace to generate a persistent name. "region"s have 2 generic attributes "size", and "mapping"s where: - size: the block-data-window accessible capacity or the span of the spa-range in the case of pm. - mappingN: a tuple describing a dimm's contribution to the region's capacity in the format (<nmemX>,<dpa>,<size>). For a PMEM-region there will be at least one mapping per dimm in the interleave set. For a BLK-region there is only "mapping0" listing the starting dimm offset of the block-data-window and the available capacity of that window (matches "size" above). The max number of mappings per "region" is hard coded per the constraints of sysfs attribute groups. That said the number of mappings per region should never exceed the maximum number of possible dimms in the system. If the current number turns out to not be enough then the "mappings" attribute clarifies how many there are supposed to be. "32 should be enough for anybody...". Cc: Neil Brown <neilb(a)suse.de&gt; Cc: <linux-acpi(a)vger.kernel.org&gt; Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; Cc: Robert Moore <robert.moore(a)intel.com&gt; Cc: Rafael J. Wysocki <rafael.j.wysocki(a)intel.com&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/Makefile | 1 drivers/block/nd/acpi.c | 130 ++++++++++++++++++ drivers/block/nd/libnd.h | 25 +++ drivers/block/nd/nd-private.h | 3 drivers/block/nd/nd.h | 11 + drivers/block/nd/region_devs.c | 294 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 463 insertions(+), 1 deletion(-) create mode 100644 drivers/block/nd/region_devs.c diff --git a/drivers/block/nd/Makefile b/drivers/block/nd/Makefile index 842ba13253fd..6010469c4d4c 100644 --- a/drivers/block/nd/Makefile +++ b/drivers/block/nd/Makefile @@ -22,3 +22,4 @@ libnd-y := core.o libnd-y += bus.o libnd-y += dimm_devs.o libnd-y += dimm.o +libnd-y += region_devs.o diff --git a/drivers/block/nd/acpi.c b/drivers/block/nd/acpi.c index bb0c2c764e78..41d0bb732b3e 100644 --- a/drivers/block/nd/acpi.c +++ b/drivers/block/nd/acpi.c @@ -751,12 +751,136 @@ static void nd_acpi_init_dsms(struct acpi_nfit_desc *acpi_desc) set_bit(i, &nd_desc->dsm_mask); } +static ssize_t spa_index_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_region *nd_region = to_nd_region(dev); + struct nfit_spa *nfit_spa = nd_region_provider_data(nd_region); + + return sprintf(buf, "%d\n", nfit_spa->spa->spa_index); +} +static DEVICE_ATTR_RO(spa_index); + +static struct attribute *nd_acpi_region_attributes[] = { + &dev_attr_spa_index.attr, + NULL, +}; + +static struct attribute_group nd_acpi_region_attribute_group = { + .name = "nfit", + .attrs = nd_acpi_region_attributes, +}; + +static const struct attribute_group *nd_acpi_region_attribute_groups[] = { + &nd_region_attribute_group, + &nd_mapping_attribute_group, + &nd_acpi_region_attribute_group, + NULL, +}; + +static int nd_acpi_register_region(struct acpi_nfit_desc *acpi_desc, + struct nfit_spa *nfit_spa) +{ + static struct nd_mapping nd_mappings[ND_MAX_MAPPINGS]; + struct acpi_nfit_spa *spa = nfit_spa->spa; + struct nfit_memdev *nfit_memdev; + struct nd_region_desc ndr_desc; + int spa_type, count = 0; + struct resource res; + u16 spa_index; + + spa_type = nfit_spa_type(spa); + spa_index = spa->spa_index; + if (spa_index == 0) { + dev_dbg(acpi_desc->dev, "%s: detected invalid spa index\n", + __func__); + return 0; + } + + memset(&res, 0, sizeof(res)); + memset(&nd_mappings, 0, sizeof(nd_mappings)); + memset(&ndr_desc, 0, sizeof(ndr_desc)); + res.start = spa->spa_base; + res.end = res.start + spa->spa_length - 1; + ndr_desc.res = &res; + ndr_desc.provider_data = nfit_spa; + ndr_desc.attr_groups = nd_acpi_region_attribute_groups; + list_for_each_entry(nfit_memdev, &acpi_desc->memdevs, list) { + struct acpi_nfit_memdev *memdev = nfit_memdev->memdev; + struct nd_mapping *nd_mapping; + struct nd_dimm *nd_dimm; + + if (memdev->spa_index != spa_index) + continue; + if (count >= ND_MAX_MAPPINGS) { + dev_err(acpi_desc->dev, "spa%d exceeds max mappings %d\n", + spa_index, ND_MAX_MAPPINGS); + return -ENXIO; + } + nd_dimm = nd_acpi_dimm_by_handle(acpi_desc, memdev->nfit_handle); + if (!nd_dimm) { + dev_err(acpi_desc->dev, "spa%d dimm: %#x not found\n", + spa_index, memdev->nfit_handle); + return -ENODEV; + } + nd_mapping = &nd_mappings[count++]; + nd_mapping->nd_dimm = nd_dimm; + if (spa_type == NFIT_SPA_PM || spa_type == NFIT_SPA_VOLATILE) { + nd_mapping->start = memdev->region_dpa; + nd_mapping->size = memdev->region_len; + } else if (spa_type == NFIT_SPA_DCR) { + struct nfit_mem *nfit_mem; + int blk_valid = 1; + + nfit_mem = nd_dimm_provider_data(nd_dimm); + if (!nfit_mem || !nfit_mem->bdw) { + dev_dbg(acpi_desc->dev, "%s: spa%d missing bdw\n", + nd_dimm_name(nd_dimm), spa_index); + blk_valid = 0; + } else { + nd_mapping->size = nfit_mem->bdw->blk_capacity; + nd_mapping->start = nfit_mem->bdw->blk_offset; + } + + ndr_desc.nd_mapping = nd_mapping; + ndr_desc.num_mappings = blk_valid; + if (!nd_blk_region_create(acpi_desc->nd_bus, &ndr_desc)) + return -ENOMEM; + } + } + + ndr_desc.nd_mapping = nd_mappings; + ndr_desc.num_mappings = count; + if (spa_type == NFIT_SPA_PM) { + if (!nd_pmem_region_create(acpi_desc->nd_bus, &ndr_desc)) + return -ENOMEM; + } else if (spa_type == NFIT_SPA_VOLATILE) { + if (!nd_volatile_region_create(acpi_desc->nd_bus, &ndr_desc)) + return -ENOMEM; + } + return 0; +} + +static int nd_acpi_register_regions(struct acpi_nfit_desc *acpi_desc) +{ + struct nfit_spa *nfit_spa; + + list_for_each_entry(nfit_spa, &acpi_desc->spas, list) { + int rc = nd_acpi_register_region(acpi_desc, nfit_spa); + + if (rc) + return rc; + } + return 0; +} + int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) { struct device *dev = acpi_desc->dev; const void *end; u8 *data, sum; acpi_size i; + int rc; INIT_LIST_HEAD(&acpi_desc->spas); INIT_LIST_HEAD(&acpi_desc->dcrs); @@ -790,7 +914,11 @@ int nd_acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) nd_acpi_init_dsms(acpi_desc); - return nd_acpi_register_dimms(acpi_desc); + rc = nd_acpi_register_dimms(acpi_desc); + if (rc) + return rc; + + return nd_acpi_register_regions(acpi_desc); } EXPORT_SYMBOL_GPL(nd_acpi_nfit_init); diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h index 3b91df914b76..630a703d1316 100644 --- a/drivers/block/nd/libnd.h +++ b/drivers/block/nd/libnd.h @@ -25,11 +25,14 @@ enum { ND_CMD_MAX_ELEM = 4, ND_CMD_MAX_ENVELOPE = 16, ND_CMD_ARS_QUERY_MAX = SZ_4K, + ND_MAX_MAPPINGS = 32, }; extern struct attribute_group nd_bus_attribute_group; extern struct attribute_group nd_dimm_attribute_group; extern struct attribute_group nd_device_attribute_group; +extern struct attribute_group nd_region_attribute_group; +extern struct attribute_group nd_mapping_attribute_group; struct nd_dimm; struct nd_bus_descriptor; @@ -37,6 +40,12 @@ typedef int (*ndctl_fn)(struct nd_bus_descriptor *nd_desc, struct nd_dimm *nd_dimm, unsigned int cmd, void *buf, unsigned int buf_len); +struct nd_mapping { + struct nd_dimm *nd_dimm; + u64 start; + u64 size; +}; + struct nd_bus_descriptor { const struct attribute_group **attr_groups; unsigned long dsm_mask; @@ -51,15 +60,25 @@ struct nd_cmd_desc { int out_sizes[ND_CMD_MAX_ELEM]; }; +struct nd_region_desc { + struct resource *res; + struct nd_mapping *nd_mapping; + u16 num_mappings; + const struct attribute_group **attr_groups; + void *provider_data; +}; + struct nd_bus; struct nd_bus *nd_bus_register(struct device *parent, struct nd_bus_descriptor *nfit_desc); void nd_bus_unregister(struct nd_bus *nd_bus); struct nd_bus *to_nd_bus(struct device *dev); struct nd_dimm *to_nd_dimm(struct device *dev); +struct nd_region *to_nd_region(struct device *dev); struct nd_bus_descriptor *to_nd_desc(struct nd_bus *nd_bus); const char *nd_dimm_name(struct nd_dimm *nd_dimm); void *nd_dimm_provider_data(struct nd_dimm *nd_dimm); +void *nd_region_provider_data(struct nd_region *nd_region); struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, const struct attribute_group **groups, unsigned long flags, unsigned long *dsm_mask); @@ -71,4 +90,10 @@ u32 nd_cmd_out_size(struct nd_dimm *nd_dimm, int cmd, const struct nd_cmd_desc *desc, int idx, const u32 *in_field, const u32 *out_field); int nd_bus_validate_dimm_count(struct nd_bus *nd_bus, int dimm_count); +struct nd_region *nd_pmem_region_create(struct nd_bus *nd_bus, + struct nd_region_desc *ndr_desc); +struct nd_region *nd_blk_region_create(struct nd_bus *nd_bus, + struct nd_region_desc *ndr_desc); +struct nd_region *nd_volatile_region_create(struct nd_bus *nd_bus, + struct nd_region_desc *ndr_desc); #endif /* __LIBND_H__ */ diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index 35ab0d476d15..838b6f958c00 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -42,5 +42,8 @@ void __exit nd_dimm_exit(void); int nd_bus_create_ndctl(struct nd_bus *nd_bus); void nd_bus_destroy_ndctl(struct nd_bus *nd_bus); void nd_synchronize(void); +int nd_bus_register_dimms(struct nd_bus *nd_bus); +int nd_bus_register_regions(struct nd_bus *nd_bus); +int nd_match_dimm(struct device *dev, void *data); bool is_nd_dimm(struct device *dev); #endif /* __ND_PRIVATE_H__ */ diff --git a/drivers/block/nd/nd.h b/drivers/block/nd/nd.h index 1a5a081ce640..cae83de12c45 100644 --- a/drivers/block/nd/nd.h +++ b/drivers/block/nd/nd.h @@ -15,6 +15,7 @@ #include <linux/device.h> #include <linux/mutex.h> #include <linux/ndctl.h> +#include "libnd.h" struct nd_dimm_drvdata { struct device *dev; @@ -22,6 +23,16 @@ struct nd_dimm_drvdata { void *data; }; +struct nd_region { + struct device dev; + u16 ndr_mappings; + u64 ndr_size; + u64 ndr_start; + int id; + void *provider_data; + struct nd_mapping mapping[0]; +}; + enum nd_async_mode { ND_SYNC, ND_ASYNC, diff --git a/drivers/block/nd/region_devs.c b/drivers/block/nd/region_devs.c new file mode 100644 index 000000000000..12a5415acfcc --- /dev/null +++ b/drivers/block/nd/region_devs.c @@ -0,0 +1,294 @@ +/* + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#include <linux/slab.h> +#include <linux/io.h> +#include "nd-private.h" +#include "nd.h" + +static DEFINE_IDA(region_ida); + +static void nd_region_release(struct device *dev) +{ + struct nd_region *nd_region = to_nd_region(dev); + u16 i; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_dimm *nd_dimm = nd_mapping->nd_dimm; + + put_device(&nd_dimm->dev); + } + ida_simple_remove(&region_ida, nd_region->id); + kfree(nd_region); +} + +static struct device_type nd_blk_device_type = { + .name = "nd_blk", + .release = nd_region_release, +}; + +static struct device_type nd_pmem_device_type = { + .name = "nd_pmem", + .release = nd_region_release, +}; + +static struct device_type nd_volatile_device_type = { + .name = "nd_volatile", + .release = nd_region_release, +}; + +static bool is_nd_pmem(struct device *dev) +{ + return dev ? dev->type == &nd_pmem_device_type : false; +} + +struct nd_region *to_nd_region(struct device *dev) +{ + struct nd_region *nd_region = container_of(dev, struct nd_region, dev); + + WARN_ON(dev->type->release != nd_region_release); + return nd_region; +} +EXPORT_SYMBOL_GPL(to_nd_region); + +static ssize_t size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_region *nd_region = to_nd_region(dev); + unsigned long long size = 0; + + if (is_nd_pmem(dev)) { + size = nd_region->ndr_size; + } else if (nd_region->ndr_mappings == 1) { + struct nd_mapping *nd_mapping = &nd_region->mapping[0]; + + size = nd_mapping->size; + } + + return sprintf(buf, "%llu\n", size); +} +static DEVICE_ATTR_RO(size); + +static ssize_t mappings_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_region *nd_region = to_nd_region(dev); + + return sprintf(buf, "%d\n", nd_region->ndr_mappings); +} +static DEVICE_ATTR_RO(mappings); + +static struct attribute *nd_region_attributes[] = { + &dev_attr_size.attr, + &dev_attr_mappings.attr, + NULL, +}; + +struct attribute_group nd_region_attribute_group = { + .attrs = nd_region_attributes, +}; +EXPORT_SYMBOL_GPL(nd_region_attribute_group); + +static ssize_t mappingN(struct device *dev, char *buf, int n) +{ + struct nd_region *nd_region = to_nd_region(dev); + struct nd_mapping *nd_mapping; + struct nd_dimm *nd_dimm; + + if (n >= nd_region->ndr_mappings) + return -ENXIO; + nd_mapping = &nd_region->mapping[n]; + nd_dimm = nd_mapping->nd_dimm; + + return sprintf(buf, "%s,%llu,%llu\n", dev_name(&nd_dimm->dev), + nd_mapping->start, nd_mapping->size); +} + +#define REGION_MAPPING(idx) \ +static ssize_t mapping##idx##_show(struct device *dev, \ + struct device_attribute *attr, char *buf) \ +{ \ + return mappingN(dev, buf, idx); \ +} \ +static DEVICE_ATTR_RO(mapping##idx) + +/* + * 32 should be enough for a while, even in the presence of socket + * interleave a 32-way interleave set is a degenerate case. + */ +REGION_MAPPING(0); +REGION_MAPPING(1); +REGION_MAPPING(2); +REGION_MAPPING(3); +REGION_MAPPING(4); +REGION_MAPPING(5); +REGION_MAPPING(6); +REGION_MAPPING(7); +REGION_MAPPING(8); +REGION_MAPPING(9); +REGION_MAPPING(10); +REGION_MAPPING(11); +REGION_MAPPING(12); +REGION_MAPPING(13); +REGION_MAPPING(14); +REGION_MAPPING(15); +REGION_MAPPING(16); +REGION_MAPPING(17); +REGION_MAPPING(18); +REGION_MAPPING(19); +REGION_MAPPING(20); +REGION_MAPPING(21); +REGION_MAPPING(22); +REGION_MAPPING(23); +REGION_MAPPING(24); +REGION_MAPPING(25); +REGION_MAPPING(26); +REGION_MAPPING(27); +REGION_MAPPING(28); +REGION_MAPPING(29); +REGION_MAPPING(30); +REGION_MAPPING(31); + +static umode_t nd_mapping_visible(struct kobject *kobj, struct attribute *a, int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + struct nd_region *nd_region = to_nd_region(dev); + + if (n < nd_region->ndr_mappings) + return a->mode; + return 0; +} + +static struct attribute *nd_mapping_attributes[] = { + &dev_attr_mapping0.attr, + &dev_attr_mapping1.attr, + &dev_attr_mapping2.attr, + &dev_attr_mapping3.attr, + &dev_attr_mapping4.attr, + &dev_attr_mapping5.attr, + &dev_attr_mapping6.attr, + &dev_attr_mapping7.attr, + &dev_attr_mapping8.attr, + &dev_attr_mapping9.attr, + &dev_attr_mapping10.attr, + &dev_attr_mapping11.attr, + &dev_attr_mapping12.attr, + &dev_attr_mapping13.attr, + &dev_attr_mapping14.attr, + &dev_attr_mapping15.attr, + &dev_attr_mapping16.attr, + &dev_attr_mapping17.attr, + &dev_attr_mapping18.attr, + &dev_attr_mapping19.attr, + &dev_attr_mapping20.attr, + &dev_attr_mapping21.attr, + &dev_attr_mapping22.attr, + &dev_attr_mapping23.attr, + &dev_attr_mapping24.attr, + &dev_attr_mapping25.attr, + &dev_attr_mapping26.attr, + &dev_attr_mapping27.attr, + &dev_attr_mapping28.attr, + &dev_attr_mapping29.attr, + &dev_attr_mapping30.attr, + &dev_attr_mapping31.attr, + NULL, +}; + +struct attribute_group nd_mapping_attribute_group = { + .is_visible = nd_mapping_visible, + .attrs = nd_mapping_attributes, +}; +EXPORT_SYMBOL_GPL(nd_mapping_attribute_group); + +void *nd_region_provider_data(struct nd_region *nd_region) +{ + return nd_region->provider_data; +} +EXPORT_SYMBOL_GPL(nd_region_provider_data); + +static noinline struct nd_region *nd_region_create(struct nd_bus *nd_bus, + struct nd_region_desc *ndr_desc, struct device_type *dev_type) +{ + struct nd_region *nd_region; + struct device *dev; + u16 i; + + for (i = 0; i < ndr_desc->num_mappings; i++) { + struct nd_mapping *nd_mapping = &ndr_desc->nd_mapping[i]; + struct nd_dimm *nd_dimm = nd_mapping->nd_dimm; + + if ((nd_mapping->start | nd_mapping->size) % SZ_4K) { + dev_err(&nd_bus->dev, "%pf: %s mapping%d is not 4K aligned\n", + __builtin_return_address(0), + dev_name(&nd_dimm->dev), i); + + return NULL; + } + } + + nd_region = kzalloc(sizeof(struct nd_region) + + sizeof(struct nd_mapping) * ndr_desc->num_mappings, + GFP_KERNEL); + if (!nd_region) + return NULL; + nd_region->id = ida_simple_get(&region_ida, 0, 0, GFP_KERNEL); + if (nd_region->id < 0) { + kfree(nd_region); + return NULL; + } + + memcpy(nd_region->mapping, ndr_desc->nd_mapping, + sizeof(struct nd_mapping) * ndr_desc->num_mappings); + for (i = 0; i < ndr_desc->num_mappings; i++) { + struct nd_mapping *nd_mapping = &ndr_desc->nd_mapping[i]; + struct nd_dimm *nd_dimm = nd_mapping->nd_dimm; + + get_device(&nd_dimm->dev); + } + nd_region->ndr_mappings = ndr_desc->num_mappings; + nd_region->provider_data = ndr_desc->provider_data; + dev = &nd_region->dev; + dev_set_name(dev, "region%d", nd_region->id); + dev->parent = &nd_bus->dev; + dev->type = dev_type; + dev->groups = ndr_desc->attr_groups; + nd_region->ndr_size = resource_size(ndr_desc->res); + nd_region->ndr_start = ndr_desc->res->start; + nd_device_register(dev); + + return nd_region; +} + +struct nd_region *nd_pmem_region_create(struct nd_bus *nd_bus, + struct nd_region_desc *ndr_desc) +{ + return nd_region_create(nd_bus, ndr_desc, &nd_pmem_device_type); +} +EXPORT_SYMBOL_GPL(nd_pmem_region_create); + +struct nd_region *nd_blk_region_create(struct nd_bus *nd_bus, + struct nd_region_desc *ndr_desc) +{ + if (ndr_desc->num_mappings > 1) + return NULL; + return nd_region_create(nd_bus, ndr_desc, &nd_blk_device_type); +} +EXPORT_SYMBOL_GPL(nd_blk_region_create); + +struct nd_region *nd_volatile_region_create(struct nd_bus *nd_bus, + struct nd_region_desc *ndr_desc) +{ + return nd_region_create(nd_bus, ndr_desc, &nd_volatile_device_type); +} +EXPORT_SYMBOL_GPL(nd_volatile_region_create);

In preparation for the pmem driver attaching to pmem-namespaces emitted by libnd, convert it to use an ida instead of an always increasing atomic index. This provides a bit of stability to pmem device names in the presence of driver re-bind events. Cc: Christoph Hellwig <hch(a)lst.de&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/pmem.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/drivers/block/pmem.c b/drivers/block/pmem.c index eabf4a8d0085..e3cf9142b172 100644 --- a/drivers/block/pmem.c +++ b/drivers/block/pmem.c @@ -34,10 +34,11 @@ struct pmem_device { phys_addr_t phys_addr; void *virt_addr; size_t size; + int id; }; static int pmem_major; -static atomic_t pmem_index; +static DEFINE_IDA(pmem_ida); static void pmem_do_bvec(struct pmem_device *pmem, struct page *page, unsigned int len, unsigned int off, int rw, @@ -122,20 +123,26 @@ static struct pmem_device *pmem_alloc(struct device *dev, struct resource *res) { struct pmem_device *pmem; struct gendisk *disk; - int idx, err; + int err; err = -ENOMEM; pmem = kzalloc(sizeof(*pmem), GFP_KERNEL); if (!pmem) goto out; + pmem->id = ida_simple_get(&pmem_ida, 0, 0, GFP_KERNEL); + if (pmem->id < 0) { + err = pmem->id; + goto out_free_dev; + } + pmem->phys_addr = res->start; pmem->size = resource_size(res); err = -EINVAL; if (!request_mem_region(pmem->phys_addr, pmem->size, "pmem")) { dev_warn(dev, "could not reserve region [0x%pa:0x%zx]\n", &pmem->phys_addr, pmem->size); - goto out_free_dev; + goto out_free_ida; } /* @@ -159,15 +166,13 @@ static struct pmem_device *pmem_alloc(struct device *dev, struct resource *res) if (!disk) goto out_free_queue; - idx = atomic_inc_return(&pmem_index) - 1; - disk->major = pmem_major; - disk->first_minor = PMEM_MINORS * idx; + disk->first_minor = PMEM_MINORS * pmem->id; disk->fops = &pmem_fops; disk->private_data = pmem; disk->queue = pmem->pmem_queue; disk->flags = GENHD_FL_EXT_DEVT; - sprintf(disk->disk_name, "pmem%d", idx); + sprintf(disk->disk_name, "pmem%d", pmem->id); disk->driverfs_dev = dev; set_capacity(disk, pmem->size >> 9); pmem->pmem_disk = disk; @@ -182,6 +187,8 @@ out_unmap: iounmap(pmem->virt_addr); out_release_region: release_mem_region(pmem->phys_addr, pmem->size); +out_free_ida: + ida_simple_remove(&pmem_ida, pmem->id); out_free_dev: kfree(pmem); out: @@ -195,6 +202,7 @@ static void pmem_free(struct pmem_device *pmem) blk_cleanup_queue(pmem->pmem_queue); iounmap(pmem->virt_addr); release_mem_region(pmem->phys_addr, pmem->size); + ida_simple_remove(&pmem_ida, pmem->id); kfree(pmem); }

On Wed, Apr 29, 2015 at 11:25 AM, Toshi Kani <toshi.kani(a)hp.com&gt; wrote: I'll take a look at making this better, but it will never be bulletproof. For the same reason that root=UUID=<uuid> is preferred over root=/dev/sda userspace should never rely on consistent pmem device names from boot to boot.

On 4/29/2015 2:53 PM, Toshi Kani wrote: What's the right answer for this in the long run? The NFIT has information like "Memory Device Physical ID" that is an SMBIOS type 17 handle. SMBIOS could have some naming information, like it does for NICs. There is also a "Memory Device Region ID". Is that combination enough to give us a unique identifier? Of course, I'm assuming that information is consistent across reboots and ideally across configuration changes, like adding more NVDIMMs. For NICs, that's where the SMBIOS consistent naming information is helpful. Is that what we need for devices that don't have namespaces and labels? -- ljk

nd_pmem attaches to persistent memory regions and namespaces emitted by the nd subsystem, and, same as the original pmem driver, presents the system-physical-address range as a block device. The existing e820-type-12 to pmem setup is converted to a full libnd bus that emits an nd_namespace_io device. Cc: Andy Lutomirski <luto(a)amacapital.net&gt; Cc: Boaz Harrosh <boaz(a)plexistor.com&gt; Cc: H. Peter Anvin <hpa(a)zytor.com&gt; Cc: Jens Axboe <axboe(a)fb.com&gt; Cc: Ingo Molnar <mingo(a)kernel.org&gt; Cc: Christoph Hellwig <hch(a)lst.de&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- arch/x86/kernel/pmem.c | 2 - drivers/block/Kconfig | 11 ----- drivers/block/Makefile | 1 drivers/block/nd/Kconfig | 27 ++++++++++++ drivers/block/nd/Makefile | 6 +++ drivers/block/nd/e820.c | 100 +++++++++++++++++++++++++++++++++++++++++++++ drivers/block/nd/pmem.c | 47 ++++++++++----------- 7 files changed, 157 insertions(+), 37 deletions(-) create mode 100644 drivers/block/nd/e820.c rename drivers/block/{pmem.c => nd/pmem.c} (88%) diff --git a/arch/x86/kernel/pmem.c b/arch/x86/kernel/pmem.c index 3420c874ddc5..279328c42f87 100644 --- a/arch/x86/kernel/pmem.c +++ b/arch/x86/kernel/pmem.c @@ -13,7 +13,7 @@ static __init void register_pmem_device(struct resource *res) struct platform_device *pdev; int error; - pdev = platform_device_alloc("pmem", PLATFORM_DEVID_AUTO); + pdev = platform_device_alloc("e820_pmem", PLATFORM_DEVID_AUTO); if (!pdev) return; diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig index dfe40e5ca9bd..1cef4ffb16c5 100644 --- a/drivers/block/Kconfig +++ b/drivers/block/Kconfig @@ -406,17 +406,6 @@ config BLK_DEV_RAM_DAX and will prevent RAM block device backing store memory from being allocated from highmem (only a problem for highmem systems). -config BLK_DEV_PMEM - tristate "Persistent memory block device support" - help - Saying Y here will allow you to use a contiguous range of reserved - memory as one or more persistent block devices. - - To compile this driver as a module, choose M here: the module will be - called 'pmem'. - - If unsure, say N. - config CDROM_PKTCDVD tristate "Packet writing on CD/DVD media" depends on !UML diff --git a/drivers/block/Makefile b/drivers/block/Makefile index 07a6acecf4d8..964d8eb2c16f 100644 --- a/drivers/block/Makefile +++ b/drivers/block/Makefile @@ -14,7 +14,6 @@ obj-$(CONFIG_PS3_VRAM) += ps3vram.o obj-$(CONFIG_ATARI_FLOPPY) += ataflop.o obj-$(CONFIG_AMIGA_Z2RAM) += z2ram.o obj-$(CONFIG_BLK_DEV_RAM) += brd.o -obj-$(CONFIG_BLK_DEV_PMEM) += pmem.o obj-$(CONFIG_BLK_DEV_LOOP) += loop.o obj-$(CONFIG_BLK_CPQ_DA) += cpqarray.o obj-$(CONFIG_BLK_CPQ_CISS_DA) += cciss.o diff --git a/drivers/block/nd/Kconfig b/drivers/block/nd/Kconfig index d2d84451e82c..c5eaf195734d 100644 --- a/drivers/block/nd/Kconfig +++ b/drivers/block/nd/Kconfig @@ -68,4 +68,31 @@ config NFIT_TEST Say N unless you are doing development of the 'nd' subsystem. +config ND_E820 + tristate "E820: Support the E820-type-12 PMEM convention" + depends on X86_PMEM_LEGACY + default m if X86_PMEM_LEGACY + select LIBND + help + Prior to ACPI 6 some platforms advertised peristent memory + via type-12 e820 memory ranges. Create a libnd bus and + attach an instance of the pmem driver to these ranges. + +config BLK_DEV_PMEM + tristate "PMEM: Persistent memory block device support" + depends on LIBND + default LIBND + help + Memory ranges for PMEM are described by either an NFIT + (NVDIMM Firmware Interface Table, see CONFIG_NFIT_ACPI), a + non-standard OEM-specific E820 memory type (type-12, see + CONFIG_X86_PMEM_LEGACY), or it is manually specified by the + 'memmap=nn[KMG]!ss[KMG]' kernel command line (see + Documentation/kernel-parameters.txt). This driver converts + these persistent memory ranges into block devices that are + capable of DAX (direct-access) file system mappings. See + Documentation/blockdev/nd.txt for more details. + + Say Y if you want to use a NVDIMM described by NFIT + endif diff --git a/drivers/block/nd/Makefile b/drivers/block/nd/Makefile index 0fb0891e1817..ebb212af9f15 100644 --- a/drivers/block/nd/Makefile +++ b/drivers/block/nd/Makefile @@ -14,10 +14,16 @@ endif obj-$(CONFIG_LIBND) += libnd.o obj-$(CONFIG_ND_ACPI) += nd_acpi.o +obj-$(CONFIG_ND_E820) += nd_e820.o obj-$(CONFIG_NFIT_TEST) += test/ +obj-$(CONFIG_BLK_DEV_PMEM) += nd_pmem.o nd_acpi-y := acpi.o +nd_e820-y := e820.o + +nd_pmem-y := pmem.o + libnd-y := core.o libnd-y += bus.o libnd-y += dimm_devs.o diff --git a/drivers/block/nd/e820.c b/drivers/block/nd/e820.c new file mode 100644 index 000000000000..f4db8c54248e --- /dev/null +++ b/drivers/block/nd/e820.c @@ -0,0 +1,100 @@ +/* + * libnd e820 support + * + * Copyright (c) 2014-2015, Intel Corporation. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ +#include <linux/platform_device.h> +#include <linux/module.h> +#include "libnd.h" + +static const struct attribute_group *nd_e820_attribute_groups[] = { + &nd_bus_attribute_group, + NULL, +}; + +static const struct attribute_group *nd_e820_region_attribute_groups[] = { + &nd_region_attribute_group, + &nd_device_attribute_group, + NULL, +}; + +static int nd_e820_probe(struct platform_device *pdev) +{ + struct nd_bus_descriptor *nd_desc; + struct nd_region_desc ndr_desc; + struct nd_bus *nd_bus; + struct resource *res; + + if (WARN_ON(pdev->num_resources > 1)) + return -ENXIO; + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + if (!res) + return -ENXIO; + + nd_desc = devm_kzalloc(&pdev->dev, sizeof(*nd_desc), GFP_KERNEL); + if (!nd_desc) + return -ENOMEM; + + nd_desc->attr_groups = nd_e820_attribute_groups; + nd_desc->provider_name = "e820"; + nd_bus = nd_bus_register(&pdev->dev, nd_desc); + if (!nd_bus) + return -ENXIO; + + memset(&ndr_desc, 0, sizeof(ndr_desc)); + ndr_desc.res = res; + ndr_desc.attr_groups = nd_e820_region_attribute_groups; + if (!nd_pmem_region_create(nd_bus, &ndr_desc)) { + nd_bus_unregister(nd_bus); + return -ENXIO; + } + + platform_set_drvdata(pdev, nd_bus); + + return 0; +} + +static int nd_e820_remove(struct platform_device *pdev) +{ + struct nd_bus *nd_bus = platform_get_drvdata(pdev); + + nd_bus_unregister(nd_bus); + + return 0; +} + +static struct platform_driver nd_e820_driver = { + .probe = nd_e820_probe, + .remove = nd_e820_remove, + .driver = { + .owner = THIS_MODULE, + .name = "e820_pmem", + }, +}; + +MODULE_ALIAS("platform:e820_pmem*"); + +static int __init nd_e820_init(void) +{ + return platform_driver_register(&nd_e820_driver); +} +module_init(nd_e820_init); + +static void nd_e820_exit(void) +{ + platform_driver_unregister(&nd_e820_driver); +} +module_exit(nd_e820_exit); + +MODULE_AUTHOR("Intel Corporation"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/block/pmem.c b/drivers/block/nd/pmem.c similarity index 88% rename from drivers/block/pmem.c rename to drivers/block/nd/pmem.c index e3cf9142b172..c9dd8d7eca3a 100644 --- a/drivers/block/pmem.c +++ b/drivers/block/nd/pmem.c @@ -1,7 +1,7 @@ /* * Persistent Memory Driver * - * Copyright (c) 2014, Intel Corporation. + * Copyright (c) 2014-2015, Intel Corporation. * Copyright (c) 2015, Christoph Hellwig <hch(a)lst.de&gt;. * Copyright (c) 2015, Boaz Harrosh <boaz(a)plexistor.com&gt;. * @@ -23,6 +23,7 @@ #include <linux/module.h> #include <linux/moduleparam.h> #include <linux/slab.h> +#include <linux/nd.h> #define PMEM_MINORS 16 @@ -206,42 +207,37 @@ static void pmem_free(struct pmem_device *pmem) kfree(pmem); } -static int pmem_probe(struct platform_device *pdev) +static int nd_pmem_probe(struct device *dev) { + struct nd_namespace_io *nsio = to_nd_namespace_io(dev); struct pmem_device *pmem; - struct resource *res; - if (WARN_ON(pdev->num_resources > 1)) - return -ENXIO; - - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!res) - return -ENXIO; - - pmem = pmem_alloc(&pdev->dev, res); + pmem = pmem_alloc(dev, &nsio->res); if (IS_ERR(pmem)) return PTR_ERR(pmem); - platform_set_drvdata(pdev, pmem); + dev_set_drvdata(dev, pmem); return 0; } -static int pmem_remove(struct platform_device *pdev) +static int nd_pmem_remove(struct device *dev) { - struct pmem_device *pmem = platform_get_drvdata(pdev); + struct pmem_device *pmem = dev_get_drvdata(dev); pmem_free(pmem); return 0; } -static struct platform_driver pmem_driver = { - .probe = pmem_probe, - .remove = pmem_remove, - .driver = { - .owner = THIS_MODULE, - .name = "pmem", +MODULE_ALIAS("pmem"); +MODULE_ALIAS_ND_DEVICE(ND_DEVICE_NAMESPACE_IO); +static struct nd_device_driver nd_pmem_driver = { + .probe = nd_pmem_probe, + .remove = nd_pmem_remove, + .drv = { + .name = "pmem", }, + .type = ND_DRIVER_NAMESPACE_IO, }; static int __init pmem_init(void) @@ -252,16 +248,19 @@ static int __init pmem_init(void) if (pmem_major < 0) return pmem_major; - error = platform_driver_register(&pmem_driver); - if (error) + error = nd_driver_register(&nd_pmem_driver); + if (error) { unregister_blkdev(pmem_major, "pmem"); - return error; + return error; + } + + return 0; } module_init(pmem_init); static void pmem_exit(void) { - platform_driver_unregister(&pmem_driver); + driver_unregister(&nd_pmem_driver.drv); unregister_blkdev(pmem_major, "pmem"); } module_exit(pmem_exit);

On platforms that have firmware support for reading/writing per-dimm label space, a portion of the dimm may be accessible via an interleave set PMEM mapping in addition to the dimm's BLK (block-data-window aperture(s)) interface. A label, stored in a "configuration data region" on the dimm, disambiguates which dimm addresses are accessed through which exclusive interface. Add infrastructure that allows the kernel to block modifications to a label in the set while any member dimm is active. Note that this is meant only for enforcing "no modifications of active labels" via the coarse ioctl command. Adding/deleting namespaces from an active interleave set will only be possible via sysfs. Another aspect of tracking interleave sets is tracking their integrity when DIMMs in a set are physically re-ordered. For this purpose we generate an "interleave-set cookie" that can be recorded in a label and validated against the current configuration. It is the bus provider implementation's responsibility to calculate the interleave set cookie and attach it to a given region. Cc: Neil Brown <neilb(a)suse.de&gt; Cc: <linux-acpi(a)vger.kernel.org&gt; Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; Cc: Robert Moore <robert.moore(a)intel.com&gt; Cc: Rafael J. Wysocki <rafael.j.wysocki(a)intel.com&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/acpi.c | 90 ++++++++++++++++++++++++++++++++++++++++ drivers/block/nd/bus.c | 41 ++++++++++++++++++ drivers/block/nd/core.c | 47 +++++++++++++++++++++ drivers/block/nd/dimm_devs.c | 19 ++++++++ drivers/block/nd/libnd.h | 6 +++ drivers/block/nd/nd-private.h | 11 ++++- drivers/block/nd/nd.h | 4 ++ drivers/block/nd/region_devs.c | 85 ++++++++++++++++++++++++++++++++++++++ 8 files changed, 299 insertions(+), 4 deletions(-) diff --git a/drivers/block/nd/acpi.c b/drivers/block/nd/acpi.c index c3dda74f73d7..d34cefe38e2f 100644 --- a/drivers/block/nd/acpi.c +++ b/drivers/block/nd/acpi.c @@ -15,6 +15,7 @@ #include <linux/ndctl.h> #include <linux/list.h> #include <linux/acpi.h> +#include <linux/sort.h> #include "acpi_nfit.h" #include "libnd.h" @@ -779,6 +780,90 @@ static const struct attribute_group *nd_acpi_region_attribute_groups[] = { NULL, }; +/* enough info to uniquely specify an interleave set */ +struct nfit_set_info { + struct nfit_set_info_map { + u64 region_spa_offset; + u32 serial_number; + u32 pad; + } mapping[0]; +}; + +static size_t sizeof_nfit_set_info(int num_mappings) +{ + return sizeof(struct nfit_set_info) + + num_mappings * sizeof(struct nfit_set_info_map); +} + +static int cmp_map(const void *m0, const void *m1) +{ + const struct nfit_set_info_map *map0 = m0; + const struct nfit_set_info_map *map1 = m1; + + return memcmp(&map0->region_spa_offset, &map1->region_spa_offset, + sizeof(u64)); +} + +/* Retrieve the nth entry referencing this spa */ +static struct acpi_nfit_memdev *memdev_from_spa( + struct acpi_nfit_desc *acpi_desc, u16 spa_index, int n) +{ + struct nfit_memdev *nfit_memdev; + + list_for_each_entry(nfit_memdev, &acpi_desc->memdevs, list) + if (nfit_memdev->memdev->spa_index == spa_index) + if (n-- == 0) + return nfit_memdev->memdev; + return NULL; +} + +static int nd_acpi_init_interleave_set(struct acpi_nfit_desc *acpi_desc, + struct nd_region_desc *ndr_desc, struct acpi_nfit_spa *spa) +{ + u16 num_mappings = ndr_desc->num_mappings; + int i, spa_type = nfit_spa_type(spa); + struct device *dev = acpi_desc->dev; + struct nd_interleave_set *nd_set; + struct nfit_set_info *info; + + if (spa_type == NFIT_SPA_PM || spa_type == NFIT_SPA_VOLATILE) + /* pass */; + else + return 0; + + nd_set = devm_kzalloc(dev, sizeof(*nd_set), GFP_KERNEL); + if (!nd_set) + return -ENOMEM; + + info = devm_kzalloc(dev, sizeof_nfit_set_info(num_mappings), GFP_KERNEL); + if (!info) + return -ENOMEM; + for (i = 0; i < num_mappings; i++) { + struct nd_mapping *nd_mapping = &ndr_desc->nd_mapping[i]; + struct nfit_set_info_map *map = &info->mapping[i]; + struct nd_dimm *nd_dimm = nd_mapping->nd_dimm; + struct nfit_mem *nfit_mem = nd_dimm_provider_data(nd_dimm); + struct acpi_nfit_memdev *memdev = memdev_from_spa(acpi_desc, + spa->spa_index, i); + + if (!memdev || !nfit_mem->dcr) { + dev_err(dev, "%s: failed to find DCR\n", __func__); + return -ENODEV; + } + + map->region_spa_offset = memdev->region_spa_offset; + map->serial_number = nfit_mem->dcr->serial_number; + } + + sort(&info->mapping[0], num_mappings, sizeof(struct nfit_set_info_map), + cmp_map, NULL); + nd_set->cookie = nd_fletcher64(info, sizeof_nfit_set_info(num_mappings)); + ndr_desc->nd_set = nd_set; + devm_kfree(dev, info); + + return 0; +} + static int nd_acpi_register_region(struct acpi_nfit_desc *acpi_desc, struct nfit_spa *nfit_spa) { @@ -786,7 +871,7 @@ static int nd_acpi_register_region(struct acpi_nfit_desc *acpi_desc, struct acpi_nfit_spa *spa = nfit_spa->spa; struct nfit_memdev *nfit_memdev; struct nd_region_desc ndr_desc; - int spa_type, count = 0; + int spa_type, count = 0, rc; struct resource res; u16 spa_index; @@ -852,6 +937,9 @@ static int nd_acpi_register_region(struct acpi_nfit_desc *acpi_desc, ndr_desc.nd_mapping = nd_mappings; ndr_desc.num_mappings = count; + rc = nd_acpi_init_interleave_set(acpi_desc, &ndr_desc, spa); + if (rc) + return rc; if (spa_type == NFIT_SPA_PM) { if (!nd_pmem_region_create(acpi_desc->nd_bus, &ndr_desc)) return -ENOMEM; diff --git a/drivers/block/nd/bus.c b/drivers/block/nd/bus.c index 46568d182559..8afb8d4a7e81 100644 --- a/drivers/block/nd/bus.c +++ b/drivers/block/nd/bus.c @@ -78,7 +78,10 @@ static int nd_bus_probe(struct device *dev) if (!try_module_get(provider)) return -ENXIO; + nd_region_probe_start(nd_bus, dev); rc = nd_drv->probe(dev); + nd_region_probe_end(nd_bus, dev, rc); + dev_dbg(&nd_bus->dev, "%s.probe(%s) = %d\n", dev->driver->name, dev_name(dev), rc); if (rc != 0) @@ -94,6 +97,8 @@ static int nd_bus_remove(struct device *dev) int rc; rc = nd_drv->remove(dev); + nd_region_notify_remove(nd_bus, dev, rc); + dev_dbg(&nd_bus->dev, "%s.remove(%s) = %d\n", dev->driver->name, dev_name(dev), rc); module_put(provider); @@ -359,6 +364,33 @@ u32 nd_cmd_out_size(struct nd_dimm *nd_dimm, int cmd, } EXPORT_SYMBOL_GPL(nd_cmd_out_size); +static void wait_nd_bus_probe_idle(struct nd_bus *nd_bus) +{ + do { + if (nd_bus->probe_active == 0) + break; + nd_bus_unlock(&nd_bus->dev); + wait_event(nd_bus->probe_wait, nd_bus->probe_active == 0); + nd_bus_lock(&nd_bus->dev); + } while (true); +} + +/* set_config requires an idle interleave set */ +static int nd_cmd_clear_to_send(struct nd_dimm *nd_dimm, unsigned int cmd) +{ + struct nd_bus *nd_bus; + + if (!nd_dimm || cmd != ND_CMD_SET_CONFIG_DATA) + return 0; + + nd_bus = walk_to_nd_bus(&nd_dimm->dev); + wait_nd_bus_probe_idle(nd_bus); + + if (atomic_read(&nd_dimm->busy)) + return -EBUSY; + return 0; +} + static int __nd_ioctl(struct nd_bus *nd_bus, struct nd_dimm *nd_dimm, int read_only, unsigned int ioctl_cmd, unsigned long arg) { @@ -469,11 +501,18 @@ static int __nd_ioctl(struct nd_bus *nd_bus, struct nd_dimm *nd_dimm, goto out; } + nd_bus_lock(&nd_bus->dev); + rc = nd_cmd_clear_to_send(nd_dimm, cmd); + if (rc) + goto out_unlock; + rc = nd_desc->ndctl(nd_desc, nd_dimm, cmd, buf, buf_len); if (rc < 0) - goto out; + goto out_unlock; if (copy_to_user(p, buf, buf_len)) rc = -EFAULT; + out_unlock: + nd_bus_unlock(&nd_bus->dev); out: vfree(buf); return rc; diff --git a/drivers/block/nd/core.c b/drivers/block/nd/core.c index 646e424ae36c..603970d0ef3a 100644 --- a/drivers/block/nd/core.c +++ b/drivers/block/nd/core.c @@ -24,6 +24,51 @@ LIST_HEAD(nd_bus_list); DEFINE_MUTEX(nd_bus_list_mutex); static DEFINE_IDA(nd_ida); +void nd_bus_lock(struct device *dev) +{ + struct nd_bus *nd_bus = walk_to_nd_bus(dev); + + if (!nd_bus) + return; + mutex_lock(&nd_bus->reconfig_mutex); +} +EXPORT_SYMBOL(nd_bus_lock); + +void nd_bus_unlock(struct device *dev) +{ + struct nd_bus *nd_bus = walk_to_nd_bus(dev); + + if (!nd_bus) + return; + mutex_unlock(&nd_bus->reconfig_mutex); +} +EXPORT_SYMBOL(nd_bus_unlock); + +bool is_nd_bus_locked(struct device *dev) +{ + struct nd_bus *nd_bus = walk_to_nd_bus(dev); + + if (!nd_bus) + return false; + return mutex_is_locked(&nd_bus->reconfig_mutex); +} +EXPORT_SYMBOL(is_nd_bus_locked); + +u64 nd_fletcher64(void __iomem *addr, size_t len) +{ + u32 lo32 = 0; + u64 hi32 = 0; + int i; + + for (i = 0; i < len; i += 4) { + lo32 = readl(addr + i); + hi32 += lo32; + } + + return hi32 << 32 | lo32; +} +EXPORT_SYMBOL_GPL(nd_fletcher64); + static void nd_bus_release(struct device *dev) { struct nd_bus *nd_bus = container_of(dev, struct nd_bus, dev); @@ -142,7 +187,9 @@ struct nd_bus *__nd_bus_register(struct device *parent, if (!nd_bus) return NULL; INIT_LIST_HEAD(&nd_bus->list); + init_waitqueue_head(&nd_bus->probe_wait); nd_bus->id = ida_simple_get(&nd_ida, 0, 0, GFP_KERNEL); + mutex_init(&nd_bus->reconfig_mutex); if (nd_bus->id < 0) { kfree(nd_bus); return NULL; diff --git a/drivers/block/nd/dimm_devs.c b/drivers/block/nd/dimm_devs.c index 33b6d5336096..8981adc59ba4 100644 --- a/drivers/block/nd/dimm_devs.c +++ b/drivers/block/nd/dimm_devs.c @@ -185,7 +185,24 @@ static ssize_t commands_show(struct device *dev, } static DEVICE_ATTR_RO(commands); +static ssize_t state_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct nd_dimm *nd_dimm = to_nd_dimm(dev); + + /* + * The state may be in the process of changing, userspace should + * quiesce probing if it wants a static answer + */ + nd_bus_lock(dev); + nd_bus_unlock(dev); + return sprintf(buf, "%s\n", atomic_read(&nd_dimm->busy) + ? "active" : "idle"); +} +static DEVICE_ATTR_RO(state); + static struct attribute *nd_dimm_attributes[] = { + &dev_attr_state.attr, &dev_attr_commands.attr, NULL, }; @@ -213,7 +230,7 @@ struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, nd_dimm->provider_data = provider_data; nd_dimm->flags = flags; nd_dimm->dsm_mask = dsm_mask; - + atomic_set(&nd_dimm->busy, 0); dev = &nd_dimm->dev; dev_set_name(dev, "nmem%d", nd_dimm->id); dev->parent = &nd_bus->dev; diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h index 8c6f07696f30..deb29eff5b61 100644 --- a/drivers/block/nd/libnd.h +++ b/drivers/block/nd/libnd.h @@ -60,11 +60,16 @@ struct nd_cmd_desc { int out_sizes[ND_CMD_MAX_ELEM]; }; +struct nd_interleave_set { + u64 cookie; +}; + struct nd_region_desc { struct resource *res; struct nd_mapping *nd_mapping; u16 num_mappings; const struct attribute_group **attr_groups; + struct nd_interleave_set *nd_set; void *provider_data; }; @@ -98,4 +103,5 @@ struct nd_region *nd_blk_region_create(struct nd_bus *nd_bus, struct nd_region_desc *ndr_desc); struct nd_region *nd_volatile_region_create(struct nd_bus *nd_bus, struct nd_region_desc *ndr_desc); +u64 nd_fletcher64(void __iomem *addr, size_t len); #endif /* __LIBND_H__ */ diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index 131fc66ce7ab..5d8249be3415 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -13,6 +13,8 @@ #ifndef __ND_PRIVATE_H__ #define __ND_PRIVATE_H__ #include <linux/device.h> +#include <linux/sizes.h> +#include <linux/mutex.h> #include "libnd.h" extern struct list_head nd_bus_list; @@ -21,10 +23,12 @@ extern int nd_dimm_major; struct nd_bus { struct nd_bus_descriptor *nd_desc; + wait_queue_head_t probe_wait; struct module *module; struct list_head list; struct device dev; - int id; + int id, probe_active; + struct mutex reconfig_mutex; }; struct nd_dimm { @@ -32,6 +36,7 @@ struct nd_dimm { void *provider_data; unsigned long *dsm_mask; struct device dev; + atomic_t busy; int id; }; @@ -45,10 +50,14 @@ int __init nd_dimm_init(void); int __init nd_region_init(void); void nd_dimm_exit(void); int nd_region_exit(void); +void nd_region_probe_start(struct nd_bus *nd_bus, struct device *dev); +void nd_region_probe_end(struct nd_bus *nd_bus, struct device *dev, int rc); +void nd_region_notify_remove(struct nd_bus *nd_bus, struct device *dev, int rc); int nd_bus_create_ndctl(struct nd_bus *nd_bus); void nd_bus_destroy_ndctl(struct nd_bus *nd_bus); void nd_synchronize(void); int nd_bus_register_dimms(struct nd_bus *nd_bus); int nd_bus_register_regions(struct nd_bus *nd_bus); +int nd_bus_init_interleave_sets(struct nd_bus *nd_bus); int nd_match_dimm(struct device *dev, void *data); #endif /* __ND_PRIVATE_H__ */ diff --git a/drivers/block/nd/nd.h b/drivers/block/nd/nd.h index 23469513f4c0..c69707dbd272 100644 --- a/drivers/block/nd/nd.h +++ b/drivers/block/nd/nd.h @@ -35,6 +35,7 @@ struct nd_region { u64 ndr_start; int id; void *provider_data; + struct nd_interleave_set *nd_set; struct nd_mapping mapping[0]; }; @@ -50,4 +51,7 @@ int nd_dimm_init_config_data(struct nd_dimm_drvdata *ndd); struct nd_region *to_nd_region(struct device *dev); int nd_region_to_namespace_type(struct nd_region *nd_region); int nd_region_register_namespaces(struct nd_region *nd_region, int *err); +void nd_bus_lock(struct device *dev); +void nd_bus_unlock(struct device *dev); +bool is_nd_bus_locked(struct device *dev); #endif /* __ND_H__ */ diff --git a/drivers/block/nd/region_devs.c b/drivers/block/nd/region_devs.c index 49ebce0c97be..c1cea42d0473 100644 --- a/drivers/block/nd/region_devs.c +++ b/drivers/block/nd/region_devs.c @@ -10,7 +10,10 @@ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * General Public License for more details. */ +#include <linux/scatterlist.h> +#include <linux/sched.h> #include <linux/slab.h> +#include <linux/sort.h> #include <linux/io.h> #include "nd-private.h" #include "nd.h" @@ -133,6 +136,21 @@ static ssize_t nstype_show(struct device *dev, } static DEVICE_ATTR_RO(nstype); +static ssize_t set_cookie_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_region *nd_region = to_nd_region(dev); + struct nd_interleave_set *nd_set = nd_region->nd_set; + + if (is_nd_pmem(dev) && nd_set) + /* pass, should be precluded by nd_region_visible */; + else + return -ENXIO; + + return sprintf(buf, "%#llx\n", nd_set->cookie); +} +static DEVICE_ATTR_RO(set_cookie); + static ssize_t init_namespaces_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -149,15 +167,81 @@ static struct attribute *nd_region_attributes[] = { &dev_attr_size.attr, &dev_attr_nstype.attr, &dev_attr_mappings.attr, + &dev_attr_set_cookie.attr, &dev_attr_init_namespaces.attr, NULL, }; +static umode_t nd_region_visible(struct kobject *kobj, struct attribute *a, int n) +{ + struct device *dev = container_of(kobj, typeof(*dev), kobj); + struct nd_region *nd_region = to_nd_region(dev); + struct nd_interleave_set *nd_set = nd_region->nd_set; + + if (a != &dev_attr_set_cookie.attr) + return a->mode; + + if (is_nd_pmem(dev) && nd_set) + return a->mode; + + return 0; +} + struct attribute_group nd_region_attribute_group = { .attrs = nd_region_attributes, + .is_visible = nd_region_visible, }; EXPORT_SYMBOL_GPL(nd_region_attribute_group); +/* + * Upon successful probe/remove, take/release a reference on the + * associated interleave set (if present) + */ +static void nd_region_notify_driver_action(struct nd_bus *nd_bus, + struct device *dev, int rc, bool probe) +{ + if (rc) + return; + + if (is_nd_pmem(dev) || is_nd_blk(dev)) { + struct nd_region *nd_region = to_nd_region(dev); + int i; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_dimm *nd_dimm = nd_mapping->nd_dimm; + + if (probe) + atomic_inc(&nd_dimm->busy); + else + atomic_dec(&nd_dimm->busy); + } + } +} + +void nd_region_probe_start(struct nd_bus *nd_bus, struct device *dev) +{ + nd_bus_lock(&nd_bus->dev); + nd_bus->probe_active++; + nd_bus_unlock(&nd_bus->dev); +} + +void nd_region_probe_end(struct nd_bus *nd_bus, struct device *dev, int rc) +{ + nd_bus_lock(&nd_bus->dev); + nd_region_notify_driver_action(nd_bus, dev, rc, true); + if (--nd_bus->probe_active == 0) + wake_up(&nd_bus->probe_wait); + nd_bus_unlock(&nd_bus->dev); +} + +void nd_region_notify_remove(struct nd_bus *nd_bus, struct device *dev, int rc) +{ + nd_bus_lock(dev); + nd_region_notify_driver_action(nd_bus, dev, rc, false); + nd_bus_unlock(dev); +} + static ssize_t mappingN(struct device *dev, char *buf, int n) { struct nd_region *nd_region = to_nd_region(dev); @@ -317,6 +401,7 @@ static noinline struct nd_region *nd_region_create(struct nd_bus *nd_bus, } nd_region->ndr_mappings = ndr_desc->num_mappings; nd_region->provider_data = ndr_desc->provider_data; + nd_region->nd_set = ndr_desc->nd_set; dev = &nd_region->dev; dev_set_name(dev, "region%d", nd_region->id); dev->parent = &nd_bus->dev;

A complete label set is a PMEM-label per dimm where all the UUIDs match and the interleave set cookie matches an active interleave set. Present a sysfs ABI for manipulation of a PMEM-namespace's 'alt_name', 'uuid', and 'size' attributes. A later patch will make these settings persistent by writing back the label. Note that PMEM allocations grow forwards from the start of an interleave set (lowest dimm-physical-address (DPA)). BLK-namespaces that alias with a PMEM interleave set will grow allocations backward from the highest DPA. Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; Cc: Neil Brown <neilb(a)suse.de&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/bus.c | 6 drivers/block/nd/core.c | 64 ++ drivers/block/nd/dimm.c | 2 drivers/block/nd/dimm_devs.c | 127 +++++ drivers/block/nd/label.c | 54 ++ drivers/block/nd/label.h | 3 drivers/block/nd/libnd.h | 2 drivers/block/nd/namespace_devs.c | 1024 +++++++++++++++++++++++++++++++++++++ drivers/block/nd/nd-private.h | 14 + drivers/block/nd/nd.h | 33 + drivers/block/nd/pmem.c | 22 + drivers/block/nd/region_devs.c | 145 +++++ include/linux/nd.h | 24 + include/uapi/linux/ndctl.h | 4 14 files changed, 1512 insertions(+), 12 deletions(-) diff --git a/drivers/block/nd/bus.c b/drivers/block/nd/bus.c index 8afb8d4a7e81..819259e92468 100644 --- a/drivers/block/nd/bus.c +++ b/drivers/block/nd/bus.c @@ -364,8 +364,10 @@ u32 nd_cmd_out_size(struct nd_dimm *nd_dimm, int cmd, } EXPORT_SYMBOL_GPL(nd_cmd_out_size); -static void wait_nd_bus_probe_idle(struct nd_bus *nd_bus) +void wait_nd_bus_probe_idle(struct device *dev) { + struct nd_bus *nd_bus = walk_to_nd_bus(dev); + do { if (nd_bus->probe_active == 0) break; @@ -384,7 +386,7 @@ static int nd_cmd_clear_to_send(struct nd_dimm *nd_dimm, unsigned int cmd) return 0; nd_bus = walk_to_nd_bus(&nd_dimm->dev); - wait_nd_bus_probe_idle(nd_bus); + wait_nd_bus_probe_idle(&nd_bus->dev); if (atomic_read(&nd_dimm->busy)) return -EBUSY; diff --git a/drivers/block/nd/core.c b/drivers/block/nd/core.c index 603970d0ef3a..cf64b7a50d3a 100644 --- a/drivers/block/nd/core.c +++ b/drivers/block/nd/core.c @@ -13,6 +13,7 @@ #include <linux/export.h> #include <linux/module.h> #include <linux/device.h> +#include <linux/ctype.h> #include <linux/ndctl.h> #include <linux/mutex.h> #include <linux/slab.h> @@ -106,6 +107,69 @@ struct nd_bus *walk_to_nd_bus(struct device *nd_dev) return NULL; } +static bool is_uuid_sep(char sep) +{ + if (sep == '\n' || sep == '-' || sep == ':' || sep == '\0') + return true; + return false; +} + +static int nd_uuid_parse(struct device *dev, u8 *uuid_out, const char *buf, + size_t len) +{ + const char *str = buf; + u8 uuid[16]; + int i; + + for (i = 0; i < 16; i++) { + if (!isxdigit(str[0]) || !isxdigit(str[1])) { + dev_dbg(dev, "%s: pos: %d buf[%zd]: %c buf[%zd]: %c\n", + __func__, i, str - buf, str[0], + str + 1 - buf, str[1]); + return -EINVAL; + } + + uuid[i] = (hex_to_bin(str[0]) << 4) | hex_to_bin(str[1]); + str += 2; + if (is_uuid_sep(*str)) + str++; + } + + memcpy(uuid_out, uuid, sizeof(uuid)); + return 0; +} + +/** + * nd_uuid_store: common implementation for writing 'uuid' sysfs attributes + * @dev: container device for the uuid property + * @uuid_out: uuid buffer to replace + * @buf: raw sysfs buffer to parse + * + * Enforce that uuids can only be changed while the device is disabled + * (driver detached) + * LOCKING: expects device_lock() is held on entry + */ +int nd_uuid_store(struct device *dev, u8 **uuid_out, const char *buf, + size_t len) +{ + u8 uuid[16]; + int rc; + + if (dev->driver) + return -EBUSY; + + rc = nd_uuid_parse(dev, uuid, buf, len); + if (rc) + return rc; + + kfree(*uuid_out); + *uuid_out = kmemdup(uuid, sizeof(uuid), GFP_KERNEL); + if (!(*uuid_out)) + return -ENOMEM; + + return 0; +} + static ssize_t commands_show(struct device *dev, struct device_attribute *attr, char *buf) { diff --git a/drivers/block/nd/dimm.c b/drivers/block/nd/dimm.c index 5477176c5de0..e2f964308672 100644 --- a/drivers/block/nd/dimm.c +++ b/drivers/block/nd/dimm.c @@ -86,7 +86,7 @@ static int nd_dimm_remove(struct device *dev) nd_bus_lock(dev); dev_set_drvdata(dev, NULL); for_each_dpa_resource_safe(ndd, res, _r) - __release_region(&ndd->dpa, res->start, resource_size(res)); + nd_dimm_free_dpa(ndd, res); nd_bus_unlock(dev); free_data(ndd); diff --git a/drivers/block/nd/dimm_devs.c b/drivers/block/nd/dimm_devs.c index 3fbd0d0502eb..b242d3ae6d12 100644 --- a/drivers/block/nd/dimm_devs.c +++ b/drivers/block/nd/dimm_devs.c @@ -159,6 +159,14 @@ struct nd_dimm *to_nd_dimm(struct device *dev) } EXPORT_SYMBOL_GPL(to_nd_dimm); +struct nd_dimm_drvdata *to_ndd(struct nd_mapping *nd_mapping) +{ + struct nd_dimm *nd_dimm = nd_mapping->nd_dimm; + + return dev_get_drvdata(&nd_dimm->dev); +} +EXPORT_SYMBOL(to_ndd); + const char *nd_dimm_name(struct nd_dimm *nd_dimm) { return dev_name(&nd_dimm->dev); @@ -247,6 +255,125 @@ struct nd_dimm *nd_dimm_create(struct nd_bus *nd_bus, void *provider_data, } EXPORT_SYMBOL_GPL(nd_dimm_create); +/** + * nd_pmem_available_dpa - for the given dimm+region account unallocated dpa + * @nd_mapping: container of dpa-resource-root + labels + * @nd_region: constrain available space check to this reference region + * @overlap: calculate available space assuming this level of overlap + * + * Validate that a PMEM label, if present, aligns with the start of an + * interleave set and truncate the available size at the lowest BLK + * overlap point. + * + * The expectation is that this routine is called multiple times as it + * probes for the largest BLK encroachment for any single member DIMM of + * the interleave set. Once that value is determined the PMEM-limit for + * the set can be established. + */ +resource_size_t nd_pmem_available_dpa(struct nd_region *nd_region, + struct nd_mapping *nd_mapping, resource_size_t *overlap) +{ + resource_size_t map_end, busy = 0, available, blk_start; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + struct resource *res; + const char *reason; + + if (!ndd) + return 0; + + map_end = nd_mapping->start + nd_mapping->size - 1; + blk_start = max(nd_mapping->start, map_end + 1 - *overlap); + for_each_dpa_resource(ndd, res) + if (res->start >= nd_mapping->start && res->start < map_end) { + if (strncmp(res->name, "blk", 3) == 0) + blk_start = min(blk_start, res->start); + else if (res->start != nd_mapping->start) { + reason = "misaligned to iset"; + goto err; + } else { + if (busy) { + reason = "duplicate overlapping PMEM reservations?"; + goto err; + } + busy += resource_size(res); + continue; + } + } else if (res->end >= nd_mapping->start && res->end <= map_end) { + if (strncmp(res->name, "blk", 3) == 0) { + /* + * If a BLK allocation overlaps the start of + * PMEM the entire interleave set may now only + * be used for BLK. + */ + blk_start = nd_mapping->start; + } else { + reason = "misaligned to iset"; + goto err; + } + } else if (nd_mapping->start > res->start + && nd_mapping->start < res->end) { + /* total eclipse of the mapping */ + busy += nd_mapping->size; + blk_start = nd_mapping->start; + } + + *overlap = map_end + 1 - blk_start; + available = blk_start - nd_mapping->start; + if (busy < available) + return available - busy; + return 0; + + err: + /* + * Something is wrong, PMEM must align with the start of the + * interleave set, and there can only be one allocation per set. + */ + nd_dbg_dpa(nd_region, ndd, res, "%s\n", reason); + return 0; +} + +void nd_dimm_free_dpa(struct nd_dimm_drvdata *ndd, struct resource *res) +{ + WARN_ON_ONCE(!is_nd_bus_locked(ndd->dev)); + kfree(res->name); + __release_region(&ndd->dpa, res->start, resource_size(res)); +} + +struct resource *nd_dimm_allocate_dpa(struct nd_dimm_drvdata *ndd, + struct nd_label_id *label_id, resource_size_t start, + resource_size_t n) +{ + char *name = kmemdup(label_id, sizeof(*label_id), GFP_KERNEL); + struct resource *res; + + if (!name) + return NULL; + + WARN_ON_ONCE(!is_nd_bus_locked(ndd->dev)); + res = __request_region(&ndd->dpa, start, n, name, 0); + if (!res) + kfree(name); + return res; +} + +/** + * nd_dimm_allocated_dpa - sum up the dpa currently allocated to this label_id + * @nd_dimm: container of dpa-resource-root + labels + * @label_id: dpa resource name of the form {pmem|blk}-<human readable uuid> + */ +resource_size_t nd_dimm_allocated_dpa(struct nd_dimm_drvdata *ndd, + struct nd_label_id *label_id) +{ + resource_size_t allocated = 0; + struct resource *res; + + for_each_dpa_resource(ndd, res) + if (strcmp(res->name, label_id->id) == 0) + allocated += resource_size(res); + + return allocated; +} + static int count_dimms(struct device *dev, void *c) { int *count = c; diff --git a/drivers/block/nd/label.c b/drivers/block/nd/label.c index e791ea8bbdde..b55fa2a6f872 100644 --- a/drivers/block/nd/label.c +++ b/drivers/block/nd/label.c @@ -228,7 +228,7 @@ static bool preamble_current(struct nd_dimm_drvdata *ndd, return true; } -static char *nd_label_gen_id(struct nd_label_id *label_id, u8 *uuid, u32 flags) +char *nd_label_gen_id(struct nd_label_id *label_id, u8 *uuid, u32 flags) { if (!label_id || !uuid) return NULL; @@ -289,3 +289,55 @@ int nd_label_reserve_dpa(struct nd_dimm_drvdata *ndd) return 0; } + +int nd_label_active_count(struct nd_dimm_drvdata *ndd) +{ + struct nd_namespace_index __iomem *nsindex; + unsigned long *free; + u32 nslot, slot; + int count = 0; + + if (!preamble_current(ndd, &nsindex, &free, &nslot)) + return 0; + + for_each_clear_bit_le(slot, free, nslot) { + struct nd_namespace_label __iomem *nd_label; + + nd_label = nd_label_base(ndd) + slot; + + if (!slot_valid(nd_label, slot)) { + dev_dbg(ndd->dev, + "%s: slot%d invalid slot: %d dpa: %lx rawsize: %lx\n", + __func__, slot, readl(&nd_label->slot), + (unsigned long) readq(&nd_label->dpa), + (unsigned long) readq(&nd_label->rawsize)); + continue; + } + count++; + } + return count; +} + +struct nd_namespace_label __iomem *nd_label_active( + struct nd_dimm_drvdata *ndd, int n) +{ + struct nd_namespace_index __iomem *nsindex; + unsigned long *free; + u32 nslot, slot; + + if (!preamble_current(ndd, &nsindex, &free, &nslot)) + return NULL; + + for_each_clear_bit_le(slot, free, nslot) { + struct nd_namespace_label __iomem *nd_label; + + nd_label = nd_label_base(ndd) + slot; + if (slot != readl(&nd_label->slot)) + continue; + + if (n-- == 0) + return nd_label_base(ndd) + slot; + } + + return NULL; +} diff --git a/drivers/block/nd/label.h b/drivers/block/nd/label.h index 79ed885a43c0..4436624f4146 100644 --- a/drivers/block/nd/label.h +++ b/drivers/block/nd/label.h @@ -126,4 +126,7 @@ void nd_label_copy(struct nd_dimm_drvdata *ndd, struct nd_namespace_index *dst, struct nd_namespace_index *src); size_t sizeof_namespace_index(struct nd_dimm_drvdata *ndd); +int nd_label_active_count(struct nd_dimm_drvdata *ndd); +struct nd_namespace_label __iomem *nd_label_active( + struct nd_dimm_drvdata *ndd, int n); #endif /* __LABEL_H__ */ diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h index deb29eff5b61..832dcfebbb49 100644 --- a/drivers/block/nd/libnd.h +++ b/drivers/block/nd/libnd.h @@ -40,8 +40,10 @@ typedef int (*ndctl_fn)(struct nd_bus_descriptor *nd_desc, struct nd_dimm *nd_dimm, unsigned int cmd, void *buf, unsigned int buf_len); +struct nd_namespace_label; struct nd_mapping { struct nd_dimm *nd_dimm; + struct nd_namespace_label **labels; u64 start; u64 size; }; diff --git a/drivers/block/nd/namespace_devs.c b/drivers/block/nd/namespace_devs.c index 6861327f4245..fb23904df4c7 100644 --- a/drivers/block/nd/namespace_devs.c +++ b/drivers/block/nd/namespace_devs.c @@ -14,8 +14,11 @@ #include <linux/device.h> #include <linux/slab.h> #include <linux/nd.h> +#include "nd-private.h" #include "nd.h" +#include <asm-generic/io-64-nonatomic-lo-hi.h> + static void namespace_io_release(struct device *dev) { struct nd_namespace_io *nsio = to_nd_namespace_io(dev); @@ -23,11 +26,50 @@ static void namespace_io_release(struct device *dev) kfree(nsio); } +static void namespace_pmem_release(struct device *dev) +{ + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + kfree(nspm->alt_name); + kfree(nspm->uuid); + kfree(nspm); +} + +static void namespace_blk_release(struct device *dev) +{ + /* TODO: blk namespace support */ +} + static struct device_type namespace_io_device_type = { .name = "nd_namespace_io", .release = namespace_io_release, }; +static struct device_type namespace_pmem_device_type = { + .name = "nd_namespace_pmem", + .release = namespace_pmem_release, +}; + +static struct device_type namespace_blk_device_type = { + .name = "nd_namespace_blk", + .release = namespace_blk_release, +}; + +static bool is_namespace_pmem(struct device *dev) +{ + return dev ? dev->type == &namespace_pmem_device_type : false; +} + +static bool is_namespace_blk(struct device *dev) +{ + return dev ? dev->type == &namespace_blk_device_type : false; +} + +static bool is_namespace_io(struct device *dev) +{ + return dev ? dev->type == &namespace_io_device_type : false; +} + static ssize_t type_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -37,13 +79,674 @@ static ssize_t type_show(struct device *dev, } static DEVICE_ATTR_RO(type); +static ssize_t __alt_name_store(struct device *dev, const char *buf, + const size_t len) +{ + char *input, *pos, *alt_name, **ns_altname; + ssize_t rc; + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + ns_altname = &nspm->alt_name; + } else if (is_namespace_blk(dev)) { + /* TODO: blk namespace support */ + return -ENXIO; + } else + return -ENXIO; + + if (dev->driver) + return -EBUSY; + + input = kmemdup(buf, len + 1, GFP_KERNEL); + if (!input) + return -ENOMEM; + + input[len] = '\0'; + pos = strim(input); + if (strlen(pos) + 1 > NSLABEL_NAME_LEN) { + rc = -EINVAL; + goto out; + } + + alt_name = kzalloc(NSLABEL_NAME_LEN, GFP_KERNEL); + if (!alt_name) { + rc = -ENOMEM; + goto out; + } + kfree(*ns_altname); + *ns_altname = alt_name; + sprintf(*ns_altname, "%s", pos); + rc = len; + +out: + kfree(input); + return rc; +} + +static ssize_t alt_name_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + ssize_t rc; + + device_lock(dev); + nd_bus_lock(dev); + wait_nd_bus_probe_idle(dev); + rc = __alt_name_store(dev, buf, len); + dev_dbg(dev, "%s: %s (%zd)\n", __func__, rc < 0 ? "fail" : "success", rc); + nd_bus_unlock(dev); + device_unlock(dev); + + return rc; +} + +static ssize_t alt_name_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + char *ns_altname; + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + ns_altname = nspm->alt_name; + } else if (is_namespace_blk(dev)) { + /* TODO: blk namespace support */ + return -ENXIO; + } else + return -ENXIO; + + return sprintf(buf, "%s\n", ns_altname ? ns_altname : ""); +} +static DEVICE_ATTR_RW(alt_name); + +static int scan_free(struct nd_region *nd_region, + struct nd_mapping *nd_mapping, struct nd_label_id *label_id, + resource_size_t n) +{ + bool is_blk = strncmp(label_id->id, "blk", 3) == 0; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + int rc = 0; + + while (n) { + struct resource *res, *last; + resource_size_t new_start; + + last = NULL; + for_each_dpa_resource(ndd, res) + if (strcmp(res->name, label_id->id) == 0) + last = res; + res = last; + if (!res) + return 0; + + if (n >= resource_size(res)) { + n -= resource_size(res); + nd_dbg_dpa(nd_region, ndd, res, "delete %d\n", rc); + nd_dimm_free_dpa(ndd, res); + /* retry with last resource deleted */ + continue; + } + + /* + * Keep BLK allocations relegated to high DPA as much as + * possible + */ + if (is_blk) + new_start = res->start + n; + else + new_start = res->start; + + rc = adjust_resource(res, new_start, resource_size(res) - n); + nd_dbg_dpa(nd_region, ndd, res, "shrink %d\n", rc); + break; + } + + return rc; +} + +/** + * shrink_dpa_allocation - for each dimm in region free n bytes for label_id + * @nd_region: the set of dimms to reclaim @n bytes from + * @label_id: unique identifier for the namespace consuming this dpa range + * @n: number of bytes per-dimm to release + * + * Assumes resources are ordered. Starting from the end try to + * adjust_resource() the allocation to @n, but if @n is larger than the + * allocation delete it and find the 'new' last allocation in the label + * set. + */ +static int shrink_dpa_allocation(struct nd_region *nd_region, + struct nd_label_id *label_id, resource_size_t n) +{ + int i; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + int rc; + + rc = scan_free(nd_region, nd_mapping, label_id, n); + if (rc) + return rc; + } + + return 0; +} + +static resource_size_t init_dpa_allocation(struct nd_label_id *label_id, + struct nd_region *nd_region, struct nd_mapping *nd_mapping, + resource_size_t n) +{ + bool is_blk = strncmp(label_id->id, "blk", 3) == 0; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + resource_size_t first_dpa; + struct resource *res; + int rc = 0; + + /* allocate blk from highest dpa first */ + if (is_blk) + first_dpa = nd_mapping->start + nd_mapping->size - n; + else + first_dpa = nd_mapping->start; + + /* first resource allocation for this label-id or dimm */ + res = nd_dimm_allocate_dpa(ndd, label_id, first_dpa, n); + if (!res) + rc = -EBUSY; + + nd_dbg_dpa(nd_region, ndd, res, "init %d\n", rc); + return rc ? n : 0; +} + +static bool space_valid(bool is_pmem, struct nd_label_id *label_id, + struct resource *res) +{ + /* + * For BLK-space any space is valid, for PMEM-space, it must be + * contiguous with an existing allocation. + */ + if (!is_pmem) + return true; + if (!res || strcmp(res->name, label_id->id) == 0) + return true; + return false; +} + +enum alloc_loc { + ALLOC_ERR = 0, ALLOC_BEFORE, ALLOC_MID, ALLOC_AFTER, +}; + +static resource_size_t scan_allocate(struct nd_region *nd_region, + struct nd_mapping *nd_mapping, struct nd_label_id *label_id, + resource_size_t n) +{ + resource_size_t mapping_end = nd_mapping->start + nd_mapping->size - 1; + bool is_pmem = strncmp(label_id->id, "pmem", 4) == 0; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + const resource_size_t to_allocate = n; + struct resource *res; + int first; + + retry: + first = 0; + for_each_dpa_resource(ndd, res) { + resource_size_t allocate, available = 0, free_start, free_end; + struct resource *next = res->sibling, *new_res = NULL; + enum alloc_loc loc = ALLOC_ERR; + const char *action; + int rc = 0; + + /* ignore resources outside this nd_mapping */ + if (res->start > mapping_end) + continue; + if (res->end < nd_mapping->start) + continue; + + /* space at the beginning of the mapping */ + if (!first++ && res->start > nd_mapping->start) { + free_start = nd_mapping->start; + available = res->start - free_start; + if (space_valid(is_pmem, label_id, NULL)) + loc = ALLOC_BEFORE; + } + + /* space between allocations */ + if (!loc && next) { + free_start = res->start + resource_size(res); + free_end = min(mapping_end, next->start - 1); + if (space_valid(is_pmem, label_id, res) + && free_start < free_end) { + available = free_end + 1 - free_start; + loc = ALLOC_MID; + } + } + + /* space at the end of the mapping */ + if (!loc && !next) { + free_start = res->start + resource_size(res); + free_end = mapping_end; + if (space_valid(is_pmem, label_id, res) + && free_start < free_end) { + available = free_end + 1 - free_start; + loc = ALLOC_AFTER; + } + } + + if (!loc || !available) + continue; + allocate = min(available, n); + switch (loc) { + case ALLOC_BEFORE: + if (strcmp(res->name, label_id->id) == 0) { + /* adjust current resource up */ + if (is_pmem) + return n; + rc = adjust_resource(res, res->start - allocate, + resource_size(res) + allocate); + action = "cur grow up"; + } else + action = "allocate"; + break; + case ALLOC_MID: + if (strcmp(next->name, label_id->id) == 0) { + /* adjust next resource up */ + if (is_pmem) + return n; + rc = adjust_resource(next, next->start + - allocate, resource_size(next) + + allocate); + new_res = next; + action = "next grow up"; + } else if (strcmp(res->name, label_id->id) == 0) { + action = "grow down"; + } else + action = "allocate"; + break; + case ALLOC_AFTER: + if (strcmp(res->name, label_id->id) == 0) + action = "grow down"; + else + action = "allocate"; + break; + default: + return n; + } + + if (strcmp(action, "allocate") == 0) { + /* BLK allocate bottom up */ + if (!is_pmem) + free_start += available - allocate; + else if (free_start != nd_mapping->start) + return n; + + new_res = nd_dimm_allocate_dpa(ndd, label_id, + free_start, allocate); + if (!new_res) + rc = -EBUSY; + } else if (strcmp(action, "grow down") == 0) { + /* adjust current resource down */ + rc = adjust_resource(res, res->start, resource_size(res) + + allocate); + } + + if (!new_res) + new_res = res; + + nd_dbg_dpa(nd_region, ndd, new_res, "%s(%d) %d\n", + action, loc, rc); + + if (rc) + return n; + + n -= allocate; + if (n) { + /* + * Retry scan with newly inserted resources. + * For example, if we did an ALLOC_BEFORE + * insertion there may also have been space + * available for an ALLOC_AFTER insertion, so we + * need to check this same resource again + */ + goto retry; + } else + return 0; + } + + if (is_pmem && n == to_allocate) + return init_dpa_allocation(label_id, nd_region, nd_mapping, n); + return n; +} + +/** + * grow_dpa_allocation - for each dimm allocate n bytes for @label_id + * @nd_region: the set of dimms to allocate @n more bytes from + * @label_id: unique identifier for the namespace consuming this dpa range + * @n: number of bytes per-dimm to add to the existing allocation + * + * Assumes resources are ordered. For BLK regions, first consume + * BLK-only available DPA free space, then consume PMEM-aliased DPA + * space starting at the highest DPA. For PMEM regions start + * allocations from the start of an interleave set and end at the first + * BLK allocation or the end of the interleave set, whichever comes + * first. + */ +static int grow_dpa_allocation(struct nd_region *nd_region, + struct nd_label_id *label_id, resource_size_t n) +{ + int i; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + int rc; + + rc = scan_allocate(nd_region, nd_mapping, label_id, n); + if (rc) + return rc; + } + + return 0; +} + +static void nd_namespace_pmem_set_size(struct nd_region *nd_region, + struct nd_namespace_pmem *nspm, resource_size_t size) +{ + struct resource *res = &nspm->nsio.res; + + res->start = nd_region->ndr_start; + res->end = nd_region->ndr_start + size - 1; +} + +static ssize_t __size_store(struct device *dev, unsigned long long val) +{ + resource_size_t allocated = 0, available = 0; + struct nd_region *nd_region = to_nd_region(dev->parent); + struct nd_mapping *nd_mapping; + struct nd_dimm_drvdata *ndd; + struct nd_label_id label_id; + u32 flags = 0, remainder; + u8 *uuid = NULL; + int rc, i; + + if (dev->driver) + return -EBUSY; + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + uuid = nspm->uuid; + } else if (is_namespace_blk(dev)) { + /* TODO: blk namespace support */ + return -ENXIO; + } + + /* + * We need a uuid for the allocation-label and dimm(s) on which + * to store the label. + */ + if (!uuid || nd_region->ndr_mappings == 0) + return -ENXIO; + + div_u64_rem(val, SZ_4K * nd_region->ndr_mappings, &remainder); + if (remainder) { + dev_dbg(dev, "%llu is not %dK aligned\n", val, + (SZ_4K * nd_region->ndr_mappings) / SZ_1K); + return -EINVAL; + } + + nd_label_gen_id(&label_id, uuid, flags); + for (i = 0; i < nd_region->ndr_mappings; i++) { + nd_mapping = &nd_region->mapping[i]; + ndd = to_ndd(nd_mapping); + + /* + * All dimms in an interleave set, or the base dimm for a blk + * region, need to be enabled for the size to be changed. + */ + if (!ndd) + return -ENXIO; + + allocated += nd_dimm_allocated_dpa(ndd, &label_id); + } + available = nd_region_available_dpa(nd_region); + + if (val > available + allocated) + return -ENOSPC; + + if (val == allocated) + return 0; + + val = div_u64(val, nd_region->ndr_mappings); + allocated = div_u64(allocated, nd_region->ndr_mappings); + if (val < allocated) + rc = shrink_dpa_allocation(nd_region, &label_id, allocated - val); + else + rc = grow_dpa_allocation(nd_region, &label_id, val - allocated); + + if (rc) + return rc; + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + nd_namespace_pmem_set_size(nd_region, nspm, + val * nd_region->ndr_mappings); + } + + return rc; +} + +static ssize_t size_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + unsigned long long val; + u8 **uuid = NULL; + int rc; + + rc = kstrtoull(buf, 0, &val); + if (rc) + return rc; + + device_lock(dev); + nd_bus_lock(dev); + wait_nd_bus_probe_idle(dev); + rc = __size_store(dev, val); + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + uuid = &nspm->uuid; + } else if (is_namespace_blk(dev)) { + /* TODO: blk namespace support */ + rc = -ENXIO; + } + + if (rc == 0 && val == 0 && uuid) { + /* setting size zero == 'delete namespace' */ + kfree(*uuid); + *uuid = NULL; + } + + dev_dbg(dev, "%s: %llx %s (%d)\n", __func__, val, rc < 0 + ? "fail" : "success", rc); + + nd_bus_unlock(dev); + device_unlock(dev); + + return rc ? rc : len; +} + +static ssize_t size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + return sprintf(buf, "%llu\n", (unsigned long long) + resource_size(&nspm->nsio.res)); + } else if (is_namespace_blk(dev)) { + /* TODO: blk namespace support */ + return -ENXIO; + } else if (is_namespace_io(dev)) { + struct nd_namespace_io *nsio = to_nd_namespace_io(dev); + + return sprintf(buf, "%llu\n", (unsigned long long) + resource_size(&nsio->res)); + } else + return -ENXIO; +} +static DEVICE_ATTR(size, S_IRUGO, size_show, size_store); + +static ssize_t uuid_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + u8 *uuid; + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + uuid = nspm->uuid; + } else if (is_namespace_blk(dev)) { + /* TODO: blk namespace support */ + return -ENXIO; + } else + return -ENXIO; + + if (uuid) + return sprintf(buf, "%pUb\n", uuid); + return sprintf(buf, "\n"); +} + +/** + * namespace_update_uuid - check for a unique uuid and whether we're "renaming" + * @nd_region: parent region so we can updates all dimms in the set + * @dev: namespace type for generating label_id + * @new_uuid: incoming uuid + * @old_uuid: reference to the uuid storage location in the namespace object + */ +static int namespace_update_uuid(struct nd_region *nd_region, + struct device *dev, u8 *new_uuid, u8 **old_uuid) +{ + u32 flags = is_namespace_blk(dev) ? NSLABEL_FLAG_LOCAL : 0; + struct nd_label_id old_label_id; + struct nd_label_id new_label_id; + int i, rc; + + rc = nd_is_uuid_unique(dev, new_uuid) ? 0 : -EINVAL; + if (rc) { + kfree(new_uuid); + return rc; + } + + if (*old_uuid == NULL) + goto out; + + nd_label_gen_id(&old_label_id, *old_uuid, flags); + nd_label_gen_id(&new_label_id, new_uuid, flags); + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + struct resource *res; + + for_each_dpa_resource(ndd, res) + if (strcmp(res->name, old_label_id.id) == 0) + sprintf((void *) res->name, "%s", + new_label_id.id); + } + kfree(*old_uuid); + out: + *old_uuid = new_uuid; + return 0; +} + +static ssize_t uuid_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + struct nd_region *nd_region = to_nd_region(dev->parent); + u8 *uuid = NULL; + u8 **ns_uuid; + ssize_t rc; + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + ns_uuid = &nspm->uuid; + } else if (is_namespace_blk(dev)) { + /* TODO: blk namespace support */ + return -ENXIO; + } else + return -ENXIO; + + device_lock(dev); + nd_bus_lock(dev); + wait_nd_bus_probe_idle(dev); + rc = nd_uuid_store(dev, &uuid, buf, len); + if (rc >= 0) + rc = namespace_update_uuid(nd_region, dev, uuid, ns_uuid); + dev_dbg(dev, "%s: result: %zd wrote: %s%s", __func__, + rc, buf, buf[len - 1] == '\n' ? "" : "\n"); + nd_bus_unlock(dev); + device_unlock(dev); + + return rc ? rc : len; +} +static DEVICE_ATTR_RW(uuid); + +static ssize_t resource_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct resource *res; + + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + res = &nspm->nsio.res; + } else if (is_namespace_io(dev)) { + struct nd_namespace_io *nsio = to_nd_namespace_io(dev); + + res = &nsio->res; + } else + return -ENXIO; + + /* no address to convey if the namespace has no allocation */ + if (resource_size(res) == 0) + return -ENXIO; + return sprintf(buf, "%#llx\n", (unsigned long long) res->start); +} +static DEVICE_ATTR_RO(resource); + static struct attribute *nd_namespace_attributes[] = { &dev_attr_type.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, + &dev_attr_resource.attr, + &dev_attr_alt_name.attr, NULL, }; +static umode_t nd_namespace_attr_visible(struct kobject *kobj, struct attribute *a, int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + + if (a == &dev_attr_resource.attr) { + if (is_namespace_blk(dev)) + return 0; + return a->mode; + } + + if (is_namespace_pmem(dev) || is_namespace_blk(dev)) { + if (a == &dev_attr_size.attr) + return S_IWUSR; + return a->mode; + } + + if (a == &dev_attr_type.attr || a == &dev_attr_size.attr) + return a->mode; + + return 0; +} + static struct attribute_group nd_namespace_attribute_group = { .attrs = nd_namespace_attributes, + .is_visible = nd_namespace_attr_visible, }; static const struct attribute_group *nd_namespace_attribute_groups[] = { @@ -80,23 +783,326 @@ static struct device **create_namespace_io(struct nd_region *nd_region) return devs; } +static bool has_uuid_at_pos(struct nd_region *nd_region, u8 *uuid, u64 cookie, u16 pos) +{ + struct nd_namespace_label __iomem *found = NULL; + int i; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_namespace_label __iomem *nd_label; + u8 label_uuid[NSLABEL_UUID_LEN]; + u8 *found_uuid = NULL; + int l; + + for_each_label(l, nd_label, nd_mapping->labels) { + u64 isetcookie = readq(&nd_label->isetcookie); + u16 position = readw(&nd_label->position); + u16 nlabel = readw(&nd_label->nlabel); + + if (isetcookie != cookie) + continue; + + memcpy_fromio(label_uuid, nd_label->uuid, + NSLABEL_UUID_LEN); + if (memcmp(label_uuid, uuid, NSLABEL_UUID_LEN) != 0) + continue; + + if (found_uuid) { + dev_dbg(to_ndd(nd_mapping)->dev, + "%s duplicate entry for uuid\n", + __func__); + return false; + } + found_uuid = label_uuid; + if (nlabel != nd_region->ndr_mappings) + continue; + if (position != pos) + continue; + found = nd_label; + break; + } + if (found) + break; + } + return found != NULL; +} + +static int select_pmem_uuid(struct nd_region *nd_region, u8 *pmem_uuid) +{ + struct nd_namespace_label __iomem *select = NULL; + int i; + + if (!pmem_uuid) + return -ENODEV; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_namespace_label __iomem *nd_label; + u64 hw_start, hw_end, pmem_start, pmem_end; + int l; + + for_each_label(l, nd_label, nd_mapping->labels) { + u8 label_uuid[NSLABEL_UUID_LEN]; + + memcpy_fromio(label_uuid, nd_label->uuid, + NSLABEL_UUID_LEN); + if (memcmp(label_uuid, pmem_uuid, NSLABEL_UUID_LEN) == 0) + break; + } + + if (!nd_label) { + WARN_ON(1); + return -EINVAL; + } + + select = nd_label; + /* + * Check that this label is compliant with the dpa + * range published in NFIT + */ + hw_start = nd_mapping->start; + hw_end = hw_start + nd_mapping->size; + pmem_start = readq(&select->dpa); + pmem_end = pmem_start + readq(&select->rawsize); + if (pmem_start == hw_start && pmem_end <= hw_end) + /* pass */; + else + return -EINVAL; + + nd_set_label(nd_mapping->labels, select, 0); + nd_set_label(nd_mapping->labels, (void __iomem *) NULL, 1); + } + return 0; +} + +/** + * find_pmem_label_set - validate interleave set labelling, retrieve label0 + * @nd_region: region with mappings to validate + */ +static int find_pmem_label_set(struct nd_region *nd_region, + struct nd_namespace_pmem *nspm) +{ + u64 cookie = nd_region_interleave_set_cookie(nd_region); + struct nd_namespace_label __iomem *nd_label; + u8 select_uuid[NSLABEL_UUID_LEN]; + resource_size_t size = 0; + u8 *pmem_uuid = NULL; + int rc = -ENODEV, l; + u16 i; + + if (cookie == 0) + return -ENXIO; + + /* + * Find a complete set of labels by uuid. By definition we can start + * with any mapping as the reference label + */ + for_each_label(l, nd_label, nd_region->mapping[0].labels) { + u64 isetcookie = readq(&nd_label->isetcookie); + u8 label_uuid[NSLABEL_UUID_LEN]; + + if (isetcookie != cookie) + continue; + + memcpy_fromio(label_uuid, nd_label->uuid, + NSLABEL_UUID_LEN); + for (i = 0; nd_region->ndr_mappings; i++) + if (!has_uuid_at_pos(nd_region, label_uuid, cookie, i)) + break; + if (i < nd_region->ndr_mappings) { + /* + * Give up if we don't find an instance of a + * uuid at each position (from 0 to + * nd_region->ndr_mappings - 1), or if we find a + * dimm with two instances of the same uuid. + */ + rc = -EINVAL; + goto err; + } else if (pmem_uuid) { + /* + * If there is more than one valid uuid set, we + * need userspace to clean this up. + */ + rc = -EBUSY; + goto err; + } + memcpy(select_uuid, label_uuid, NSLABEL_UUID_LEN); + pmem_uuid = select_uuid; + } + + /* + * Fix up each mapping's 'labels' to have the validated pmem label for + * that position at labels[0], and NULL at labels[1]. In the process, + * check that the namespace aligns with interleave-set. We know + * that it does not overlap with any blk namespaces by virtue of + * the dimm being enabled (i.e. nd_label_reserve_dpa() + * succeeded). + */ + rc = select_pmem_uuid(nd_region, pmem_uuid); + if (rc) + goto err; + + /* Calculate total size and populate namespace properties from label0 */ + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_namespace_label __iomem *label0; + + label0 = nd_get_label(nd_mapping->labels, 0); + size += readq(&label0->rawsize); + if (readl(&label0->position) != 0) + continue; + WARN_ON(nspm->alt_name || nspm->uuid); + nspm->alt_name = kmemdup((void __force *) label0->name, + NSLABEL_NAME_LEN, GFP_KERNEL); + nspm->uuid = kmemdup((void __force *) label0->uuid, + NSLABEL_UUID_LEN, GFP_KERNEL); + } + + if (!nspm->alt_name || !nspm->uuid) { + rc = -ENOMEM; + goto err; + } + + nd_namespace_pmem_set_size(nd_region, nspm, size); + + return 0; + err: + switch (rc) { + case -EINVAL: + dev_dbg(&nd_region->dev, "%s: invalid label(s)\n", __func__); + break; + case -ENODEV: + dev_dbg(&nd_region->dev, "%s: label not found\n", __func__); + break; + default: + dev_dbg(&nd_region->dev, "%s: unexpected err: %d\n", __func__, rc); + break; + } + return rc; +} + +static struct device **create_namespace_pmem(struct nd_region *nd_region) +{ + struct nd_namespace_pmem *nspm; + struct device *dev, **devs; + struct resource *res; + int rc; + + nspm = kzalloc(sizeof(*nspm), GFP_KERNEL); + if (!nspm) + return NULL; + + dev = &nspm->nsio.dev; + dev->type = &namespace_pmem_device_type; + res = &nspm->nsio.res; + res->name = dev_name(&nd_region->dev); + res->flags = IORESOURCE_MEM; + rc = find_pmem_label_set(nd_region, nspm); + if (rc == -ENODEV) { + int i; + + /* Pass, try to permit namespace creation... */ + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + + kfree(nd_mapping->labels); + nd_mapping->labels = NULL; + } + + /* Publish a zero-sized namespace for userspace to configure. */ + nd_namespace_pmem_set_size(nd_region, nspm, 0); + + rc = 0; + } else if (rc) + goto err; + + devs = kcalloc(2, sizeof(struct device *), GFP_KERNEL); + if (!devs) + goto err; + + devs[0] = dev; + return devs; + + err: + namespace_pmem_release(&nspm->nsio.dev); + return NULL; +} + +static int init_active_labels(struct nd_region *nd_region) +{ + int i; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + int count, j; + + /* + * If the dimm is disabled then prevent the region from + * being activated if it aliases DPA. + */ + if (!ndd) { + struct nd_dimm *nd_dimm = nd_mapping->nd_dimm; + + if ((nd_dimm->flags & NDD_ALIASING) == 0) + return 0; + dev_dbg(&nd_region->dev, "%s: is disabled, failing probe\n", + dev_name(&nd_mapping->nd_dimm->dev)); + return -ENXIO; + } + + count = nd_label_active_count(ndd); + dev_dbg(ndd->dev, "%s: %d\n", __func__, count); + if (!count) + continue; + nd_mapping->labels = kcalloc(count + 1, + sizeof(struct nd_namespace_label *), GFP_KERNEL); + if (!nd_mapping->labels) + return -ENOMEM; + for (j = 0; j < count; j++) { + struct nd_namespace_label __iomem *label; + + label = nd_label_active(ndd, j); + nd_set_label(nd_mapping->labels, label, j); + } + } + + return 0; +} + int nd_region_register_namespaces(struct nd_region *nd_region, int *err) { struct device **devs = NULL; - int i; + int i, rc = 0, type; *err = 0; - switch (nd_region_to_namespace_type(nd_region)) { + nd_bus_lock(&nd_region->dev); + rc = init_active_labels(nd_region); + if (rc) { + nd_bus_unlock(&nd_region->dev); + return rc; + } + + type = nd_region_to_namespace_type(nd_region); + switch (type) { case ND_DEVICE_NAMESPACE_IO: devs = create_namespace_io(nd_region); break; + case ND_DEVICE_NAMESPACE_PMEM: + devs = create_namespace_pmem(nd_region); + break; default: break; } + nd_bus_unlock(&nd_region->dev); - if (!devs) - return -ENODEV; + if (!devs) { + rc = -ENODEV; + goto err; + } + nd_region->ns_seed = devs[0]; for (i = 0; devs[i]; i++) { struct device *dev = devs[i]; @@ -108,4 +1114,14 @@ int nd_region_register_namespaces(struct nd_region *nd_region, int *err) kfree(devs); return i; + + err: + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + + kfree(nd_mapping->labels); + nd_mapping->labels = NULL; + } + + return rc; } diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index 5d8249be3415..f09d7784dc11 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -60,4 +60,18 @@ int nd_bus_register_dimms(struct nd_bus *nd_bus); int nd_bus_register_regions(struct nd_bus *nd_bus); int nd_bus_init_interleave_sets(struct nd_bus *nd_bus); int nd_match_dimm(struct device *dev, void *data); +struct nd_label_id; +char *nd_label_gen_id(struct nd_label_id *label_id, u8 *uuid, u32 flags); +bool nd_is_uuid_unique(struct device *dev, u8 *uuid); +struct nd_region; +struct nd_dimm_drvdata; +struct nd_mapping; +resource_size_t nd_pmem_available_dpa(struct nd_region *nd_region, + struct nd_mapping *nd_mapping, resource_size_t *overlap); +resource_size_t nd_region_available_dpa(struct nd_region *nd_region); +struct resource *nd_dimm_allocate_dpa(struct nd_dimm_drvdata *ndd, + struct nd_label_id *label_id, resource_size_t start, + resource_size_t n); +resource_size_t nd_dimm_allocated_dpa(struct nd_dimm_drvdata *ndd, + struct nd_label_id *label_id); #endif /* __ND_PRIVATE_H__ */ diff --git a/drivers/block/nd/nd.h b/drivers/block/nd/nd.h index 832103a5e3f7..c93675172889 100644 --- a/drivers/block/nd/nd.h +++ b/drivers/block/nd/nd.h @@ -15,6 +15,7 @@ #include <linux/device.h> #include <linux/mutex.h> #include <linux/ndctl.h> +#include <linux/types.h> #include "libnd.h" #include "label.h" @@ -59,12 +60,37 @@ static inline struct nd_namespace_index __iomem *to_next_namespace_index( (unsigned long long) (res ? resource_size(res) : 0), \ (unsigned long long) (res ? res->start : 0), ##arg) +/* sparse helpers */ +static inline void nd_set_label(struct nd_namespace_label **labels, + struct nd_namespace_label __iomem *label, int idx) +{ + labels[idx] = (void __force *) label; +} + +static inline struct nd_namespace_label __iomem *nd_get_label( + struct nd_namespace_label **labels, int idx) +{ + struct nd_namespace_label __iomem *label = NULL; + + if (labels) + label = (struct nd_namespace_label __iomem *) labels[idx]; + + return label; +} + +#define for_each_label(l, label, labels) \ + for (l = 0; (label = nd_get_label(labels, l)); l++) + +#define for_each_dpa_resource(ndd, res) \ + for (res = (ndd)->dpa.child; res; res = res->sibling) + #define for_each_dpa_resource_safe(ndd, res, next) \ for (res = (ndd)->dpa.child, next = res ? res->sibling : NULL; \ res; res = next, next = next ? next->sibling : NULL) struct nd_region { struct device dev; + struct device *ns_seed; u16 ndr_mappings; u64 ndr_size; u64 ndr_start; @@ -88,15 +114,22 @@ enum nd_async_mode { ND_ASYNC, }; +void wait_nd_bus_probe_idle(struct device *dev); void nd_device_register(struct device *dev); void nd_device_unregister(struct device *dev, enum nd_async_mode mode); +int nd_uuid_store(struct device *dev, u8 **uuid_out, const char *buf, + size_t len); +struct nd_dimm; +struct nd_dimm_drvdata *to_ndd(struct nd_mapping *nd_mapping); int nd_dimm_init_nsarea(struct nd_dimm_drvdata *ndd); int nd_dimm_init_config_data(struct nd_dimm_drvdata *ndd); struct nd_region *to_nd_region(struct device *dev); int nd_region_to_namespace_type(struct nd_region *nd_region); int nd_region_register_namespaces(struct nd_region *nd_region, int *err); +u64 nd_region_interleave_set_cookie(struct nd_region *nd_region); void nd_bus_lock(struct device *dev); void nd_bus_unlock(struct device *dev); bool is_nd_bus_locked(struct device *dev); int nd_label_reserve_dpa(struct nd_dimm_drvdata *ndd); +void nd_dimm_free_dpa(struct nd_dimm_drvdata *ndd, struct resource *res); #endif /* __ND_H__ */ diff --git a/drivers/block/nd/pmem.c b/drivers/block/nd/pmem.c index c9dd8d7eca3a..32e7e74ee25a 100644 --- a/drivers/block/nd/pmem.c +++ b/drivers/block/nd/pmem.c @@ -24,6 +24,7 @@ #include <linux/moduleparam.h> #include <linux/slab.h> #include <linux/nd.h> +#include "nd.h" #define PMEM_MINORS 16 @@ -209,9 +210,27 @@ static void pmem_free(struct pmem_device *pmem) static int nd_pmem_probe(struct device *dev) { + struct nd_region *nd_region = to_nd_region(dev->parent); struct nd_namespace_io *nsio = to_nd_namespace_io(dev); struct pmem_device *pmem; + if (resource_size(&nsio->res) < ND_MIN_NAMESPACE_SIZE) { + resource_size_t size = resource_size(&nsio->res); + + dev_dbg(dev, "%s: size: %pa, too small must be at least %#x\n", + __func__, &size, ND_MIN_NAMESPACE_SIZE); + return -ENODEV; + } + + if (nd_region_to_namespace_type(nd_region) == ND_DEVICE_NAMESPACE_PMEM) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + if (!nspm->uuid) { + dev_dbg(dev, "%s: uuid not set\n", __func__); + return -ENODEV; + } + } + pmem = pmem_alloc(dev, &nsio->res); if (IS_ERR(pmem)) return PTR_ERR(pmem); @@ -231,13 +250,14 @@ static int nd_pmem_remove(struct device *dev) MODULE_ALIAS("pmem"); MODULE_ALIAS_ND_DEVICE(ND_DEVICE_NAMESPACE_IO); +MODULE_ALIAS_ND_DEVICE(ND_DEVICE_NAMESPACE_PMEM); static struct nd_device_driver nd_pmem_driver = { .probe = nd_pmem_probe, .remove = nd_pmem_remove, .drv = { .name = "pmem", }, - .type = ND_DRIVER_NAMESPACE_IO, + .type = ND_DRIVER_NAMESPACE_IO | ND_DRIVER_NAMESPACE_PMEM, }; static int __init pmem_init(void) diff --git a/drivers/block/nd/region_devs.c b/drivers/block/nd/region_devs.c index c1cea42d0473..e74b3d4534b4 100644 --- a/drivers/block/nd/region_devs.c +++ b/drivers/block/nd/region_devs.c @@ -15,6 +15,7 @@ #include <linux/slab.h> #include <linux/sort.h> #include <linux/io.h> +#include <linux/nd.h> #include "nd-private.h" #include "nd.h" @@ -99,6 +100,58 @@ int nd_region_to_namespace_type(struct nd_region *nd_region) return 0; } +EXPORT_SYMBOL(nd_region_to_namespace_type); + +static int is_uuid_busy(struct device *dev, void *data) +{ + struct nd_region *nd_region = to_nd_region(dev->parent); + u8 *uuid = data; + + switch (nd_region_to_namespace_type(nd_region)) { + case ND_DEVICE_NAMESPACE_PMEM: { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + + if (!nspm->uuid) + break; + if (memcmp(uuid, nspm->uuid, NSLABEL_UUID_LEN) == 0) + return -EBUSY; + break; + } + case ND_DEVICE_NAMESPACE_BLK: { + /* TODO: blk namespace support */ + break; + } + default: + break; + } + + return 0; +} + +static int is_namespace_uuid_busy(struct device *dev, void *data) +{ + if (is_nd_pmem(dev) || is_nd_blk(dev)) + return device_for_each_child(dev, data, is_uuid_busy); + return 0; +} + +/** + * nd_is_uuid_unique - verify that no other namespace has @uuid + * @dev: any device on a nd_bus + * @uuid: uuid to check + */ +bool nd_is_uuid_unique(struct device *dev, u8 *uuid) +{ + struct nd_bus *nd_bus = walk_to_nd_bus(dev); + + if (!nd_bus) + return false; + WARN_ON_ONCE(!is_nd_bus_locked(&nd_bus->dev)); + if (device_for_each_child(&nd_bus->dev, uuid, + is_namespace_uuid_busy) != 0) + return false; + return true; +} static ssize_t size_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -151,6 +204,60 @@ static ssize_t set_cookie_show(struct device *dev, } static DEVICE_ATTR_RO(set_cookie); +resource_size_t nd_region_available_dpa(struct nd_region *nd_region) +{ + resource_size_t blk_max_overlap = 0, available, overlap; + int i; + + WARN_ON(!is_nd_bus_locked(&nd_region->dev)); + + retry: + available = 0; + overlap = blk_max_overlap; + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + + /* if a dimm is disabled the available capacity is zero */ + if (!ndd) + return 0; + + if (is_nd_pmem(&nd_region->dev)) { + available += nd_pmem_available_dpa(nd_region, + nd_mapping, &overlap); + if (overlap > blk_max_overlap) { + blk_max_overlap = overlap; + goto retry; + } + } else if (is_nd_blk(&nd_region->dev)) { + /* TODO: BLK Namespace support */ + } + } + + return available; +} + +static ssize_t available_size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_region *nd_region = to_nd_region(dev); + unsigned long long available = 0; + + /* + * Flush in-flight updates and grab a snapshot of the available + * size. Of course, this value is potentially invalidated the + * memory nd_bus_lock() is dropped, but that's userspace's + * problem to not race itself. + */ + nd_bus_lock(dev); + wait_nd_bus_probe_idle(dev); + available = nd_region_available_dpa(nd_region); + nd_bus_unlock(dev); + + return sprintf(buf, "%llu\n", available); +} +static DEVICE_ATTR_RO(available_size); + static ssize_t init_namespaces_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -163,11 +270,29 @@ static ssize_t init_namespaces_show(struct device *dev, } static DEVICE_ATTR_RO(init_namespaces); +static ssize_t namespace_seed_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_region *nd_region = to_nd_region(dev); + ssize_t rc; + + nd_bus_lock(dev); + if (nd_region->ns_seed) + rc = sprintf(buf, "%s\n", dev_name(nd_region->ns_seed)); + else + rc = sprintf(buf, "\n"); + nd_bus_unlock(dev); + return rc; +} +static DEVICE_ATTR_RO(namespace_seed); + static struct attribute *nd_region_attributes[] = { &dev_attr_size.attr, &dev_attr_nstype.attr, &dev_attr_mappings.attr, &dev_attr_set_cookie.attr, + &dev_attr_available_size.attr, + &dev_attr_namespace_seed.attr, &dev_attr_init_namespaces.attr, NULL, }; @@ -177,12 +302,17 @@ static umode_t nd_region_visible(struct kobject *kobj, struct attribute *a, int struct device *dev = container_of(kobj, typeof(*dev), kobj); struct nd_region *nd_region = to_nd_region(dev); struct nd_interleave_set *nd_set = nd_region->nd_set; + int type = nd_region_to_namespace_type(nd_region); - if (a != &dev_attr_set_cookie.attr) + if (a != &dev_attr_set_cookie.attr && a != &dev_attr_available_size.attr) return a->mode; - if (is_nd_pmem(dev) && nd_set) - return a->mode; + if ((type == ND_DEVICE_NAMESPACE_PMEM + || type == ND_DEVICE_NAMESPACE_BLK) + && a == &dev_attr_available_size.attr) + return a->mode; + else if (is_nd_pmem(dev) && nd_set) + return a->mode; return 0; } @@ -193,6 +323,15 @@ struct attribute_group nd_region_attribute_group = { }; EXPORT_SYMBOL_GPL(nd_region_attribute_group); +u64 nd_region_interleave_set_cookie(struct nd_region *nd_region) +{ + struct nd_interleave_set *nd_set = nd_region->nd_set; + + if (nd_set) + return nd_set->cookie; + return 0; +} + /* * Upon successful probe/remove, take/release a reference on the * associated interleave set (if present) diff --git a/include/linux/nd.h b/include/linux/nd.h index da70e9962197..255c38a83083 100644 --- a/include/linux/nd.h +++ b/include/linux/nd.h @@ -28,16 +28,40 @@ static inline struct nd_device_driver *to_nd_device_driver( return container_of(drv, struct nd_device_driver, drv); }; +/** + * struct nd_namespace_io - infrastructure for loading an nd_pmem instance + * @dev: namespace device created by the nd region driver + * @res: struct resource conversion of a NFIT SPA table + */ struct nd_namespace_io { struct device dev; struct resource res; }; +/** + * struct nd_namespace_pmem - namespace device for dimm-backed interleaved memory + * @nsio: device and system physical address range to drive + * @alt_name: namespace name supplied in the dimm label + * @uuid: namespace name supplied in the dimm label + */ +struct nd_namespace_pmem { + struct nd_namespace_io nsio; + char *alt_name; + u8 *uuid; +}; + static inline struct nd_namespace_io *to_nd_namespace_io(struct device *dev) { return container_of(dev, struct nd_namespace_io, dev); } +static inline struct nd_namespace_pmem *to_nd_namespace_pmem(struct device *dev) +{ + struct nd_namespace_io *nsio = to_nd_namespace_io(dev); + + return container_of(nsio, struct nd_namespace_pmem, nsio); +} + #define MODULE_ALIAS_ND_DEVICE(type) \ MODULE_ALIAS("nd:t" __stringify(type) "*") #define ND_DEVICE_MODALIAS_FMT "nd:t%d" diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h index 624a19d9e6e4..0b4dcabb248a 100644 --- a/include/uapi/linux/ndctl.h +++ b/include/uapi/linux/ndctl.h @@ -190,4 +190,8 @@ enum nd_driver_flags { ND_DRIVER_NAMESPACE_PMEM = 1 << ND_DEVICE_NAMESPACE_PMEM, ND_DRIVER_NAMESPACE_BLK = 1 << ND_DEVICE_NAMESPACE_BLK, }; + +enum { + ND_MIN_NAMESPACE_SIZE = 0x00400000, +}; #endif /* __NDCTL_H__ */

After 'uuid', 'size', and optionally 'alt_name' have been set to valid values the labels on the dimms can be updated. Write procedure is: 1/ Allocate and write new labels in the "next" index 2/ Free the old labels in the working copy 3/ Write the bitmap and the label space on the dimm 4/ Write the index to make the update valid Label ranges directly mirror the dpa resource values for the given label_id of the namespace. Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; Cc: Neil Brown <neilb(a)suse.de&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/dimm_devs.c | 49 ++++++ drivers/block/nd/label.c | 327 +++++++++++++++++++++++++++++++++++++ drivers/block/nd/label.h | 6 + drivers/block/nd/namespace_devs.c | 82 ++++++++- drivers/block/nd/nd.h | 3 5 files changed, 453 insertions(+), 14 deletions(-) diff --git a/drivers/block/nd/dimm_devs.c b/drivers/block/nd/dimm_devs.c index 4aa5654354ac..358b2a06d680 100644 --- a/drivers/block/nd/dimm_devs.c +++ b/drivers/block/nd/dimm_devs.c @@ -132,6 +132,55 @@ int nd_dimm_init_config_data(struct nd_dimm_drvdata *ndd) return rc; } +int nd_dimm_set_config_data(struct nd_dimm_drvdata *ndd, size_t offset, + void *buf, size_t len) +{ + int rc = validate_dimm(ndd); + size_t max_cmd_size, buf_offset; + struct nd_cmd_set_config_hdr *cmd; + struct nd_bus *nd_bus = walk_to_nd_bus(ndd->dev); + struct nd_bus_descriptor *nd_desc = nd_bus->nd_desc; + + if (rc) + return rc; + + if (!ndd->data) + return -ENXIO; + + if (offset + len > ndd->nsarea.config_size) + return -ENXIO; + + max_cmd_size = min_t(u32, PAGE_SIZE, len); + max_cmd_size = min_t(u32, max_cmd_size, ndd->nsarea.max_xfer); + cmd = kzalloc(max_cmd_size + sizeof(*cmd) + sizeof(u32), GFP_KERNEL); + if (!cmd) + return -ENOMEM; + + for (buf_offset = 0; len; len -= cmd->in_length, + buf_offset += cmd->in_length) { + size_t cmd_size; + u32 *status; + + cmd->in_offset = offset + buf_offset; + cmd->in_length = min(max_cmd_size, len); + memcpy(cmd->in_buf, buf + buf_offset, cmd->in_length); + + /* status is output in the last 4-bytes of the command buffer */ + cmd_size = sizeof(*cmd) + cmd->in_length + sizeof(u32); + status = ((void *) cmd) + cmd_size - sizeof(u32); + + rc = nd_desc->ndctl(nd_desc, to_nd_dimm(ndd->dev), + ND_CMD_SET_CONFIG_DATA, cmd, cmd_size); + if (rc || *status) { + rc = rc ? rc : -ENXIO; + break; + } + } + kfree(cmd); + + return rc; +} + static void nd_dimm_release(struct device *dev) { struct nd_dimm *nd_dimm = to_nd_dimm(dev); diff --git a/drivers/block/nd/label.c b/drivers/block/nd/label.c index b55fa2a6f872..78898b642191 100644 --- a/drivers/block/nd/label.c +++ b/drivers/block/nd/label.c @@ -12,6 +12,7 @@ */ #include <linux/device.h> #include <linux/ndctl.h> +#include <linux/slab.h> #include <linux/io.h> #include <linux/nd.h> #include "nd-private.h" @@ -57,6 +58,11 @@ size_t sizeof_namespace_index(struct nd_dimm_drvdata *ndd) return ndd->nsindex_size; } +static int nd_dimm_num_label_slots(struct nd_dimm_drvdata *ndd) +{ + return ndd->nsarea.config_size / 129; +} + int nd_label_validate(struct nd_dimm_drvdata *ndd) { /* @@ -202,23 +208,30 @@ static struct nd_namespace_label __iomem *nd_label_base(struct nd_dimm_drvdata * return base + 2 * sizeof_namespace_index(ndd); } +static int to_slot(struct nd_dimm_drvdata *ndd, + struct nd_namespace_label __iomem *nd_label) +{ + return nd_label - nd_label_base(ndd); +} + #define for_each_clear_bit_le(bit, addr, size) \ for ((bit) = find_next_zero_bit_le((addr), (size), 0); \ (bit) < (size); \ (bit) = find_next_zero_bit_le((addr), (size), (bit) + 1)) /** - * preamble_current - common variable initialization for nd_label_* routines + * preamble_index - common variable initialization for nd_label_* routines * @nd_dimm: dimm container for the relevant label set + * @idx: namespace_index index * @nsindex: on return set to the currently active namespace index * @free: on return set to the free label bitmap in the index * @nslot: on return set to the number of slots in the label space */ -static bool preamble_current(struct nd_dimm_drvdata *ndd, +static bool preamble_index(struct nd_dimm_drvdata *ndd, int idx, struct nd_namespace_index **nsindex, unsigned long **free, u32 *nslot) { - *nsindex = to_current_namespace_index(ndd); + *nsindex = to_namespace_index(ndd, idx); if (*nsindex == NULL) return false; @@ -237,6 +250,22 @@ char *nd_label_gen_id(struct nd_label_id *label_id, u8 *uuid, u32 flags) return label_id->id; } +static bool preamble_current(struct nd_dimm_drvdata *ndd, + struct nd_namespace_index **nsindex, + unsigned long **free, u32 *nslot) +{ + return preamble_index(ndd, ndd->ns_current, nsindex, + free, nslot); +} + +static bool preamble_next(struct nd_dimm_drvdata *ndd, + struct nd_namespace_index **nsindex, + unsigned long **free, u32 *nslot) +{ + return preamble_index(ndd, ndd->ns_next, nsindex, + free, nslot); +} + static bool slot_valid(struct nd_namespace_label __iomem *nd_label, u32 slot) { /* check that we are written where we expect to be written */ @@ -341,3 +370,295 @@ struct nd_namespace_label __iomem *nd_label_active( return NULL; } + +u32 nd_label_alloc_slot(struct nd_dimm_drvdata *ndd) +{ + struct nd_namespace_index __iomem *nsindex; + unsigned long *free; + u32 nslot, slot; + + if (!preamble_next(ndd, &nsindex, &free, &nslot)) + return UINT_MAX; + + WARN_ON(!is_nd_bus_locked(ndd->dev)); + + slot = find_next_bit_le(free, nslot, 0); + if (slot == nslot) + return UINT_MAX; + + clear_bit_le(slot, free); + + return slot; +} + +bool nd_label_free_slot(struct nd_dimm_drvdata *ndd, u32 slot) +{ + struct nd_namespace_index __iomem *nsindex; + unsigned long *free; + u32 nslot; + + if (!preamble_next(ndd, &nsindex, &free, &nslot)) + return false; + + WARN_ON(!is_nd_bus_locked(ndd->dev)); + + if (slot < nslot) + return !test_and_set_bit_le(slot, free); + return false; +} + +u32 nd_label_nfree(struct nd_dimm_drvdata *ndd) +{ + struct nd_namespace_index __iomem *nsindex; + unsigned long *free; + u32 nslot; + + WARN_ON(!is_nd_bus_locked(ndd->dev)); + + if (!preamble_next(ndd, &nsindex, &free, &nslot)) + return 0; + + return bitmap_weight(free, nslot); +} + +static int nd_label_write_index(struct nd_dimm_drvdata *ndd, int index, u32 seq, + unsigned long flags) +{ + struct nd_namespace_index *nsindex = to_namespace_index(ndd, index); + unsigned long offset; + u64 checksum; + u32 nslot; + int rc; + + if (flags & ND_NSINDEX_INIT) + nslot = nd_dimm_num_label_slots(ndd); + else + nslot = readl(&nsindex->nslot); + + memcpy_toio(nsindex->sig, NSINDEX_SIGNATURE, NSINDEX_SIG_LEN); + writel(0, &nsindex->flags); + writel(seq, &nsindex->seq); + offset = (unsigned long) nsindex + - (unsigned long) to_namespace_index(ndd, 0); + writeq(offset, &nsindex->myoff); + writeq(sizeof_namespace_index(ndd), &nsindex->mysize); + offset = (unsigned long) to_namespace_index(ndd, + nd_label_next_nsindex(index)) + - (unsigned long) to_namespace_index(ndd, 0); + writeq(offset, &nsindex->otheroff); + offset = (unsigned long) nd_label_base(ndd) + - (unsigned long) to_namespace_index(ndd, 0); + writeq(offset, &nsindex->labeloff); + writel(nslot, &nsindex->nslot); + writew(1, &nsindex->major); + writew(1, &nsindex->minor); + writeq(0, &nsindex->checksum); + if (flags & ND_NSINDEX_INIT) { + unsigned long *free = (unsigned long __force *) nsindex->free; + u32 nfree = ALIGN(nslot, BITS_PER_LONG); + int last_bits, i; + + memset_io(nsindex->free, 0xff, nfree / 8); + for (i = 0, last_bits = nfree - nslot; i < last_bits; i++) + clear_bit_le(nslot + i, free); + } + checksum = nd_fletcher64(nsindex, sizeof_namespace_index(ndd)); + writeq(checksum, &nsindex->checksum); + rc = nd_dimm_set_config_data(ndd, readq(&nsindex->myoff), + nsindex, sizeof_namespace_index(ndd)); + if (rc < 0) + return rc; + + if (flags & ND_NSINDEX_INIT) + return 0; + + /* copy the index we just wrote to the new 'next' */ + WARN_ON(index != ndd->ns_next); + nd_label_copy(ndd, to_current_namespace_index(ndd), nsindex); + ndd->ns_current = nd_label_next_nsindex(ndd->ns_current); + ndd->ns_next = nd_label_next_nsindex(ndd->ns_next); + WARN_ON(ndd->ns_current == ndd->ns_next); + + return 0; +} + +static unsigned long nd_label_offset(struct nd_dimm_drvdata *ndd, + struct nd_namespace_label __iomem *nd_label) +{ + return (unsigned long) nd_label + - (unsigned long) to_namespace_index(ndd, 0); +} + +static int __pmem_label_update(struct nd_region *nd_region, + struct nd_mapping *nd_mapping, struct nd_namespace_pmem *nspm, + int pos) +{ + u64 cookie = nd_region_interleave_set_cookie(nd_region), rawsize; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + struct nd_namespace_label __iomem *victim_label; + struct nd_namespace_label __iomem *nd_label; + struct nd_namespace_index __iomem *nsindex; + unsigned long *free; + u32 nslot, slot; + size_t offset; + int rc; + + if (!preamble_next(ndd, &nsindex, &free, &nslot)) + return -ENXIO; + + /* allocate and write the label to the staging (next) index */ + slot = nd_label_alloc_slot(ndd); + if (slot == UINT_MAX) + return -ENXIO; + dev_dbg(ndd->dev, "%s: allocated: %d\n", __func__, slot); + + nd_label = nd_label_base(ndd) + slot; + memset_io(nd_label, 0, sizeof(struct nd_namespace_label)); + memcpy_toio(nd_label->uuid, nspm->uuid, NSLABEL_UUID_LEN); + if (nspm->alt_name) + memcpy_toio(nd_label->name, nspm->alt_name, NSLABEL_NAME_LEN); + writel(NSLABEL_FLAG_UPDATING, &nd_label->flags); + writew(nd_region->ndr_mappings, &nd_label->nlabel); + writew(pos, &nd_label->position); + writeq(cookie, &nd_label->isetcookie); + rawsize = div_u64(resource_size(&nspm->nsio.res), + nd_region->ndr_mappings); + writeq(rawsize, &nd_label->rawsize); + writeq(nd_mapping->start, &nd_label->dpa); + writel(slot, &nd_label->slot); + + /* update label */ + offset = nd_label_offset(ndd, nd_label); + rc = nd_dimm_set_config_data(ndd, offset, nd_label, + sizeof(struct nd_namespace_label)); + if (rc < 0) + return rc; + + /* Garbage collect the previous label */ + victim_label = nd_get_label(nd_mapping->labels, 0); + if (victim_label) { + slot = to_slot(ndd, victim_label); + nd_label_free_slot(ndd, slot); + dev_dbg(ndd->dev, "%s: free: %d\n", __func__, slot); + } + + /* update index */ + rc = nd_label_write_index(ndd, ndd->ns_next, + nd_inc_seq(readl(&nsindex->seq)), 0); + if (rc < 0) + return rc; + + nd_set_label(nd_mapping->labels, nd_label, 0); + + return 0; +} + +static int init_labels(struct nd_mapping *nd_mapping) +{ + int i; + struct nd_namespace_index __iomem *nsindex; + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + + if (!nd_mapping->labels) + nd_mapping->labels = kcalloc(2, sizeof(void *), GFP_KERNEL); + + if (!nd_mapping->labels) + return -ENOMEM; + + if (ndd->ns_current == -1 || ndd->ns_next == -1) + /* pass */; + else + return 0; + + nsindex = to_namespace_index(ndd, 0); + memset_io(nsindex, 0, ndd->nsarea.config_size); + for (i = 0; i < 2; i++) { + int rc = nd_label_write_index(ndd, i, i*2, ND_NSINDEX_INIT); + + if (rc) + return rc; + } + ndd->ns_next = 1; + ndd->ns_current = 0; + + return 0; +} + +static int del_labels(struct nd_mapping *nd_mapping, u8 *uuid) +{ + struct nd_dimm_drvdata *ndd = to_ndd(nd_mapping); + struct nd_namespace_label __iomem *nd_label; + struct nd_namespace_index __iomem *nsindex; + u8 label_uuid[NSLABEL_UUID_LEN]; + int l, num_freed = 0; + unsigned long *free; + u32 nslot, slot; + + if (!uuid) + return 0; + + /* no index || no labels == nothing to delete */ + if (!preamble_next(ndd, &nsindex, &free, &nslot) + || !nd_mapping->labels) + return 0; + + for_each_label(l, nd_label, nd_mapping->labels) { + int j; + + memcpy_fromio(label_uuid, nd_label->uuid, NSLABEL_UUID_LEN); + if (memcmp(label_uuid, uuid, NSLABEL_UUID_LEN) != 0) + continue; + slot = to_slot(ndd, nd_label); + nd_label_free_slot(ndd, slot); + dev_dbg(ndd->dev, "%s: free: %d\n", __func__, slot); + for (j = l; nd_get_label(nd_mapping->labels, j + 1); j++) { + struct nd_namespace_label __iomem *next_label; + + next_label = nd_get_label(nd_mapping->labels, j + 1); + nd_set_label(nd_mapping->labels, next_label, j); + } + nd_set_label(nd_mapping->labels, NULL, j); + num_freed++; + } + + if (num_freed > l) { + /* + * num_freed will only ever be > l when we delete the last + * label + */ + kfree(nd_mapping->labels); + nd_mapping->labels = NULL; + dev_dbg(ndd->dev, "%s: no more labels\n", __func__); + } + + return nd_label_write_index(ndd, ndd->ns_next, + nd_inc_seq(readl(&nsindex->seq)), 0); +} + +int nd_pmem_namespace_label_update(struct nd_region *nd_region, + struct nd_namespace_pmem *nspm, resource_size_t size) +{ + int i; + + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + int rc; + + if (size == 0) { + rc = del_labels(nd_mapping, nspm->uuid); + if (rc) + return rc; + continue; + } + + rc = init_labels(nd_mapping); + if (rc) + return rc; + + rc = __pmem_label_update(nd_region, nd_mapping, nspm, i); + if (rc) + return rc; + } + + return 0; +} diff --git a/drivers/block/nd/label.h b/drivers/block/nd/label.h index 4436624f4146..e17958941e34 100644 --- a/drivers/block/nd/label.h +++ b/drivers/block/nd/label.h @@ -34,6 +34,7 @@ enum { BTTINFO_MAJOR_VERSION = 1, ND_LABEL_MIN_SIZE = 512 * 129, /* see sizeof_namespace_index() */ ND_LABEL_ID_SIZE = 50, + ND_NSINDEX_INIT = 0x1, }; static const char NSINDEX_SIGNATURE[] = "NAMESPACE_INDEX\0"; @@ -129,4 +130,9 @@ size_t sizeof_namespace_index(struct nd_dimm_drvdata *ndd); int nd_label_active_count(struct nd_dimm_drvdata *ndd); struct nd_namespace_label __iomem *nd_label_active( struct nd_dimm_drvdata *ndd, int n); +u32 nd_label_nfree(struct nd_dimm_drvdata *ndd); +struct nd_region; +struct nd_namespace_pmem; +int nd_pmem_namespace_label_update(struct nd_region *nd_region, + struct nd_namespace_pmem *nspm, resource_size_t size); #endif /* __LABEL_H__ */ diff --git a/drivers/block/nd/namespace_devs.c b/drivers/block/nd/namespace_devs.c index e229755509a2..c04f55fb7c7c 100644 --- a/drivers/block/nd/namespace_devs.c +++ b/drivers/block/nd/namespace_devs.c @@ -151,20 +151,52 @@ static resource_size_t nd_namespace_blk_size(struct nd_namespace_blk *nsblk) return size; } +static int nd_namespace_label_update(struct nd_region *nd_region, struct device *dev) +{ + dev_WARN_ONCE(dev, dev->driver, + "namespace must be idle during label update\n"); + if (dev->driver) + return 0; + + /* + * Only allow label writes that will result in a valid namespace + * or deletion of an existing namespace. + */ + if (is_namespace_pmem(dev)) { + struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); + struct resource *res = &nspm->nsio.res; + resource_size_t size = resource_size(res); + + if (size == 0 && nspm->uuid) + /* delete allocation */; + else if (!nspm->uuid) + return 0; + + return nd_pmem_namespace_label_update(nd_region, nspm, size); + } else if (is_namespace_blk(dev)) { + /* TODO: implement blk labels */ + return 0; + } else + return -ENXIO; +} + static ssize_t alt_name_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { + struct nd_region *nd_region = to_nd_region(dev->parent); ssize_t rc; device_lock(dev); nd_bus_lock(dev); wait_nd_bus_probe_idle(dev); rc = __alt_name_store(dev, buf, len); + if (rc >= 0) + rc = nd_namespace_label_update(nd_region, dev); dev_dbg(dev, "%s: %s (%zd)\n", __func__, rc < 0 ? "fail" : "success", rc); nd_bus_unlock(dev); device_unlock(dev); - return rc; + return rc < 0 ? rc : len; } static ssize_t alt_name_show(struct device *dev, @@ -707,6 +739,7 @@ static ssize_t __size_store(struct device *dev, unsigned long long val) static ssize_t size_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { + struct nd_region *nd_region = to_nd_region(dev->parent); unsigned long long val; u8 **uuid = NULL; int rc; @@ -719,6 +752,8 @@ static ssize_t size_store(struct device *dev, nd_bus_lock(dev); wait_nd_bus_probe_idle(dev); rc = __size_store(dev, val); + if (rc >= 0) + rc = nd_namespace_label_update(nd_region, dev); if (is_namespace_pmem(dev)) { struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev); @@ -742,7 +777,7 @@ static ssize_t size_store(struct device *dev, nd_bus_unlock(dev); device_unlock(dev); - return rc ? rc : len; + return rc < 0 ? rc : len; } static ssize_t size_show(struct device *dev, @@ -802,17 +837,34 @@ static int namespace_update_uuid(struct nd_region *nd_region, u32 flags = is_namespace_blk(dev) ? NSLABEL_FLAG_LOCAL : 0; struct nd_label_id old_label_id; struct nd_label_id new_label_id; - int i, rc; + int i; - rc = nd_is_uuid_unique(dev, new_uuid) ? 0 : -EINVAL; - if (rc) { - kfree(new_uuid); - return rc; - } + if (!nd_is_uuid_unique(dev, new_uuid)) + return -EINVAL; if (*old_uuid == NULL) goto out; + /* + * If we've already written a label with this uuid, then it's + * too late to rename because we can't reliably update the uuid + * without losing the old namespace. Userspace must delete this + * namespace to abandon the old uuid. + */ + for (i = 0; i < nd_region->ndr_mappings; i++) { + struct nd_mapping *nd_mapping = &nd_region->mapping[i]; + + /* + * This check by itself is sufficient because old_uuid + * would be NULL above if this uuid did not exist in the + * currently written set. + * + * FIXME: can we delete uuid with zero dpa allocated? + */ + if (nd_mapping->labels) + return -EBUSY; + } + nd_label_gen_id(&old_label_id, *old_uuid, flags); nd_label_gen_id(&new_label_id, new_uuid, flags); for (i = 0; i < nd_region->ndr_mappings; i++) { @@ -856,12 +908,16 @@ static ssize_t uuid_store(struct device *dev, rc = nd_uuid_store(dev, &uuid, buf, len); if (rc >= 0) rc = namespace_update_uuid(nd_region, dev, uuid, ns_uuid); + if (rc >= 0) + rc = nd_namespace_label_update(nd_region, dev); + else + kfree(uuid); dev_dbg(dev, "%s: result: %zd wrote: %s%s", __func__, rc, buf, buf[len - 1] == '\n' ? "" : "\n"); nd_bus_unlock(dev); device_unlock(dev); - return rc ? rc : len; + return rc < 0 ? rc : len; } static DEVICE_ATTR_RW(uuid); @@ -905,6 +961,7 @@ static ssize_t sector_size_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { struct nd_namespace_blk *nsblk = to_nd_namespace_blk(dev); + struct nd_region *nd_region = to_nd_region(dev->parent); ssize_t rc; if (!is_namespace_blk(dev)) @@ -914,8 +971,11 @@ static ssize_t sector_size_store(struct device *dev, nd_bus_lock(dev); rc = nd_sector_size_store(dev, buf, &nsblk->lbasize, ns_lbasize_supported); - dev_dbg(dev, "%s: result: %zd wrote: %s%s", __func__, - rc, buf, buf[len - 1] == '\n' ? "" : "\n"); + if (rc >= 0) + rc = nd_namespace_label_update(nd_region, dev); + dev_dbg(dev, "%s: result: %zd %s: %s%s", __func__, + rc, rc < 0 ? "tried" : "wrote", buf, + buf[len - 1] == '\n' ? "" : "\n"); nd_bus_unlock(dev); device_unlock(dev); diff --git a/drivers/block/nd/nd.h b/drivers/block/nd/nd.h index 63d4c6c09275..8eaf0ae82db4 100644 --- a/drivers/block/nd/nd.h +++ b/drivers/block/nd/nd.h @@ -110,6 +110,7 @@ static inline unsigned nd_inc_seq(unsigned seq) return next[seq & 3]; } + enum nd_async_mode { ND_SYNC, ND_ASYNC, @@ -128,6 +129,8 @@ struct nd_dimm; struct nd_dimm_drvdata *to_ndd(struct nd_mapping *nd_mapping); int nd_dimm_init_nsarea(struct nd_dimm_drvdata *ndd); int nd_dimm_init_config_data(struct nd_dimm_drvdata *ndd); +int nd_dimm_set_config_data(struct nd_dimm_drvdata *ndd, size_t offset, + void *buf, size_t len); struct nd_region *to_nd_region(struct device *dev); int nd_region_to_namespace_type(struct nd_region *nd_region); int nd_region_register_namespaces(struct nd_region *nd_region, int *err);

Block devices from an nd bus, in addition to accepting "struct bio" based requests, also have the capability to perform byte-aligned accesses. By default only the bio/block interface is used. However, if another driver can make effective use of the byte-aligned capability it can claim/disable the block interface and use the byte-aligned "nd_io" interface. The BTT driver is the intended first consumer of this mechanism to allow layering atomic sector update guarantees on top of nd_io capable nd-bus-block-devices. Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; Cc: Neil Brown <neilb(a)suse.de&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- drivers/block/nd/Kconfig | 3 drivers/block/nd/Makefile | 1 drivers/block/nd/btt.h | 45 ++++ drivers/block/nd/btt_devs.c | 442 +++++++++++++++++++++++++++++++++++++++++ drivers/block/nd/bus.c | 128 ++++++++++++ drivers/block/nd/core.c | 79 +++++++ drivers/block/nd/nd-private.h | 28 +++ drivers/block/nd/nd.h | 94 +++++++++ drivers/block/nd/pmem.c | 29 +++ include/uapi/linux/ndctl.h | 2 10 files changed, 847 insertions(+), 4 deletions(-) create mode 100644 drivers/block/nd/btt.h create mode 100644 drivers/block/nd/btt_devs.c diff --git a/drivers/block/nd/Kconfig b/drivers/block/nd/Kconfig index c5eaf195734d..15896db4de37 100644 --- a/drivers/block/nd/Kconfig +++ b/drivers/block/nd/Kconfig @@ -95,4 +95,7 @@ config BLK_DEV_PMEM Say Y if you want to use a NVDIMM described by NFIT +config ND_BTT_DEVS + def_bool y + endif diff --git a/drivers/block/nd/Makefile b/drivers/block/nd/Makefile index d588f691163c..0c6d64b7a69d 100644 --- a/drivers/block/nd/Makefile +++ b/drivers/block/nd/Makefile @@ -32,3 +32,4 @@ libnd-y += region_devs.o libnd-y += region.o libnd-y += namespace_devs.o libnd-y += label.o +libnd-$(CONFIG_ND_BTT_DEVS) += btt_devs.o diff --git a/drivers/block/nd/btt.h b/drivers/block/nd/btt.h new file mode 100644 index 000000000000..e8f6d8e0ddd3 --- /dev/null +++ b/drivers/block/nd/btt.h @@ -0,0 +1,45 @@ +/* + * Block Translation Table library + * Copyright (c) 2014-2015, Intel Corporation. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ + +#ifndef _LINUX_BTT_H +#define _LINUX_BTT_H + +#include <linux/types.h> + +#define BTT_SIG_LEN 16 +#define BTT_SIG "BTT_ARENA_INFO\0" + +struct btt_sb { + u8 signature[BTT_SIG_LEN]; + u8 uuid[16]; + u8 parent_uuid[16]; + __le32 flags; + __le16 version_major; + __le16 version_minor; + __le32 external_lbasize; + __le32 external_nlba; + __le32 internal_lbasize; + __le32 internal_nlba; + __le32 nfree; + __le32 infosize; + __le64 nextoff; + __le64 dataoff; + __le64 mapoff; + __le64 logoff; + __le64 info2off; + u8 padding[3968]; + __le64 checksum; +}; + +#endif diff --git a/drivers/block/nd/btt_devs.c b/drivers/block/nd/btt_devs.c new file mode 100644 index 000000000000..e6f0b8b999d8 --- /dev/null +++ b/drivers/block/nd/btt_devs.c @@ -0,0 +1,442 @@ +/* + * Copyright(c) 2013-2015 Intel Corporation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ +#include <linux/device.h> +#include <linux/genhd.h> +#include <linux/sizes.h> +#include <linux/slab.h> +#include <linux/fs.h> +#include <linux/mm.h> +#include "nd-private.h" +#include "btt.h" +#include "nd.h" + +static DEFINE_IDA(btt_ida); + +static void nd_btt_release(struct device *dev) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + + dev_dbg(dev, "%s\n", __func__); + WARN_ON(nd_btt->backing_dev); + ndio_del_claim(nd_btt->ndio_claim); + ida_simple_remove(&btt_ida, nd_btt->id); + kfree(nd_btt->uuid); + kfree(nd_btt); +} + +static struct device_type nd_btt_device_type = { + .name = "nd_btt", + .release = nd_btt_release, +}; + +bool is_nd_btt(struct device *dev) +{ + return dev->type == &nd_btt_device_type; +} + +struct nd_btt *to_nd_btt(struct device *dev) +{ + struct nd_btt *nd_btt = container_of(dev, struct nd_btt, dev); + + WARN_ON(!is_nd_btt(dev)); + return nd_btt; +} +EXPORT_SYMBOL(to_nd_btt); + +static const unsigned long btt_lbasize_supported[] = { 512, 4096, 0 }; + +static ssize_t sector_size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + + return nd_sector_size_show(nd_btt->lbasize, btt_lbasize_supported, buf); +} + +static ssize_t sector_size_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + ssize_t rc; + + device_lock(dev); + nd_bus_lock(dev); + rc = nd_sector_size_store(dev, buf, &nd_btt->lbasize, + btt_lbasize_supported); + dev_dbg(dev, "%s: result: %zd wrote: %s%s", __func__, + rc, buf, buf[len - 1] == '\n' ? "" : "\n"); + nd_bus_unlock(dev); + device_unlock(dev); + + return rc ? rc : len; +} +static DEVICE_ATTR_RW(sector_size); + +static ssize_t uuid_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + + if (nd_btt->uuid) + return sprintf(buf, "%pUb\n", nd_btt->uuid); + return sprintf(buf, "\n"); +} + +static ssize_t uuid_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + ssize_t rc; + + device_lock(dev); + rc = nd_uuid_store(dev, &nd_btt->uuid, buf, len); + dev_dbg(dev, "%s: result: %zd wrote: %s%s", __func__, + rc, buf, buf[len - 1] == '\n' ? "" : "\n"); + device_unlock(dev); + + return rc ? rc : len; +} +static DEVICE_ATTR_RW(uuid); + +static ssize_t backing_dev_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + char name[BDEVNAME_SIZE]; + + if (nd_btt->backing_dev) + return sprintf(buf, "/dev/%s\n", + bdevname(nd_btt->backing_dev, name)); + else + return sprintf(buf, "\n"); +} + +static const fmode_t nd_btt_devs_mode = FMODE_READ | FMODE_WRITE | FMODE_EXCL; + +static void nd_btt_ndio_notify_remove(struct nd_io_claim *ndio_claim) +{ + char bdev_name[BDEVNAME_SIZE]; + struct nd_btt *nd_btt; + + if (!ndio_claim || !ndio_claim->holder) + return; + + nd_btt = to_nd_btt(ndio_claim->holder); + WARN_ON_ONCE(!is_nd_bus_locked(&nd_btt->dev)); + dev_dbg(&nd_btt->dev, "%pf: %s: release /dev/%s\n", + __builtin_return_address(0), __func__, + bdevname(nd_btt->backing_dev, bdev_name)); + blkdev_put(nd_btt->backing_dev, nd_btt_devs_mode); + nd_btt->backing_dev = NULL; + + /* + * Once we've had our backing device removed we need to be fully + * reconfigured. The bus will have already created a new seed + * for this purpose, so now is a good time to clean up this + * stale nd_btt instance. + */ + if (nd_btt->dev.driver) + nd_device_unregister(&nd_btt->dev, ND_ASYNC); + else { + ndio_del_claim(ndio_claim); + nd_btt->ndio_claim = NULL; + } +} + +static ssize_t __backing_dev_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + struct nd_bus *nd_bus = walk_to_nd_bus(dev); + struct nd_btt *nd_btt = to_nd_btt(dev); + char bdev_name[BDEVNAME_SIZE]; + struct block_device *bdev; + struct nd_io *ndio; + char *path; + + if (dev->driver) { + dev_dbg(dev, "%s: -EBUSY\n", __func__); + return -EBUSY; + } + + path = kstrndup(buf, len, GFP_KERNEL); + if (!path) + return -ENOMEM; + + /* detach the backing device */ + if (strcmp(strim(path), "") == 0) { + if (!nd_btt->backing_dev) + goto out; + nd_btt_ndio_notify_remove(nd_btt->ndio_claim); + goto out; + } else if (nd_btt->backing_dev) { + dev_dbg(dev, "backing_dev already set\n"); + len = -EBUSY; + goto out; + } + + bdev = blkdev_get_by_path(strim(path), nd_btt_devs_mode, nd_btt); + if (IS_ERR(bdev)) { + dev_dbg(dev, "open '%s' failed: %ld\n", strim(path), + PTR_ERR(bdev)); + len = PTR_ERR(bdev); + goto out; + } + + if (get_capacity(bdev->bd_disk) < SZ_16M / 512) { + blkdev_put(bdev, nd_btt_devs_mode); + len = -ENXIO; + goto out; + } + + ndio = ndio_lookup(nd_bus, bdevname(bdev->bd_contains, bdev_name)); + if (!ndio) { + dev_dbg(dev, "%s does not have an ndio interface\n", + strim(path)); + blkdev_put(bdev, nd_btt_devs_mode); + len = -ENXIO; + goto out; + } + + nd_btt->ndio_claim = ndio_add_claim(ndio, &nd_btt->dev, + nd_btt_ndio_notify_remove); + if (!nd_btt->ndio_claim) { + blkdev_put(bdev, nd_btt_devs_mode); + len = -ENOMEM; + goto out; + } + + WARN_ON_ONCE(!is_nd_bus_locked(&nd_btt->dev)); + nd_btt->backing_dev = bdev; + + out: + kfree(path); + return len; +} + +static ssize_t backing_dev_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + ssize_t rc; + + nd_bus_lock(dev); + device_lock(dev); + rc = __backing_dev_store(dev, attr, buf, len); + dev_dbg(dev, "%s: result: %zd wrote: %s%s", __func__, + rc, buf, buf[len - 1] == '\n' ? "" : "\n"); + device_unlock(dev); + nd_bus_unlock(dev); + + return rc; +} +static DEVICE_ATTR_RW(backing_dev); + +static bool is_nd_btt_idle(struct device *dev) +{ + struct nd_bus *nd_bus = walk_to_nd_bus(dev); + struct nd_btt *nd_btt = to_nd_btt(dev); + + if (nd_bus->nd_btt == nd_btt || dev->driver || nd_btt->backing_dev) + return false; + return true; +} + +static ssize_t delete_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + /* return 1 if can be deleted */ + return sprintf(buf, "%d\n", is_nd_btt_idle(dev)); +} + +static ssize_t delete_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + unsigned long val; + + /* write 1 to delete */ + if (kstrtoul(buf, 0, &val) != 0 || val != 1) + return -EINVAL; + + /* prevent deletion while this btt is active, or is the current seed */ + if (!is_nd_btt_idle(dev)) + return -EBUSY; + + /* + * userspace raced itself if device goes active here and it gets + * to keep the pieces + */ + nd_device_unregister(dev, ND_ASYNC); + + return len; +} +static DEVICE_ATTR_RW(delete); + +static struct attribute *nd_btt_attributes[] = { + &dev_attr_sector_size.attr, + &dev_attr_backing_dev.attr, + &dev_attr_delete.attr, + &dev_attr_uuid.attr, + NULL, +}; + +static struct attribute_group nd_btt_attribute_group = { + .attrs = nd_btt_attributes, +}; + +static const struct attribute_group *nd_btt_attribute_groups[] = { + &nd_btt_attribute_group, + &nd_device_attribute_group, + NULL, +}; + +static struct nd_btt *__nd_btt_create(struct nd_bus *nd_bus, + unsigned long lbasize, u8 *uuid) +{ + struct nd_btt *nd_btt = kzalloc(sizeof(*nd_btt), GFP_KERNEL); + struct device *dev; + + if (!nd_btt) + return NULL; + nd_btt->id = ida_simple_get(&btt_ida, 0, 0, GFP_KERNEL); + if (nd_btt->id < 0) { + kfree(nd_btt); + return NULL; + } + + nd_btt->lbasize = lbasize; + if (uuid) + uuid = kmemdup(uuid, 16, GFP_KERNEL); + nd_btt->uuid = uuid; + dev = &nd_btt->dev; + dev_set_name(dev, "btt%d", nd_btt->id); + dev->parent = &nd_bus->dev; + dev->type = &nd_btt_device_type; + dev->groups = nd_btt_attribute_groups; + return nd_btt; +} + +struct nd_btt *nd_btt_create(struct nd_bus *nd_bus) +{ + struct nd_btt *nd_btt = __nd_btt_create(nd_bus, 0, NULL); + + if (!nd_btt) + return NULL; + nd_device_register(&nd_btt->dev); + return nd_btt; +} + +/* + * nd_btt_sb_checksum: compute checksum for btt info block + * + * Returns a fletcher64 checksum of everything in the given info block + * except the last field (since that's where the checksum lives). + */ +u64 nd_btt_sb_checksum(struct btt_sb *btt_sb) +{ + u64 sum, sum_save; + + sum_save = btt_sb->checksum; + btt_sb->checksum = 0; + sum = nd_fletcher64(btt_sb, sizeof(*btt_sb)); + btt_sb->checksum = sum_save; + return sum; +} +EXPORT_SYMBOL(nd_btt_sb_checksum); + +static int nd_btt_autodetect(struct nd_bus *nd_bus, struct nd_io *ndio, + struct block_device *bdev) +{ + char name[BDEVNAME_SIZE]; + struct nd_btt *nd_btt; + struct btt_sb *btt_sb; + u64 offset, checksum; + u32 lbasize; + u8 *uuid; + int rc; + + btt_sb = kzalloc(sizeof(*btt_sb), GFP_KERNEL); + if (!btt_sb) + return -ENODEV; + + offset = nd_partition_offset(bdev); + rc = ndio->rw_bytes(ndio, btt_sb, offset + SZ_4K, sizeof(*btt_sb), READ); + if (rc) + goto out_free_sb; + + if (get_capacity(bdev->bd_disk) < SZ_16M / 512) + goto out_free_sb; + + if (memcmp(btt_sb->signature, BTT_SIG, BTT_SIG_LEN) != 0) + goto out_free_sb; + + checksum = le64_to_cpu(btt_sb->checksum); + btt_sb->checksum = 0; + if (checksum != nd_btt_sb_checksum(btt_sb)) + goto out_free_sb; + btt_sb->checksum = cpu_to_le64(checksum); + + uuid = kmemdup(btt_sb->uuid, 16, GFP_KERNEL); + if (!uuid) + goto out_free_sb; + + lbasize = le32_to_cpu(btt_sb->external_lbasize); + nd_btt = __nd_btt_create(nd_bus, lbasize, uuid); + if (!nd_btt) + goto out_free_uuid; + + device_initialize(&nd_btt->dev); + nd_btt->ndio_claim = ndio_add_claim(ndio, &nd_btt->dev, + nd_btt_ndio_notify_remove); + if (!nd_btt->ndio_claim) + goto out_free_btt; + + nd_btt->backing_dev = bdev; + dev_dbg(&nd_btt->dev, "%s: activate %s\n", __func__, + bdevname(bdev, name)); + __nd_device_register(&nd_btt->dev); + kfree(btt_sb); + return 0; + + out_free_btt: + kfree(nd_btt); + out_free_uuid: + kfree(uuid); + out_free_sb: + kfree(btt_sb); + + return -ENODEV; +} + +void nd_btt_notify_ndio(struct nd_bus *nd_bus, struct nd_io *ndio) +{ + struct disk_part_iter piter; + struct hd_struct *part; + + disk_part_iter_init(&piter, ndio->disk, DISK_PITER_INCL_PART0); + while ((part = disk_part_iter_next(&piter))) { + struct block_device *bdev; + int rc; + + bdev = bdget_disk(ndio->disk, part->partno); + if (!bdev) + continue; + if (blkdev_get(bdev, nd_btt_devs_mode, nd_bus) != 0) + continue; + rc = nd_btt_autodetect(nd_bus, ndio, bdev); + if (rc) + blkdev_put(bdev, nd_btt_devs_mode); + /* no need to scan further in the case of whole disk btt */ + if (rc == 0 && part->partno == 0) + break; + } + disk_part_iter_exit(&piter); +} diff --git a/drivers/block/nd/bus.c b/drivers/block/nd/bus.c index 6c272f245f4e..27785f3fc4ca 100644 --- a/drivers/block/nd/bus.c +++ b/drivers/block/nd/bus.c @@ -16,6 +16,7 @@ #include <linux/module.h> #include <linux/fcntl.h> #include <linux/async.h> +#include <linux/genhd.h> #include <linux/ndctl.h> #include <linux/sched.h> #include <linux/slab.h> @@ -40,6 +41,8 @@ static int to_nd_device_type(struct device *dev) return ND_DEVICE_REGION_BLK; else if (is_nd_pmem(dev->parent) || is_nd_blk(dev->parent)) return nd_region_to_namespace_type(to_nd_region(dev->parent)); + else if (is_nd_btt(dev)) + return ND_DEVICE_BTT; return 0; } @@ -84,6 +87,21 @@ static int nd_bus_probe(struct device *dev) dev_dbg(&nd_bus->dev, "%s.probe(%s) = %d\n", dev->driver->name, dev_name(dev), rc); + + /* check if our btt-seed has sprouted, and plant another */ + if (rc == 0 && is_nd_btt(dev) && dev == &nd_bus->nd_btt->dev) { + const char *sep = "", *name = "", *status = "failed"; + + nd_bus->nd_btt = nd_btt_create(nd_bus); + if (nd_bus->nd_btt) { + status = "succeeded"; + sep = ": "; + name = dev_name(&nd_bus->nd_btt->dev); + } + dev_dbg(&nd_bus->dev, "btt seed creation %s%s%s\n", + status, sep, name); + } + if (rc != 0) module_put(provider); return rc; @@ -144,14 +162,19 @@ static void nd_async_device_unregister(void *d, async_cookie_t cookie) put_device(dev); } -void nd_device_register(struct device *dev) +void __nd_device_register(struct device *dev) { dev->bus = &nd_bus_type; - device_initialize(dev); get_device(dev); async_schedule_domain(nd_async_device_register, dev, &nd_async_domain); } + +void nd_device_register(struct device *dev) +{ + device_initialize(dev); + __nd_device_register(dev); +} EXPORT_SYMBOL(nd_device_register); void nd_device_unregister(struct device *dev, enum nd_async_mode mode) @@ -200,6 +223,107 @@ int __nd_driver_register(struct nd_device_driver *nd_drv, struct module *owner, } EXPORT_SYMBOL(__nd_driver_register); +/** + * nd_register_ndio() - register byte-aligned access capability for an nd-bdev + * @disk: child gendisk of the ndio namepace device + * @ndio: initialized ndio instance to register + * + * LOCKING: hold nd_bus_lock() over the creation of ndio->disk and the + * subsequent nd_region_ndio event + */ +int nd_register_ndio(struct nd_io *ndio) +{ + struct nd_bus *nd_bus; + struct device *dev; + + if (!ndio || !ndio->dev || !ndio->disk || !list_empty(&ndio->list) + || !ndio->rw_bytes || !list_empty(&ndio->claims)) { + pr_debug("%s bad parameters from %pf\n", __func__, + __builtin_return_address(0)); + return -EINVAL; + } + + dev = ndio->dev; + nd_bus = walk_to_nd_bus(dev); + if (!nd_bus) + return -EINVAL; + + WARN_ON_ONCE(!is_nd_bus_locked(&nd_bus->dev)); + list_add(&ndio->list, &nd_bus->ndios); + + /* TODO: generic infrastructure for 3rd party ndio claimers */ + nd_btt_notify_ndio(nd_bus, ndio); + + return 0; +} +EXPORT_SYMBOL(nd_register_ndio); + +/** + * __nd_unregister_ndio() - try to remove an ndio interface + * @ndio: interface to remove + */ +static int __nd_unregister_ndio(struct nd_io *ndio) +{ + struct nd_io_claim *ndio_claim, *_n; + struct nd_bus *nd_bus; + LIST_HEAD(claims); + + nd_bus = walk_to_nd_bus(ndio->dev); + if (!nd_bus || list_empty(&ndio->list)) + return -ENXIO; + + spin_lock(&ndio->lock); + list_splice_init(&ndio->claims, &claims); + spin_unlock(&ndio->lock); + + list_for_each_entry_safe(ndio_claim, _n, &claims, list) + ndio_claim->notify_remove(ndio_claim); + + list_del_init(&ndio->list); + + return 0; +} + +int nd_unregister_ndio(struct nd_io *ndio) +{ + struct device *dev = ndio->dev; + int rc; + + nd_bus_lock(dev); + rc = __nd_unregister_ndio(ndio); + nd_bus_unlock(dev); + + /* + * Flush in case ->notify_remove() kicked off asynchronous device + * unregistration + */ + nd_synchronize(); + + return rc; +} +EXPORT_SYMBOL(nd_unregister_ndio); + +static struct nd_io *__ndio_lookup(struct nd_bus *nd_bus, const char *diskname) +{ + struct nd_io *ndio; + + list_for_each_entry(ndio, &nd_bus->ndios, list) + if (strcmp(diskname, ndio->disk->disk_name) == 0) + return ndio; + + return NULL; +} + +struct nd_io *ndio_lookup(struct nd_bus *nd_bus, const char *diskname) +{ + struct nd_io *ndio; + + WARN_ON_ONCE(!is_nd_bus_locked(&nd_bus->dev)); + ndio = __ndio_lookup(nd_bus, diskname); + + return ndio; +} + static ssize_t modalias_show(struct device *dev, struct device_attribute *attr, char *buf) { diff --git a/drivers/block/nd/core.c b/drivers/block/nd/core.c index 3ec38289be58..dca48f9c1580 100644 --- a/drivers/block/nd/core.c +++ b/drivers/block/nd/core.c @@ -55,6 +55,62 @@ bool is_nd_bus_locked(struct device *dev) } EXPORT_SYMBOL(is_nd_bus_locked); +void nd_init_ndio(struct nd_io *ndio, nd_rw_bytes_fn rw_bytes, + struct device *dev, struct gendisk *disk, unsigned long align) +{ + memset(ndio, 0, sizeof(*ndio)); + INIT_LIST_HEAD(&ndio->claims); + INIT_LIST_HEAD(&ndio->list); + spin_lock_init(&ndio->lock); + ndio->dev = dev; + ndio->disk = disk; + ndio->align = align; + ndio->rw_bytes = rw_bytes; +} +EXPORT_SYMBOL(nd_init_ndio); + +void ndio_del_claim(struct nd_io_claim *ndio_claim) +{ + struct nd_io *ndio; + struct device *holder; + + if (!ndio_claim) + return; + ndio = ndio_claim->parent; + holder = ndio_claim->holder; + + dev_dbg(holder, "%s: drop %s\n", __func__, dev_name(ndio->dev)); + spin_lock(&ndio->lock); + list_del(&ndio_claim->list); + spin_unlock(&ndio->lock); + put_device(ndio->dev); + kfree(ndio_claim); + put_device(holder); +} + +struct nd_io_claim *ndio_add_claim(struct nd_io *ndio, struct device *holder, + ndio_notify_remove_fn notify_remove) +{ + struct nd_io_claim *ndio_claim = kzalloc(sizeof(*ndio_claim), GFP_KERNEL); + + if (!ndio_claim) + return NULL; + + INIT_LIST_HEAD(&ndio_claim->list); + ndio_claim->parent = ndio; + get_device(ndio->dev); + + spin_lock(&ndio->lock); + list_add(&ndio_claim->list, &ndio->claims); + spin_unlock(&ndio->lock); + + ndio_claim->holder = holder; + ndio_claim->notify_remove = notify_remove; + get_device(holder); + + return ndio_claim; +} + u64 nd_fletcher64(void __iomem *addr, size_t len) { u32 lo32 = 0; @@ -74,6 +130,8 @@ static void nd_bus_release(struct device *dev) { struct nd_bus *nd_bus = container_of(dev, struct nd_bus, dev); + WARN_ON(!list_empty(&nd_bus->ndios)); + ida_simple_remove(&nd_ida, nd_bus->id); kfree(nd_bus); } @@ -270,10 +328,28 @@ static ssize_t wait_probe_show(struct device *dev, } static DEVICE_ATTR_RO(wait_probe); +static ssize_t btt_seed_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nd_bus *nd_bus = to_nd_bus(dev); + ssize_t rc; + + nd_bus_lock(dev); + if (nd_bus->nd_btt) + rc = sprintf(buf, "%s\n", dev_name(&nd_bus->nd_btt->dev)); + else + rc = sprintf(buf, "\n"); + nd_bus_unlock(dev); + + return rc; +} +static DEVICE_ATTR_RO(btt_seed); + static struct attribute *nd_bus_attributes[] = { &dev_attr_commands.attr, &dev_attr_wait_probe.attr, &dev_attr_provider.attr, + &dev_attr_btt_seed.attr, NULL, }; @@ -290,6 +366,7 @@ struct nd_bus *__nd_bus_register(struct device *parent, if (!nd_bus) return NULL; + INIT_LIST_HEAD(&nd_bus->ndios); INIT_LIST_HEAD(&nd_bus->list); init_waitqueue_head(&nd_bus->probe_wait); nd_bus->id = ida_simple_get(&nd_ida, 0, 0, GFP_KERNEL); @@ -318,6 +395,8 @@ struct nd_bus *__nd_bus_register(struct device *parent, list_add_tail(&nd_bus->list, &nd_bus_list); mutex_unlock(&nd_bus_list_mutex); + nd_bus->nd_btt = nd_btt_create(nd_bus); + return nd_bus; err: put_device(&nd_bus->dev); diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index c7b29c6f21cb..4ab4765dc9ee 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -22,14 +22,21 @@ extern struct list_head nd_bus_list; extern struct mutex nd_bus_list_mutex; extern int nd_dimm_major; +struct block_device; +struct nd_io_claim; +struct nd_btt; +struct nd_io; + struct nd_bus { struct nd_bus_descriptor *nd_desc; wait_queue_head_t probe_wait; struct module *module; + struct list_head ndios; struct list_head list; struct device dev; int id, probe_active; struct mutex reconfig_mutex; + struct nd_btt *nd_btt; }; struct nd_dimm { @@ -41,9 +48,29 @@ struct nd_dimm { int id; }; +struct nd_io *ndio_lookup(struct nd_bus *nd_bus, const char *diskname); bool is_nd_dimm(struct device *dev); bool is_nd_blk(struct device *dev); bool is_nd_pmem(struct device *dev); +#if IS_ENABLED(CONFIG_ND_BTT_DEVS) +bool is_nd_btt(struct device *dev); +struct nd_btt *nd_btt_create(struct nd_bus *nd_bus); +void nd_btt_notify_ndio(struct nd_bus *nd_bus, struct nd_io *ndio); +#else +static inline bool is_nd_btt(struct device *dev) +{ + return false; +} + +static inline struct nd_btt *nd_btt_create(struct nd_bus *nd_bus) +{ + return NULL; +} + +static inline void nd_btt_notify_ndio(struct nd_bus *nd_bus, struct nd_io *ndio) +{ +} +#endif struct nd_bus *walk_to_nd_bus(struct device *nd_dev); int __init nd_bus_init(void); void nd_bus_exit(void); @@ -62,6 +89,7 @@ void nd_synchronize(void); int nd_bus_register_dimms(struct nd_bus *nd_bus); int nd_bus_register_regions(struct nd_bus *nd_bus); int nd_bus_init_interleave_sets(struct nd_bus *nd_bus); +void __nd_device_register(struct device *dev); int nd_match_dimm(struct device *dev, void *data); struct nd_label_id; char *nd_label_gen_id(struct nd_label_id *label_id, u8 *uuid, u32 flags); diff --git a/drivers/block/nd/nd.h b/drivers/block/nd/nd.h index 8eaf0ae82db4..c6ed26d4dcad 100644 --- a/drivers/block/nd/nd.h +++ b/drivers/block/nd/nd.h @@ -12,13 +12,19 @@ */ #ifndef __ND_H__ #define __ND_H__ +#include <linux/genhd.h> #include <linux/device.h> #include <linux/mutex.h> #include <linux/ndctl.h> #include <linux/types.h> +#include <linux/fs.h> #include "libnd.h" #include "label.h" +enum { + SECTOR_SHIFT = 9, +}; + struct nd_dimm_drvdata { struct device *dev; int nsindex_size; @@ -111,6 +117,84 @@ static inline unsigned nd_inc_seq(unsigned seq) return next[seq & 3]; } +struct nd_io; +/** + * nd_rw_bytes_fn() - access bytes relative to the "whole disk" namespace device + * @ndio: per-namespace context + * @buf: source / target for the write / read + * @offset: offset relative to the start of the namespace device + * @n: num bytes to access + * @flags: READ, WRITE, and other REQ_* flags + * + * Note: Implementations may assume that offset + n never crosses ndio->align + */ +typedef int (*nd_rw_bytes_fn)(struct nd_io *ndio, void *buf, size_t offset, + size_t n, unsigned long flags); +#define nd_data_dir(flags) (flags & 1) + +/** + * struct nd_io - info for byte-aligned access to nd devices + * @rw_bytes: operation to perform byte-aligned access + * @align: a single ->rw_bytes() request may not cross this alignment + * @gendisk: whole disk block device for the namespace + * @list: for the core to cache a list of "ndio"s for later association + * @dev: namespace device + * @claims: list of clients using this interface + * @lock: protect @claims mutation + */ +struct nd_io { + nd_rw_bytes_fn rw_bytes; + unsigned long align; + struct gendisk *disk; + struct list_head list; + struct device *dev; + struct list_head claims; + spinlock_t lock; +}; + +struct nd_io_claim; +typedef void (*ndio_notify_remove_fn)(struct nd_io_claim *ndio_claim); + +/** + * struct nd_io_claim - instance of a claim on a parent ndio + * @notify_remove: ndio is going away, release resources + * @holder: object that has claimed this ndio + * @parent: ndio in use + * @holder: holder device + * @list: claim peers + * + * An ndio may be claimed multiple times, consider the case of a btt + * instance per partition on a namespace. + */ +struct nd_io_claim { + struct nd_io *parent; + ndio_notify_remove_fn notify_remove; + struct list_head list; + struct device *holder; +}; + +struct nd_btt { + struct device dev; + struct nd_io *ndio; + struct block_device *backing_dev; + unsigned long lbasize; + u8 *uuid; + u64 offset; + int id; + struct nd_io_claim *ndio_claim; +}; + +static inline u64 nd_partition_offset(struct block_device *bdev) +{ + struct hd_struct *p; + + if (bdev == bdev->bd_contains) + return 0; + + p = bdev->bd_part; + return ((u64) p->start_sect) << SECTOR_SHIFT; +} + enum nd_async_mode { ND_SYNC, ND_ASYNC, @@ -125,12 +209,22 @@ ssize_t nd_sector_size_show(unsigned long current_lbasize, const unsigned long *supported, char *buf); ssize_t nd_sector_size_store(struct device *dev, const char *buf, unsigned long *current_lbasize, const unsigned long *supported); +int nd_register_ndio(struct nd_io *ndio); +int nd_unregister_ndio(struct nd_io *ndio); +void nd_init_ndio(struct nd_io *ndio, nd_rw_bytes_fn rw_bytes, + struct device *dev, struct gendisk *disk, unsigned long align); +void ndio_del_claim(struct nd_io_claim *ndio_claim); +struct nd_io_claim *ndio_add_claim(struct nd_io *ndio, struct device *holder, + ndio_notify_remove_fn notify_remove); struct nd_dimm; struct nd_dimm_drvdata *to_ndd(struct nd_mapping *nd_mapping); int nd_dimm_init_nsarea(struct nd_dimm_drvdata *ndd); int nd_dimm_init_config_data(struct nd_dimm_drvdata *ndd); int nd_dimm_set_config_data(struct nd_dimm_drvdata *ndd, size_t offset, void *buf, size_t len); +struct nd_btt *to_nd_btt(struct device *dev); +struct btt_sb; +u64 nd_btt_sb_checksum(struct btt_sb *btt_sb); struct nd_region *to_nd_region(struct device *dev); int nd_region_to_namespace_type(struct nd_region *nd_region); int nd_region_register_namespaces(struct nd_region *nd_region, int *err); diff --git a/drivers/block/nd/pmem.c b/drivers/block/nd/pmem.c index 32e7e74ee25a..900dad61a6b9 100644 --- a/drivers/block/nd/pmem.c +++ b/drivers/block/nd/pmem.c @@ -31,6 +31,7 @@ struct pmem_device { struct request_queue *pmem_queue; struct gendisk *pmem_disk; + struct nd_io ndio; /* One contiguous memory region per device */ phys_addr_t phys_addr; @@ -100,6 +101,26 @@ static int pmem_rw_page(struct block_device *bdev, sector_t sector, return 0; } +static int pmem_rw_bytes(struct nd_io *ndio, void *buf, size_t offset, + size_t n, unsigned long flags) +{ + struct pmem_device *pmem = container_of(ndio, typeof(*pmem), ndio); + int rw = nd_data_dir(flags); + + if (unlikely(offset + n > pmem->size)) { + dev_WARN_ONCE(ndio->dev, 1, "%s: request out of range\n", + __func__); + return -EFAULT; + } + + if (rw == READ) + memcpy(buf, pmem->virt_addr + offset, n); + else + memcpy(pmem->virt_addr + offset, buf, n); + + return 0; +} + static long pmem_direct_access(struct block_device *bdev, sector_t sector, void **kaddr, unsigned long *pfn, long size) { @@ -179,8 +200,6 @@ static struct pmem_device *pmem_alloc(struct device *dev, struct resource *res) set_capacity(disk, pmem->size >> 9); pmem->pmem_disk = disk; - add_disk(disk); - return pmem; out_free_queue: @@ -235,7 +254,12 @@ static int nd_pmem_probe(struct device *dev) if (IS_ERR(pmem)) return PTR_ERR(pmem); + nd_bus_lock(dev); + add_disk(pmem->pmem_disk); dev_set_drvdata(dev, pmem); + nd_init_ndio(&pmem->ndio, pmem_rw_bytes, dev, pmem->pmem_disk, 0); + nd_register_ndio(&pmem->ndio); + nd_bus_unlock(dev); return 0; } @@ -244,6 +268,7 @@ static int nd_pmem_remove(struct device *dev) { struct pmem_device *pmem = dev_get_drvdata(dev); + nd_unregister_ndio(&pmem->ndio); pmem_free(pmem); return 0; } diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h index 0b4dcabb248a..e595751c613d 100644 --- a/include/uapi/linux/ndctl.h +++ b/include/uapi/linux/ndctl.h @@ -181,6 +181,7 @@ static inline const char *nd_dimm_cmd_name(unsigned cmd) #define ND_DEVICE_NAMESPACE_IO 4 /* legacy persistent memory */ #define ND_DEVICE_NAMESPACE_PMEM 5 /* persistent memory namespace (may alias) */ #define ND_DEVICE_NAMESPACE_BLK 6 /* block-data-window namespace (may alias) */ +#define ND_DEVICE_BTT 7 /* block-translation table device */ enum nd_driver_flags { ND_DRIVER_DIMM = 1 << ND_DEVICE_DIMM, @@ -189,6 +190,7 @@ enum nd_driver_flags { ND_DRIVER_NAMESPACE_IO = 1 << ND_DEVICE_NAMESPACE_IO, ND_DRIVER_NAMESPACE_PMEM = 1 << ND_DEVICE_NAMESPACE_PMEM, ND_DRIVER_NAMESPACE_BLK = 1 << ND_DEVICE_NAMESPACE_BLK, + ND_DRIVER_BTT = 1 << ND_DEVICE_BTT, }; enum {

From: Vishal Verma <vishal.l.verma(a)linux.intel.com&gt; BTT stands for Block Translation Table, and is a way to provide power fail sector atomicity semantics for block devices that have the ability to perform byte granularity IO. It relies on the ->rw_bytes() capability of provided nd namespace devices. The BTT works as a stacked blocked device, and reserves a chunk of space from the backing device for its accounting metadata. BLK namespaces may mandate use of a BTT and expect the bus to initialize a BTT if not already present. Otherwise if a BTT is desired for other namespaces (or partitions of a namespace) a BTT may be manually configured. Cc: Andy Lutomirski <luto(a)amacapital.net&gt; Cc: Boaz Harrosh <boaz(a)plexistor.com&gt; Cc: H. Peter Anvin <hpa(a)zytor.com&gt; Cc: Jens Axboe <axboe(a)fb.com&gt; Cc: Ingo Molnar <mingo(a)kernel.org&gt; Cc: Christoph Hellwig <hch(a)lst.de&gt; Cc: Neil Brown <neilb(a)suse.de&gt; Cc: Jeff Moyer <jmoyer(a)redhat.com&gt; Cc: Dave Chinner <david(a)fromorbit.com&gt; Cc: Greg KH <gregkh(a)linuxfoundation.org&gt; [jmoyer: fix nmi watchdog timeout in btt_map_init] [jmoyer: move btt initialization to module load path] [jmoyer: fix memory leak in the btt initialization path] [jmoyer: Don't overwrite corrupted arenas] Signed-off-by: Vishal Verma <vishal.l.verma(a)linux.intel.com&gt; Signed-off-by: Dan Williams <dan.j.williams(a)intel.com&gt; --- Documentation/blockdev/btt.txt | 273 ++++++++ drivers/block/nd/Kconfig | 20 + drivers/block/nd/Makefile | 3 drivers/block/nd/acpi.c | 1 drivers/block/nd/btt.c | 1423 ++++++++++++++++++++++++++++++++++++++++ drivers/block/nd/btt.h | 140 ++++ drivers/block/nd/btt_devs.c | 3 drivers/block/nd/libnd.h | 1 drivers/block/nd/nd-private.h | 1 drivers/block/nd/nd.h | 10 drivers/block/nd/region.c | 67 ++ drivers/block/nd/region_devs.c | 10 12 files changed, 1948 insertions(+), 4 deletions(-) create mode 100644 Documentation/blockdev/btt.txt create mode 100644 drivers/block/nd/btt.c diff --git a/Documentation/blockdev/btt.txt b/Documentation/blockdev/btt.txt new file mode 100644 index 000000000000..95134d5ec4a0 --- /dev/null +++ b/Documentation/blockdev/btt.txt @@ -0,0 +1,273 @@ +BTT - Block Translation Table +============================= + + +1. Introduction +--------------- + +Persistent memory based storage is able to perform IO at byte (or more +accurately, cache line) granularity. However, we often want to expose such +storage as traditional block devices. The block drivers for persistent memory +will do exactly this. However, they do not provide any atomicity guarantees. +Traditional SSDs typically provide protection against torn sectors in hardware, +using stored energy in capacitors to complete in-flight block writes, or perhaps +in firmware. We don't have this luxury with persistent memory - if a write is in +progress, and we experience a power failure, the block will contain a mix of old +and new data. Applications may not be prepared to handle such a scenario. + +The Block Translation Table (BTT) provides atomic sector update semantics for +persistent memory devices, so that applications that rely on sector writes not +being torn can continue to do so. The BTT manifests itself as a stacked block +device, and reserves a portion of the underlying storage for its metadata. At +the heart of it, is an indirection table that re-maps all the blocks on the +volume. It can be thought of as an extremely simple file system that only +provides atomic sector updates. + + +2. Static Layout +---------------- + +The underlying storage on which a BTT can be laid out is not limited in any way. +The BTT, however, splits the available space into chunks of up to 512 GiB, +called "Arenas". + +Each arena follows the same layout for its metadata, and all references in an +arena are internal to it (with the exception of one field that points to the +next arena). The following depicts the "On-disk" metadata layout: + + + Backing Store +-------> Arena ++---------------+ | +------------------+ +| | | | Arena info block | +| Arena 0 +---+ | 4K | +| 512G | +------------------+ +| | | | ++---------------+ | | +| | | | +| Arena 1 | | Data Blocks | +| 512G | | | +| | | | ++---------------+ | | +| . | | | +| . | | | +| . | | | +| | | | +| | | | ++---------------+ +------------------+ + | | + | BTT Map | + | | + | | + +------------------+ + | | + | BTT Flog | + | | + +------------------+ + | Info block copy | + | 4K | + +------------------+ + + +3. Theory of Operation +---------------------- + + +a. The BTT Map +-------------- + +The map is a simple lookup/indirection table that maps an LBA to an internal +block. Each map entry is 32 bits. The two most significant bits are special +flags, and the remaining form the internal block number. + +Bit Description +31 : TRIM flag - marks if the block was trimmed or discarded +30 : ERROR flag - marks an error block. Cleared on write. +29 - 0 : Mappings to internal 'postmap' blocks + + +Some of the terminology that will be subsequently used: + +External LBA : LBA as made visible to upper layers. +ABA : Arena Block Address - Block offset/number within an arena +Premap ABA : The block offset into an arena, which was decided upon by range + checking the External LBA +Postmap ABA : The block number in the "Data Blocks" area obtained after + indirection from the map +nfree : The number of free blocks that are maintained at any given time. + This is the number of concurrent writes that can happen to the + arena. + + +For example, after adding a BTT, we surface a disk of 1024G. We get a read for +the external LBA at 768G. This falls into the second arena, and of the 512G +worth of blocks that this arena contributes, this block is at 256G. Thus, the +premap ABA is 256G. We now refer to the map, and find out the mapping for block +'X' (256G) points to block 'Y', say '64'. Thus the postmap ABA is 64. + + +b. The BTT Flog +--------------- + +The BTT provides sector atomicity by making every write an "allocating write", +i.e. Every write goes to a "free" block. A running list of free blocks is +maintained in the form of the BTT flog. 'Flog' is a combination of the words +"free list" and "log". The flog contains 'nfree' entries, and an entry contains: + +lba : The premap ABA that is being written to +old_map : The old postmap ABA - after 'this' write completes, this will be a + free block. +new_map : The new postmap ABA. The map will up updated to reflect this + lba->postmap_aba mapping, but we log it here in case we have to + recover. +seq : Sequence number to mark which of the 2 sections of this flog entry is + valid/newest. It cycles between 01->10->11->01 (binary) under normal + operation, with 00 indicating an uninitialized state. +lba' : alternate lba entry +old_map': alternate old postmap entry +new_map': alternate new postmap entry +seq' : alternate sequence number. + +Each of the above fields is 32-bit, making one entry 16 bytes. Flog updates are +done such that for any entry being written, it: +a. overwrites the 'old' section in the entry based on sequence numbers +b. writes the new entry such that the sequence number is written last. + + +c. The concept of lanes +----------------------- + +While 'nfree' describes the number of concurrent IOs an arena can process +concurrently, 'nlanes' is the number of IOs the BTT device as a whole can +process. + nlanes = min(nfree, num_cpus) +A lane number is obtained at the start of any IO, and is used for indexing into +all the on-disk and in-memory data structures for the duration of the IO. It is +protected by a spinlock. + + +d. In-memory data structure: Read Tracking Table (RTT) +------------------------------------------------------ + +Consider a case where we have two threads, one doing reads and the other, +writes. We can hit a condition where the writer thread grabs a free block to do +a new IO, but the (slow) reader thread is still reading from it. In other words, +the reader consulted a map entry, and started reading the corresponding block. A +writer started writing to the same external LBA, and finished the write updating +the map for that external LBA to point to its new postmap ABA. At this point the +internal, postmap block that the reader is (still) reading has been inserted +into the list of free blocks. If another write comes in for the same LBA, it can +grab this free block, and start writing to it, causing the reader to read +incorrect data. To prevent this, we introduce the RTT. + +The RTT is a simple, per arena table with 'nfree' entries. Every reader inserts +into rtt[lane_number], the postmap ABA it is reading, and clears it after the +read is complete. Every writer thread, after grabbing a free block, checks the +RTT for its presence. If the postmap free block is in the RTT, it waits till the +reader clears the RTT entry, and only then starts writing to it. + + +e. In-memory data structure: map locks +-------------------------------------- + +Consider a case where two writer threads are writing to the same LBA. There can +be a race in the following sequence of steps: + +free[lane] = map[premap_aba] +map[premap_aba] = postmap_aba + +Both threads can update their respective free[lane] with the same old, freed +postmap_aba. This has made the layout inconsistent by losing a free entry, and +at the same time, duplicating another free entry for two lanes. + +To solve this, we could have a single map lock (per arena) that has to be taken +before performing the above sequence, but we feel that could be too contentious. +Instead we use an array of (nfree) map_locks that is indexed by +(premap_aba modulo nfree). + + +f. Reconstruction from the Flog +------------------------------- + +On startup, we analyze the BTT flog to create our list of free blocks. We walk +through all the entries, and for each lane, of the set of two possible +'sections', we always look at the most recent one only (based on the sequence +number). The reconstruction rules/steps are simple: +- Read map[log_entry.lba]. +- If log_entry.new matches the map entry, then log_entry.old is free. +- If log_entry.new does not match the map entry, then log_entry.new is free. + (This case can only be caused by power-fails/unsafe shutdowns) + + +g. Summarizing - Read and Write flows +------------------------------------- + +Read: + +1. Convert external LBA to arena number + pre-map ABA +2. Get a lane (and take lane_lock) +3. Read map to get the entry for this pre-map ABA +4. Enter post-map ABA into RTT[lane] +5. If TRIM flag set in map, return zeroes, and end IO (go to step 8) +6. If ERROR flag set in map, end IO with EIO (go to step 8) +7. Read data from this block +8. Remove post-map ABA entry from RTT[lane] +9. Release lane (and lane_lock) + +Write: + +1. Convert external LBA to Arena number + pre-map ABA +2. Get a lane (and take lane_lock) +3. Use lane to index into in-memory free list and obtain a new block, next flog + index, next sequence number +4. Scan the RTT to check if free block is present, and spin/wait if it is. +5. Write data to this free block +6. Read map to get the existing post-map ABA entry for this pre-map ABA +7. Write flog entry: [premap_aba / old postmap_aba / new postmap_aba / seq_num] +8. Write new post-map ABA into map. +9. Write old post-map entry into the free list +10. Calculate next sequence number and write into the free list entry +11. Release lane (and lane_lock) + + +4. Error Handling +================= + +An arena would be in an error state if any of the metadata is corrupted +irrecoverably, either due to a bug or a media error. The following conditions +indicate an error: +- Info block checksum does not match (and recovering from the copy also fails) +- All internal available blocks are not uniquely and entirely addressed by the + sum of mapped blocks and free blocks (from the BTT flog). +- Rebuilding free list from the flog reveals missing/duplicate/impossible + entries +- A map entry is out of bounds + +If any of these error conditions are encountered, the arena is put into a read +only state using a flag in the info block. + + +5. In-kernel usage +================== + +Any block driver that supports byte granularity IO to the storage may register +with the BTT. It will have to provide the rw_bytes interface in its +block_device_operations struct: + + int (*rw_bytes)(struct gendisk *, void *, size_t, off_t, int rw); + +It may register with the BTT after it adds its own gendisk, using btt_init: + + struct btt *btt_init(struct gendisk *disk, unsigned long long rawsize, + u32 lbasize, u8 uuid[], int maxlane); + +note that maxlane is the maximum amount of concurrency the driver wishes to +allow the BTT to use. + +The BTT 'disk' appears as a stacked block device that grabs the underlying block +device in the O_EXCL mode. + +When the driver wishes to remove the backing disk, it should similarly call +btt_fini using the same struct btt* handle that was provided to it by btt_init. + + void btt_fini(struct btt *btt); + diff --git a/drivers/block/nd/Kconfig b/drivers/block/nd/Kconfig index 15896db4de37..612bf2b14283 100644 --- a/drivers/block/nd/Kconfig +++ b/drivers/block/nd/Kconfig @@ -93,9 +93,25 @@ config BLK_DEV_PMEM capable of DAX (direct-access) file system mappings. See Documentation/blockdev/nd.txt for more details. - Say Y if you want to use a NVDIMM described by NFIT + Say Y if you want to use a NVDIMM described by ACPI, E820, etc... config ND_BTT_DEVS - def_bool y + bool + +config ND_BTT + tristate "BTT: Block Translation Table (atomic sector updates)" + depends on LIBND + default LIBND + select ND_BTT_DEVS + +config ND_MAX_REGIONS + int "Maximum number of regions supported by the sub-system" + default 64 + ---help--- + A 'region' corresponds to an individual DIMM or an interleave + set of DIMMs. A typical maximally configured system may have + up to 32 DIMMs. + + Leave the default of 64 if you are unsure. endif diff --git a/drivers/block/nd/Makefile b/drivers/block/nd/Makefile index 0c6d64b7a69d..7d778b4523d4 100644 --- a/drivers/block/nd/Makefile +++ b/drivers/block/nd/Makefile @@ -17,6 +17,7 @@ obj-$(CONFIG_ND_ACPI) += nd_acpi.o obj-$(CONFIG_ND_E820) += nd_e820.o obj-$(CONFIG_NFIT_TEST) += test/ obj-$(CONFIG_BLK_DEV_PMEM) += nd_pmem.o +obj-$(CONFIG_ND_BTT) += nd_btt.o nd_acpi-y := acpi.o @@ -24,6 +25,8 @@ nd_e820-y := e820.o nd_pmem-y := pmem.o +nd_btt-y := btt.o + libnd-y := core.o libnd-y += bus.o libnd-y += dimm_devs.o diff --git a/drivers/block/nd/acpi.c b/drivers/block/nd/acpi.c index d34cefe38e2f..5b9997fbc344 100644 --- a/drivers/block/nd/acpi.c +++ b/drivers/block/nd/acpi.c @@ -926,6 +926,7 @@ static int nd_acpi_register_region(struct acpi_nfit_desc *acpi_desc, } else { nd_mapping->size = nfit_mem->bdw->blk_capacity; nd_mapping->start = nfit_mem->bdw->blk_offset; + ndr_desc.num_lanes = nfit_mem->bdw->num_bdw; } ndr_desc.nd_mapping = nd_mapping; diff --git a/drivers/block/nd/btt.c b/drivers/block/nd/btt.c new file mode 100644 index 000000000000..abcefb7aeed1 --- /dev/null +++ b/drivers/block/nd/btt.c @@ -0,0 +1,1423 @@ +/* + * Block Translation Table + * Copyright (c) 2014-2015, Intel Corporation. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ +#include <linux/highmem.h> +#include <linux/debugfs.h> +#include <linux/blkdev.h> +#include <linux/module.h> +#include <linux/device.h> +#include <linux/mutex.h> +#include <linux/hdreg.h> +#include <linux/genhd.h> +#include <linux/sizes.h> +#include <linux/ndctl.h> +#include <linux/fs.h> +#include <linux/nd.h> +#include "btt.h" +#include "nd.h" + +enum log_ent_request { + LOG_NEW_ENT = 0, + LOG_OLD_ENT +}; + +static int btt_major; + +static int nd_btt_rw_bytes(struct nd_btt *nd_btt, void *buf, size_t offset, + size_t n, unsigned long flags) +{ + struct nd_io *ndio = nd_btt->ndio; + + if (unlikely(nd_data_dir(flags) == WRITE) + && bdev_read_only(nd_btt->backing_dev)) + return -EACCES; + + return ndio->rw_bytes(ndio, buf, offset + nd_btt->offset, n, flags); +} + +static int arena_rw_bytes(struct arena_info *arena, void *buf, size_t n, + size_t offset, unsigned long flags) +{ + /* yes, FIXME, 'offset' and 'n' are swapped */ + return nd_btt_rw_bytes(arena->nd_btt, buf, offset, n, flags); +} + +static int btt_info_write(struct arena_info *arena, struct btt_sb *super) +{ + int ret; + + ret = arena_rw_bytes(arena, super, sizeof(struct btt_sb), + arena->info2off, WRITE); + if (ret) + return ret; + + return arena_rw_bytes(arena, super, sizeof(struct btt_sb), + arena->infooff, WRITE); +} + +static int btt_info_read(struct arena_info *arena, struct btt_sb *super) +{ + WARN_ON(!super); + return arena_rw_bytes(arena, super, sizeof(struct btt_sb), + arena->infooff, READ); + +} + +/* + * 'raw' version of btt_map write + * Assumptions: + * mapping is in little-endian + * mapping contains 'E' and 'Z' flags as desired + */ +static int __btt_map_write(struct arena_info *arena, u32 lba, __le32 mapping) +{ + u64 ns_off = arena->mapoff + (lba * MAP_ENT_SIZE); + + WARN_ON(lba >= arena->external_nlba); + return arena_rw_bytes(arena, &mapping, MAP_ENT_SIZE, ns_off, WRITE); +} + +static int btt_map_write(struct arena_info *arena, u32 lba, u32 mapping, + u32 z_flag, u32 e_flag) +{ + u32 ze; + __le32 mapping_le; + + /* + * This 'mapping' is supposed to be just the LBA mapping, without + * any flags set, so strip the flag bits. + */ + mapping &= MAP_LBA_MASK; + + ze = (z_flag << 1) + e_flag; + switch (ze) { + case 0: + /* + * We want to set neither of the Z or E flags, and + * in the actual layout, this means setting the bit + * positions of both to '1' to indicate a 'normal' + * map entry + */ + mapping |= MAP_ENT_NORMAL; + break; + case 1: + mapping |= (1 << MAP_ERR_SHIFT); + break; + case 2: + mapping |= (1 << MAP_TRIM_SHIFT); + break; + default: + /* + * The case where Z and E are both sent in as '1' could be + * construed as a valid 'normal' case, but we decide not to, + * to avoid confusion + */ + WARN_ONCE(1, "Invalid use of Z and E flags\n"); + return -EIO; + } + + mapping_le = cpu_to_le32(mapping); + return __btt_map_write(arena, lba, mapping_le); +} + +static int btt_map_read(struct arena_info *arena, u32 lba, u32 *mapping, + int *trim, int *error) +{ + int ret; + __le32 in; + u32 raw_mapping, postmap, ze, z_flag, e_flag; + u64 ns_off = arena->mapoff + (lba * MAP_ENT_SIZE); + + WARN_ON(lba >= arena->external_nlba); + + ret = arena_rw_bytes(arena, &in, MAP_ENT_SIZE, ns_off, READ); + if (ret) + return ret; + + raw_mapping = le32_to_cpu(in); + + z_flag = (raw_mapping & MAP_TRIM_MASK) >> MAP_TRIM_SHIFT; + e_flag = (raw_mapping & MAP_ERR_MASK) >> MAP_ERR_SHIFT; + ze = (z_flag << 1) + e_flag; + postmap = raw_mapping & MAP_LBA_MASK; + + /* Reuse the {z,e}_flag variables for *trim and *error */ + z_flag = 0; + e_flag = 0; + + switch (ze) { + case 0: + /* Initial state. Return postmap = premap */ + *mapping = lba; + break; + case 1: + *mapping = postmap; + e_flag = 1; + break; + case 2: + *mapping = postmap; + z_flag = 1; + break; + case 3: + *mapping = postmap; + break; + default: + return -EIO; + } + + if (trim) + *trim = z_flag; + if (error) + *error = e_flag; + + return ret; +} + +static int btt_log_read_pair(struct arena_info *arena, u32 lane, + struct log_entry *ent) +{ + WARN_ON(!ent); + return arena_rw_bytes(arena, ent, 2 * LOG_ENT_SIZE, + arena->logoff + (2 * lane * LOG_ENT_SIZE), READ); +} + +static struct dentry *debugfs_root; + +static void arena_debugfs_init(struct arena_info *a, struct dentry *parent, + int idx) +{ + char dirname[32]; + struct dentry *d; + + /* If for some reason, parent bttN was not created, exit */ + if (!parent) + return; + + snprintf(dirname, 32, "arena%d", idx); + d = debugfs_create_dir(dirname, parent); + if (IS_ERR_OR_NULL(d)) + return; + a->debugfs_dir = d; + + debugfs_create_x64("size", S_IRUGO, d, &a->size); + debugfs_create_x64("external_lba_start", S_IRUGO, d, + &a->external_lba_start); + debugfs_create_x32("internal_nlba", S_IRUGO, d, &a->internal_nlba); + debugfs_create_u32("internal_lbasize", S_IRUGO, d, + &a->internal_lbasize); + debugfs_create_x32("external_nlba", S_IRUGO, d, &a->external_nlba); + debugfs_create_u32("external_lbasize", S_IRUGO, d, + &a->external_lbasize); + debugfs_create_u32("nfree", S_IRUGO, d, &a->nfree); + debugfs_create_u16("version_major", S_IRUGO, d, &a->version_major); + debugfs_create_u16("version_minor", S_IRUGO, d, &a->version_minor); + debugfs_create_x64("nextoff", S_IRUGO, d, &a->nextoff); + debugfs_create_x64("infooff", S_IRUGO, d, &a->infooff); + debugfs_create_x64("dataoff", S_IRUGO, d, &a->dataoff); + debugfs_create_x64("mapoff", S_IRUGO, d, &a->mapoff); + debugfs_create_x64("logoff", S_IRUGO, d, &a->logoff); + debugfs_create_x64("info2off", S_IRUGO, d, &a->info2off); + debugfs_create_x32("flags", S_IRUGO, d, &a->flags); +} + +static void btt_debugfs_init(struct btt *btt) +{ + int i = 0; + struct arena_info *arena; + + btt->debugfs_dir = debugfs_create_dir(dev_name(&btt->nd_btt->dev), + debugfs_root); + if (IS_ERR_OR_NULL(btt->debugfs_dir)) + return; + + list_for_each_entry(arena, &btt->arena_list, list) { + arena_debugfs_init(arena, btt->debugfs_dir, i); + i++; + } +} + +/* + * This function accepts two log entries, and uses the + * sequence number to find the 'older' entry. + * It also updates the sequence number in this old entry to + * make it the 'new' one if the mark_flag is set. + * Finally, it returns which of the entries was the older one. + * + * TODO The logic feels a bit kludge-y. make it better.. + */ +static int btt_log_get_old(struct log_entry *ent) +{ + int old; + + /* + * the first ever time this is seen, the entry goes into [0] + * the next time, the following logic works out to put this + * (next) entry into [1] + */ + if (ent[0].seq == 0) { + ent[0].seq = cpu_to_le32(1); + return 0; + } + + if (ent[0].seq == ent[1].seq) + return -EINVAL; + if (le32_to_cpu(ent[0].seq) + le32_to_cpu(ent[1].seq) > 5) + return -EINVAL; + + if (le32_to_cpu(ent[0].seq) < le32_to_cpu(ent[1].seq)) { + if (le32_to_cpu(ent[1].seq) - le32_to_cpu(ent[0].seq) == 1) + old = 0; + else + old = 1; + } else { + if (le32_to_cpu(ent[0].seq) - le32_to_cpu(ent[1].seq) == 1) + old = 1; + else + old = 0; + } + + return old; +} + +static struct device *to_dev(struct arena_info *arena) +{ + return &arena->nd_btt->dev; +} + +/* + * This function copies the desired (old/new) log entry into ent if + * it is not NULL. It returns the sub-slot number (0 or 1) + * where the desired log entry was found. Negative return values + * indicate errors. + */ +static int btt_log_read(struct arena_info *arena, u32 lane, + struct log_entry *ent, int old_flag) +{ + int ret; + int old_ent, ret_ent; + struct log_entry log[2]; + + ret = btt_log_read_pair(arena, lane, log); + if (ret) + return -EIO; + + old_ent = btt_log_get_old(log); + if (old_ent < 0 || old_ent > 1) { + dev_info(to_dev(arena), + "log corruption (%d): lane %d seq [%d, %d]\n", + old_ent, lane, log[0].seq, log[1].seq); + /* TODO set error state? */ + return -EIO; + } + + ret_ent = (old_flag ? old_ent : (1 - old_ent)); + + if (ent != NULL) + memcpy(ent, &log[ret_ent], LOG_ENT_SIZE); + + return ret_ent; +} + +/* + * This function commits a log entry to media + * It does _not_ prepare the freelist entry for the next write + * btt_flog_write is the wrapper for updating the freelist elements + */ +static int __btt_log_write(struct arena_info *arena, u32 lane, + u32 sub, struct log_entry *ent) +{ + int ret; + /* + * Ignore the padding in log_entry for calculating log_half. + * The entry is 'committed' when we write the sequence number, + * and we want to ensure that that is the last thing written. + * We don't bother writing the padding as that would be extra + * media wear and write amplification + */ + unsigned int log_half = (LOG_ENT_SIZE - 2 * sizeof(u64)) / 2; + u64 ns_off = arena->logoff + (((2 * lane) + sub) * LOG_ENT_SIZE); + void *src = ent; + + /* split the 16B write into atomic, durable halves */ + ret = arena_rw_bytes(arena, src, log_half, ns_off, WRITE); + if (ret) + return ret; + + ns_off += log_half; + src += log_half; + return arena_rw_bytes(arena, src, log_half, ns_off, WRITE); +} + +static int btt_flog_write(struct arena_info *arena, u32 lane, u32 sub, + struct log_entry *ent) +{ + int ret; + + ret = __btt_log_write(arena, lane, sub, ent); + if (ret) + return ret; + + /* prepare the next free entry */ + arena->freelist[lane].sub = 1 - arena->freelist[lane].sub; + if (++(arena->freelist[lane].seq) == 4) + arena->freelist[lane].seq = 1; + arena->freelist[lane].block = le32_to_cpu(ent->old_map); + + return ret; +} + +/* + * This function initializes the BTT map to a state with all externally + * exposed blocks having an identity mapping, and the TRIM flag set + */ +static int btt_map_init(struct arena_info *arena) +{ + int ret = -EINVAL; + void *zerobuf; + size_t offset = 0; + size_t chunk_size = SZ_2M; + size_t mapsize = arena->logoff - arena->mapoff; + + zerobuf = kzalloc(chunk_size, GFP_KERNEL); + if (!zerobuf) + return -ENOMEM; + + while (mapsize) { + size_t size = min(mapsize, chunk_size); + + ret = arena_rw_bytes(arena, zerobuf, size, + arena->mapoff + offset, WRITE); + if (ret) + goto free; + + offset += size; + mapsize -= size; + cond_resched(); + } + + free: + kfree(zerobuf); + return ret; +} + +/* + * This function initializes the BTT log with 'fake' entries pointing + * to the initial reserved set of blocks as being free + */ +static int btt_log_init(struct arena_info *arena) +{ + int ret; + u32 i; + struct log_entry log, zerolog; + + memset(&zerolog, 0, sizeof(zerolog)); + + for (i = 0; i < arena->nfree; i++) { + log.lba = cpu_to_le32(i); + log.old_map = cpu_to_le32(arena->external_nlba + i); + log.new_map = cpu_to_le32(arena->external_nlba + i); + log.seq = cpu_to_le32(LOG_SEQ_INIT); + ret = __btt_log_write(arena, i, 0, &log); + if (ret) + return ret; + ret = __btt_log_write(arena, i, 1, &zerolog); + if (ret) + return ret; + } + + return 0; +} + +static int btt_freelist_init(struct arena_info *arena) +{ + int old, new, ret; + u32 i, map_entry; + struct log_entry log_new, log_old; + + arena->freelist = kcalloc(arena->nfree, sizeof(struct free_entry), + GFP_KERNEL); + if (!arena->freelist) + return -ENOMEM; + + for (i = 0; i < arena->nfree; i++) { + old = btt_log_read(arena, i, &log_old, LOG_OLD_ENT); + if (old < 0) + return old; + + new = btt_log_read(arena, i, &log_new, LOG_NEW_ENT); + if (new < 0) + return new; + + /* sub points to the next one to be overwritten */ + arena->freelist[i].sub = 1 - new; + arena->freelist[i].seq = nd_inc_seq(le32_to_cpu(log_new.seq)); + arena->freelist[i].block = le32_to_cpu(log_new.old_map); + + /* This implies a newly created or untouched flog entry */ + if (log_new.old_map == log_new.new_map) + continue; + + /* Check if map recovery is needed */ + ret = btt_map_read(arena, le32_to_cpu(log_new.lba), &map_entry, + NULL, NULL); + if (ret) + return ret; + if ((le32_to_cpu(log_new.new_map) != map_entry) && + (le32_to_cpu(log_new.old_map) == map_entry)) { + /* + * Last transaction wrote the flog, but wasn't able + * to complete the map write. So fix up the map. + */ + ret = btt_map_write(arena, le32_to_cpu(log_new.lba), + le32_to_cpu(log_new.new_map), 0, 0); + if (ret) + return ret; + } + + } + + return 0; +} + +static int btt_rtt_init(struct arena_info *arena) +{ + arena->rtt = kcalloc(arena->nfree, sizeof(u32), GFP_KERNEL); + if (arena->rtt == NULL) + return -ENOMEM; + + return 0; +} + +static int btt_maplocks_init(struct arena_info *arena) +{ + u32 i; + + arena->map_locks = kcalloc(arena->nfree, sizeof(struct aligned_lock), + GFP_KERNEL); + if (!arena->map_locks) + return -ENOMEM; + + for (i = 0; i < arena->nfree; i++) + spin_lock_init(&arena->map_locks[i].lock); + + return 0; +} + +static struct arena_info *alloc_arena(struct btt *btt, size_t size, + size_t start, size_t arena_off) +{ + struct arena_info *arena; + u64 logsize, mapsize, datasize; + u64 available = size; + + arena = kzalloc(sizeof(struct arena_info), GFP_KERNEL); + if (!arena) + return NULL; + arena->nd_btt = btt->nd_btt; + + if (!size) + return arena; + + arena->size = size; + arena->external_lba_start = start; + arena->external_lbasize = btt->lbasize; + arena->internal_lbasize = roundup(arena->external_lbasize, + INT_LBASIZE_ALIGNMENT); + arena->nfree = BTT_DEFAULT_NFREE; + arena->version_major = 1; + arena->version_minor = 1; + + if (available % BTT_PG_SIZE) + available -= (available % BTT_PG_SIZE); + + /* Two pages are reserved for the super block and its copy */ + available -= 2 * BTT_PG_SIZE; + + /* The log takes a fixed amount of space based on nfree */ + logsize = roundup(2 * arena->nfree * sizeof(struct log_entry), + BTT_PG_SIZE); + available -= logsize; + + /* Calculate optimal split between map and data area */ + arena->internal_nlba = div_u64(available - BTT_PG_SIZE, + arena->internal_lbasize + MAP_ENT_SIZE); + arena->external_nlba = arena->internal_nlba - arena->nfree; + + mapsize = roundup((arena->external_nlba * MAP_ENT_SIZE), BTT_PG_SIZE); + datasize = available - mapsize; + + /* 'Absolute' values, relative to start of storage space */ + arena->infooff = arena_off; + arena->dataoff = arena->infooff + BTT_PG_SIZE; + arena->mapoff = arena->dataoff + datasize; + arena->logoff = arena->mapoff + mapsize; + arena->info2off = arena->logoff + logsize; + return arena; +} + +static void free_arenas(struct btt *btt) +{ + struct arena_info *arena, *next; + + list_for_each_entry_safe(arena, next, &btt->arena_list, list) { + list_del(&arena->list); + kfree(arena->rtt); + kfree(arena->map_locks); + kfree(arena->freelist); + debugfs_remove_recursive(arena->debugfs_dir); + kfree(arena); + } +} + +/* + * This function checks if the metadata layout is valid and error free + */ +static int arena_is_valid(struct arena_info *arena, struct btt_sb *super, + u8 *uuid) +{ + u64 checksum; + + if (memcmp(super->uuid, uuid, 16)) + return 0; + + checksum = le64_to_cpu(super->checksum); + super->checksum = 0; + if (checksum != nd_btt_sb_checksum(super)) + return 0; + super->checksum = cpu_to_le64(checksum); + + /* TODO: figure out action for this */ + if ((le32_to_cpu(super->flags) & IB_FLAG_ERROR_MASK) != 0) + dev_info(to_dev(arena), "Found arena with an error flag\n"); + + return 1; +} + +/* + * This function reads an existing valid btt superblock and + * populates the corresponding arena_info struct + */ +static void parse_arena_meta(struct arena_info *arena, struct btt_sb *super, + u64 arena_off) +{ + arena->internal_nlba = le32_to_cpu(super->internal_nlba); + arena->internal_lbasize = le32_to_cpu(super->internal_lbasize); + arena->external_nlba = le32_to_cpu(super->external_nlba); + arena->external_lbasize = le32_to_cpu(super->external_lbasize); + arena->nfree = le32_to_cpu(super->nfree); + arena->version_major = le16_to_cpu(super->version_major); + arena->version_minor = le16_to_cpu(super->version_minor); + + arena->nextoff = (super->nextoff == 0) ? 0 : (arena_off + + le64_to_cpu(super->nextoff)); + arena->infooff = arena_off; + arena->dataoff = arena_off + le64_to_cpu(super->dataoff); + arena->mapoff = arena_off + le64_to_cpu(super->mapoff); + arena->logoff = arena_off + le64_to_cpu(super->logoff); + arena->info2off = arena_off + le64_to_cpu(super->info2off); + + arena->size = (super->nextoff > 0) ? (le64_to_cpu(super->nextoff)) : + (arena->info2off - arena->infooff + BTT_PG_SIZE); + + arena->flags = le32_to_cpu(super->flags); +} + +static int discover_arenas(struct btt *btt) +{ + int ret = 0; + struct arena_info *arena; + struct btt_sb *super; + size_t remaining = btt->rawsize; + u64 cur_nlba = 0; + size_t cur_off = 0; + int num_arenas = 0; + + super = kzalloc(sizeof(*super), GFP_KERNEL); + if (!super) + return -ENOMEM; + + while (remaining) { + /* Alloc memory for arena */ + arena = alloc_arena(btt, 0, 0, 0); + if (!arena) { + ret = -ENOMEM; + goto out_super; + } + + arena->infooff = cur_off; + ret = btt_info_read(arena, super); + if (ret) + goto out; + + if (!arena_is_valid(arena, super, btt->nd_btt->uuid)) { + if (remaining == btt->rawsize) { + btt->init_state = INIT_NOTFOUND; + dev_info(to_dev(arena), "No existing arenas\n"); + goto out; + } else { + dev_info(to_dev(arena), + "Found corrupted metadata!\n"); + ret = -ENODEV; + goto out; + } + } + + arena->external_lba_start = cur_nlba; + parse_arena_meta(arena, super, cur_off); + + ret = btt_freelist_init(arena); + if (ret) + goto out; + + ret = btt_rtt_init(arena); + if (ret) + goto out; + + ret = btt_maplocks_init(arena); + if (ret) + goto out; + + list_add_tail(&arena->list, &btt->arena_list); + + remaining -= arena->size; + cur_off += arena->size; + cur_nlba += arena->external_nlba; + num_arenas++; + + if (arena->nextoff == 0) + break; + } + btt->num_arenas = num_arenas; + btt->nlba = cur_nlba; + btt->init_state = INIT_READY; + + kfree(super); + return ret; + + out: + kfree(arena); + free_arenas(btt); + out_super: + kfree(super); + return ret; +} + +static int create_arenas(struct btt *btt) +{ + size_t remaining = btt->rawsize; + size_t cur_off = 0; + + while (remaining) { + struct arena_info *arena; + size_t arena_size = min_t(u64, ARENA_MAX_SIZE, remaining); + + remaining -= arena_size; + if (arena_size < ARENA_MIN_SIZE) + break; + + arena = alloc_arena(btt, arena_size, btt->nlba, cur_off); + if (!arena) { + free_arenas(btt); + return -ENOMEM; + } + btt->nlba += arena->external_nlba; + if (remaining >= ARENA_MIN_SIZE) + arena->nextoff = arena->size; + else + arena->nextoff = 0; + cur_off += arena_size; + list_add_tail(&arena->list, &btt->arena_list); + } + + return 0; +} + +/* + * This function completes arena initialization by writing + * all the metadata. + * It is only called for an uninitialized arena when a write + * to that arena occurs for the first time. + */ +static int btt_arena_write_layout(struct arena_info *arena, u8 *uuid) +{ + int ret; + struct btt_sb *super; + + ret = btt_map_init(arena); + if (ret) + return ret; + + ret = btt_log_init(arena); + if (ret) + return ret; + + super = kzalloc(sizeof(struct btt_sb), GFP_NOIO); + if (!super) + return -ENOMEM; + + strncpy(super->signature, BTT_SIG, BTT_SIG_LEN); + memcpy(super->uuid, uuid, 16); + super->flags = cpu_to_le32(arena->flags); + super->version_major = cpu_to_le16(arena->version_major); + super->version_minor = cpu_to_le16(arena->version_minor); + super->external_lbasize = cpu_to_le32(arena->external_lbasize); + super->external_nlba = cpu_to_le32(arena->external_nlba); + super->internal_lbasize = cpu_to_le32(arena->internal_lbasize); + super->internal_nlba = cpu_to_le32(arena->internal_nlba); + super->nfree = cpu_to_le32(arena->nfree); + super->infosize = cpu_to_le32(sizeof(struct btt_sb)); + + /* TODO: make these relative to arena start. For now we get this + * since each file = 1 arena = 1 dimm, but will change */ + super->nextoff = cpu_to_le64(arena->nextoff); + /* + * Subtract arena->infooff (arena start) so numbers are relative + * to 'this' arena + */ + super->dataoff = cpu_to_le64(arena->dataoff - arena->infooff); + super->mapoff = cpu_to_le64(arena->mapoff - arena->infooff); + super->logoff = cpu_to_le64(arena->logoff - arena->infooff); + super->info2off = cpu_to_le64(arena->info2off - arena->infooff); + + super->flags = 0; + super->checksum = cpu_to_le64(nd_btt_sb_checksum(super)); + + ret = btt_info_write(arena, super); + + kfree(super); + return ret; +} + +/* + * This function completes the initialization for the BTT namespace + * such that it is ready to accept IOs + */ +static int btt_meta_init(struct btt *btt) +{ + int ret = 0; + struct arena_info *arena; + + mutex_lock(&btt->init_lock); + list_for_each_entry(arena, &btt->arena_list, list) { + ret = btt_arena_write_layout(arena, btt->nd_btt->uuid); + if (ret) + goto unlock; + + ret = btt_freelist_init(arena); + if (ret) + goto unlock; + + ret = btt_rtt_init(arena); + if (ret) + goto unlock; + + ret = btt_maplocks_init(arena); + if (ret) + goto unlock; + } + + btt->init_state = INIT_READY; + + unlock: + mutex_unlock(&btt->init_lock); + return ret; +} + +/* + * This function calculates the arena in which the given LBA lies + * by doing a linear walk. This is acceptable since we expect only + * a few arenas. If we have backing devices that get much larger, + * we can construct a balanced binary tree of arenas at init time + * so that this range search becomes faster. + */ +static int lba_to_arena(struct btt *btt, sector_t sector, __u32 *premap, + struct arena_info **arena) +{ + struct arena_info *arena_list; + __u64 lba = div_u64(sector << SECTOR_SHIFT, btt->lbasize); + + list_for_each_entry(arena_list, &btt->arena_list, list) { + if (lba < arena_list->external_nlba) { + *arena = arena_list; + *premap = lba; + return 0; + } + lba -= arena_list->external_nlba; + } + + return -EIO; +} + +/* + * The following (lock_map, unlock_map) are mostly just to improve + * readability, since they index into an array of locks + */ +static void lock_map(struct arena_info *arena, u32 premap) +{ + u32 idx = (premap * MAP_ENT_SIZE / L1_CACHE_BYTES) % arena->nfree; + + spin_lock(&arena->map_locks[idx].lock); +} + +static void unlock_map(struct arena_info *arena, u32 premap) +{ + u32 idx = (premap * MAP_ENT_SIZE / L1_CACHE_BYTES) % arena->nfree; + + spin_unlock(&arena->map_locks[idx].lock); +} + +static u64 to_namespace_offset(struct arena_info *arena, u64 lba) +{ + return arena->dataoff + ((u64)lba * arena->internal_lbasize); +} + +static int btt_data_read(struct arena_info *arena, struct page *page, + unsigned int off, u32 lba, u32 len) +{ + int ret; + u64 nsoff = to_namespace_offset(arena, lba); + void *mem = kmap_atomic(page); + + ret = arena_rw_bytes(arena, mem + off, len, nsoff, READ); + kunmap_atomic(mem); + + return ret; +} + +static int btt_data_write(struct arena_info *arena, u32 lba, + struct page *page, unsigned int off, u32 len) +{ + int ret; + u64 nsoff = to_namespace_offset(arena, lba); + void *mem = kmap_atomic(page); + + ret = arena_rw_bytes(arena, mem + off, len, nsoff, WRITE); + kunmap_atomic(mem); + + return ret; +} + +static void zero_fill_data(struct page *page, unsigned int off, u32 len) +{ + void *mem = kmap_atomic(page); + + memset(mem + off, 0, len); + kunmap_atomic(mem); +} + +static int btt_read_pg(struct btt *btt, struct page *page, unsigned int off, + sector_t sector, unsigned int len) +{ + int ret = 0; + int t_flag, e_flag; + struct arena_info *arena = NULL; + u32 lane = 0, premap, postmap; + + while (len) { + u32 cur_len; + + lane = nd_region_acquire_lane(btt->nd_region); + + ret = lba_to_arena(btt, sector, &premap, &arena); + if (ret) + goto out_lane; + + cur_len = min(arena->external_lbasize, len); + + ret = btt_map_read(arena, premap, &postmap, &t_flag, &e_flag); + if (ret) + goto out_lane; + + /* + * We loop to make sure that the post map LBA didn't change + * from under us between writing the RTT and doing the actual + * read. + */ + while (1) { + u32 new_map; + + if (t_flag) { + zero_fill_data(page, off, cur_len); + goto out_lane; + } + + if (e_flag) { + ret = -EIO; + goto out_lane; + } + + arena->rtt[lane] = RTT_VALID | postmap; + /* + * Barrier to make sure this write is not reordered + * to do the verification map_read before the RTT store + */ + barrier(); + + ret = btt_map_read(arena, premap, &new_map, &t_flag, + &e_flag); + if (ret) + goto out_rtt; + + if (postmap == new_map) + break; + + postmap = new_map; + } + + ret = btt_data_read(arena, page, off, postmap, cur_len); + if (ret) + goto out_rtt; + + arena->rtt[lane] = RTT_INVALID; + nd_region_release_lane(btt->nd_region, lane); + + len -= cur_len; + off += cur_len; + sector += arena->external_lbasize >> SECTOR_SHIFT; + } + + return 0; + + out_rtt: + arena->rtt[lane] = RTT_INVALID; + out_lane: + nd_region_release_lane(btt->nd_region, lane); + return ret; +} + +static int btt_write_pg(struct btt *btt, sector_t sector, struct page *page, + unsigned int off, unsigned int len) +{ + int ret = 0; + struct arena_info *arena = NULL; + u32 premap = 0, old_postmap, new_postmap, lane = 0, i; + struct log_entry log; + int sub; + + while (len) { + u32 cur_len; + + lane = nd_region_acquire_lane(btt->nd_region); + + ret = lba_to_arena(btt, sector, &premap, &arena); + if (ret) + goto out_lane; + cur_len = min(arena->external_lbasize, len); + + if ((arena->flags & IB_FLAG_ERROR_MASK) != 0) { + ret = -EIO; + goto out_lane; + } + + new_postmap = arena->freelist[lane].block; + + /* Wait if the new block is being read from */ + for (i = 0; i < arena->nfree; i++) + while (arena->rtt[i] == (RTT_VALID | new_postmap)) + cpu_relax(); + + + if (new_postmap >= arena->internal_nlba) { + ret = -EIO; + goto out_lane; + } else + ret = btt_data_write(arena, new_postmap, page, + off, cur_len); + if (ret) + goto out_lane; + + lock_map(arena, premap); + ret = btt_map_read(arena, premap, &old_postmap, NULL, NULL); + if (ret) + goto out_map; + if (old_postmap >= arena->internal_nlba) { + ret = -EIO; + goto out_map; + } + + log.lba = cpu_to_le32(premap); + log.old_map = cpu_to_le32(old_postmap); + log.new_map = cpu_to_le32(new_postmap); + log.seq = cpu_to_le32(arena->freelist[lane].seq); + sub = arena->freelist[lane].sub; + ret = btt_flog_write(arena, lane, sub, &log); + if (ret) + goto out_map; + + ret = btt_map_write(arena, premap, new_postmap, 0, 0); + if (ret) + goto out_map; + + unlock_map(arena, premap); + nd_region_release_lane(btt->nd_region, lane); + + len -= cur_len; + off += cur_len; + sector += arena->external_lbasize >> SECTOR_SHIFT; + } + + return 0; + + out_map: + unlock_map(arena, premap); + out_lane: + nd_region_release_lane(btt->nd_region, lane); + return ret; +} + +static int btt_do_bvec(struct btt *btt, struct page *page, + unsigned int len, unsigned int off, int rw, + sector_t sector) +{ + int ret; + + if (rw == READ) { + ret = btt_read_pg(btt, page, off, sector, len); + flush_dcache_page(page); + } else { + flush_dcache_page(page); + ret = btt_write_pg(btt, sector, page, off, len); + } + + return ret; +} + +static void btt_make_request(struct request_queue *q, struct bio *bio) +{ + struct block_device *bdev = bio->bi_bdev; + struct btt *btt = q->queuedata; + int rw; + struct bio_vec bvec; + sector_t sector; + struct bvec_iter iter; + int err = 0; + + sector = bio->bi_iter.bi_sector; + if (bio_end_sector(bio) > get_capacity(bdev->bd_disk)) { + err = -EIO; + goto out; + } + + BUG_ON(bio->bi_rw & REQ_DISCARD); + + rw = bio_rw(bio); + if (rw == READA) + rw = READ; + + bio_for_each_segment(bvec, bio, iter) { + unsigned int len = bvec.bv_len; + + BUG_ON(len > PAGE_SIZE); + /* Make sure len is in multiples of lbasize. */ + /* XXX is this right? */ + BUG_ON(len < btt->lbasize); + BUG_ON(len % btt->lbasize); + + err = btt_do_bvec(btt, bvec.bv_page, len, bvec.bv_offset, + rw, sector); + if (err) { + dev_info(&btt->nd_btt->dev, + "io error in %s sector %lld, len %d,\n", + (rw == READ) ? "READ" : "WRITE", + (unsigned long long) sector, len); + goto out; + } + sector += len >> SECTOR_SHIFT; + } + +out: + bio_endio(bio, err); +} + +static int btt_getgeo(struct block_device *bd, struct hd_geometry *geo) +{ + /* some standard values */ + geo->heads = 1 << 6; + geo->sectors = 1 << 5; + geo->cylinders = get_capacity(bd->bd_disk) >> 11; + return 0; +} + +static const struct block_device_operations btt_fops = { + .owner = THIS_MODULE, + /* TODO: Disable rw_page till lazy init is reworked */ + /*.rw_page = btt_rw_page, */ + .getgeo = btt_getgeo, +}; + +static int btt_blk_init(struct btt *btt) +{ + int ret; + + /* create a new disk and request queue for btt */ + btt->btt_queue = blk_alloc_queue(GFP_KERNEL); + if (!btt->btt_queue) + return -ENOMEM; + + btt->btt_disk = alloc_disk(0); + if (!btt->btt_disk) { + ret = -ENOMEM; + goto out_free_queue; + } + + sprintf(btt->btt_disk->disk_name, "nd%d", btt->nd_btt->id); + btt->btt_disk->driverfs_dev = &btt->nd_btt->dev; + btt->btt_disk->major = btt_major; + btt->btt_disk->first_minor = 0; + btt->btt_disk->fops = &btt_fops; + btt->btt_disk->private_data = btt; + btt->btt_disk->queue = btt->btt_queue; + btt->btt_disk->flags = GENHD_FL_EXT_DEVT; + + blk_queue_make_request(btt->btt_queue, btt_make_request); + blk_queue_max_hw_sectors(btt->btt_queue, 1024); + blk_queue_bounce_limit(btt->btt_queue, BLK_BOUNCE_ANY); + blk_queue_logical_block_size(btt->btt_queue, btt->lbasize); + btt->btt_queue->queuedata = btt; + + set_capacity(btt->btt_disk, btt->nlba * btt->lbasize >> SECTOR_SHIFT); + add_disk(btt->btt_disk); + + return 0; + +out_free_queue: + blk_cleanup_queue(btt->btt_queue); + return ret; +} + +static void btt_blk_cleanup(struct btt *btt) +{ + del_gendisk(btt->btt_disk); + put_disk(btt->btt_disk); + blk_cleanup_queue(btt->btt_queue); +} + +/** + * btt_init - initialize a block translation table for the given device + * @nd_btt: device with BTT geometry and backing device info + * @rawsize: raw size in bytes of the backing device + * @lbasize: lba size of the backing device + * @uuid: A uuid for the backing device - this is stored on media + * @maxlane: maximum number of parallel requests the device can handle + * + * Initialize a Block Translation Table on a backing device to provide + * single sector power fail atomicity. + * + * Context: + * Might sleep. + * + * Returns: + * Pointer to a new struct btt on success, NULL on failure. + */ +static struct btt *btt_init(struct nd_btt *nd_btt, unsigned long long rawsize, + u32 lbasize, u8 *uuid, struct nd_region *nd_region) +{ + int ret; + struct btt *btt; + struct device *dev = &nd_btt->dev; + + btt = kzalloc(sizeof(struct btt), GFP_KERNEL); + if (!btt) + return NULL; + + btt->nd_btt = nd_btt; + btt->rawsize = rawsize; + btt->lbasize = lbasize; + INIT_LIST_HEAD(&btt->arena_list); + mutex_init(&btt->init_lock); + btt->nd_region = nd_region; + + ret = discover_arenas(btt); + if (ret) { + dev_err(dev, "init: error in arena_discover: %d\n", ret); + goto out_free; + } + + if (btt->init_state != INIT_READY) { + btt->num_arenas = (rawsize / ARENA_MAX_SIZE) + + ((rawsize % ARENA_MAX_SIZE) ? 1 : 0); + dev_dbg(dev, "init: %d arenas for %llu rawsize\n", + btt->num_arenas, rawsize); + + ret = create_arenas(btt); + if (ret) { + dev_info(dev, "init: create_arenas: %d\n", ret); + goto out_free; + } + + ret = btt_meta_init(btt); + if (ret) { + dev_err(dev, "init: error in meta_init: %d\n", ret); + return NULL; + } + } + + ret = btt_blk_init(btt); + if (ret) { + dev_err(dev, "init: error in blk_init: %d\n", ret); + goto out_free; + } + + btt_debugfs_init(btt); + + return btt; + + out_free: + kfree(btt); + return NULL; +} + +/** + * btt_fini - de-initialize a BTT + * @btt: the BTT handle that was generated by btt_init + * + * De-initialize a Block Translation Table on device removal + * + * Context: + * Might sleep. + */ +static void btt_fini(struct btt *btt) +{ + if (btt) { + btt_blk_cleanup(btt); + free_arenas(btt); + debugfs_remove_recursive(btt->debugfs_dir); + kfree(btt); + } +} + +static int link_btt(struct nd_btt *nd_btt) +{ + struct block_device *bdev = nd_btt->backing_dev; + struct kobject *dir = &part_to_dev(bdev->bd_part)->kobj; + + return sysfs_create_link(dir, &nd_btt->dev.kobj, "nd_btt"); +} + +static void unlink_btt(struct nd_btt *nd_btt) +{ + struct block_device *bdev = nd_btt->backing_dev; + struct kobject *dir; + + /* if backing_dev was deleted first we may have nothing to unlink */ + if (!nd_btt->backing_dev) + return; + + dir = &part_to_dev(bdev->bd_part)->kobj; + sysfs_remove_link(dir, "nd_btt"); +} + +static int nd_btt_probe(struct device *dev) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + struct nd_io_claim *ndio_claim = nd_btt->ndio_claim; + struct nd_region *nd_region; + struct block_device *bdev; + struct btt *btt; + size_t rawsize; + int rc; + + if (!ndio_claim || !nd_btt->uuid || !nd_btt->backing_dev + || !nd_btt->lbasize) + return -ENODEV; + + rc = link_btt(nd_btt); + if (rc) + return rc; + + bdev = nd_btt->backing_dev; + /* the first 4K of a device is padding */ + nd_btt->offset = nd_partition_offset(bdev) + SZ_4K; + rawsize = (bdev->bd_part->nr_sects << SECTOR_SHIFT) - SZ_4K; + if (rawsize < ARENA_MIN_SIZE) { + rc = -ENXIO; + goto err_btt; + } + nd_btt->ndio = nd_btt->ndio_claim->parent; + nd_region = to_nd_region(nd_btt->ndio->dev->parent); + btt = btt_init(nd_btt, rawsize, nd_btt->lbasize, nd_btt->uuid, + nd_region); + if (!btt) { + rc = -ENOMEM; + goto err_btt; + } + btt->backing_dev = bdev; + dev_set_drvdata(dev, btt); + + return 0; + err_btt: + unlink_btt(nd_btt); + return rc; +} + +static int nd_btt_remove(struct device *dev) +{ + struct nd_btt *nd_btt = to_nd_btt(dev); + struct btt *btt = dev_get_drvdata(dev); + + btt_fini(btt); + unlink_btt(nd_btt); + + return 0; +} + +static struct nd_device_driver nd_btt_driver = { + .probe = nd_btt_probe, + .remove = nd_btt_remove, + .drv = { + .name = "nd_btt", + }, + .type = ND_DRIVER_BTT, +}; + +static int __init nd_btt_init(void) +{ + int rc; + + BUILD_BUG_ON(sizeof(struct btt_sb) != SZ_4K); + + btt_major = register_blkdev(0, "btt"); + if (btt_major < 0) + return btt_major; + + debugfs_root = debugfs_create_dir("btt", NULL); + if (IS_ERR_OR_NULL(debugfs_root)) { + rc = -ENXIO; + goto err_debugfs; + } + + rc = nd_driver_register(&nd_btt_driver); + if (rc < 0) + goto err_driver; + return 0; + + err_driver: + debugfs_remove_recursive(debugfs_root); + err_debugfs: + unregister_blkdev(btt_major, "btt"); + + return rc; +} + +static void __exit nd_btt_exit(void) +{ + driver_unregister(&nd_btt_driver.drv); + debugfs_remove_recursive(debugfs_root); + unregister_blkdev(btt_major, "btt"); +} + +MODULE_ALIAS_ND_DEVICE(ND_DEVICE_BTT); +MODULE_AUTHOR("Vishal Verma <vishal.l.verma(a)linux.intel.com&gt;&quot;); +MODULE_LICENSE("GPL v2"); +module_init(nd_btt_init); +module_exit(nd_btt_exit); diff --git a/drivers/block/nd/btt.h b/drivers/block/nd/btt.h index e8f6d8e0ddd3..d4e67c75c91f 100644 --- a/drivers/block/nd/btt.h +++ b/drivers/block/nd/btt.h @@ -19,6 +19,39 @@ #define BTT_SIG_LEN 16 #define BTT_SIG "BTT_ARENA_INFO\0" +#define MAP_ENT_SIZE 4 +#define MAP_TRIM_SHIFT 31 +#define MAP_TRIM_MASK (1 << MAP_TRIM_SHIFT) +#define MAP_ERR_SHIFT 30 +#define MAP_ERR_MASK (1 << MAP_ERR_SHIFT) +#define MAP_LBA_MASK (~((1 << MAP_TRIM_SHIFT) | (1 << MAP_ERR_SHIFT))) +#define MAP_ENT_NORMAL 0xC0000000 +#define LOG_ENT_SIZE sizeof(struct log_entry) +#define ARENA_MIN_SIZE (1UL << 24) /* 16 MB */ +#define ARENA_MAX_SIZE (1ULL << 39) /* 512 GB */ +#define RTT_VALID (1UL << 31) +#define RTT_INVALID 0 +#define INT_LBASIZE_ALIGNMENT 256 +#define BTT_PG_SIZE 4096 +#define BTT_DEFAULT_NFREE ND_MAX_LANES +#define LOG_SEQ_INIT 1 + +#define IB_FLAG_ERROR 0x00000001 +#define IB_FLAG_ERROR_MASK 0x00000001 + +enum btt_init_state { + INIT_UNCHECKED = 0, + INIT_NOTFOUND, + INIT_READY +}; + +struct log_entry { + __le32 lba; + __le32 old_map; + __le32 new_map; + __le32 seq; + __le64 padding[2]; +}; struct btt_sb { u8 signature[BTT_SIG_LEN]; @@ -42,4 +75,111 @@ struct btt_sb { __le64 checksum; }; +struct free_entry { + u32 block; + u8 sub; + u8 seq; +}; + +struct aligned_lock { + union { + spinlock_t lock; + u8 cacheline_padding[L1_CACHE_BYTES]; + }; +}; + +/** + * struct arena_info - handle for an arena + * @size: Size in bytes this arena occupies on the raw device. + * This includes arena metadata. + * @external_lba_start: The first external LBA in this arena. + * @internal_nlba: Number of internal blocks available in the arena + * including nfree reserved blocks + * @internal_lbasize: Internal and external lba sizes may be different as + * we can round up 'odd' external lbasizes such as 520B + * to be aligned. + * @external_nlba: Number of blocks contributed by the arena to the number + * reported to upper layers. (internal_nlba - nfree) + * @external_lbasize: LBA size as exposed to upper layers. + * @nfree: A reserve number of 'free' blocks that is used to + * handle incoming writes. + * @version_major: Metadata layout version major. + * @version_minor: Metadata layout version minor. + * @nextoff: Offset in bytes to the start of the next arena. + * @infooff: Offset in bytes to the info block of this arena. + * @dataoff: Offset in bytes to the data area of this arena. + * @mapoff: Offset in bytes to the map area of this arena. + * @logoff: Offset in bytes to the log area of this arena. + * @info2off: Offset in bytes to the backup info block of this arena. + * @freelist: Pointer to in-memory list of free blocks + * @rtt: Pointer to in-memory "Read Tracking Table" + * @map_locks: Spinlocks protecting concurrent map writes + * @nd_btt: Pointer to parent nd_btt structure. + * @list: List head for list of arenas + * @debugfs_dir: Debugfs dentry + * @flags: Arena flags - may signify error states. + * + * arena_info is a per-arena handle. Once an arena is narrowed down for an + * IO, this struct is passed around for the duration of the IO. + */ +struct arena_info { + u64 size; /* Total bytes for this arena */ + u64 external_lba_start; + u32 internal_nlba; + u32 internal_lbasize; + u32 external_nlba; + u32 external_lbasize; + u32 nfree; + u16 version_major; + u16 version_minor; + /* Byte offsets to the different on-media structures */ + u64 nextoff; + u64 infooff; + u64 dataoff; + u64 mapoff; + u64 logoff; + u64 info2off; + /* Pointers to other in-memory structures for this arena */ + struct free_entry *freelist; + u32 *rtt; + struct aligned_lock *map_locks; + struct nd_btt *nd_btt; + struct list_head list; + struct dentry *debugfs_dir; + /* Arena flags */ + u32 flags; +}; + +/** + * struct btt - handle for a BTT instance + * @btt_disk: Pointer to the gendisk for BTT device + * @btt_queue: Pointer to the request queue for the BTT device + * @arena_list: Head of the list of arenas + * @debugfs_dir: Debugfs dentry + * @backing_dev: Backing block device for the BTT + * @nd_btt: Parent nd_btt struct + * @nlba: Number of logical blocks exposed to the upper layers + * after removing the amount of space needed by metadata + * @rawsize: Total size in bytes of the available backing device + * @lbasize: LBA size as requested and presented to upper layers + * @lanes: Per-lane spinlocks + * @init_lock: Mutex used for the BTT initialization + * @init_state: Flag describing the initialization state for the BTT + * @num_arenas: Number of arenas in the BTT instance + */ +struct btt { + struct gendisk *btt_disk; + struct request_queue *btt_queue; + struct list_head arena_list; + struct dentry *debugfs_dir; + struct block_device *backing_dev; + struct nd_btt *nd_btt; + u64 nlba; + unsigned long long rawsize; + u32 lbasize; + struct nd_region *nd_region; + struct mutex init_lock; + int init_state; + int num_arenas; +}; #endif diff --git a/drivers/block/nd/btt_devs.c b/drivers/block/nd/btt_devs.c index e6f0b8b999d8..0746db70973c 100644 --- a/drivers/block/nd/btt_devs.c +++ b/drivers/block/nd/btt_devs.c @@ -342,7 +342,8 @@ struct nd_btt *nd_btt_create(struct nd_bus *nd_bus) */ u64 nd_btt_sb_checksum(struct btt_sb *btt_sb) { - u64 sum, sum_save; + u64 sum; + __le64 sum_save; sum_save = btt_sb->checksum; btt_sb->checksum = 0; diff --git a/drivers/block/nd/libnd.h b/drivers/block/nd/libnd.h index 3f6b5e09cd67..e188840ed2b9 100644 --- a/drivers/block/nd/libnd.h +++ b/drivers/block/nd/libnd.h @@ -76,6 +76,7 @@ struct nd_region_desc { const struct attribute_group **attr_groups; struct nd_interleave_set *nd_set; void *provider_data; + int num_lanes; }; struct nd_bus; diff --git a/drivers/block/nd/nd-private.h b/drivers/block/nd/nd-private.h index 4ab4765dc9ee..68e9ec824dc8 100644 --- a/drivers/block/nd/nd-private.h +++ b/drivers/block/nd/nd-private.h @@ -76,6 +76,7 @@ int __init nd_bus_init(void); void nd_bus_exit(void); int __init nd_dimm_init(void); int __init nd_region_init(void); +void __init nd_region_init_locks(void); void nd_dimm_exit(void); int nd_region_exit(void); void nd_region_probe_start(struct nd_bus *nd_bus, struct device *dev); diff --git a/drivers/block/nd/nd.h b/drivers/block/nd/nd.h index c6ed26d4dcad..a29fb7409925 100644 --- a/drivers/block/nd/nd.h +++ b/drivers/block/nd/nd.h @@ -22,6 +22,12 @@ #include "label.h" enum { + /* + * Limits the maximum number of block apertures a dimm can + * support and is an input to the geometry/on-disk-format of a + * BTT instance + */ + ND_MAX_LANES = 256, SECTOR_SHIFT = 9, }; @@ -101,7 +107,7 @@ struct nd_region { u16 ndr_mappings; u64 ndr_size; u64 ndr_start; - int id; + int id, num_lanes; void *provider_data; struct nd_interleave_set *nd_set; struct nd_mapping mapping[0]; @@ -226,6 +232,8 @@ struct nd_btt *to_nd_btt(struct device *dev); struct btt_sb; u64 nd_btt_sb_checksum(struct btt_sb *btt_sb); struct nd_region *to_nd_region(struct device *dev); +unsigned int nd_region_acquire_lane(struct nd_region *nd_region); +void nd_region_release_lane(struct nd_region *nd_region, unsigned int lane); int nd_region_to_namespace_type(struct nd_region *nd_region); int nd_region_register_namespaces(struct nd_region *nd_region, int *err); u64 nd_region_interleave_set_cookie(struct nd_region *nd_region); diff --git a/drivers/block/nd/region.c b/drivers/block/nd/region.c index 9d1fd45d78a1..dd5a885cea11 100644 --- a/drivers/block/nd/region.c +++ b/drivers/block/nd/region.c @@ -15,6 +15,72 @@ #include <linux/nd.h> #include "nd.h" +static struct { + struct { + int count[CONFIG_ND_MAX_REGIONS]; + spinlock_t lock[CONFIG_ND_MAX_REGIONS]; + } lane[NR_CPUS]; +} nd_percpu_lane; + +static void __init nd_region_init_locks(void) +{ + int i, j; + + for (i = 0; i < NR_CPUS; i++) + for (j = 0; j < CONFIG_ND_MAX_REGIONS; j++) + spin_lock_init(&nd_percpu_lane.lane[i].lock[j]); +} + +/** + * nd_region_acquire_lane - allocate and lock a lane + * @nd_region: region id and number of lanes possible + * + * A lane correlates to a BLK-data-window and/or a log slot in the BTT. + * We optimize for the common case where there are 256 lanes, one + * per-cpu. For larger systems we need to lock to share lanes. For now + * this implementation assumes the cost of maintaining an allocator for + * free lanes is on the order of the lock hold time, so it implements a + * static lane = cpu % num_lanes mapping. + * + * In the case of a BTT instance on top of a BLK namespace a lane may be + * acquired recursively. We lock on the first instance. + * + * In the case of a BTT instance on top of PMEM, we only acquire a lane + * for the BTT metadata updates. + */ +unsigned int nd_region_acquire_lane(struct nd_region *nd_region) +{ + unsigned int cpu, lane; + + cpu = get_cpu(); + + if (nd_region->num_lanes < NR_CPUS) { + unsigned int id = nd_region->id; + + lane = cpu % nd_region->num_lanes; + if (nd_percpu_lane.lane[cpu].count[id]++ == 0) + spin_lock(&nd_percpu_lane.lane[lane].lock[id]); + } else + lane = cpu; + + return lane; +} +EXPORT_SYMBOL(nd_region_acquire_lane); + +void nd_region_release_lane(struct nd_region *nd_region, unsigned int lane) +{ + if (nd_region->num_lanes < NR_CPUS) { + unsigned int cpu = get_cpu(); + unsigned int id = nd_region->id; + + if (--nd_percpu_lane.lane[cpu].count[id] == 0) + spin_unlock(&nd_percpu_lane.lane[lane].lock[id]); + put_cpu(); + } + put_cpu(); +} +EXPORT_SYMBOL(nd_region_release_lane); + static int nd_region_probe(struct device *dev) { int err; @@ -76,6 +142,7 @@ static struct nd_device_driver nd_region_driver = { int __init nd_region_init(void) { + nd_region_init_locks(); return nd_driver_register(&nd_region_driver); } diff --git a/drivers/block/nd/region_devs.c b/drivers/block/nd/region_devs.c index bcdd8e1e21a2..268d9ef67f9c 100644 --- a/drivers/block/nd/region_devs.c +++ b/drivers/block/nd/region_devs.c @@ -538,6 +538,12 @@ static noinline struct nd_region *nd_region_create(struct nd_bus *nd_bus, if (nd_region->id < 0) { kfree(nd_region); return NULL; + } else if (nd_region->id >= CONFIG_ND_MAX_REGIONS) { + dev_err(&nd_bus->dev, "max region limit %d reached\n", + CONFIG_ND_MAX_REGIONS); + ida_simple_remove(&region_ida, nd_region->id); + kfree(nd_region); + return NULL; } memcpy(nd_region->mapping, ndr_desc->nd_mapping, @@ -551,6 +557,7 @@ static noinline struct nd_region *nd_region_create(struct nd_bus *nd_bus, nd_region->ndr_mappings = ndr_desc->num_mappings; nd_region->provider_data = ndr_desc->provider_data; nd_region->nd_set = ndr_desc->nd_set; + nd_region->num_lanes = ndr_desc->num_lanes; ida_init(&nd_region->ns_ida); dev = &nd_region->dev; dev_set_name(dev, "region%d", nd_region->id); @@ -567,6 +574,7 @@ static noinline struct nd_region *nd_region_create(struct nd_bus *nd_bus, struct nd_region *nd_pmem_region_create(struct nd_bus *nd_bus, struct nd_region_desc *ndr_desc) { + ndr_desc->num_lanes = ND_MAX_LANES; return nd_region_create(nd_bus, ndr_desc, &nd_pmem_device_type); } EXPORT_SYMBOL_GPL(nd_pmem_region_create); @@ -576,6 +584,7 @@ struct nd_region *nd_blk_region_create(struct nd_bus *nd_bus, { if (ndr_desc->num_mappings > 1) return NULL; + ndr_desc->num_lanes = min(ndr_desc->num_lanes, ND_MAX_LANES); return nd_region_create(nd_bus, ndr_desc, &nd_blk_device_type); } EXPORT_SYMBOL_GPL(nd_blk_region_create); @@ -583,6 +592,7 @@ EXPORT_SYMBOL_GPL(nd_blk_region_create); struct nd_region *nd_volatile_region_create(struct nd_bus *nd_bus, struct nd_region_desc *ndr_desc) { + ndr_desc->num_lanes = ND_MAX_LANES; return nd_region_create(nd_bus, ndr_desc, &nd_volatile_device_type); } EXPORT_SYMBOL_GPL(nd_volatile_region_create);

On Tue, Apr 28, 2015 at 11:26 AM, Dan Williams <dan.j.williams(a)intel.com&gt; wrote: This describes how it works, not what it is. How about: This driver exposes NVDIMM BLK regions as block devices. BLK regions are regions of NVDIMM storage that are sector-addressable, not byte-addressible, and do not support DAX. Say Y if you want to... --Andy

On Tue, Apr 28, 2015 at 3:30 PM, Dan Williams <dan.j.williams(a)intel.com&gt; wrote: Right, I was assuming incorrectly that the sector-atomic thing was a necessary part of BLK, or at least of this implementation. Anyway, I think my point stands: let's describe what these drivers do from a user's perspective, not how they work. --Andy -- Andy Lutomirski AMA Capital Management, LLC

On Wed, Apr 29, 2015 at 10:10 AM, Dan Williams <dan.j.williams(a)intel.com&gt; wrote: Sure. --Andy