7:15 a.m.
Hi Dan,
I've got 4 NVDIMMs in an interleave set in a configuration that supports labels.
I'm running a 4.12 kernel with the latest ndctl.
I had three namespaces configured and all seemed well. When I configured the
fourth one, I made a mistake in the name so I hit control-c. I wasn't sure what
state I was in but according to what I could see with ndctl, it had created the
namespace but not enabled it, so I enabled it manually with ndctl and that
seemed ok.
Then I tried to use ndctl create-namespace to change the name, which failed
because the namespace was enabled so I disabled it and tried again. At some
point, not really sure where, I got this kernel warning:
# [ 5224.196085] nd namespace4.3: failed to track label: 4
(details in the attached file)
At this point I rebooted the system. When it came back up, nmem0 was disabled.
I dumped the labels (also attached) and I see that nmem0 has some extra labels
that correspond to the namespace that I was struggling with.
I think my troubles started with the control-c. It doesn't look like ndctl traps
signals when creating namespaces so perhaps we can get into an inconsistent
state.
It also seems like that kernel warning is a bit more important than a
WARN_ONCE would imply. I think that was the beginning of the end of my
configuration. It might have been better to just panic.
I was trying to figure out if I could fix my configuration without
losing the good namespaces but I don't see a way. The check-labels option
isn't very helpful because I think it only looks at the info blocks,
which are fine, even though the labels on nmem0 are not. The destroy-namespace
option doesn't help because it only works with a good namespace.
I'm going to wipe my nvdimms and start over. I suspect the problem is
reproducible but it could depend on the timing of the control-c, unless
the root cause was actually trying to rename a namespace. Maybe I'll try
that again but not today.
-- ljk
7:35 a.m.
On Mon, Jul 24, 2017 at 4:15 PM, Linda Knippers <linda.knippers(a)hpe.com> wrote:
Hi Dan,
I've got 4 NVDIMMs in an interleave set in a configuration that supports labels.
I'm running a 4.12 kernel with the latest ndctl.
I had three namespaces configured and all seemed well. When I configured the
fourth one, I made a mistake in the name so I hit control-c. I wasn't sure what
state I was in but according to what I could see with ndctl, it had created the
namespace but not enabled it, so I enabled it manually with ndctl and that
seemed ok.
Then I tried to use ndctl create-namespace to change the name, which failed
because the namespace was enabled so I disabled it and tried again. At some
point, not really sure where, I got this kernel warning:
# [ 5224.196085] nd namespace4.3: failed to track label: 4
(details in the attached file)
At this point I rebooted the system. When it came back up, nmem0 was disabled.
I dumped the labels (also attached) and I see that nmem0 has some extra labels
that correspond to the namespace that I was struggling with.
I think my troubles started with the control-c. It doesn't look like ndctl traps
signals when creating namespaces so perhaps we can get into an inconsistent
state.
It also seems like that kernel warning is a bit more important than a
WARN_ONCE would imply. I think that was the beginning of the end of my
configuration. It might have been better to just panic.
In general if the system is even remotely recoverable we don't panic.
In this case it is recoverable. The WARN_ONCE() is really there as a
loud, "this is a kernel bug, but we'll do our best to keep going".
I was trying to figure out if I could fix my configuration without
losing the good namespaces but I don't see a way. The check-labels option
isn't very helpful because I think it only looks at the info blocks,
which are fine, even though the labels on nmem0 are not. The destroy-namespace
option doesn't help because it only works with a good namespace.
I'm going to wipe my nvdimms and start over. I suspect the problem is
reproducible but it could depend on the timing of the control-c, unless
the root cause was actually trying to rename a namespace. Maybe I'll try
that again but not today.
The recovery method when the labels are corrupted is:
ndctl disable-region all
ndctl zero-labels all
ndctl enable-region all
...and that should get you back to square one.
If you are able to reproduce I'd like to see the state of the DIMM
label areas. You can dump them in json format with the following:
ndctl read-labels -j all
7:43 a.m.
On 07/24/2017 07:35 PM, Dan Williams wrote:
On Mon, Jul 24, 2017 at 4:15 PM, Linda Knippers
<linda.knippers(a)hpe.com> wrote:
> Hi Dan,
>
> I've got 4 NVDIMMs in an interleave set in a configuration that supports labels.
> I'm running a 4.12 kernel with the latest ndctl.
>
> I had three namespaces configured and all seemed well. When I configured the
> fourth one, I made a mistake in the name so I hit control-c. I wasn't sure what
> state I was in but according to what I could see with ndctl, it had created the
> namespace but not enabled it, so I enabled it manually with ndctl and that
> seemed ok.
>
> Then I tried to use ndctl create-namespace to change the name, which failed
> because the namespace was enabled so I disabled it and tried again. At some
> point, not really sure where, I got this kernel warning:
>
> # [ 5224.196085] nd namespace4.3: failed to track label: 4
>
> (details in the attached file)
>
> At this point I rebooted the system. When it came back up, nmem0 was disabled.
> I dumped the labels (also attached) and I see that nmem0 has some extra labels
> that correspond to the namespace that I was struggling with.
>
> I think my troubles started with the control-c. It doesn't look like ndctl traps
> signals when creating namespaces so perhaps we can get into an inconsistent
> state.
>
> It also seems like that kernel warning is a bit more important than a
> WARN_ONCE would imply. I think that was the beginning of the end of my
> configuration. It might have been better to just panic.
In general if the system is even remotely recoverable we don't panic.
In this case it is recoverable. The WARN_ONCE() is really there as a
loud, "this is a kernel bug, but we'll do our best to keep going".
Keeping going is ok unless you're risking data.
> I was trying to figure out if I could fix my configuration
without
> losing the good namespaces but I don't see a way. The check-labels option
> isn't very helpful because I think it only looks at the info blocks,
> which are fine, even though the labels on nmem0 are not. The destroy-namespace
> option doesn't help because it only works with a good namespace.
>
> I'm going to wipe my nvdimms and start over. I suspect the problem is
> reproducible but it could depend on the timing of the control-c, unless
> the root cause was actually trying to rename a namespace. Maybe I'll try
> that again but not today.
The recovery method when the labels are corrupted is:
ndctl disable-region all
ndctl zero-labels all
ndctl enable-region all
...and that should get you back to square one.
Right, but that blows away all my namespaces. I was hoping to find a way
to just fix up (delete) what appeared to be extraneous labels.
If you are able to reproduce I'd like to see the state of the DIMM
label areas. You can dump them in json format with the following:
ndctl read-labels -j all
That was in one of the attachments.
-- ljk
7:57 a.m.
On Mon, Jul 24, 2017 at 4:43 PM, Linda Knippers <linda.knippers(a)hpe.com> wrote:
On 07/24/2017 07:35 PM, Dan Williams wrote:
> On Mon, Jul 24, 2017 at 4:15 PM, Linda Knippers <linda.knippers(a)hpe.com>
wrote:
>> Hi Dan,
>>
>> I've got 4 NVDIMMs in an interleave set in a configuration that supports
labels.
>> I'm running a 4.12 kernel with the latest ndctl.
>>
>> I had three namespaces configured and all seemed well. When I configured the
>> fourth one, I made a mistake in the name so I hit control-c. I wasn't sure
what
>> state I was in but according to what I could see with ndctl, it had created the
>> namespace but not enabled it, so I enabled it manually with ndctl and that
>> seemed ok.
>>
>> Then I tried to use ndctl create-namespace to change the name, which failed
>> because the namespace was enabled so I disabled it and tried again. At some
>> point, not really sure where, I got this kernel warning:
>>
>> # [ 5224.196085] nd namespace4.3: failed to track label: 4
>>
>> (details in the attached file)
>>
>> At this point I rebooted the system. When it came back up, nmem0 was disabled.
>> I dumped the labels (also attached) and I see that nmem0 has some extra labels
>> that correspond to the namespace that I was struggling with.
>>
>> I think my troubles started with the control-c. It doesn't look like ndctl
traps
>> signals when creating namespaces so perhaps we can get into an inconsistent
>> state.
>>
>> It also seems like that kernel warning is a bit more important than a
>> WARN_ONCE would imply. I think that was the beginning of the end of my
>> configuration. It might have been better to just panic.
>
> In general if the system is even remotely recoverable we don't panic.
> In this case it is recoverable. The WARN_ONCE() is really there as a
> loud, "this is a kernel bug, but we'll do our best to keep going".
Keeping going is ok unless you're risking data.
>> I was trying to figure out if I could fix my configuration without
>> losing the good namespaces but I don't see a way. The check-labels option
>> isn't very helpful because I think it only looks at the info blocks,
>> which are fine, even though the labels on nmem0 are not. The destroy-namespace
>> option doesn't help because it only works with a good namespace.
>>
>> I'm going to wipe my nvdimms and start over. I suspect the problem is
>> reproducible but it could depend on the timing of the control-c, unless
>> the root cause was actually trying to rename a namespace. Maybe I'll try
>> that again but not today.
>
> The recovery method when the labels are corrupted is:
>
> ndctl disable-region all
> ndctl zero-labels all
> ndctl enable-region all
>
> ...and that should get you back to square one.
Right, but that blows away all my namespaces. I was hoping to find a way
to just fix up (delete) what appeared to be extraneous labels.
That's something I've been thinking about as well, like a 'slot'
option to ndctl zero-labels to just try to limit the zeroing to single
bad slot. That said, since the kernel was in charge of the label area
it's a bug that it generated a layout that itself couldn't parse.
> If you are able to reproduce I'd like to see the state of the
DIMM
> label areas. You can dump them in json format with the following:
>
> ndctl read-labels -j all
That was in one of the attachments.
Ah, sorry, I overlooked that.
12:40 a.m.
Hi, Linda and Dan,
Linda Knippers <linda.knippers(a)hpe.com> writes:
Hi Dan,
I've got 4 NVDIMMs in an interleave set in a configuration that supports labels.
I'm running a 4.12 kernel with the latest ndctl.
I had three namespaces configured and all seemed well. When I configured the
fourth one, I made a mistake in the name so I hit control-c. I wasn't sure what
state I was in but according to what I could see with ndctl, it had created the
namespace but not enabled it, so I enabled it manually with ndctl and that
seemed ok.
Then I tried to use ndctl create-namespace to change the name, which failed
because the namespace was enabled so I disabled it and tried again. At some
point, not really sure where, I got this kernel warning:
# [ 5224.196085] nd namespace4.3: failed to track label: 4
I think I know how to reproduce this part reliably. Simply try to
create multiple namespaces in a single region at the same time:
# ndctl create-namespace -r regionN -m memory & ndctl create-namespace -r regionN -m
memory
That will lead to the dev_WARN_ONCE Linda mentioned. Then, the DIMM
will have an invalid label layout. On reboot, the dimm will be disabled
(these messages are printed when I reboot in this state):
[ 24.311419] nvdimm nmem1: nvdimm_init_config_data: len: 131072 rc: 0
[ 24.311420] nvdimm nmem1: config data size: 131072
[ 24.311421] nvdimm nmem1: __nd_label_validate: nsindex0 labelsize 1 invalid
[ 24.311422] nvdimm nmem1: __nd_label_validate: nsindex1 labelsize 1 invalid
[ 24.311425] nvdimm nmem1: : pmem-9221e8a3: 0x1f80000000 @ 0x10000000 reserve
[ 24.311427] nvdimm nmem1: : null: 0x0 @ 0x0 reserve
[ 24.311428] nvdimm nmem1: nvdimm_drvdata_release
[ 24.311430] nd_bus ndbus0: nvdimm.probe(nmem1) = -16
[ 24.311442] nvdimm: probe of nmem1 failed with error -16
Trying to enable nmem1 will result in EBUSY, since we're trying to
reserve address 0 (see the null entry above).
Unlike Linda's case, I can recover by zeroing the label space. However,
I don't have interleave enabled.
I've attached the result of read-labels for nmem1 below.
-Jeff
# ndctl read-labels -j nmem1
{
"dev":"nmem1",
"index":[
{
"signature":"NAMESPACE_INDEX",
"major":1,
"minor":2,
"labelsize":256,
"seq":1,
"nslot":510
},
{
"signature":"NAMESPACE_INDEX",
"major":1,
"minor":2,
"labelsize":256,
"seq":2,
"nslot":510
}
],
"label":[
{
"uuid":"9221e8a3-f43a-4204-86b1-e4bcd977ae27",
"name":"",
"slot":0,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"4c812805-e736-4876-bab2-eab15a847a9f",
"name":"",
"slot":1,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"4c812805-e736-4876-bab2-eab15a847a9f",
"name":"",
"slot":2,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"f7be9e94-1ba7-4f52-9090-12bcc08839fb",
"name":"",
"slot":3,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"f7be9e94-1ba7-4f52-9090-12bcc08839fb",
"name":"",
"slot":4,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"f7be9e94-1ba7-4f52-9090-12bcc08839fb",
"name":"",
"slot":5,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"b8bf5176-cc34-4b39-8d03-3a912e715366",
"name":"",
"slot":6,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"bd340ce8-5774-402b-b1ac-b82d590665d7",
"name":"",
"slot":7,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
}
]
}
read 1 nmem
4:12 a.m.
On 08/16/2017 12:40 PM, Jeff Moyer wrote:
Hi, Linda and Dan,
Linda Knippers <linda.knippers(a)hpe.com> writes:
> Hi Dan,
>
> I've got 4 NVDIMMs in an interleave set in a configuration that supports labels.
> I'm running a 4.12 kernel with the latest ndctl.
>
> I had three namespaces configured and all seemed well. When I configured the
> fourth one, I made a mistake in the name so I hit control-c. I wasn't sure what
> state I was in but according to what I could see with ndctl, it had created the
> namespace but not enabled it, so I enabled it manually with ndctl and that
> seemed ok.
>
> Then I tried to use ndctl create-namespace to change the name, which failed
> because the namespace was enabled so I disabled it and tried again. At some
> point, not really sure where, I got this kernel warning:
>
> # [ 5224.196085] nd namespace4.3: failed to track label: 4
I think I know how to reproduce this part reliably. Simply try to
create multiple namespaces in a single region at the same time:
# ndctl create-namespace -r regionN -m memory & ndctl create-namespace -r regionN -m
memory
That will lead to the dev_WARN_ONCE Linda mentioned. Then, the DIMM
will have an invalid label layout. On reboot, the dimm will be disabled
(these messages are printed when I reboot in this state):
[ 24.311419] nvdimm nmem1: nvdimm_init_config_data: len: 131072 rc: 0
[ 24.311420] nvdimm nmem1: config data size: 131072
[ 24.311421] nvdimm nmem1: __nd_label_validate: nsindex0 labelsize 1 invalid
[ 24.311422] nvdimm nmem1: __nd_label_validate: nsindex1 labelsize 1 invalid
[ 24.311425] nvdimm nmem1: : pmem-9221e8a3: 0x1f80000000 @ 0x10000000 reserve
[ 24.311427] nvdimm nmem1: : null: 0x0 @ 0x0 reserve
[ 24.311428] nvdimm nmem1: nvdimm_drvdata_release
[ 24.311430] nd_bus ndbus0: nvdimm.probe(nmem1) = -16
[ 24.311442] nvdimm: probe of nmem1 failed with error -16
Trying to enable nmem1 will result in EBUSY, since we're trying to
reserve address 0 (see the null entry above).
Unlike Linda's case, I can recover by zeroing the label space.
If you have to zero your labels, it's not really recovering. Or are you
able to recreate labels and not lose data that might have been in those
pmem ranges?
However, I don't have interleave enabled.
Perhaps Maurice can try this with interleave enabled.
-- ljk
I've attached the result of read-labels for nmem1 below.
-Jeff
# ndctl read-labels -j nmem1
{
"dev":"nmem1",
"index":[
{
"signature":"NAMESPACE_INDEX",
"major":1,
"minor":2,
"labelsize":256,
"seq":1,
"nslot":510
},
{
"signature":"NAMESPACE_INDEX",
"major":1,
"minor":2,
"labelsize":256,
"seq":2,
"nslot":510
}
],
"label":[
{
"uuid":"9221e8a3-f43a-4204-86b1-e4bcd977ae27",
"name":"",
"slot":0,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"4c812805-e736-4876-bab2-eab15a847a9f",
"name":"",
"slot":1,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"4c812805-e736-4876-bab2-eab15a847a9f",
"name":"",
"slot":2,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"f7be9e94-1ba7-4f52-9090-12bcc08839fb",
"name":"",
"slot":3,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"f7be9e94-1ba7-4f52-9090-12bcc08839fb",
"name":"",
"slot":4,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"f7be9e94-1ba7-4f52-9090-12bcc08839fb",
"name":"",
"slot":5,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"b8bf5176-cc34-4b39-8d03-3a912e715366",
"name":"",
"slot":6,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
},
{
"uuid":"bd340ce8-5774-402b-b1ac-b82d590665d7",
"name":"",
"slot":7,
"position":0,
"nlabel":1,
"isetcookie":62413126465469009,
"lbasize":0,
"dpa":268435456,
"rawsize":135291469824,
"type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb",
"abstraction_guid":"00000000-0000-0000-0000-000000000000"
}
]
}
read 1 nmem
4:30 a.m.
Linda Knippers <linda.knippers(a)hpe.com> writes:
On 08/16/2017 12:40 PM, Jeff Moyer wrote:
> [ 24.311419] nvdimm nmem1: nvdimm_init_config_data: len: 131072 rc: 0
> [ 24.311420] nvdimm nmem1: config data size: 131072
> [ 24.311421] nvdimm nmem1: __nd_label_validate: nsindex0 labelsize 1 invalid
> [ 24.311422] nvdimm nmem1: __nd_label_validate: nsindex1 labelsize 1 invalid
> [ 24.311425] nvdimm nmem1: : pmem-9221e8a3: 0x1f80000000 @ 0x10000000 reserve
> [ 24.311427] nvdimm nmem1: : null: 0x0 @ 0x0 reserve
> [ 24.311428] nvdimm nmem1: nvdimm_drvdata_release
> [ 24.311430] nd_bus ndbus0: nvdimm.probe(nmem1) = -16
> [ 24.311442] nvdimm: probe of nmem1 failed with error -16
>
> Trying to enable nmem1 will result in EBUSY, since we're trying to
> reserve address 0 (see the null entry above).
>
> Unlike Linda's case, I can recover by zeroing the label space.
If you have to zero your labels, it's not really recovering. Or are you
able to recreate labels and not lose data that might have been in those
pmem ranges?
Good point. Due to the corrupted label, I lose everything on that DIMM.
I think Dan mentioned maybe adding the ability to clear out bad labels.
We'd be better off not getting into this state in the first place,
obviously. ;-)
-Jeff
1401
days inactive
1424
days old
6 comments
3 participants
participants (3)
-
Dan Williams -
Jeff Moyer -
Linda Knippers