3:29 a.m.
Hello List,
We're running into something we haven't seen before... our OSTs (not
our MDT) seem to be out of inodes and we're seeing the following errors
in the OST logs:
LustreError: 12393:0:(filter.c:3953:filter_precreate()) create failed
rc = -28
LustreError: 12393:0:(filter.c:3953:filter_precreate()) Skipped 3 previous similar
messages
Here's the space count for this filesystem:
[root@warp e2fsprogs]# lfs df -i /lustre
UUID Inodes IUsed IFree IUse% Mounted on
data-MDT0000_UUID 159907840 44093755 115814085 28% /lustre[MDT:0]
data-OST0000_UUID 7629312 7629312 0 100% /lustre[OST:0]
data-OST0001_UUID 7629312 7629312 0 100% /lustre[OST:1]
data-OST0002_UUID 7629312 7629312 0 100% /lustre[OST:2]
data-OST0003_UUID 7629312 7629312 0 100% /lustre[OST:3]
data-OST0004_UUID 7629312 7629312 0 100% /lustre[OST:4]
data-OST0005_UUID 7629312 7629312 0 100% /lustre[OST:5]
data-OST0006_UUID 7629312 7629312 0 100% /lustre[OST:6]
data-OST0007_UUID 7629312 7629312 0 100% /lustre[OST:7]
data-OST0008_UUID 7629312 7629312 0 100% /lustre[OST:8]
data-OST0009_UUID 7629312 7629312 0 100% /lustre[OST:9]
data-OST000a_UUID 7629312 7629312 0 100% /lustre[OST:10]
data-OST000b_UUID 7629312 7629312 0 100% /lustre[OST:11]
data-OST000c_UUID 7629312 7629312 0 100% /lustre[OST:12]
data-OST000d_UUID 7629312 7629312 0 100% /lustre[OST:13]
data-OST000e_UUID 7629312 7629312 0 100% /lustre[OST:14]
data-OST000f_UUID 7629312 7629312 0 100% /lustre[OST:15]
data-OST0010_UUID 7629312 7629312 0 100% /lustre[OST:16]
data-OST0011_UUID 7629312 7629312 0 100% /lustre[OST:17]
data-OST0012_UUID 7629312 7629312 0 100% /lustre[OST:18]
data-OST0013_UUID 7629312 7629312 0 100% /lustre[OST:19]
data-OST0014_UUID 7629312 7629312 0 100% /lustre[OST:20]
data-OST0015_UUID 7629312 7629312 0 100% /lustre[OST:21]
data-OST0016_UUID 7629312 7629312 0 100% /lustre[OST:22]
filesystem summary: 159907840 44093755 115814085 28% /lustre
[root@warp e2fsprogs]# lfs df -h /lustre
UUID bytes Used Available Use% Mounted on
data-MDT0000_UUID 228.7G 8.3G 205.2G 4% /lustre[MDT:0]
data-OST0000_UUID 1.8T 646.2G 1.1T 37% /lustre[OST:0]
data-OST0001_UUID 1.8T 658.4G 1.1T 37% /lustre[OST:1]
data-OST0002_UUID 1.8T 667.4G 1.1T 38% /lustre[OST:2]
data-OST0003_UUID 1.8T 668.5G 1.1T 38% /lustre[OST:3]
data-OST0004_UUID 1.8T 667.8G 1.1T 38% /lustre[OST:4]
data-OST0005_UUID 1.8T 663.6G 1.1T 38% /lustre[OST:5]
data-OST0006_UUID 1.8T 659.2G 1.1T 37% /lustre[OST:6]
data-OST0007_UUID 1.8T 647.2G 1.1T 37% /lustre[OST:7]
data-OST0008_UUID 1.8T 648.6G 1.1T 37% /lustre[OST:8]
data-OST0009_UUID 1.8T 659.6G 1.1T 37% /lustre[OST:9]
data-OST000a_UUID 1.8T 673.2G 1.1T 38% /lustre[OST:10]
data-OST000b_UUID 1.8T 645.9G 1.1T 37% /lustre[OST:11]
data-OST000c_UUID 1.8T 665.7G 1.1T 38% /lustre[OST:12]
data-OST000d_UUID 1.8T 650.8G 1.1T 37% /lustre[OST:13]
data-OST000e_UUID 1.8T 646.2G 1.1T 37% /lustre[OST:14]
data-OST000f_UUID 1.8T 642.1G 1.1T 36% /lustre[OST:15]
data-OST0010_UUID 1.8T 645.6G 1.1T 37% /lustre[OST:16]
data-OST0011_UUID 1.8T 642.7G 1.1T 36% /lustre[OST:17]
data-OST0012_UUID 1.8T 640.4G 1.1T 36% /lustre[OST:18]
data-OST0013_UUID 1.8T 652.4G 1.1T 37% /lustre[OST:19]
data-OST0014_UUID 1.8T 640.3G 1.1T 36% /lustre[OST:20]
data-OST0015_UUID 1.8T 645.8G 1.1T 37% /lustre[OST:21]
data-OST0016_UUID 1.8T 643.7G 1.1T 36% /lustre[OST:22]
filesystem summary: 41.8T 14.7T 25.0T 37% /lustre
We use a default stripe count of "4" so the numbers seem to add up:
4x 44,093,755 mdt_inodes / 23 OSTs = 7,668,479 inodes per OST
This seems to be in the ballpark of what the OSTs are actually reporting.
Does this make sense or am I misunderstanding what I'm seeing?
This filesystem was originally deployed with Lustre 1.8.4. We used the
similar formatting options and stripecount (4) then. We consumed
91,269,360 inodes on the MDT (we ran it out of inodes) with that
filesystem without running out of inodes on the OSTs. Has something
changed with how the inode count is chosen for the OSTs since then?
We reformatted everything as part of the upgrade from 1.8.4 Here's how
we formatted things for Lustre 2.1.4.
for i in {b,c,e,f,g} ; do mkfs.lustre --reformat --fsname data
--param ost.quota_type=ug --mgsnode=mdt-3-40.ib@o2ib,mdt-3-40.coma-ib@o2ib1 --ost
/dev/sd${i}1 ; done
And this is what we used for 1.8.4:
mkfs.lustre --fsname=data --ost --reformat --mgsnode=172.16.1.3@o2ib
--mountfsoptions="stripe=256,errors=remount-ro,extents,mballoc"
--mkfsoptions="-E stride=32" /dev/sdd1
As you can see; neither of these explicitly change the inode count on
the OSTs (but 1.8.4 seemed to support our usage patterns).
Any pointers or suggestions as to what changed or what we missed would
be greatly appreciated. My suspicion is that we'll need to move our
data and reformat our OSTs to fix this though. Is that our only option
here?
Thanks much.
-Ed Walter
Carnegie Mellon University