6:34 a.m.
I am setting up a small Lustre file system to use with our clusters. It
is based on Dell R415 (MDS) and Dell R515's for the OST's.
Each R515 has 14 disks. Two are configured as RAID 1 for the OS. The
other 12 are 2TB Near Line SAS, currently configured as a single RAID 6
array. There are four R515's as OST's.
They are connected to a DDR2 Infiniband network, along with all the
cluster nodes.
All servers are currently set up with Lustre 2.3.
I am looking for advice on how the RAID setup should be configured in
terms of stripe size, read caching, read/write policy, etc.
Right now I have a test file system configured with each OSS running
with a single OST configured from the 12 disk as RAID 6. Is there an
advantage to partitioning these in to smaller virtual disks and having
multiple OST's on each OSS?
--
Ray Muno
Information Technology Supervisor
University of Minnesota
Aerospace Engineering and Mechanics Mechanical Engineering
110 Union St. S.E. 111 Church Street SE
Minneapolis, MN 55455 Minneapolis, MN 55455
9:57 p.m.
Hi,
there were several discussions of the topic on the Lustre mailing lists, e.g.
"[Lustre-discuss]
clarification on mkfs.lustre options, 30.7.2010", "Re: [Lustre-discuss] mkfs
options/tuning for RAID
based OSTs, 20.10.2010"
As far as I understand, everything derives from Lustre's basic 1MB rpc size. In the
end, the number of
data disks in a RAID should be a power of 2, so Raid6 = 4+2, 8+2, etc.
According to this, your Raids have two disks too many...
Also, in the quoted discussions, Andreas Dilger made a strong point about why partitioning
is evil for
performance.
In fact I have just tested a number of new servers which came with Raid 6, each 10 disks,
but a "strip
size" (LSI-controller-speak) of 512kB. I have since recreated the Raids, changing
this strip size to
128kB (8 x 128 kB = 1MB), and the results as seen by ost-survey ( from lustre-iokit) show
three times
more throughput for 100MB files. (A bit too good for my taste, but at least it points to
the right
direction.)
Cheers,
Thomas
On 03.04.2013 21:34, Ray Muno wrote:
I am setting up a small Lustre file system to use with our clusters.
It is based on Dell R415 (MDS)
and Dell R515's for the OST's.
Each R515 has 14 disks. Two are configured as RAID 1 for the OS. The other 12 are 2TB
Near Line SAS,
currently configured as a single RAID 6 array. There are four R515's as OST's.
They are connected to a DDR2 Infiniband network, along with all the cluster nodes.
All servers are currently set up with Lustre 2.3.
I am looking for advice on how the RAID setup should be configured in terms of stripe
size, read
caching, read/write policy, etc.
Right now I have a test file system configured with each OSS running with a single OST
configured from
the 12 disk as RAID 6. Is there an advantage to partitioning these in to smaller virtual
disks and
having multiple OST's on each OSS?
11:58 p.m.
On 04/04/2013 05:57 AM, Thomas Roth wrote:
Hi,
there were several discussions of the topic on the Lustre mailing lists, e.g.
"[Lustre-discuss]
clarification on mkfs.lustre options, 30.7.2010", "Re: [Lustre-discuss] mkfs
options/tuning for RAID
based OSTs, 20.10.2010"
As far as I understand, everything derives from Lustre's basic 1MB rpc size. In the
end, the number of
data disks in a RAID should be a power of 2, so Raid6 = 4+2, 8+2, etc.
According to this, your Raids have two disks too many...
Also, in the quoted discussions, Andreas Dilger made a strong point about why
partitioning is evil for
performance.
In fact I have just tested a number of new servers which came with Raid 6, each 10 disks,
but a "strip
size" (LSI-controller-speak) of 512kB. I have since recreated the Raids, changing
this strip size to
128kB (8 x 128 kB = 1MB), and the results as seen by ost-survey ( from lustre-iokit) show
three times
more throughput for 100MB files. (A bit too good for my taste, but at least it points to
the right
direction.)
Cheers,
Thomas
I found some of those discussions last night after I posted.
The "power of 2" aspect has finally sunk in.
With 12 disks, I can do two sets of RAID6 4+2 or one set of 8+2 and have
two hot spares. From a space efficiency standpoint, it looks like I
end up with a similar amount of usable disk space. The 8+2 with the hot
spares would seem to be the better choice.
Is there any reason I would pick the two sets of RAID6 4+2 over the one
RAID6 8+2? The latter seems to be the better choice.
--
Ray Muno
Information Technology Supervisor
University of Minnesota
Aerospace Engineering and Mechanics Mechanical Engineering
110 Union St. S.E. 111 Church Street SE
Minneapolis, MN 55455 Minneapolis, MN 55455
12:51 a.m.
On Thu, 04 Apr 2013 07:58:11 -0500
Ray Muno <muno(a)aem.umn.edu> wrote:
Is there any reason I would pick the two sets of RAID6 4+2 over the
one
RAID6 8+2? The latter seems to be the better choice.
If you are going with RAID6 I wouldn't bother with hot spares. The
2*4+2 will have theoretically better performance since there are more
drives involved (instead of 2 sitting idle with a single 8+2).
If you are using an LSI controller there may be other considerations
like "tray protection". If a single tray fails, how many drives in
your raid group are in a single tray?
3454
days inactive
3455
days old
3 comments
3 participants
participants (3)
-
chas williams - CONTRACTOR -
Ray Muno -
Thomas Roth