Per the discussion on the meeting yesterday, I’m going to send you the instructions to set up Lustre to run the performance benchmark.
1. Please check out ZFS on linux code and apply the following two patches:
2. Rebuild and install SPL and ZFS RPMs; be sure that devel packages are installed as well that will be required to build Lustre.
3. It turned out that most of Alex’s patches have already been landed into lustre 2.9; to check out the latest Lustre code, run the command:
git clone git://git.whamcloud.com/fs/lustre-release.git
4. Apply the following two patches to Lustre:
5. Rebuild Lustre and make sure USEROBJ_ACCOUNTING is enabled in Lustre:
nfsroot: lustre$ sh autogen.sh
nfsroot: lustre$ ./configure --with-linux=/lib/modules/`name -r`/build/ --with-zfs —without-ldiskfs
nfsroot: lustre$ grep USEROBJ config.h
#define HAVE_DMU_USEROBJ_ACCOUNTING 1
nfsroot: lustre$ make rpms
if things go okay, you are going to see a bunch of RPM packages generated. Please install them on test nodes.
6. Start Lustre. Usually I use a configure file so that it’s easier to edit and change the environment settings. I can start it by the following command:
root@c1: tests# pwd
root@c1: tests# ls cfg/titan.sh
root@c1: tests# FSTYPE=zfs NAME=titan bash llmount.sh
I started Lustre from one of client node.
I’ve attached my config file for your reference, so you can adjust the settings for your own testing env, most likely you need to change the following variable in the file:
CLIENTS=“” -> list of your client nodes, you only need 1 client initially
mds_HOST -> the name of your MDS node
MDSCOUNT -> number of MDS target, a MDS node can have multiple targets, but let’s set it to 1 for now
MDSDEV1 -> the target drive. Now that you use ram device, you can set it to ‘/dev/ram0’ for example
mgs_HOST -> set it to the same value of mds_HOST
ost_HOST -> the node where OST targets resides. I would suggest a separate node from the MDS
OSTCOUNT -> let’s set it to 1 for now
OSTDEV1 -> the corresponding target drive for OST 1
Please make sure that passwordless ssh access has turned out on those nodes. You can adjust the setting by changing
PDSH='pdsh -Rssh -S -w’
in the config file.
7. Run mds-survey that is included in the lustre-iokit RPM package
root@c1: tests# NAME=titan FSTYPE=zfs ONLY=1 bash mds-survey.sh
This test will only create objects on the MDT side, which is the exact issue this project is addressing. Lustre stores data on OSTs.
If everything goes okay, you’re going to see results like follows:
== mds-survey test 1: Metadata survey with zero-stripe == 14:03:19 (1474664599)
mdt 1 file 50000 dir 2 thr 2 create 49959.98 [ 49959.98, 49959.98] ... destroy 33335.87 [ 33335.87, 33335.87]
mdt 1 file 50000 dir 2 thr 4 create 51926.53 [ 51926.53, 51926.53] … destroy 50550.29 [ 50550.29, 50550.29]
The above results show creating speed at ~50K files per second for 2 threads to create files into 2 directories; and ~52K files per second for 4 threads to create files into 2 directories. This is not bad as I ran the test on one node with KVMs, and my CPU is core i7.
For detailed information about mds-survey, please take a look at http://doc.lustre.org/lustre_manual.pdf and chapter 25.5 is about mds-survey. This is a bash script so you can also read the source code.
8. To unmount Lustre
root@c1: tests# NAME=titan FSTYPE=zfs bash llmountcleanup.sh
#1: since you’re going to set up an developing env, and most likely you only need to change ZFS, so you don’t have to compile ZFS RPMs every time. After you have made a change to ZFS, you can just remove corresponding zfs modules under /lib/modules/`name -r`/extra/zfs, and then go to $ZFS_SRC/module, and then run command 'make install’ to install zfs modules, and then run ‘modprobe -r zfs’ and then ‘depmod -a; modprobe zfs’ to reload the modules. You need to unmount Lustre before doing those.
#2: If you want some immediate help, please ping me on Skype(jinshan.xiong), or google chat(jinshan.xiong(a)gmail.com)