I tried creating a directory /benchmarks/TestDFSIO under / directory and linked it as:ln -s /benchmarks /mnt/lustre.But I am getting this error:# bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO -read -nrFiles 10 -filesize 1000TestDFSIO.0.0.413/05/10 15:42:38 INFO fs.TestDFSIO: nrFiles = 1013/05/10 15:42:38 INFO fs.TestDFSIO: fileSize (MB) = 113/05/10 15:42:38 INFO fs.TestDFSIO: bufferSize = 100000013/05/10 15:42:38 INFO fs.TestDFSIO: creating control file: 1 mega bytes, 10 files13/05/10 15:42:38 INFO util.NativeCodeLoader: Loaded the native-hadoop library13/05/10 15:42:38 INFO fs.TestDFSIO: created control files for: 10 files13/05/10 15:42:39 INFO mapred.FileInputFormat: Total input paths to process : 1013/05/10 15:42:39 INFO mapred.JobClient: Running job: job_201305101207_000513/05/10 15:42:40 INFO mapred.JobClient: map 0% reduce 0%13/05/10 15:43:12 INFO mapred.JobClient: Task Id : attempt_201305101207_0005_m_000000_0, Status : FAILEDjava.io.FileNotFoundException: File file:/benchmarks/TestDFSIO/io_control/in_file_test_io_5 does not exist.at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)at org.apache.hadoop.mapred.Child$4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:416)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)at org.apache.hadoop.mapred.Child.main(Child.java:249)13/05/10 15:43:12 WARN mapred.JobClient: Error reading task outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0005_m_000000_0&filter=stdout13/05/10 15:43:12 WARN mapred.JobClient: Error reading task outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0005_m_000000_0&filter=stderr13/05/10 15:43:12 INFO mapred.JobClient: Task Id : attempt_201305101207_0005_m_000001_0, Status : FAILEDjava.io.FileNotFoundException: File file:/benchmarks/TestDFSIO/io_control/in_file_test_io_0 does not exist.at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)On Thu, May 9, 2013 at 8:32 PM, linux freaker <linuxfreaker@gmail.com> wrote:
Do I need to create benchmark folder under /mnt/lustre?On Thu, May 9, 2013 at 8:24 PM, Diep, Minh <minh.diep@intel.com> wrote:
You should be able to use the exact same command as in hdfs but make sure you create a symlink from /benchmark or /benchmarks (don't remember with the s or not) to lustre FS.
HTH-Minh
From: linux freaker <linuxfreaker@gmail.com>
Date: Thursday, May 9, 2013 1:16 AM
To: "hpdd-discuss@lists.01.org" <hpdd-discuss@lists.01.org>
Subject: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?
Any suggestion?