Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?

I believe you tried –read before did a –write. From: linux freaker <linuxfreaker(a)gmail.com&gt; Date: Friday, May 10, 2013 9:33 AM To: Minh Diep <minh.diep(a)intel.com&gt; Cc: &quot;hpdd-discuss(a)lists.01.org&quot; <hpdd-discuss(a)lists.01.org&gt; Subject: Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre? I have already tried it in first stage but it didn't work. On 10 May 2013 21:37, "Diep, Minh" <minh.diep(a)intel.com&gt; wrote: > Try this > Rm –f /benchmarks/* > Bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO –write –nrFiles 10 > –fileSize 1000 > > From: linux freaker <linuxfreaker(a)gmail.com&gt; > Date: Friday, May 10, 2013 8:51 AM > To: Minh Diep <minh.diep(a)intel.com&gt;, &quot;hpdd-discuss(a)lists.01.org&quot; < > hpdd-discuss(a)lists.01.org&gt; > Subject: Re: [HPDD-discuss] How to run TestDfsio on hadoop running on > lustre? > > I can see the error regarding org.apache.fs. am I missing anything? > On 10 May 2013 16:09, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote: > >> I tried running this way too but no luck: >> >> [root@lustreclient1 hadoop]# bin/hadoop jar hadoop-test-1.1.1.jar >> TestDFSIO -write -nrFiles 10 -fileSize 1000 -resFile >> /mnt/lustre/benchmarks/TestDFSIO/io_control >> TestDFSIO.0.0.4 >> 13/05/10 16:05:29 INFO fs.TestDFSIO: nrFiles = 10 >> 13/05/10 16:05:29 INFO fs.TestDFSIO: fileSize (MB) = 1000 >> 13/05/10 16:05:29 INFO fs.TestDFSIO: bufferSize = 1000000 >> 13/05/10 16:05:30 INFO fs.TestDFSIO: creating control file: 1000 mega >> bytes, 10 files >> 13/05/10 16:05:30 INFO util.NativeCodeLoader: Loaded the native-hadoop >> library >> 13/05/10 16:05:30 INFO fs.TestDFSIO: created control files for: 10 files >> 13/05/10 16:05:30 INFO mapred.FileInputFormat: Total input paths to >> process : 10 >> 13/05/10 16:05:30 INFO mapred.JobClient: Running job: >> job_201305101207_0007 >> 13/05/10 16:05:31 INFO mapred.JobClient: map 0% reduce 0% >> 13/05/10 16:06:01 INFO mapred.JobClient: map 20% reduce 0% >> 13/05/10 16:06:07 INFO mapred.JobClient: map 40% reduce 0% >> 13/05/10 16:06:08 INFO mapred.JobClient: map 40% reduce 13% >> 13/05/10 16:06:12 INFO mapred.JobClient: map 50% reduce 13% >> 13/05/10 16:06:13 INFO mapred.JobClient: map 60% reduce 13% >> 13/05/10 16:06:18 INFO mapred.JobClient: map 60% reduce 16% >> 13/05/10 16:06:24 INFO mapred.JobClient: map 60% reduce 20% >> 13/05/10 16:06:33 INFO mapred.JobClient: Task Id : >> attempt_201305101207_0007_m_000004_0, Status : FAILED >> java.io.FileNotFoundException: File >> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_6 does not exist. >> at >> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) >> at >> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) >> at >> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) >> at >> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) >> at >> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) >> at >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:416) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) >> at org.apache.hadoop.mapred.Child.main(Child.java:249) >> >> 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output >> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >> 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output >> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >> 13/05/10 16:06:33 INFO mapred.JobClient: Task Id : >> attempt_201305101207_0007_m_000005_0, Status : FAILED >> java.io.FileNotFoundException: File >> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_9 does not exist. >> at >> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) >> at >> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) >> at >> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) >> at >> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) >> at >> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) >> at >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:416) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) >> at org.apache.hadoop.mapred.Child.main(Child.java:249) >> >> 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output >> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >> 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output >> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >> 13/05/10 16:06:37 INFO mapred.JobClient: map 80% reduce 20% >> >> >> >> On Fri, May 10, 2013 at 3:47 PM, linux freaker <linuxfreaker(a)gmail.com&gt;wrote: >> >>> I tried creating a directory /benchmarks/TestDFSIO under / directory >>> and linked it as: >>> ln -s /benchmarks /mnt/lustre. >>> But I am getting this error: >>> >>> >>> # bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO -read -nrFiles 10 >>> -filesize 1000 >>> TestDFSIO.0.0.4 >>> 13/05/10 15:42:38 INFO fs.TestDFSIO: nrFiles = 10 >>> 13/05/10 15:42:38 INFO fs.TestDFSIO: fileSize (MB) = 1 >>> 13/05/10 15:42:38 INFO fs.TestDFSIO: bufferSize = 1000000 >>> 13/05/10 15:42:38 INFO fs.TestDFSIO: creating control file: 1 mega >>> bytes, 10 files >>> 13/05/10 15:42:38 INFO util.NativeCodeLoader: Loaded the native-hadoop >>> library >>> 13/05/10 15:42:38 INFO fs.TestDFSIO: created control files for: 10 files >>> 13/05/10 15:42:39 INFO mapred.FileInputFormat: Total input paths to >>> process : 10 >>> 13/05/10 15:42:39 INFO mapred.JobClient: Running job: >>> job_201305101207_0005 >>> 13/05/10 15:42:40 INFO mapred.JobClient: map 0% reduce 0% >>> 13/05/10 15:43:12 INFO mapred.JobClient: Task Id : >>> attempt_201305101207_0005_m_000000_0, Status : FAILED >>> java.io.FileNotFoundException: File >>> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_5 does not exist. >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) >>> at >>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >>> at >>> org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) >>> at >>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) >>> at >>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) >>> at >>> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) >>> at >>> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) >>> at >>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) >>> at >>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) >>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:416) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) >>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>> >>> 13/05/10 15:43:12 WARN mapred.JobClient: Error reading task output >>> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >>> 13/05/10 15:43:12 WARN mapred.JobClient: Error reading task output >>> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >>> 13/05/10 15:43:12 INFO mapred.JobClient: Task Id : >>> attempt_201305101207_0005_m_000001_0, Status : FAILED >>> java.io.FileNotFoundException: File >>> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_0 does not exist. >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) >>> at >>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >>> at >>> org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) >>> at >>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) >>> at >>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) >>> at >>> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) >>> at >>> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) >>> at >>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) >>> >>> >>> >>> On Thu, May 9, 2013 at 8:32 PM, linux freaker <linuxfreaker(a)gmail.com&gt;wrote: >>> >>>> Do I need to create benchmark folder under /mnt/lustre? >>>> >>>> >>>> On Thu, May 9, 2013 at 8:24 PM, Diep, Minh <minh.diep(a)intel.com&gt;wrote: >>>> >>>>> You should be able to use the exact same command as in hdfs but >>>>> make sure you create a symlink from /benchmark or /benchmarks (don't >>>>> remember with the s or not) to lustre FS. >>>>> >>>>> HTH >>>>> -Minh >>>>> >>>>> From: linux freaker <linuxfreaker(a)gmail.com&gt; >>>>> Date: Thursday, May 9, 2013 1:16 AM >>>>> To: &quot;hpdd-discuss(a)lists.01.org&quot; <hpdd-discuss(a)lists.01.org&gt; >>>>> Subject: [HPDD-discuss] How to run TestDfsio on hadoop running on >>>>> lustre? >>>>> >>>>> Any suggestion? >>>>> >>>> >>>> >>> >>