Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?

Try this Rm –f /benchmarks/* Bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO –write –nrFiles 10 –fileSize 1000 From: linux freaker <linuxfreaker(a)gmail.com&gt; Date: Friday, May 10, 2013 8:51 AM To: Minh Diep <minh.diep(a)intel.com&gt;, &quot;hpdd-discuss(a)lists.01.org&quot; < hpdd-discuss(a)lists.01.org&gt; Subject: Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre? I can see the error regarding org.apache.fs. am I missing anything? On 10 May 2013 16:09, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote: > I tried running this way too but no luck: > > [root@lustreclient1 hadoop]# bin/hadoop jar hadoop-test-1.1.1.jar > TestDFSIO -write -nrFiles 10 -fileSize 1000 -resFile > /mnt/lustre/benchmarks/TestDFSIO/io_control > TestDFSIO.0.0.4 > 13/05/10 16:05:29 INFO fs.TestDFSIO: nrFiles = 10 > 13/05/10 16:05:29 INFO fs.TestDFSIO: fileSize (MB) = 1000 > 13/05/10 16:05:29 INFO fs.TestDFSIO: bufferSize = 1000000 > 13/05/10 16:05:30 INFO fs.TestDFSIO: creating control file: 1000 mega > bytes, 10 files > 13/05/10 16:05:30 INFO util.NativeCodeLoader: Loaded the native-hadoop > library > 13/05/10 16:05:30 INFO fs.TestDFSIO: created control files for: 10 files > 13/05/10 16:05:30 INFO mapred.FileInputFormat: Total input paths to > process : 10 > 13/05/10 16:05:30 INFO mapred.JobClient: Running job: > job_201305101207_0007 > 13/05/10 16:05:31 INFO mapred.JobClient: map 0% reduce 0% > 13/05/10 16:06:01 INFO mapred.JobClient: map 20% reduce 0% > 13/05/10 16:06:07 INFO mapred.JobClient: map 40% reduce 0% > 13/05/10 16:06:08 INFO mapred.JobClient: map 40% reduce 13% > 13/05/10 16:06:12 INFO mapred.JobClient: map 50% reduce 13% > 13/05/10 16:06:13 INFO mapred.JobClient: map 60% reduce 13% > 13/05/10 16:06:18 INFO mapred.JobClient: map 60% reduce 16% > 13/05/10 16:06:24 INFO mapred.JobClient: map 60% reduce 20% > 13/05/10 16:06:33 INFO mapred.JobClient: Task Id : > attempt_201305101207_0007_m_000004_0, Status : FAILED > java.io.FileNotFoundException: File > file:/benchmarks/TestDFSIO/io_control/in_file_test_io_6 does not exist. > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) > at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) > at > org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) > at > org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output > http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... > 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output > http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... > 13/05/10 16:06:33 INFO mapred.JobClient: Task Id : > attempt_201305101207_0007_m_000005_0, Status : FAILED > java.io.FileNotFoundException: File > file:/benchmarks/TestDFSIO/io_control/in_file_test_io_9 does not exist. > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) > at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) > at > org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) > at > org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output > http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... > 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task output > http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... > 13/05/10 16:06:37 INFO mapred.JobClient: map 80% reduce 20% > > > > On Fri, May 10, 2013 at 3:47 PM, linux freaker <linuxfreaker(a)gmail.com&gt;wrote: > >> I tried creating a directory /benchmarks/TestDFSIO under / directory and >> linked it as: >> ln -s /benchmarks /mnt/lustre. >> But I am getting this error: >> >> >> # bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO -read -nrFiles 10 >> -filesize 1000 >> TestDFSIO.0.0.4 >> 13/05/10 15:42:38 INFO fs.TestDFSIO: nrFiles = 10 >> 13/05/10 15:42:38 INFO fs.TestDFSIO: fileSize (MB) = 1 >> 13/05/10 15:42:38 INFO fs.TestDFSIO: bufferSize = 1000000 >> 13/05/10 15:42:38 INFO fs.TestDFSIO: creating control file: 1 mega >> bytes, 10 files >> 13/05/10 15:42:38 INFO util.NativeCodeLoader: Loaded the native-hadoop >> library >> 13/05/10 15:42:38 INFO fs.TestDFSIO: created control files for: 10 files >> 13/05/10 15:42:39 INFO mapred.FileInputFormat: Total input paths to >> process : 10 >> 13/05/10 15:42:39 INFO mapred.JobClient: Running job: >> job_201305101207_0005 >> 13/05/10 15:42:40 INFO mapred.JobClient: map 0% reduce 0% >> 13/05/10 15:43:12 INFO mapred.JobClient: Task Id : >> attempt_201305101207_0005_m_000000_0, Status : FAILED >> java.io.FileNotFoundException: File >> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_5 does not exist. >> at >> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) >> at >> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) >> at >> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) >> at >> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) >> at >> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) >> at >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:416) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) >> at org.apache.hadoop.mapred.Child.main(Child.java:249) >> >> 13/05/10 15:43:12 WARN mapred.JobClient: Error reading task output >> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >> 13/05/10 15:43:12 WARN mapred.JobClient: Error reading task output >> http://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_2... >> 13/05/10 15:43:12 INFO mapred.JobClient: Task Id : >> attempt_201305101207_0005_m_000001_0, Status : FAILED >> java.io.FileNotFoundException: File >> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_0 does not exist. >> at >> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) >> at >> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) >> at >> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) >> at >> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) >> at >> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) >> >> >> >> On Thu, May 9, 2013 at 8:32 PM, linux freaker <linuxfreaker(a)gmail.com&gt;wrote: >> >>> Do I need to create benchmark folder under /mnt/lustre? >>> >>> >>> On Thu, May 9, 2013 at 8:24 PM, Diep, Minh <minh.diep(a)intel.com&gt; wrote: >>> >>>> You should be able to use the exact same command as in hdfs but make >>>> sure you create a symlink from /benchmark or /benchmarks (don't remember >>>> with the s or not) to lustre FS. >>>> >>>> HTH >>>> -Minh >>>> >>>> From: linux freaker <linuxfreaker(a)gmail.com&gt; >>>> Date: Thursday, May 9, 2013 1:16 AM >>>> To: &quot;hpdd-discuss(a)lists.01.org&quot; <hpdd-discuss(a)lists.01.org&gt; >>>> Subject: [HPDD-discuss] How to run TestDfsio on hadoop running on >>>> lustre? >>>> >>>> Any suggestion? >>>> >>> >>> >> >