hadoop--基壓測試

在安裝完叢集後,我們都需要先對叢集做一些測試,下面講解測試讀寫的性能

寫性能

包名:

Apache:

hadoop-mapreduce-client-jobclient-2.7.5-tests.jar

CDH:

hadoop-mapreduce-client-jobclient-3.0.0-cdh6.2.0-tests.jar

包路徑:

/home/hadoop-jrq/bigdata/hadoop-2.7.5/share/hadoop/mapreduce

CDH路徑:

/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars

執行指令:

hadoop jar hadoop-mapreduce-client-jobclient-2.7.5-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB

CDH同理 寫10個128M的資料檔案

結果:

INFO fs.TestDFSIO: ----- TestDFSIO ----- : write

INFO fs.TestDFSIO: Date & time: Thu May 02 11:45:23 CST 2019

INFO fs.TestDFSIO: Number of files: 10 --十個檔案

INFO fs.TestDFSIO: Total MBytes processed: 1280.0 --總大小1280M

INFO fs.TestDFSIO: Throughput mb/sec: 10.69751115716984 --吞吐量 每秒10.69M

INFO fs.TestDFSIO: Average IO rate mb/sec: 14.91699504852295 --平均IO情況

INFO fs.TestDFSIO: IO rate std deviation: 11.160882132355928

INFO fs.TestDFSIO: Test exec time sec: 52.315 --總運作時間

讀性能

指令:

hadoop jar hadoop-mapreduce-client-jobclient-2.7.5-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB

結果:

INFO fs.TestDFSIO: ----- TestDFSIO ----- : read

INFO fs.TestDFSIO: Date & time: Thu May 02 11:56:36 CST 2019

INFO fs.TestDFSIO: Number of files: 10 --檔案數

INFO fs.TestDFSIO: Total MBytes processed: 1280.0 --總大小

INFO fs.TestDFSIO: Throughput mb/sec: 16.001000062503905 --吞吐量

INFO fs.TestDFSIO: Average IO rate mb/sec: 17.202795028686523 --平均IO情況

INFO fs.TestDFSIO: IO rate std deviation: 4.881590515873911

INFO fs.TestDFSIO: Test exec time sec: 49.116 --總時間

删除測試資料

hadoop jar hadoop-mapreduce-client-jobclient-2.7.2-tests.jar TestDFSIO -clean

測試排序程式

1.使用RandomWriter來産生随機數，每個節點運作10個Map任務，每個Map産生大約1G大小的二進制随機數

hadoop jar hadoop-mapreduce-examples-2.7.5.jar randomwriter random-data

2.執行Sort程式

hadoop jar hadoop-mapreduce-examples-2.7.5.jar sort random-data sorted-data

3.驗證資料是否真正排好序了

hadoop jar hadoop-mapreduce-examples-2.7.5.jar testmapredsort -sortInput random-data -sortOutput sorted-data