編譯hadoop eclipse的插件（hadoop1.0）

在hadoop-1.0中，不像0.20.2版本，有現成的eclipse-plugin源碼包，而是在hadoop_home/src

/contrib/eclipse-plugin目錄下放置了eclipse插件的源碼，這篇文章

，我想詳細記錄一下自己是如何編譯此源碼生成适用于hadoop1.0的eclipse插件

1、安裝環境

作業系統：ubuntu14.4

軟體：

eclipse

java

hadoop 1.0

2、編譯步驟

（1）首先下載下傳ant與ivy的安裝包

将安裝包解壓縮到指定的目錄，然後将ivy包中的ivy-2.2.0.jar包放到ant安裝目錄的lib目錄下，然後在/etc/profile中添加以下内容以設定配置環境：

export ant_home=/home/wu/opt/apache-ant-1.8.3 export path=”anthome/bin:

path”

（2）終端轉到hadoop安裝目錄下，執行ant compile，結果如下：

……………………

compile:

[echo] contrib: vaidya

[javac]

/home/wu/opt/hadoop-1.0.1/src/contrib/build-contrib.xml:185: warning:

‘includeantruntime’ was not set, defaulting to build.sysclasspath=last;

set to false for repeatable builds

[javac] compiling 14 source files to /home/wu/opt/hadoop-1.0.1/build/contrib/vaidya/classes

note:

/home/wu/opt/hadoop-1.0.1/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/statistics/job/jobstatistics.java

uses unchecked or unsafe operations.

[javac] note: recompile with -xlint:unchecked for details.

compile-ant-tasks:

/home/wu/opt/hadoop-1.0.1/build.xml:2170: warning: ‘includeantruntime’

was not set, defaulting to build.sysclasspath=last; set to false for

repeatable builds

[javac] compiling 5 source files to /home/wu/opt/hadoop-1.0.1/build/ant

build successful

total time: 12 minutes 29 seconds

可以看到編譯成功！花的時間比較長，可以泡壺茶休息一下～～

（3）再将終端定位到hadoop_home/src/contrib/eclipse-plugin，然後執行下面的指令：

ant -declipse.home=/home/wu/opt/eclipse -dversion=1.0.1 jar

編譯完成後就可以找到eclipse插件了

3、安裝步驟

（1）僞分布式的配置過程也很簡單，隻需要修改幾個檔案，在代碼的conf檔案夾内，就可以找到下面幾個配置檔案，具體過程我就不多說了，這裡列出我的配置：

core-site.xml

hdfs-site.xml

mapred-site.xml

進入conf檔案夾，修改配置檔案：hadoop-env.sh，将裡面的java_home注釋打開，并把裡面的位址配置正确

（2）運作hadoop

進入hadoop目錄，首次運作，需要格式化檔案系統，輸入指令：

輸入指令，啟動所有進出：

關閉hadoop可以用：

bin/stop-all.sh

最後驗證hadoop是否安裝成功，打開浏覽器，分别輸入：

http://localhost:50030/ （mapreduce的web頁面）

http://localhost:50070/ （hdfs的web頁面）

用jps指令看一下有幾個java程序在運作，如果是下面幾個就正常了

wu@ubuntu:~/opt/hadoop-1.0.1$ jps

4113 secondarynamenode

4318 tasktracker

3984 datanode

3429 

3803 namenode

4187 jobtracker

4415 jps

系統啟動正常後，現在來跑個程式:

出現一長串的運作過程：

****hdfs://localhost:9000/user/wu/in

15/05/29 10:51:41 info input.fileinputformat: total input paths to process : 2

15/05/29 10:51:42 info mapred.jobclient: running job: job_201505291029_0001

15/05/29 10:51:43 info mapred.jobclient: map 0% reduce 0%

15/05/29 10:52:13 info mapred.jobclient: map 100% reduce 0%

15/05/29 10:52:34 info mapred.jobclient: map 100% reduce 100%

15/05/29 10:52:39 info mapred.jobclient: job complete: job_201505291029_0001

15/05/29 10:52:39 info mapred.jobclient: counters: 29

15/05/29 10:52:39 info mapred.jobclient: job counters 

15/05/29 10:52:39 info mapred.jobclient: launched reduce tasks=1

15/05/29 10:52:39 info mapred.jobclient: slots_millis_maps=43724

15/05/29 10:52:39 info mapred.jobclient: total time spent by all reduces waiting after reserving slots (ms)=0

15/05/29 10:52:39 info mapred.jobclient: total time spent by all maps waiting after reserving slots (ms)=0

15/05/29 10:52:39 info mapred.jobclient: launched map tasks=2

15/05/29 10:52:39 info mapred.jobclient: data-local map tasks=2

15/05/29 10:52:39 info mapred.jobclient: slots_millis_reduces=20072

15/05/29 10:52:39 info mapred.jobclient: file output format counters 

15/05/29 10:52:39 info mapred.jobclient: bytes written=25

15/05/29 10:52:39 info mapred.jobclient: filesystemcounters

15/05/29 10:52:39 info mapred.jobclient: file_bytes_read=55

15/05/29 10:52:39 info mapred.jobclient: hdfs_bytes_read=239

15/05/29 10:52:39 info mapred.jobclient: file_bytes_written=64837

15/05/29 10:52:39 info mapred.jobclient: hdfs_bytes_written=25

15/05/29 10:52:39 info mapred.jobclient: file input format counters 

15/05/29 10:52:39 info mapred.jobclient: bytes read=25

15/05/29 10:52:39 info mapred.jobclient: map-reduce framework

15/05/29 10:52:39 info mapred.jobclient: map output materialized bytes=61

15/05/29 10:52:39 info mapred.jobclient: map input records=2

15/05/29 10:52:39 info mapred.jobclient: reduce shuffle bytes=61

15/05/29 10:52:39 info mapred.jobclient: spilled records=8

15/05/29 10:52:39 info mapred.jobclient: map output bytes=41

15/05/29 10:52:39 info mapred.jobclient: cpu time spent (ms)=7330

15/05/29 10:52:39 info mapred.jobclient: total committed heap usage (bytes)=247275520

15/05/29 10:52:39 info mapred.jobclient: combine input records=4

15/05/29 10:52:39 info mapred.jobclient: split_raw_bytes=214

15/05/29 10:52:39 info mapred.jobclient: reduce input records=4

15/05/29 10:52:39 info mapred.jobclient: reduce input groups=3

15/05/29 10:52:39 info mapred.jobclient: combine output records=4

15/05/29 10:52:39 info mapred.jobclient: physical memory (bytes) snapshot=338845696

15/05/29 10:52:39 info mapred.jobclient: reduce output records=3

15/05/29 10:52:39 info mapred.jobclient: virtual memory (bytes) snapshot=1139433472

15/05/29 10:52:39 info mapred.jobclient: map output records=4

檢視out檔案夾：

wu@ubuntu:~/opt/hadoop-1.0.1$ bin/hadoop dfs -cat out/*

hadoop 1

hello 2

world 1