大資料之----部署安裝編譯打包hadoop終極篇

------------------------------軟體版本-------------------------------------- RHEL6.8 hadoop2.8.1 apache-maven-3.3.9 findbugs-1.3.9 protobuf-2.5.0.tar.gz jdk-8u45

------------------------------軟體版本---------------------------------------

大資料概述 可以用“5V + 1C”來概括：

Variety 多樣化 Volume 海量 Velocity 快速 Vitality 靈活 Value 價值性 Complexity 複雜

1.Hadoop生态

宏觀: Hadoop為主的生态圈 hadoop flume........

狹義: apache hadoop  hadoop.apache.org

2.Hadoop(存儲+計算+資源和作業排程)

hadoop1.x  

    HDFS      存儲

    MapReduce 計算+資源和作業排程

hadoop2.x  企業正在用

    MapReduce 計算

    YARN      資源和作業排程平台 計算元件都會on yarn

hadoop3.x  ???

EC技術：Erasure Encoding 簡稱EC，是Hadoop3給HDFS拓展的一種新特性，用來解決存儲空間檔案。

YARN：提供YARN的時間軸服務V.2，以便使用者和開發人員可以對其進行測試，并提供回報意見。

優化Hadoop Shell腳本

重構Hadoop Client Jar包

支援随機Container

MapReduce任務級本地優化

支援多個NameNode

部分預設服務端口被改變

支援檔案系統連接配接器

DataNode内部添加了負載均衡

重構背景程式和任務對管理

2.Maven部署

blog

2.1解壓

[root@hadoop1 softwore]# pwd

/opt/softwore

[root@hadoop1 softwore]# ls

apache-maven-3.3.9-bin.zip  hadoop-2.8.1-src.tar.gz  jdk-8u45-linux-x64.gz

findbugs-1.3.9.zip          hadoop-2.8.1.tar.gz      protobuf-2.5.0.tar.gz

[root@hadoop1 softwore]# unzip apache-maven-3.3.9-bin.zip

--配置環境變量

[root@hadoop1 softwore]#

vi /etc/profile

export MAVEN_HOME=/opt/software/apache-maven-3.3.9

export MAVEN_OPTS="-Xms256m -Xmx512m"

export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop1 softwore]# source /etc/profile

[root@hadoop1 apache-maven-3.3.9]# mvn --version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)

Maven home: /opt/software/apache-maven-3.3.9

Java version: 1.8.0_45, vendor: Oracle Corporation

Java home: /usr/java/jdk1.8.0_45/jre

Default locale: zh_CN, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

Maven部署為二進制部署，解壓配置環境變量

2.2配置mavne目錄

2.3檢視配置檔案和 解壓我們準備好的倉庫檔案

3.Hadoop編譯

編譯hadoop 通過Requirements我看可以知道需要安裝jdk maven findbugs protocolBuffer cmake Zlib openssl-devel 等軟體

3.1解壓

3.2檢視pom.xml

3.3檢視BUILDING.txt

Requirements:編譯軟體環境要求

* Unix System

* JDK 1.7+

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac

* Zlib devel (if compiling native code)

* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)

* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

3.4 JDK部署

[root@hadoop1 softwore]# tar -zxvf jdk-8u45-linux-x64.gz    -C /usr/java[root@hadoop1 softwore]# ls -ld /usr/java/*

drwxr-xr-x 8 root root 4096 12月 15 2016 /usr/java/djdk1.7.0_79

drwxr-xr-x 8 uucp  143 4096 4月  11 2015 /usr/java/jdk1.8.0_45

drwxr-xr-x 8 uucp  143 4096 10月  7 2015 /usr/java/jdk1.8.0_65

[root@hadoop1 softwore]# vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45

export JRE_HOME=/usr/java/jdk1.8.0_45/jre

[root@hadoop1 softwore]# java -version

java version "1.8.0_45"

Java(TM) SE Runtime Environment (build 1.8.0_45-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

二進制版本解壓，配置環境變量即可

3.5 MAVEN

上面我們已經安裝maven和jdk檢測版本是否滿足我們編譯軟體要求

[root@hadoop1 softwore]# mvn --version

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"

3.6 FINDBUGS

[root@hadoop1 softwore]#  unzip findbugs-1.3.9.zip

[root@hadoop1 softwore]# vi /etc/profile

  export FINDBUGS_HOME=/opt/software/findbugs-1.3.9

  export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH [root@hadoop1 softwore]# source /etc/profile

[root@hadoop1 softwore]# findbugs -version

1.3.9

二進制安裝解壓，配置環境變量即可

3.7 PROTOCBUF編譯安裝

[root@hadoop1 software]# tar xvf  protobuf-2.5.0.tar.gz

[root@hadoop1 yum.repos.d]# yum install -y gcc gcc-c++ make cmake   ---編譯環境

[root@hadoop1 yum.repos.d]#cd /opt/software/protobuf-2.5.0

[root@hadoop1 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf

[root@hadoop1 protobuf-2.5.0]# make && make install

[root@hadoop1 protobuf-2.5.0]# vim /etc/profile

export PROTOC_HOME=/usr/local/protobuf

export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop1 software]# source /etc/profile

[root@hadoop1 software]# protoc --version

libprotoc 2.5.0

[root@hadoop000 local]#

編譯安裝完成，配置環境變量

3.8 OTHER 依賴包

yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool

yum install -y snappy snappy-devel bzip2 bzip2-devel \n

zo lzo-devel lzop autoconf automake

其他依賴包在系統安裝ios都包含，執行yum安裝即可

---------------------------------------------------------------------------------------------------------------------------------------

至此，編譯hadoop需要的編譯環境全部準備好，下面開啟編譯之路

3.9 編譯

[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz

[root@hadoop1 software]# cd hadoop-2.8.1-src

[root@hadoop1 hadoop-2.8.1-src]# pwd

/opt/software/hadoop-2.8.1-src

[root@hadoop1 hadoop-2.8.1-src]#

mvn clean package -Pdist,native -DskipTests –Dtar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

----編譯好軟體包儲存位置

直接編譯需要聯網找依賴包，一般網絡情況較好，需要幾個小時

提前準備好依賴包編譯方式

[root@hadoop1 hadoop-2.8.1-src]# ls pom.xml  ---根據這個檔案下載下傳jar

pom.xml

[root@hado

-rw-r--r--   1 root   root  96721446 4月  25 09:57 .m2.tar.gz   ---提前下好的倉庫軟體包

[root@hadoop1 software]#tar xvf .m2.tar.gz –C /root   ---不想download，使用提前準備好的倉庫軟體--->解壓----->root家目錄

[root@hadoop1 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests –Dtar

……..

[INFO] Apache Hadoop Azure support ........................ SUCCESS [  5.562 s]

[INFO] Apache Hadoop Client ............................... SUCCESS [ 13.396 s]

[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.969 s]

[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  6.050 s]

[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.110 s]

[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  3.862 s]

[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.029 s]

[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:01 min]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] Total time: 13:45 min

[INFO] Finished at: 2018-04-25T14:03:20+08:00

[INFO] Final Memory: 161M/479M

[root@hadoop1 hadoop-2.8.1-src]# 編譯成功後的軟體儲存路徑

[root@hadoop1 hadoop-2.8.1-src]# ls /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-*

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1.jar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-javadoc.jar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-sources.jar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-test-sources.jar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1:

bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share

[root@hadoop1 hadoop-2.8.1-src]# du -sh /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

186M    /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

總結：完成源碼的編譯好打包，得到安裝hadoop安裝包，難點在于軟體包的裝備和倉庫軟體的準備，接下來準備開始hadoop安裝之路。