HDFS配置lzo壓縮

參考 https://www.cnblogs.com/caoshouling/p/14091113.html， 做了驗證，很好的文檔。

1) 停止hdfs叢集

2）安裝配置maven

https://blog.csdn.net/hailunw/article/details/117996934

3）生成lzo壓縮程式包

3.1）安裝前置package

yum -y install  lzo-devel  zlib-devel  gcc autoconf automake libtool
           

3.2）下載下傳lzo源檔案 https://github.com/twitter/hadoop-lzo/archive/refs/heads/master.zip 到99伺服器，解壓縮。

wget https://github.com/twitter/hadoop-lzo/archive/refs/heads/master.zip
unzip master.zip 
           

3.3）安裝lzo

cd /home/user/hadoop-lzo-master
mkdir lzo
export CFLAGS=-m64
export CXXFLAGS=-m64
export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
export LIBRARY_PATH=/usr/local/hadoop/lzo/lib

mvn clean package -Dmaven.test.skip=true
cd target/native/Linux-amd64-64
tar -cBf - -C lib . | tar -xBvf - -C ~
cp ~/libgplcompression* $HADOOP_HOME/lib/native/
cp /home/user/hadoop-lzo-master/target/hadoop-lzo-0.4.21-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/
           

4）配置Hadoop

4.1）将lzo包分發到66伺服器和88伺服器

scp -r /home/user/hadoop-3.2.2/share/hadoop/common/hadoop-lzo-0.4.21-SNAPSHOT.jar [email protected]:/home/user/hadoop-3.2.2/share/hadoop/common/hadoop-lzo-0.4.21-SNAPSHOT.jar

scp -r /home/user/hadoop-3.2.2/share/hadoop/common/hadoop-lzo-0.4.21-SNAPSHOT.jar [email protected]:/home/user/hadoop-3.2.2/share/hadoop/common/hadoop-lzo-0.4.21-SNAPSHOT.jar

scp -r $HADOOP_HOME/lib/native/* [email protected]:$HADOOP_HOME/lib/native/

scp -r $HADOOP_HOME/lib/native/* [email protected]:$HADOOP_HOME/lib/native/

           

4.2）在66,88,99伺服器修改配置檔案core-site.xml,添加如下内容。

<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
           

4.3）重新啟動hdfs叢集