Apache Hadoop 2.7如何支援讀寫OSS

背景

2017.12.13日Apache Hadoop 3.0.0正式版本釋出，預設支援阿裡雲OSS對象存儲系統，作為Hadoop相容的檔案系統，後續版本号大于等于Hadoop 2.9.x系列也支援OSS。然而，低版本的Apache Hadoop官方不再支援OSS，本文将描述如何通過支援包來使Hadoop 2.7.2能夠讀寫OSS。

如何使用

下面的步驟需要在所有的Hadoop節點執行

下載下傳支援包

http://gosspublic.alicdn.com/hadoop-spark/hadoop-oss-2.7.2.tar.gz

解壓這個支援包，裡面的檔案是：

[root@apache hadoop-oss-2.7.2]# ls -lh
總用量 3.1M
-rw-r--r-- 1 root root 3.1M 2月  28 17:01 hadoop-aliyun-2.7.2.jar

這個支援包是根據Hadoop 2.7.2的版本，并打了Apache Hadoop對OSS支援的patch後編譯得到，其他的小版本對OSS的支援後續也将陸續提供。

部署

首先将檔案hadoop-aliyun-2.7.2.jar複制到$HADOOP_HOME/share/hadoop/tools/lib/目錄下；

修改$HADOOP_HOME/libexec/hadoop-config.sh檔案，在檔案的327行加下代碼：

CLASSPATH=$CLASSPATH:$TOOL_PATH

修改的目的就是将$HADOOP_HOME/share/hadoop/tools/lib/放到Hadoop的CLASSPATH裡面；下面是修改前後，這個檔案的diff供參考(hadoop-config.sh.bak是修改前的檔案):

[root@apache hadoop-2.7.2]# diff -C 3 libexec/hadoop-config.sh.bak libexec/hadoop-config.sh
*** libexec/hadoop-config.sh.bak    2019-03-01 10:35:59.629136885 +0800
--- libexec/hadoop-config.sh    2019-02-28 16:33:39.661707800 +0800
***************
*** 325,330 ****
--- 325,332 ----
  CLASSPATH=${CLASSPATH}:$HADOOP_MAPRED_HOME/$MAPRED_DIR'/*'
fi

+ CLASSPATH=$CLASSPATH:$TOOL_PATH
+
# Add the user-specified CLASSPATH via HADOOP_CLASSPATH
# Add it first or last depending on if user has
# set env-var HADOOP_USER_CLASSPATH_FIRST

增加OSS的配置

修改core-site.xml檔案，增加如下配置項：

配置項	值	說明
fs.oss.endpoint	如 oss-cn-zhangjiakou-internal.aliyuncs.com	要連接配接的endpoint
fs.oss.accessKeyId	access key id
fs.oss.accessKeySecret	access key secret
fs.oss.impl	org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem	hadoop oss檔案系統實作類，目前固定為這個
fs.oss.buffer.dir	/tmp/oss	臨時檔案目錄
fs.oss.connection.secure.enabled	false	是否enable https, 根據需要來設定，enable https會影響性能
fs.oss.connection.maximum	2048	與oss的連接配接數，根據需要設定

相關參數的解釋可以在

這裡

找到

重新開機叢集，驗證讀寫OSS

增加配置後，重新開機叢集，重新開機後，可以測試

# 測試寫
hadoop fs -mkdir oss://{your-bucket-name}/hadoop-test
# 測試讀
hadoop fs -ls oss://{your-bucket-name}/

運作teragen

[root@apache hadoop-2.7.2]# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar teragen -Dmapred.map.tasks=100 10995116 oss://{your-bucket-name}/1G-input
19/02/28 16:38:59 INFO client.RMProxy: Connecting to ResourceManager at apache/192.168.0.176:8032
19/02/28 16:39:01 INFO terasort.TeraSort: Generating 10995116 using 100
19/02/28 16:39:01 INFO mapreduce.JobSubmitter: number of splits:100
19/02/28 16:39:01 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/02/28 16:39:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1551343125387_0001
19/02/28 16:39:02 INFO impl.YarnClientImpl: Submitted application application_1551343125387_0001
19/02/28 16:39:02 INFO mapreduce.Job: The url to track the job: http://apache:8088/proxy/application_1551343125387_0001/
19/02/28 16:39:02 INFO mapreduce.Job: Running job: job_1551343125387_0001
19/02/28 16:39:09 INFO mapreduce.Job: Job job_1551343125387_0001 running in uber mode : false
19/02/28 16:39:09 INFO mapreduce.Job:  map 0% reduce 0%
19/02/28 16:39:18 INFO mapreduce.Job:  map 1% reduce 0%
19/02/28 16:39:19 INFO mapreduce.Job:  map 2% reduce 0%
19/02/28 16:39:21 INFO mapreduce.Job:  map 4% reduce 0%
19/02/28 16:39:25 INFO mapreduce.Job:  map 5% reduce 0%
19/02/28 16:39:28 INFO mapreduce.Job:  map 6% reduce 0%
......
19/02/28 16:42:36 INFO mapreduce.Job:  map 94% reduce 0%
19/02/28 16:42:38 INFO mapreduce.Job:  map 95% reduce 0%
19/02/28 16:42:41 INFO mapreduce.Job:  map 96% reduce 0%
19/02/28 16:42:44 INFO mapreduce.Job:  map 97% reduce 0%
19/02/28 16:42:45 INFO mapreduce.Job:  map 98% reduce 0%
19/02/28 16:42:46 INFO mapreduce.Job:  map 99% reduce 0%
19/02/28 16:42:48 INFO mapreduce.Job:  map 100% reduce 0%
19/02/28 16:43:11 INFO mapreduce.Job: Job job_1551343125387_0001 completed successfully
19/02/28 16:43:12 INFO mapreduce.Job: Counters: 37
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=11931190
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=8497
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=100
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
        OSS: Number of bytes read=0
        OSS: Number of bytes written=1099511600
        OSS: Number of read operations=1100
        OSS: Number of large read operations=0
        OSS: Number of write operations=500
......

運作distcp

從OSS往HDFS拷貝資料

[root@apache hadoop-2.7.2]# hadoop distcp oss://{your-bucket-name}/data hdfs:/data/input
19/03/05 09:43:59 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[oss://{your-bucket-name}/data], targetPath=hdfs:/data/input, targetPathExists=false, preserveRawXattrs=false}
19/03/05 09:43:59 INFO client.RMProxy: Connecting to ResourceManager at apache/192.168.0.176:8032
19/03/05 09:44:00 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
19/03/05 09:44:00 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
19/03/05 09:44:01 INFO client.RMProxy: Connecting to ResourceManager at apache/192.168.0.176:8032
19/03/05 09:44:01 INFO mapreduce.JobSubmitter: number of splits:24
19/03/05 09:44:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1551343125387_0008
19/03/05 09:44:01 INFO impl.YarnClientImpl: Submitted application application_1551343125387_0008
19/03/05 09:44:01 INFO mapreduce.Job: The url to track the job: http://apache:8088/proxy/application_1551343125387_0008/
19/03/05 09:44:01 INFO tools.DistCp: DistCp job-id: job_1551343125387_0008
19/03/05 09:44:01 INFO mapreduce.Job: Running job: job_1551343125387_0008
19/03/05 09:44:07 INFO mapreduce.Job: Job job_1551343125387_0008 running in uber mode : false
19/03/05 09:44:07 INFO mapreduce.Job:  map 0% reduce 0%
19/03/05 09:44:16 INFO mapreduce.Job:  map 4% reduce 0%
19/03/05 09:44:19 INFO mapreduce.Job:  map 8% reduce 0%
......
19/03/05 09:45:11 INFO mapreduce.Job:  map 96% reduce 0%
19/03/05 09:45:12 INFO mapreduce.Job:  map 100% reduce 0%
19/03/05 09:45:13 INFO mapreduce.Job: Job job_1551343125387_0008 completed successfully
19/03/05 09:45:13 INFO mapreduce.Job: Counters: 38
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=2932262
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=24152
        HDFS: Number of bytes written=1099511600
        HDFS: Number of read operations=898
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=251
        OSS: Number of bytes read=1099511600
        OSS: Number of bytes written=0
        OSS: Number of read operations=2404
        OSS: Number of large read operations=0
        OSS: Number of write operations=0
......

[root@apache hadoop-2.7.2]# hadoop fs -ls hdfs:/data
Found 1 items
drwxr-xr-x   - root supergroup          0 2019-03-05 09:45 hdfs:///data/input

從HDFS往OSS拷貝資料

[root@apache hadoop-2.7.2]# hadoop distcp hdfs:/data/input oss://{your-bucket-name}/data/output
19/03/05 09:48:06 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs:/data/input], targetPath=oss://{your-bucket-name}/data/output, targetPathExists=false, preserveRawXattrs=false}
19/03/05 09:48:06 INFO client.RMProxy: Connecting to ResourceManager at apache/192.168.0.176:8032
19/03/05 09:48:06 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
19/03/05 09:48:06 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
19/03/05 09:48:07 INFO client.RMProxy: Connecting to ResourceManager at apache/192.168.0.176:8032
19/03/05 09:48:07 INFO mapreduce.JobSubmitter: number of splits:24
19/03/05 09:48:08 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1551343125387_0009
19/03/05 09:48:08 INFO impl.YarnClientImpl: Submitted application application_1551343125387_0009
19/03/05 09:48:08 INFO mapreduce.Job: The url to track the job: http://apache:8088/proxy/application_1551343125387_0009/
19/03/05 09:48:08 INFO tools.DistCp: DistCp job-id: job_1551343125387_0009
19/03/05 09:48:08 INFO mapreduce.Job: Running job: job_1551343125387_0009
19/03/05 09:48:14 INFO mapreduce.Job: Job job_1551343125387_0009 running in uber mode : false
19/03/05 09:48:14 INFO mapreduce.Job:  map 0% reduce 0%
19/03/05 09:48:24 INFO mapreduce.Job:  map 4% reduce 0%
19/03/05 09:48:27 INFO mapreduce.Job:  map 8% reduce 0%
......
19/03/05 09:49:18 INFO mapreduce.Job:  map 92% reduce 0%
19/03/05 09:49:20 INFO mapreduce.Job:  map 96% reduce 0%
19/03/05 09:49:21 INFO mapreduce.Job:  map 100% reduce 0%
19/03/05 09:49:22 INFO mapreduce.Job: Job job_1551343125387_0009 completed successfully
19/03/05 09:49:22 INFO mapreduce.Job: Counters: 38
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=2932910
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=1099535478
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=548
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=48
        OSS: Number of bytes read=0
        OSS: Number of bytes written=1099511600
        OSS: Number of read operations=1262
        OSS: Number of large read operations=0
        OSS: Number of write operations=405
......

[root@apache hadoop-2.7.2]# hadoop fs -ls oss://{your-bucket-name}/data/output
Found 101 items
-rw-rw-rw-   1 root root          0 2019-03-05 09:48 oss://{your-bucket-name}/data/output/_SUCCESS
-rw-rw-rw-   1 root root   10995200 2019-03-05 09:48 oss://{your-bucket-name}/data/output/part-m-00000
-rw-rw-rw-   1 root root   10995100 2019-03-05 09:48 oss://{your-bucket-name}/data/output/part-m-00001
......

參考連結

https://yq.aliyun.com/articles/292792?spm=a2c4e.11155435.0.0.7ccba82fbDwfhK https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md

Apache Hadoop 2.7如何支援讀寫OSS

背景

如何使用

下載下傳支援包

部署

增加OSS的配置

重新開機叢集，驗證讀寫OSS

運作teragen

運作distcp

從OSS往HDFS拷貝資料

從HDFS往OSS拷貝資料

參考連結

繼續閱讀

Apache配置檔案中的deny和allow的使用

Apache 配置預設編碼

伺服器配置——Apache

Apache靜态檔案通路配置（書封伺服器）

apache httpd 配置

Ubuntu16.04安裝Apache+MySQL+PHP1. 安裝Apache2. 安裝MySQL3. 安裝PHP4. 安裝phpMyAdmin

淺談企業活動中進行資料分析的重要性

ubuntu14.04下安裝hbse1.0.1.1

Apache配置SSLApache配置SSL

Windows下配置Apache的SSL服務

User Defined Hadoop DataType

Apache2.4.x 配置檔案詳解Apache配置需要了解如下：開始講解：

配置apache支援PHP（win7）

Ambari介紹和架構原理

NOSQL安全攻擊

win10本地scala和spark安裝安裝scala安裝spark