Hadoop HDFS NFS GateWay部署深入詳解

0.系統版本&hadoop版本

1）系統版本

[root@WEB-W031 sbin]# cat /etc/issue

CentOS release 5.8 (Final)

1

2

2）Hadoop版本

[root@WEB-W031 /]# hadoop version

Hadoop 2.7.2

Subversion

https://git-wip-us.apache.org/repos/asf/hadoop.git

-r b165c4fe8a74265c792ce23f546c64604acf0e41

Compiled by jenkins on 2016-01-26T00:08Z

Compiled with protoc 2.5.0

From source with checksum d0fda26633fa762bff87ec759ebe689c

This command was run using /home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.2.jar

You have new mail in /var/spool/mail/root

3

4

5

6

7

8

1、配置Core-site.xml檔案

<property>

 <name>hadoop.proxyuser.hadoop.groups</name>

 <value>*</value>

 <description>

        The 'nfsserver' user is allowed to proxy all members of the 'users-group1' and

        'users-group2' groups. Note that in most cases you will need to include the

        group "root" because the user "root" (which usually belonges to "root" group) will

        generally be the user that initially executes the mount on the NFS client system.

        Set this to '*' to allow nfsserver user to proxy any group.

 </description>

</property>

 <name>hadoop.proxyuser.hadoop.hosts</name>

        This is the host where the nfs gateway is running. Set this to '*' to allow

        requests from any hosts to be proxied.

9

10

11

12

13

14

15

16

17

18

19

20

21

含義解讀：

1）hadoop.proxyuser.hadoop.groups：第二個和執行Hadoop的使用者名一緻，本測試用的名稱為Hadoop。

2）hadoop.proxyuser.hadoop.hosts：第二個和執行Hadoop的使用者名一緻，本測試用的名稱為Hadoop。

2、配置hdfs-site.xml檔案

    <property>

 <name>dfs.namenode.accesstime.precision</name>

 <value>3600000</value>

 <description>The access time for HDFS file is precise upto this value.

   The default value is 1 hour. Setting a value of 0 disables

   access times for HDFS.

 <property>

   <name>nfs.dump.dir</name>

   <value>/tmp/.hdfs-nfs</value>

 </property>

 <name>nfs.exports.allowed.hosts</name>

 <value>* rw</value>

   <name>nfs.rtmax</name>

   <value>1048576</value>

   <description>This is the maximum size in bytes of a READ request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's rsize(add rsize= # of bytes to the mount directive).</description>

   <name>nfs.wtmax</name>

   <value>65536</value>

   <description>This is the maximum size in bytes of a WRITE request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's wsize(add wsize= # of bytes to the mount directive).</description>

22

23

24

25

26

27

28

29

30

3、配置*/etc/hadoop/log4j.properties

log4j.logger.org.apache.hadoop.hdfs.nfs=DEBUG

log4j.logger.org.apache.hadoop.oncrpc=DEBUG

4、root使用者下停止portmap(Centos 5.X系統），停止nfs

[root@WEB-W031 etc]# service portmap stop

Stopping portmap: [ OK ]

[root@WEB-W031 hadoop]# service nfs stop

注意：5.X的portmap 等價于 6.X上的rpcbind

并通過:service nfs status 驗證是否已經結束。

并通過：service portmap status 驗證是否已經結束。

5、開啟Hadoop的Portmap服務（需要root權限）

[root]> /sbin/hadoop-daemon.sh –script /bin/hdfs start portmap

6、開啟Hadoop的Nfs3服務（需要執行Hadoop的使用者權限，本系統為Hadoop）

[hadoop]$ /sbin/hadoop-daemon.sh –script /bin/hdfs start nfs3

7、開啟Hadoop并驗證(需要執行Hadoop的使用者權限，本系統為Hadoop）

[root@WEB-W031 sbin]# ./start-dfs.sh

[root@WEB-W031 sbin]# jps

14728 ResourceManager

27478 Jps

5228 Portmap

14104 NameNode

14373 SecondaryNameNode

16762 org.eclipse.equinox.launcher_1.3.0.v20130327-1440.jar

13512 Nfs3

8、驗證NFS相關服務是否啟動成功

1）驗證各服務是否成功運作

[root@WEB-W031 hadoop]# rpcinfo -p $nfs_server_ip

program vers proto port

100005 2 tcp 4242 mountd

100000 2 udp 111 portmapper

100000 2 tcp 111 portmapper

100005 1 tcp 4242 mountd

100003 3 tcp 2049 nfs

100005 1 udp 4242 mountd

100005 3 udp 4242 mountd

100005 3 tcp 4242 mountd

2）顯示挂載路徑

[root@WEB-W031 hadoop]# showmount -e $nfs_server_ip

Export list for WEB-W031:

/ *

9、實施挂載

[root]>mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync $10.0.1.31:/  /hdfsnew

10、檢視挂載是否成功

[root@WEB-W031 hdfsnew]# ls -al

total 10

drwxr-xr-x  5 root   2584148964  160 Mar 11 09:43 .

drwxr-xr-x 31 root   root       4096 Mar 11 10:40 ..

drwxr-xr-x  3 root   root         96 Mar 11 09:58 testing

drwxrwx---  3 hadoop 2584148964   96 Mar  9 11:14 tmp

drwxr-xr-x  3 hadoop 2584148964   96 Mar  9 13:24 user

[root@WEB-W031 hdfsnew]# cd testing/

[root@WEB-W031 testing]# ls -al

total 2

drwxr-xr-x 3 root   root        96 Mar 11 09:58 .

drwxr-xr-x 5 root   2584148964 160 Mar 11 09:43 ..

-rw-rw-r-- 1 hadoop hadoop      18 Mar 11 10:08 dffafdadafasdfaf

[root@WEB-W031 testing]# echo  "testinglaoyang" > testlaoyang

drwxr-xr-x 4 root   root       128 Mar 11 18:05 .

-rw-r--r-- 1 root   root        15 Mar 11 18:05 testlaoyang

11、通路效果

12、結束服務

1）NFS3服務結束（必須和啟動hadoop使用者一緻）

[hadoop]$ */sbin/hadoop-daemon.sh --script */bin/hdfs stop nfs3

2）PortMap服務結束（必須為root使用者）

[root]> */sbin/hadoop-daemon.sh --script */bin/hdfs start portmap

3）Hadoop結束（必須和啟動hadoop使用者一緻）

“`

[hadoop@WEB-W031 sbin]# ./stop-dfs.sh

13、常見問題

java 程序 占用了111 端口

[root@WEB-W031 etc]# lsof -i:111

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

java 31206 root 474u IPv4 93384812 0t0 TCP *:sunrpc (LISTEN)

java 31206 root 476u IPv4 93384815 0t0 UDP *:sunrpc

“

解決方案：kill -9 31206

錯誤明細

[root@xx sbin]# mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync localhost:/ /hdfs_y

mount.nfs: Input/output error

注意：一定要注意配置檔案的細節含義。

解決方案：

http://stackoverflow.com/questions/35910751/hdfs-nfs-gateway-mount-nfs-input-output-error/35935229#35935229

參考：

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html#Allow_mounts_from_unprivileged_clients http://tianxingzhe.blog.51cto.com/3390077/1650182