大資料之---hadoop僞分布式部署(HDFS)全網終極篇

1、軟體環境

RHEL6 jdk-8u45

hadoop-2.8.1.tar.gz ssh

xx.xx.xx.xx ip位址 hadoop1

xx.xx.xx.xx ip位址 hadoop2

xx.xx.xx.xx ip位址 hadoop3

xx.xx.xx.xx ip位址 hadoop4

xx.xx.xx.xx ip位址 hadoop5

本次部署隻涉及僞分布式部署隻是要主機hadoop1

2.僞分布式部署

僞分布式部署文檔參考官方網站hadoop

Apache > Hadoop > Apache Hadoop Project Dist POM > Apache Hadoop 2.8.3 > Hadoop: Setting up a Single Node Cluster

添加hadoop使用者

[root@hadoop1 ~]# useradd hadoop

[root@hadoop1 ~]# vi /etc/sudoers

找到root ALL=(ALL) ALL，添加

上傳并解壓

[root@hadoop1 software]# rz #上傳hadoop-2.8.1.tar.gz

[root@hadoop1 software]# tar -xzvf hadoop-2.8.1.tar.gz

軟連接配接

[root@hadoop1 software]# ln -s /opt/software/hadoop-2.8.1 /opt/software/hadoop

設定環境變量

[root@hadoop1 software]# vi /etc/profile

export HADOOP_HOME=/opt/software/hadoop

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

[root@hadoop1 software]# source /etc/profile

設定使用者、使用者組

[root@hadoop1 software]# chown -R hadoop:hadoop hadoop

[root@hadoop1 software]# chown -R hadoop:hadoop hadoop/

[root@hadoop1 software]# chown -R hadoop:hadoop hadoop-2.8.1

[root@hadoop1 software]# cd hadoop

[root@hadoop1 hadoop]# rm -f .txt

[root@hadoop1 software]# su - hadoop

[root@hadoop1 hadoop]# ll

total 32

drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 bin

drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 etc

drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 include

drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 lib

drwxrwxr-x. 2 hadoop hadoop 4096 Aug 20 13:59 libexec

drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 13:59 logs

drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 sbin

drwxrwxr-x. 4 hadoop hadoop 4096 Jun 2 14:24 share*

[hadoop@hadoop1 ~]# cd /opt/software/hadoop

[hadoop@hadoop1 hadoop]# vi etc/hadoop/core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://172.16.18.133:9000</value> # 配置自己機器的IP

</property>

</configuration>

[hadoop@hadoop1 hadoop]# vi etc/hadoop/hdfs-site.xml

<name>dfs.replication</name>

<value>1</value>

配置hadoop使用者的ssh信任關系

[hadoop@hadoop1 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

[hadoop@hadoop1 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[hadoop@hadoop1 ~]# chmod 0600 ~/.ssh/authorized_keys

[root@hadoop1 ~]# ssh hadoop1 date

Wed Apr 25 15:22:58 CST 2018

[root@hadoop1 ~]# ssh localhost date

Wed Apr 25 15:23:25 CST 2018

格式化和啟動

[hadoop@hadoop1 hadoop]# bin/hdfs namenode -format

[hadoop@hadoop1 hadoop]# sbin/start-dfs.sh

ERROR:

hadoop1: Error: JAVA_HOME is not set and could not be found.

localhost: Error: JAVA_HOME is not set and could not be found.

解決方法:添加環境變量

[hadoop@hadoop1 hadoop]# vi etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_45

檢查是否成功

[hadoop@hadoop1 hadoop]# jps

8026 Jps

7485 NameNode

7597 DataNode

7807 SecondaryNameNode

通路： http://192.168.137.130:50070

修改dfs啟動的程序，以hadoop1啟動 啟動的三個程序：

namenode: hadoop1 bin/hdfs getconf -namenodes

datanode: localhost datanodes (using default slaves file) etc/hadoop/slaves

secondarynamenode: 0.0.0.0

[hadoop@hadoop1 hadoop]# echo "hadoop1" > ./etc/hadoop/slaves

[hadoop@hadoop1 hadoop]# cat ./etc/hadoop/slaves

hadoop1

[hadoop@hadoop1 hadoop]# vi ./etc/hadoop/hdfs-site.xml

<name>dfs.namenode.secondary.http-address</name>

<value>hadoop1:50090</value>

<name>dfs.namenode.secondary.https-address</name>

<value>hadoop1:50091</value>

[hadoop@hadoop1 hadoop]# sbin/stop-dfs.sh

通路： http://172.16.18.133:50070界面