HadoopHA搭建(簡版)
前言
剛搭建完叢集把以前的問題都給解決了,十分激動,于是寫下了第一篇部落格,由于之前不了解叢集的各節點分工及配置什麼的,導緻叢集搭建過程中錯誤頻出,後來看log日志檔案并且重新配置了一遍才成功,報錯了記得一定一定一定要看log檔案
準備工作
zookeeper叢集已經搭建并且配置完畢,bin/zkServer.sh status之後狀态為follower或者leader,了解每個節點的任務和工作,叢集共有四台機器:M1 M2 S1 S2
配置檔案的修改
HadoopHA在zookeeper叢集的基礎上修改hdfs-site.xml和core-site.xml兩個檔案就可以了
- hdfs-site.xml
<configuration>
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>M1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>M2:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>M1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>M2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://S1:8485;S2:8485;M2:8485/mycluster</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.permission.enable</name>
<value>false</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
</configuration>
- core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/HA/data/tmp/jn</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/HA/data/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>M1:2181,S1:2181,S2:2181</value>
</property>
</configuration>
啟動叢集
首先要啟動zookeeper叢集,在zookeeper中初始化HA的狀态,進入zookeeper安裝路徑下執行指令:
/bin/zkfc -formatZK
注意:namenode格式化一次後如果再次格式化,需删除
/opt/HA/data/tmp
目錄下的dfs檔案,否則将會報錯,再次格式化時需啟動journalNode節點,同時關閉叢集,以上操作完在主節點上啟動叢集:
/opt/hadoop/hadoop-2.9.sbin/start-dfs.sh
然後jps就能夠看到啟動的程序。
HadoopHA官網配置連結: https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html