【轉】hadoop/hbase搭建

1. 環境

作業系統：red hat 6.3，300g硬碟，雙核cpu

java：jdk1.6

hbase：hbase-0.98.1

hadoop：hadoop-2.2.0

現在我們使用3台機器(虛拟機)，來搭建hadoop環境，如下為機器清單，首先我們需要在三台機器上hosts檔案中都增加如下資訊，同時需要注意，區域網路内的機器都需要設定網卡為“靜态ip”，以防止機器重新開機後ip不斷變化。本例中所有機器使用root使用者操作。

127.0.0.1 localhost

::1 localhost

192.168.0.110 node1.hadoop.virtual

192.168.0.111 node2.hadoop.virtual

192.168.0.112 node2.hadoop.virtual

##需要注意目前的機器名

## 192.168.0.110(本機ip) 機器名

當然使用“hostname”轉義成ip并非必須的，我們也可以直接在hadoop、hbase的配置檔案使用ip，不過本人為了便于定位node所在的機器位置，就“額外”的在hosts中增加了上述配置。

2. ssh

hadoop、hbase中使用ssh實作無密碼登入，在hadoop叢集啟動時将會依次跳轉到叢集中所有的機器上啟動相應的程序，是以我們首先配置ssh授權資訊(請首先安裝ssh-keygen用戶端，關于ssh用戶端配置請參見其他文檔)：

ssh-keygen -t rsa

一路回車，均采用預設配置，無password設定。最終會在:/root/.ssh目錄下生成id_rsa.pub(如果使用其他使用者登入，則儲存在~/.ssh目錄下)，這個檔案中所包含的的就是登陸本機所需要的rsa-pub值。我們依次在上述三台機器上執行ssh-keygen指令。

我們把三台機器生成的三個rsa-pub值都依次寫入每台機器的/root/.ssh/authorized_keys檔案中，很多人忘記把本機的rsa-pub值寫入自己的authorized_keys，這會導緻本機的hadoop執行個體無法啟動。

3. jdk安裝

現在jdk1.6+版本，然後直接安裝，并在配置java_home，本執行個體中jdk安裝在/opt/app目錄下。在/etc/profile檔案的尾部增加如下配置：

## set java

java_home=/opt/app/jdk1.6.0_45

path=$path:/$java_home/bin

classpath=.:$java_home/lib/dt.jar:$java_home/lib/tools.jar

jre_home=$java_home/jre

export java_home path classpath jre_home

export ld_library_path=/usr/local/lib

配置完畢之後，執行“source

/etc/profile”使環境變量生效；如果你的機器上已經安裝了其他的jdk(比如openjdk)，你應該首先解除安裝或者禁用其啟動檔案，然後再執行上述“source”指令。(比如，使用"whereis

java"指令查找出java啟動檔案所在位置，然後删除它，它可能位于“/usr/bin/java”中)

建議三台機器中，jdk、hadoop、hbase的安裝目錄都相同，這樣排查問題和檔案配置都會非常的友善。

4. ulimit設定

因為hadoop和hbase在運作時需要打開很多的檔案，同時在面向衆多client連接配接時也會消耗檔案描述符，是以我們需要簡單的将ulimit設定的稍微大一些(預設為1024，顯然太小了)，在/etc/security/limits.conf檔案的尾部增加：

* soft nofile 20480

* hard nofile 20480

此外還需要檢測/etc/profile檔案中是否也指定了“ulimit”選項，如果有，可以考慮暫時先移除它，ulimit參數将會在root使用者重登陸後生效。關于linux下如何調整機器支撐的最大檔案打開個數以及線程的并發數，請參見其他文檔。

5. hadoop搭建與配置

通過apache網站下載下傳hadoop-2.2.tar.gz檔案包，解壓到指定目錄，本執行個體中将hadoop安裝在“/opt/app”目錄下；我們首先在一台機器上安裝hadoop并配置，稍後再把依次同步到另外兩台機器即可。

我們約定，node1.hadoop.virtual作為namenode，其他兩台作為datanode，其中node2.hadoop.virtual作為輔助master(即secondarynamenode)，本文中将不引入yarn(mapreduce)，是以我們暫時不關心resourcemanager等程序。同時約定hadoop與hbase中關于file

blocksize的設定保持一緻，為128m。我們依次配置如下檔案(${hadoop}/etc/hadoop)：

1) masters檔案：此檔案隻需要放置在namenode和secondarynamenode上，datanode上不需要此檔案；masters檔案用來指定“secondarynamenode”的位址；叢集中，通常需要一個secondarynamenode。

node2.hadoop.virtual

2) slaves檔案：用來指定datanode的位址，隻有datanode上需要此檔案；

node1.hadoop.virtual

node3.hadoop.virtual

如果你不希望node1同時作為datanode和namenode兩種角色，你可以在slaves檔案中移除它。

3) hdfs-site.xml檔案：hadoop主要有hdfs和mapreduce兩種核心特性，hdfs-site.xml用來配置hdfs特性。如下配置參數，僅供參考。

<?xml version="1.0" encoding="utf-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.replication</name>

</property>

<name>dfs.blocksize</name>

<name>dfs.client.block.write.retries</name>

<name>dfs.heartbeat.interval</name>

<name>dfs.namenode.replication.interval</name>

<name>dfs.namenode.logging.level</name>

<name>dfs.datanode.handler.count</name>

<name>dfs.namenode.handler.count</name>

<name>dfs.namenode.name.dir</name>

<value>/data/hadoop/dfs/name</value>

<name>dfs.namenode.name.dir.restore</name>

<value>false</value>

<name>dfs.datanode.data.dir</name>

<value>/data/hadoop/dfs/data</value>

<name>dfs.namenode.checkpoint.dir</name>

<value>/data/hadoop/dfs/namesecondary</value>

<!--

<name>dfs.datanode.address</name>

-->

</configuration>

4) core-site.xml：hadoop全局配置

<!--

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/core-default.xml

-->

<name>fs.default.name</name>

<value>hdfs://node1.hadoop.virtual:8020</value>

<name>file.blocksize</name>

<name>hadoop.tmp.dir</name>

<value>/var/hadoop/tmp</value>

<name>file.replication</name>

<name>hadoop.security.authorization</name>

<name>io.file.buffer.size</name>

<name>io.bytes.per.checksum</name>

<name>io.skip.checksum.errors</name>

<name>io.compression.codecs</name>

<name>io.seqfile.local.dir</name>

<value>/data/hadoop/io/local</value>

<name>fs.trash.interval</name>

<name>io.map.index.skip</name>

<!-- number of index entries to skip between each entry.

zero by default.

setting this to values larger than zero can facilitate opening large mapfiles using less memory.

-->

<name>io.map.index.interval</name>

<!-- mapfile consist of two files - data file (tuples) and index file (keys).

for every io.map.index.interval records written in the data file,

an entry (record-key, data-file-position) is written in the index file.

this is to allow for doing binary search later within the index file to look up records

by their keys and get their closest positions in the data file.

<name>io.mapfile.bloom.size</name>

<name>io.mapfile.bloom.error.rate</name>

<name>ha.zookeeper.quorum</name>

5) hadoop-env.sh： hadoop啟動腳本，這個腳本中包含了jvm參數調優以及全局性參數，為了便于參考，如下為全部資訊。

# set hadoop-specific environment variables here.

# the only required environment variable is java_home. all others are

# optional. when running a distributed configuration it is best to

# set java_home in this file, so that it is correctly defined on

# remote nodes.

# the java implementation to use.

export java_home=/opt/app/jdk1.6.0_45

# the jsvc implementation to use. jsvc is required to run secure datanodes.

#export jsvc_home=${jsvc_home}

export hadoop_conf_dir=${hadoop_conf_dir:-"/etc/hadoop"}

# extra java classpath elements. automatically insert capacity-scheduler.

for f in $hadoop_home/contrib/capacity-scheduler/*.jar; do

if [ "$hadoop_classpath" ]; then

export hadoop_classpath=$hadoop_classpath:$f

else

export hadoop_classpath=$f

done

# the maximum amount of heap to use, in mb. default is 1000.

## 1g,every damon processor,there are about 2 ~ 4 procesors,so this value should be suitable.

export hadoop_heapsize=1024

export hadoop_namenode_init_heapsize="512"

# extra java runtime options. empty by default.

## default for all ,will append to name_node/data_node opts

export hadoop_opts="$hadoop_opts -djava.net.preferipv4stack=true -xx:maxpermsize=256m -xx:survivorratio=6 -xx:+useconcmarksweepgc -xx:maxtenuringthreshold=3 -xx:cmsinitiatingoccupancyfraction=70 -xx:+usecmscompactatfullcollection"

# command specific options appended to hadoop_opts when specified

export hadoop_namenode_opts="-dhadoop.security.logger=${hadoop_security_logger:-info,rfas} -dhdfs.audit.logger=${hdfs_audit_logger:-info,nullappender} $hadoop_namenode_opts -xmx2048m"

export hadoop_datanode_opts="-dhadoop.security.logger=error,rfas $hadoop_datanode_opts -xmx2048m"

export hadoop_secondarynamenode_opts="-dhadoop.security.logger=${hadoop_security_logger:-info,rfas} -dhdfs.audit.logger=${hdfs_audit_logger:-info,nullappender} $hadoop_secondarynamenode_opts -xmx2048m"

# the following applies to multiple commands (fs, dfs, fsck, distcp etc)

export hadoop_client_opts="-xmx512m $hadoop_client_opts"

#hadoop_java_platform_opts="-xx:-useperfdata $hadoop_java_platform_opts"

# on secure datanodes, user to run the datanode as after dropping privileges

export hadoop_secure_dn_user=${hadoop_secure_dn_user}

# where log files are stored. $hadoop_home/logs by default.

export hadoop_log_dir=/data/hadoop/logs/hadoop

# where log files are stored in the secure data environment.

export hadoop_secure_dn_log_dir=${hadoop_log_dir}/${hadoop_hdfs_user}

# the directory where pid files are stored. /tmp by default.

# note: this should be set to a directory that can only be written to by

# the user that will run the hadoop daemons. otherwise there is the

# potential for a symlink attack.

export hadoop_pid_dir=/var/hadoop/pids

export hadoop_secure_dn_pid_dir=${hadoop_pid_dir}

# a string representing this instance of hadoop. $user by default.

export hadoop_ident_string=$user

配置檔案調整結束後，将此hadoop-2.2安裝檔案全部copy到其他兩台機器上(node2，node3)上；然在node1上啟動叢集，需要注意，在啟動叢集之前需要在“namenode”和“secondarynamenode”上指定format指令：

#在bin目錄下

> ./hadoop namenode -format

然後依次運作${hadoop}/sbin目錄下的“start-dfs.sh”。如果叢集啟動過程中，沒有任何異常，可以依次在每台機器上執行“jps”指令檢視基于jmx的java程序，你或許會看到如下字樣：

[[email protected] ~]# jps

19922 datanode

19757 namenode

21569 jps

[[email protected] hadoop]# jps

31627 jps

31466 secondarynamenode

31324 datanode

[[email protected] ~]# jps

15913 datanode

23508 jps

16530 nodemanager

如果某台機器上，namenode、datanode等程序啟動不符合預期，請到日志檔案中檢視原因(本執行個體中，日志位于：/data/hadoop/logs)

6. hbase搭建與配置

如果上述hadoop環境就緒後，我們就可以搭建hbase叢集了，不過此處需要提醒，并不是hbase存儲必須使用hadoop，它可以基于本地檔案系統或者其他任何hbase支援遠端檔案系統。

我們将hbase-0.98安裝檔案解壓到/opt/app目錄下，和hadoop目錄平級，便于管理。

1) hbase-core.xml：hbase核心配置

<?xml version="1.0"?>

<name>hbase.regionserver.port</name>

<name>hbase.hregion.max.filesize</name>

<name>hbase.hregion.memstore.flush.size</name>

<name>hbase.regionserver.handler.count</name>

<name>hbase.regionserver.lease.period</name>

<name>hbase.rootdir</name>

<value>hdfs://node1.hadoop.virtual:8020/hbase</value>

<name>hbase.cluster.distributed</name>

<!--

<name>hbase.zookeeper.property.clientport</name>

<name>hbase.zookeeper.property.datadir</name>

<value>../zookeeper</value>

<name>hbase.zookeeper.quorum</name>

<value>node1.hadoop.virtual,node2.hadoop.virtual,node2.hadoop.virtual</value>

<name>hbase.dynamic.jars.dir</name>

<value>hdfs://node1.hadoop.virtual:8020/hbase-extlib</value>

<name>zookeeper.session.timeout</name>

<name>zookeeper.znode.parent</name>

<value>/online-hbase</value>

hadoop并不依賴于zookeeper來儲存meta資訊，甚至hadoop本身不具有master選舉的特性；不過hbase則不同，hbase将會把meta資訊儲存在zookeeper中，以及使用zookeeper來感覺reginserver的存活狀态、hmaster選舉等等，是以在搭建hbase之前，需要一個即備的zookeeper叢集，具體zookeeper的搭建此處将不再贅言。

2) regionservers檔案：配置叢集中regionserver的清單，我們将會在如下三台機器上搭建hbase叢集

3) hbase-env.sh：hbase啟動腳本，其中包含了jvm參數選項，為了便于查閱，此處為全文。

# set environment variables here.

# this script sets variables multiple times over the course of starting an hbase process,

# so try to keep things idempotent unless you want to take an even deeper look

# into the startup scripts (bin/hbase, etc.)

# the java implementation to use. java 1.6 required.

# extra java classpath elements. optional.

export hbase_classpath="/opt/app/hbase-0.98.1/extlib/hadoop-ext-1.0.0-snapshot.jar"

# export hbase_heapsize=1000

# extra java runtime options.

# below are what we set by default. may only work with sun jvm.

# for more on why as well as other possible settings,

# see http://wiki.apache.org/hadoop/performancetuning

export hbase_opts="-xx:+useconcmarksweepgc -xx:maxpermsize=256m -xx:survivorratio=6 -xx:+useconcmarksweepgc -xx:maxtenuringthreshold=5 -xx:cmsinitiatingoccupancyfraction=70 -xx:+usecmscompactatfullcollection"

# uncomment one of the below three options to enable java garbage collection logging for the server-side processes.

# this enables basic gc logging to the .out file.

# export server_gc_opts="-verbose:gc -xx:+printgcdetails -xx:+printgcdatestamps"

# this enables basic gc logging to its own file.

# if file-path is not replaced, the log file(.gc) would still be generated in the hbase_log_dir .

# export server_gc_opts="-verbose:gc -xx:+printgcdetails -xx:+printgcdatestamps -xloggc:<file-path>"

# this enables basic gc logging to its own file with automatic log rolling. only applies to jdk 1.6.0_34+ and 1.7.0_2+.

export server_gc_opts="-verbose:gc -xx:+printgcdetails -xx:+printgcdatestamps -xloggc:/data/hadoop/logs/hbase/server-gc.log.$(date +%y%m%d%h%m) -xx:+usegclogfilerotation -xx:numberofgclogfiles=1 -xx:gclogfilesize=512m"

# uncomment one of the below three options to enable java garbage collection logging for the client processes.

# export client_gc_opts="-verbose:gc -xx:+printgcdetails -xx:+printgcdatestamps"

# export client_gc_opts="-verbose:gc -xx:+printgcdetails -xx:+printgcdatestamps -xloggc:<file-path>"

export client_gc_opts="-verbose:gc -xx:+printgcdetails -xx:+printgcdatestamps -xloggc:/data/hadoop/logs/hbase/client-gc.log.$(date +%y%m%d%h%m) -xx:+usegclogfilerotation -xx:numberofgclogfiles=1 -xx:gclogfilesize=512m"

# uncomment below if you intend to use the experimental off heap cache.

# export hbase_opts="$hbase_opts -xx:maxdirectmemorysize="

# set hbase.offheapcache.percentage in hbase-site.xml to a nonzero value.

# uncomment and adjust to enable jmx exporting

# see jmxremote.password and jmxremote.access in $jre_home/lib/management to configure remote password access.

# more details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html

export hbase_jmx_base="-dcom.sun.management.jmxremote.ssl=false -dcom.sun.management.jmxremote.authenticate=false"

export hbase_master_opts="$hbase_master_opts $hbase_jmx_base -dcom.sun.management.jmxremote.port=10101 -xmx2048m -xms2048m -xx:maxnewsize=512m"

export hbase_regionserver_opts="$hbase_regionserver_opts $hbase_jmx_base -dcom.sun.management.jmxremote.port=10102 -xmx4096m -xms4096m -xx:maxnewsize=1024m"

# export hbase_thrift_opts="$hbase_thrift_opts $hbase_jmx_base -dcom.sun.management.jmxremote.port=10103"

# export hbase_zookeeper_opts="$hbase_zookeeper_opts $hbase_jmx_base -dcom.sun.management.jmxremote.port=10104"

# export hbase_rest_opts="$hbase_rest_opts $hbase_jmx_base -dcom.sun.management.jmxremote.port=10105"

# file naming hosts on which hregionservers will run. $hbase_home/conf/regionservers by default.

# export hbase_regionservers=${hbase_home}/conf/regionservers

# uncomment and adjust to keep all the region server pages mapped to be memory resident

#hbase_regionserver_mlock=true

#hbase_regionserver_uid="hbase"

# file naming hosts on which backup hmaster will run. $hbase_home/conf/backup-masters by default.

# export hbase_backup_masters=${hbase_home}/conf/backup-masters

# extra ssh options. empty by default.

# export hbase_ssh_opts="-o connecttimeout=1 -o sendenv=hbase_conf_dir"

# where log files are stored. $hbase_home/logs by default.

export hbase_log_dir=/data/hadoop/logs/hbase

# enable remote jdwp debugging of major hbase processes. meant for core developers

# export hbase_master_opts="$hbase_master_opts -xdebug -xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"

# export hbase_regionserver_opts="$hbase_regionserver_opts -xdebug -xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"

# export hbase_thrift_opts="$hbase_thrift_opts -xdebug -xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"

# export hbase_zookeeper_opts="$hbase_zookeeper_opts -xdebug -xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"

# a string representing this instance of hbase. $user by default.

# export hbase_ident_string=$user

# the scheduling priority for daemon processes. see 'man nice'.

# export hbase_niceness=10

export hbase_pid_dir=/var/hadoop/pids

# seconds to sleep between slave commands. unset by default. this

# can be useful in large clusters, where, e.g., slave rsyncs can

# otherwise arrive faster than the master can service them.

# export hbase_slave_sleep=0.1

# tell hbase whether it should manage it's own instance of zookeeper or not.

export hbase_manages_zk=false

# the default log rolling policy is rfa, where the log file is rolled as per the size defined for the

# rfa appender. please refer to the log4j.properties file to see more details on this appender.

# in case one needs to do log rolling on a date change, one should set the environment property

# hbase_root_logger to "<desired_log level>,drfa".

# for example:

# hbase_root_logger=info,drfa

# the reason for changing default to rfa is to avoid the boundary case of filling out disk space as

# drfa doesn't put any cap on the log size. please refer to hbase-5655 for more context.

到此為之，可以把調整配置之後的hbase安裝包，在三台機器上部署，并在bin目錄下，我們将node3作為hmaster，是以需要在node3上執行“start-hbase.sh”,此後hbase叢集将會全部啟動。

可以使用jps指令檢視節點是否啟動正常，在node3上将會看到“hmaster,hregionserver”兩個程序，其他2個node上隻會看到“hregionserver”程序。

原文連結:[http://wely.iteye.com/blog/2347826]

【轉】hadoop/hbase搭建

繼續閱讀

Java小案例——随機數猜測随機數猜測

nginx location中斜線的位置的重要性

27 Best Free Eclipse Plug-ins for Java Developer to be ProductiveCode Quality PluginsText Editor PluginsDependency ManagementVersion Control Integration PluginsFramework Development Continuous Integration Related PluginsOther Utility Plugins

Java String.format方法的簡單使用

neo4j之cypher使用文檔

Ambari介紹和架構原理

GitHub連夜封殺！這份阿裡 10W 字内部 Java 字面試手冊到底有多強？

spark/scala關于【資源檔案】加載方法概述外部檔案加載方案測試資源檔案打包入jar包中小結

mybatis_入門程式Mybatis入門

AOP程式設計_Android優雅權限架構(1)概念基礎，2021金三銀四前言正文大綱正文

Effective Java 8:通用程式設計

OOM三種類型

工廠模式-三種類型

【遞歸】高效率求2的n次幂

win10本地scala和spark安裝安裝scala安裝spark

scala (3) Function 和 Method