伺服器配置資訊:CentOS6.5 最小化安裝+development tools組包,其餘元件yum安裝即可。
1、伺服器資訊如下(/etc/hosts檔案):
192.168.100.231 db01.chavin.king db01
192.168.100.232 db02.chavin.king db02
192.168.100.233 db03.chavin.king db03
192.168.100.234 db04.chavin.king db04
192.168.100.235 db05.chavin.king db05
192.168.100.236 db06.chavin.king db06
192.168.100.237 db07.chavin.king db07
2、建立普通使用者及密碼:
groupadd hadoop
useradd -g hadoop hadoop
echo "dbking588" | passwd --stdin hadoop
3、配置hadoop賬号sudo權限(/etc/sudoers):
chmod u+w /etc/sudoers
echo "hadoop ALL=(root)NOPASSWD:ALL" >> /etc/sudoers
chmod u-w /etc/sudoers
4、關閉防火牆并且禁用selinux
sed -i '/SELINUX=enforcing/d' /etc/selinux/config
sed -i '/SELINUX=disabled/d' /etc/selinux/config
echo "SELINUX=disabled" >> /etc/selinux/config
sed -e 's/SELINUX=enforcing/SELINUX=disabled/d' /etc/selinux/config
5、設定檔案打開數量及最大程序數
cp /etc/security/limits.conf /etc/security/limits.conf.bak
echo "* soft nproc 32000" >>/etc/security/limits.conf
echo "* hard nproc 32000" >>/etc/security/limits.conf
echo "* soft nofile 65535" >>/etc/security/limits.conf
echo "* hard nofile 65535" >>/etc/security/limits.conf
6、配置叢集時間同步服務
cp /etc/ntp.conf /etc/ntp.conf.bak
cp /etc/sysconfig/ntpd /etc/sysconfig/ntpd.bak
echo "restrict 192.168.100.0 mask 255.255.255.0 nomodify notrap" >> /etc/ntp.conf
echo "SYNC_HWCLOCK=yes" >> /etc/sysconfig/ntpd
service ntpd restart
0-59/10 * * * * /opt/scripts/sync_time.sh
# cat /opt/scripts/sync_time.sh
/sbin/service ntpd stop
/usr/sbin/ntpdate db01.chavin.king
/sbin/service ntpd start
1、安裝mysql資料庫(mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz)
tar -zxvf mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz -C /opt/
mv mysql-5.6.24-linux-glibc2.5-x86_64/ mysql5624
yum -y install libaio
groupadd mysql
useradd -g mysql mysql
echo "dbking588" | passwd --stdin mysql
mv /etc/my.cnf /etc/my.cnf.bak
cp support-files/my-default.cnf /etc/my.cnf
/etc/my.cnf添加如下内容:
----------------------------------------------------------------------------------------------------
basedir = /opt/mysql5624
datadir = /opt/mysql5624/data
port = 3306
skip-grant-tables
log-error=/opt/mysql5624/data/mysql.log
pid-file=/opt/mysql5624/data/mysql.pid
scripts/mysql_install_db --user=mysql
chown -R mysql:mysql /opt/mysql5624/
cp support-files/mysql.server /etc/rc.d/init.d/mysql
chkconfig --add mysql
Service mysql start
2、安裝cloudera內建postgresql資料庫
--需要以以下方式安裝postgresql資料庫:
[root@db01 postgresq-libs]# ll
total 6564
-rw-r--r-- 1 root root 2905984 Apr 16 23:58 postgresql-8.4.18-1.el6_4.x86_64.rpm
-rw-r--r-- 1 root root 205732 Apr 16 23:58 postgresql-libs-8.4.18-1.el6_4.x86_64.rpm
-rw-r--r-- 1 root root 3602880 Apr 16 23:58 postgresql-server-8.4.18-1.el6_4.x86_64.rpm
[root@db01 postgresq-libs]# rpm -ivh *.rpm
Preparing... ########################################### [100%]
1:postgresql-libs ########################################### [ 33%]
2:postgresql ########################################### [ 67%]
3:postgresql-server ########################################### [100%]
1、軟體下載下傳:
安裝版本CM 5.3.6
總下載下傳位址:http://archive.cloudera.com/cm5/
cm-5.3.6 bin檔案下載下傳位址:http://archive.cloudera.com/cm5/installer/5.3.6/
cm-5.3.6依賴rpm包:http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/
2、配置本地源
1)安裝papche伺服器:
yum -y install httpd
service httpd start
chkconfig httpd on
cd /var/www/html/
mkdir -p cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/
--将下載下傳好的cm5依賴包mv到/var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/目錄下:
[root@db01 x86_64]# ll
total 700568
-rw-r--r-- 1 root root 3989520 Apr 16 22:21 cloudera-manager-agent-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rw-r--r-- 1 root root 499418684 Apr 16 22:22 cloudera-manager-daemons-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rw-r--r-- 1 root root 7852 Apr 16 22:21 cloudera-manager-server-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rw-r--r-- 1 root root 9884 Apr 16 22:21 cloudera-manager-server-db-2-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rw-r--r-- 1 root root 693024 Apr 16 22:21 enterprise-debuginfo-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rw-r--r-- 1 root root 71204325 Apr 16 22:21 jdk-6u31-linux-amd64.rpm
-rw-r--r-- 1 root root 142039186 Apr 16 22:21 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
--建立repodata相關依賴檔案、否則安裝cm将預設查找最新版本(/var/www/html/cm5/redhat/6/x86_64/cm/5.3.6目錄下):
[root@db01 repo-libs]# ll
total 196
-rw-r--r-- 1 root root 96552 Apr 16 23:21 createrepo-0.9.9-18.el6.noarch.rpm
-rw-r--r-- 1 root root 72520 Apr 16 23:21 deltarpm-3.5-0.5.20090913git.el6.x86_64.rpm
-rw-r--r-- 1 root root 27748 Apr 16 23:21 python-deltarpm-3.5-0.5.20090913git.el6.x86_64.rpm
[root@db01 repo-libs]# rpm -ivh *.repo
[root@db01 repo-libs]# cd /var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/
[root@db01 5.3.6]# createrepo .
2)配置repo檔案
[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=http://db01.chavin.king/cm5/redhat/6/x86_64/cm/5.3.6/
enabled=1
gpgcheck=0
3、安裝CM5
./cloudera-manager-installer.bin
4、浏覽器登入,我這裡登入位址為:db01:7180,使用者名密碼預設admin/admin
1、下載下傳parcels軟體包(安裝版本CDH 5.3.6):http://archive.cloudera.com/cdh5/parcels/5.3.6/
2、上傳檔案到/opt/cloudera/parcel-repo/目錄下:
[root@db01 parcel-repo]# ll
total 725736
-rw-r--r-- 1 root root 743145472 Apr 17 12:23 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel
-rw-r--r-- 1 root root 41 Apr 17 12:21 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1
[root@db01 parcel-repo]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
[root@db01 parcel-repo]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
total 1473856
-rw-r--r-- 1 root root 1509217191 Apr 17 12:24 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
-rw-r--r-- 1 root root 41 Apr 17 12:21 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
3、重新開機cloudera服務:
[root@db01 parcel-repo]# service cloudera-scm-server status
cloudera-scm-server (pid 22823) is running...
[root@db01 parcel-repo]# service cloudera-scm-server restart
Stopping cloudera-scm-server: [ OK ]
Starting cloudera-scm-server: [ OK ]
4、配置yum源:
将db01上yum檔案同步到db02、db03、db04、db05、db06、db07/etc/yum.repo.d/上。
5、向叢集中添加主機:
執行如下指令安裝依賴包:
#yum -y install cyrus-sasl-gssapi fuse cyrus-sasl-plain libxslt fuse-libs redhat-lsb portmap bind-utils
#yum -y install libxslt fuse-libs
注意:安裝以上依賴包才可以正确安裝agent服務,否則很可能報錯,需要根據實際情況處理。
問題:
解決辦法:
# sysctl -w vm.swappiness=0
# echo "vm.swappiness=0" >>/etc/sysctl.conf
6、配置java環境變量
echo "export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera" >> /etc/profile
echo "export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH" >>/etc/profile
echo "export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH" >> /etc/profile
source /etc/profile
7、版本彙總
Cluster 1 — CDH 5
主機
db[01-07].chavin.king
元件
版本
發行版
CDH 版本
Bigtop-Tomcat(僅限 CDH 5)
0.7.0+cdh5.3.6+0
1.cdh5.3.6.p0.18
CDH 5
Crunch(僅限 CDH 5 )
0.11.0+cdh5.3.6+31
Flume NG
1.5.0+cdh5.3.6+93
MapReduce 1
2.5.0+cdh5.3.6+898
Hadoop
HDFS
HttpFS
hadoop-kms
MapReduce 2
YARN
HBase
0.98.6+cdh5.3.6+115
Lily HBase Indexer
1.5+cdh5.3.6+31
Hive
0.13.1+cdh5.3.6+397
HCatalog
Hue
3.7.0+cdh5.3.6+203
Impala
2.1.5+cdh5.3.6+0
Kite(僅限 CDH 5 )
0.15.0+cdh5.3.6+201
Llama(僅限 CDH 5 )
1.0.0+cdh5.3.6+0
Mahout
0.9+cdh5.3.6+25
Oozie
4.0.0+cdh5.3.6+349
Parquet
1.5.0+cdh5.3.6+69
Pig
0.12.0+cdh5.3.6+59
sentry
1.4.0+cdh5.3.6+155
Solr
4.4.0+cdh5.3.6+352
spark
1.2.0+cdh5.3.6+379
Sqoop
1.99.4+cdh5.3.6+32
1.4.5+cdh5.3.6+78
Whirr
0.9.0+cdh5.3.6+19
ZooKeeper
3.4.5+cdh5.3.6+91
Cloudera Manager Management Daemon
5.3.6
1.cm536.p0.244
不适用
Java 6
JAVA_HOME=/usr/java/jdk1.6.0_31 java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
不可用
Java 7
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
Cloudera Manager Agent
1.cm536.p0.244.el6
圖形界面(略),以下同此要求。
1、安裝zookeeper
2、安裝hdfs
3、安裝yarn
4、安裝hive
5、安裝hbase
進行相關基準測試。