第一、前置準備
1、基礎環境規劃
作業系統：CentOS Linux release 7.4.1708 (Core)
CDM版本: 5.12.1
CDH版本：5.12.1
MySQL版本:5.7.13
浏覽器版本: ChromeStandalone_56以上、IE10
記憶體:16G以上
CPU :8core
網絡:千兆以上
2、主機用途規劃
主機名稱 IP位址 用途 備注
cdh01.domain 172.27.9.191 cm、namendoe cdm server
cdh02.domain 172.27.9.192 Datanode、NodeManager
cdh03.domain 172.27.9.193 Datanode、NodeManager
cdh04.domain 172.27.9.194 Datanode、NodeManager
cdh05.domain 172.27.9.195 Datanode、NodeManager
cdh06.domain 172.27.9.196 Datanode、NodeManager
cdh07.domain 172.27.9.197 Datanode、NodeManager Mairadb server
注意: 一般5台,至少3台
3、設定hosts檔案
将以下檔案追加在/etc/hosts中
#cat /etc/hosts
172.27.9.191 cdh01.localdomain cdh01
172.27.9.192 cdh02.localdomain cdh02
172.27.9.193 cdh03.localdomain cdh03
172.27.9.194 cdh04.localdomain cdh04
172.27.9.195 cdh05.localdomain cdh05
172.27.9.196 cdh06.localdomain cdh06
172.27.9.197 cdh07.localdomain cdh07
4、linux目錄規劃
目錄名稱 目錄大小(推薦) 備注
/opt 100G以上 Hadoop程式目錄
/var 100G以上 日志目錄
/data/sdv1 1TB以上 Datanode目錄
/data/sdv2 1TB以上 Datanode目錄
/data/sdv..n 1TB以上 Datanode目錄
第二、Linux作業系統安裝
1、配置核心參數(所有節點)
# cat /etc/sysctl.conf
##核心預設參數
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
##打開檔案數參數(20*1024*1024)
fs.file-max= 20971520
##WEB Server參數
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1200
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_rmem=4096 87380 8388608
net.ipv4.tcp_wmem=4096 87380 8388608
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_max_tw_buckets = 5000
##TCP補充參數
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
##禁用ipv6
net.ipv6.conf.all.disable_ipv6 =1
net.ipv6.conf.default.disable_ipv6 =1
##swap使用率優化
vm.swappiness=0
執行以下指令使參數生效:
# sysctl -p
2、配置打開檔案數(所有節點)
# vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
3、關閉防火牆(所有節點)
# service iptables stop
# chkconfig iptables off
關閉SELINUX
# vi /etc/selinux/config
修改SELINUX=enforcing為 SELINUX=disabled
4、時間同步(所有節點)
#/usr/sbin/ntpdate -us 172.27.5.123;hwclock -w;
6、關閉大葉記憶體(所有節點)
#cat << EOF >> /etc/rc.d/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
EOF
7、系統預設語言(所有節點)
#env |grep LANG
如果不是en_US.UTF-8，請修改/etc/locale.conf中的LANG為”en_US.UTF-8”
注意:優化完成後，必須重新開機所有主機
安裝額外依賴包：
yum -y install libxslt-devel psmisc python-lxml
第三、前置環境
參考連結：https://www.cloudera.com/documentation/enterprise/release-notes/topics/rg_vd.html
1、CDH安裝包下載下傳
http://archive.cloudera.com/cdh5/parcels/5.12/
下載下傳對應作業系統的3個檔案
CDH-5.12.1-1.cdh5.12.1.p0.3-el6.parcel
CDH-5.12.1-1.cdh5.12.1.p0.3-el6.parcel.sha1
manifest.json
2、CM安裝包下載下傳
從http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.12.1/RPMS/x86_64/下載下傳所有包
上傳下載下傳的包至yum伺服器，并搭建本地yum源server,實作通路http://172.27.10.200/cdh/ 如下效果：
repo檔案配置
#cat /etc/yum.repos.d/cm.repo
[cm]
name=cm5.12
baseurl=http://172.27.10.200/cdh/
enabled=1
gpgcheck=0
priority=1
3、JDBC驅動下載下傳與安裝
參考連結:
http://www.cloudera.com/documentation/cdh/5-1-x/CDH5-Installation-Guide/cdh5ig_jdbc_driver_install.html
1)、下載下傳JDBC驅動
# mkdir -p /usr/share/java && cd /usr/share/java
# wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.44.tar.gz
# tar zxf mysql-connector-java-5.1.44.tar.gz
# cp mysql-connector-java-5.1.44/mysql-connector-java-5.1.44-bin.jar .
# mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar
# chmod +x mysql-connector-java.jar
4、安裝JDK環境(每個節點)
#yum -y install jdk oracle-j2sdk1.7
5、mariadb部署(cdh07)
參考連結: https://www.cloudera.com/documentation/enterprise/5-5-x/topics/install_cm_mariadb.html
1)、資料庫安裝
#yum -y install mariadb-server
2)、my.cnf配置
# cp /etc/my.cnf /etc/my.cnfbak
#cat /etc/my.cnf
[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0
key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
# systemctl enable mariadb
# systemctl list-unit-files | grep mariadb
2)、建立資料庫
建立Hive資料庫:
mysql >create database hive512 default character set latin1;
字元必須為拉丁(latin1),如果uft8可能會報錯Specified key was too long; max key length is 767 bytes
mysql >grant all privileges on hive512.* to [email protected] identified by '123456';
mysql >grant all privileges on hive512.* to hive512@'172.27.%' identified by '123456';
mysql >flush privileges;
建立report資料庫
mysql >create database report512 character set utf8;
mysql >grant all privileges on report512.* to [email protected] identified by '123456';
mysql >grant all privileges on report512.* to report512@'172.27.%' identified by '123456';
mysql >flush privileges;
建立monitor資料庫
mysql>create database monitor512 character set utf8;
mysql>grant all privileges on monitor512.* to [email protected] identified by '123456';
mysql>grant all privileges on monitor512.* to monitor512@'172.27.%' identified by '123456';
mysql>flush privileges;
建立sentry資料庫
mysql>create database sentry512 character set utf8;
mysql>grant all privileges on sentry512.* to [email protected] identified by '123456';
mysql>grant all privileges on sentry512.* to sentry512@'172.27.%' identified by '123456';
mysql>flush privileges;
第四、部署CDM-manager
1、CDM部署
#yum install cloudera-manager-server
2、建立cmf資料庫
mysql >create database cmf character set utf8;
mysql >grant all privileges on cmf.* to [email protected] identified by 'cmf123456';
mysql >grant all privileges on cmf.* to cmf@'172.27.%' identified by 'cmf123456';
mysql >flush privileges;
3、修改配置檔案
# cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak
# vi /etc/cloudera-scm-server/db.properties
4、啟動服務
# /etc/init.d/cloudera-scm-server restart
# /etc/init.d/cloudera-scm-server status
至此cdm-server部署完成
第五、部署cdm-agent
1、agent安裝
# yum -y install cloudera-manager-daemons cloudera-manager-agent
2、修改用戶端配置檔案config.ini
#cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak
#sed -i '/server_host/s/localhost/cdh01.localdomain/' /etc/cloudera-scm-agent/config.ini
3、啟動用戶端
# /etc/init.d/cloudera-scm-agent restart
# /etc/init.d/cloudera-scm-agent status
至此cdm-agent部署完成
第六、CDH部署
1、上傳檔案
上傳檔案到windcdh01節點的/opt/cloudera/parcel-repo目錄下
#cd /opt/cloudera/parcel-repo
# mv CDH-5.10.1-1.cdh5.10.1.p0.10-el6.parcel.sha1 CDH-5.10.1-1.cdh5.10.1.p0.10-el6.parcel.sha
如果不重命名CDH會識别不了版本
2、檢查目錄并授權
# cd /opt/cloudera/parcel-repo
# chown -R cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo
檢查檔案(注意使用者名和權限)
至此CDH/CDM部署完成
第七、利用web部署hadoop
1、上傳檔案
http://172.27.9.191:7180
賬号:admin 密碼:admin
隻要檔案路徑配置正确,已下載下傳就是100%,如果出現進度條1%，必須檢查檔案和用戶端配置
檢查結果要仔細看下，如果有異常，按照提示解決即可。
這裡選擇安裝元件:
配置設定角色，注意保障節點的元件數均衡：
檢查下圖紅框部分：
安裝完成後進入如下界面