第一、前置准备
1、基础环境规划
操作系统：CentOS Linux release 7.4.1708 (Core)
CDM版本: 5.12.1
CDH版本：5.12.1
MySQL版本:5.7.13
浏览器版本: ChromeStandalone_56以上、IE10
内存:16G以上
CPU :8core
网络:千兆以上
2、主机用途规划
主机名称 IP地址 用途 备注
cdh01.domain 172.27.9.191 cm、namendoe cdm server
cdh02.domain 172.27.9.192 Datanode、NodeManager
cdh03.domain 172.27.9.193 Datanode、NodeManager
cdh04.domain 172.27.9.194 Datanode、NodeManager
cdh05.domain 172.27.9.195 Datanode、NodeManager
cdh06.domain 172.27.9.196 Datanode、NodeManager
cdh07.domain 172.27.9.197 Datanode、NodeManager Mairadb server
注意: 一般5台,至少3台
3、设置hosts文件
将以下文件追加在/etc/hosts中
#cat /etc/hosts
172.27.9.191 cdh01.localdomain cdh01
172.27.9.192 cdh02.localdomain cdh02
172.27.9.193 cdh03.localdomain cdh03
172.27.9.194 cdh04.localdomain cdh04
172.27.9.195 cdh05.localdomain cdh05
172.27.9.196 cdh06.localdomain cdh06
172.27.9.197 cdh07.localdomain cdh07
4、linux目录规划
目录名称 目录大小(推荐) 备注
/opt 100G以上 Hadoop程序目录
/var 100G以上 日志目录
/data/sdv1 1TB以上 Datanode目录
/data/sdv2 1TB以上 Datanode目录
/data/sdv..n 1TB以上 Datanode目录
第二、Linux操作系统安装
1、配置内核参数(所有节点)
# cat /etc/sysctl.conf
##内核默认参数
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
##打开文件数参数(20*1024*1024)
fs.file-max= 20971520
##WEB Server参数
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1200
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_rmem=4096 87380 8388608
net.ipv4.tcp_wmem=4096 87380 8388608
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_max_tw_buckets = 5000
##TCP补充参数
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
##禁用ipv6
net.ipv6.conf.all.disable_ipv6 =1
net.ipv6.conf.default.disable_ipv6 =1
##swap使用率优化
vm.swappiness=0
执行以下命令使参数生效:
# sysctl -p
2、配置打开文件数(所有节点)
# vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
3、关闭防火墙(所有节点)
# service iptables stop
# chkconfig iptables off
关闭SELINUX
# vi /etc/selinux/config
修改SELINUX=enforcing为 SELINUX=disabled
4、时间同步(所有节点)
#/usr/sbin/ntpdate -us 172.27.5.123;hwclock -w;
6、关闭大叶内存(所有节点)
#cat << EOF >> /etc/rc.d/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
EOF
7、系统默认语言(所有节点)
#env |grep LANG
如果不是en_US.UTF-8，请修改/etc/locale.conf中的LANG为”en_US.UTF-8”
注意:优化完成后，必须重启所有主机
安装额外依赖包：
yum -y install libxslt-devel psmisc python-lxml
第三、前置环境
参考链接：https://www.cloudera.com/documentation/enterprise/release-notes/topics/rg_vd.html
1、CDH安装包下载
http://archive.cloudera.com/cdh5/parcels/5.12/
下载对应操作系统的3个文件
CDH-5.12.1-1.cdh5.12.1.p0.3-el6.parcel
CDH-5.12.1-1.cdh5.12.1.p0.3-el6.parcel.sha1
manifest.json
2、CM安装包下载
从http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.12.1/RPMS/x86_64/下载所有包
上传下载的包至yum服务器，并搭建本地yum源server,实现访问http://172.27.10.200/cdh/ 如下效果：
repo文件配置
#cat /etc/yum.repos.d/cm.repo
[cm]
name=cm5.12
baseurl=http://172.27.10.200/cdh/
enabled=1
gpgcheck=0
priority=1
3、JDBC驱动下载与安装
参考链接:
http://www.cloudera.com/documentation/cdh/5-1-x/CDH5-Installation-Guide/cdh5ig_jdbc_driver_install.html
1)、下载JDBC驱动
# mkdir -p /usr/share/java && cd /usr/share/java
# wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.44.tar.gz
# tar zxf mysql-connector-java-5.1.44.tar.gz
# cp mysql-connector-java-5.1.44/mysql-connector-java-5.1.44-bin.jar .
# mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar
# chmod +x mysql-connector-java.jar
4、安装JDK环境(每个节点)
#yum -y install jdk oracle-j2sdk1.7
5、mariadb部署(cdh07)
参考链接: https://www.cloudera.com/documentation/enterprise/5-5-x/topics/install_cm_mariadb.html
1)、数据库安装
#yum -y install mariadb-server
2)、my.cnf配置
# cp /etc/my.cnf /etc/my.cnfbak
#cat /etc/my.cnf
[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0
key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
# systemctl enable mariadb
# systemctl list-unit-files | grep mariadb
2)、创建数据库
创建Hive数据库:
mysql >create database hive512 default character set latin1;
字符必须为拉丁(latin1),如果uft8可能会报错Specified key was too long; max key length is 767 bytes
mysql >grant all privileges on hive512.* to [email protected] identified by '123456';
mysql >grant all privileges on hive512.* to hive512@'172.27.%' identified by '123456';
mysql >flush privileges;
创建report数据库
mysql >create database report512 character set utf8;
mysql >grant all privileges on report512.* to [email protected] identified by '123456';
mysql >grant all privileges on report512.* to report512@'172.27.%' identified by '123456';
mysql >flush privileges;
创建monitor数据库
mysql>create database monitor512 character set utf8;
mysql>grant all privileges on monitor512.* to [email protected] identified by '123456';
mysql>grant all privileges on monitor512.* to monitor512@'172.27.%' identified by '123456';
mysql>flush privileges;
创建sentry数据库
mysql>create database sentry512 character set utf8;
mysql>grant all privileges on sentry512.* to [email protected] identified by '123456';
mysql>grant all privileges on sentry512.* to sentry512@'172.27.%' identified by '123456';
mysql>flush privileges;
第四、部署CDM-manager
1、CDM部署
#yum install cloudera-manager-server
2、创建cmf数据库
mysql >create database cmf character set utf8;
mysql >grant all privileges on cmf.* to [email protected] identified by 'cmf123456';
mysql >grant all privileges on cmf.* to cmf@'172.27.%' identified by 'cmf123456';
mysql >flush privileges;
3、修改配置文件
# cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak
# vi /etc/cloudera-scm-server/db.properties
4、启动服务
# /etc/init.d/cloudera-scm-server restart
# /etc/init.d/cloudera-scm-server status
至此cdm-server部署完成
第五、部署cdm-agent
1、agent安装
# yum -y install cloudera-manager-daemons cloudera-manager-agent
2、修改客户端配置文件config.ini
#cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak
#sed -i '/server_host/s/localhost/cdh01.localdomain/' /etc/cloudera-scm-agent/config.ini
3、启动客户端
# /etc/init.d/cloudera-scm-agent restart
# /etc/init.d/cloudera-scm-agent status
至此cdm-agent部署完成
第六、CDH部署
1、上传文件
上传文件到windcdh01节点的/opt/cloudera/parcel-repo目录下
#cd /opt/cloudera/parcel-repo
# mv CDH-5.10.1-1.cdh5.10.1.p0.10-el6.parcel.sha1 CDH-5.10.1-1.cdh5.10.1.p0.10-el6.parcel.sha
如果不重命名CDH会识别不了版本
2、检查目录并授权
# cd /opt/cloudera/parcel-repo
# chown -R cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo
检查文件(注意用户名和权限)
至此CDH/CDM部署完成
第七、利用web部署hadoop
1、上传文件
http://172.27.9.191:7180
账号:admin 密码:admin
只要文件路径配置正确,已下载就是100%,如果出现进度条1%，必须检查文件和客户端配置
检查结果要仔细看下，如果有异常，按照提示解决即可。
这里选择安装组件:
分配角色，注意保障节点的组件数均衡：
检查下图红框部分：
安装完成后进入如下界面