转自：https://blog.csdn.net/jssg_tzw/article/details/73549284
Apache impala集成apache kudu
1.环境准备
1.1.服务器配置
| ID | IP address | Hostname | Notes |
|---|---|---|---|
| 1 | 192.168.56.181 | apollo.dt.com | Hadoop Master 2. Spark Master 3.Kudu Master 4. Impala Master 5. JDK 6. Scala |
| 2 | 192.168.56.182 | artemis.dt.com | Hadoop Slave 2. Spark Slave 3.Kudu tserver 4. Impalad 5. JDK 6. Scala |
| 3 | 192.168.56.183 | uranus.dt.com | Hadoop Slave 2. Spark Slave 3.Kudu tserver 4. Impalad 5. JDK 6. Scala |
1.2.环境配置
- 1.有关CentOS7.0安装配置JDK请参考CentOS7安装配置JDK1.8
- 2.有关Hadoop2.7.3 集群配置请参考CentOS7上Hadoop集群的搭建
- 3.有关Spark 2.10 集群搭建请参考CentOS7上Spark集群拱建
- 4.有关Scala 安装配置请参考CentOS7上安装配置Scala
- 5.有关Kudu 1.3.1集群搭建请参考CentOS7上安装配置Kudu集群
1.3. impala 安装包下载
- 下载impala安装包，下载地址：http://archive.cloudera.com/beta/impala-kudu/redhat/7/x86_64/impala-kudu/0/RPMS/x86_64/
CentOS7上Apache impala2.7集成apache kudu1.3Apache impala集成apache kudu - 下载 bigtop-utils，下载地址：http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.9.0/RPMS/noarch/
CentOS7上Apache impala2.7集成apache kudu1.3Apache impala集成apache kudu
2.下载安装依赖包
[[email protected] impala2.7]# yum -y install cyrus-sasl-plain lsb ntp
3.安装impala
3.1.安装 bigtop-utils (主从机都要安装)
[[email protected] impala2.7]# rpm -ivh bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el7.noarch.rpm
warning: bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el7.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:bigtop-utils-0.7.0+cdh5.9.0+0-1.c################################# [100%]
3.2.安装 impala-kudu-2.7.0 (主从机都要安装)
# 要加上选项--nodeps否则安装会因为缺少依赖包无法通过
[[email protected] impala2.7]# rpm -ivh impala-kudu-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm --nodeps
warning: impala-kudu-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-kudu-2.7.0+cdh5.9.0+0-1.cd################################# [100%]
3.3.安装 impala-kudu-catalog (主机安装)
[[email protected] impala2.7]# rpm -ivh impala-kudu-catalog-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-catalog-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-kudu-catalog-2.7.0+cdh5.9.################################# [100%]
3.4.安装 impala-kudu-state (主机安装)
[[email protected] impala2.7]# rpm -ivh impala-kudu-state-store-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-state-store-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-kudu-state-store-2.7.0+cdh################################# [100%]
3.5.主机安装 impala-kudu-server (主从机都要安装)
[[email protected] impala2.7]# rpm -ivh impala-kudu-server-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-server-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-kudu-server-2.7.0+cdh5.9.0################################# [100%]
3.6.安装impala-kudu-shell (主从机都要安装)
[[email protected] impala2.7]# rpm -ivh impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-kudu-shell-2.7.0+cdh5.9.0+################################# [100%]
3.7.安装 impala-kudu-udf-devel(主从机都要安装)
[[email protected] impala2.7]# rpm -ivh impala-kudu-udf-devel-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-udf-devel-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-kudu-udf-devel-2.7.0+cdh5.################################# [100%]
4.配置impala (主从机都要配置)
4.1. 配置 vim /etc/default/bigtop-utils里的JDK
# export JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_77
4.2.配置vim /etc/default/impala (主从机都要配置)
IMPALA_CATALOG_SERVICE_HOST=apollo.dt.com
IMPALA_STATE_STORE_HOST=apollo.dt.com
5.配置三台机器时间同步（主从机都要配置)
#主机：apollo.dt.com
[[email protected] kudu]# systemctl restart ntpd
#从机：artemis.dt.com
[[email protected] kudu]# systemctl restart ntpd
#从机：uranus.dt.com
[[email protected] kudu]# systemctl restart ntpd
#主机apollo.dt.com设置开机启动
[[email protected] kudu]# systemctl enable ntpd
#从机artemis.dt.com设置开机启动
[[email protected] kudu]# systemctl enable ntpd
#从机uranus.dt.com设置开机启动
[[email protected] kudu]# systemctl enable ntpd
6.把hadoop配置文件core-site.xml和hdfs-site.xml拷到/etc/impala/conf.dist/目录下并作如下改动
6.1 拷贝core-site.xml和hdfs-site.xml (主从机都要配置)
#主机apollo.dt.com
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/
#从机artemis.dt.com
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/
#从机uranus.dt.com
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[[email protected] .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/
6.2.具体配置（在hadoop原有内容上增加以下内容）
- core-site.xml
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.skip.checksum</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
- hdfs-site.xml
<!--impala configuration -->
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.block.local-path-access.user</name>
<value>impala</value>
</property>
<property>
<name>dfs.client.file-block-storage-locations.timeout.millis</name>
<value>60000</value>
</property>
7. 设置scoket path
在目录/var/run/目录下新建目录hadoop-hdfs (主从机都要配置)。
注意：该文件夹可能已经存在，应当确认用impala是否有权限进行读写。
如果已经存在，将用户impala加入该文件所属的组，并修改该文件组的权限即: chown -R 775 hadoop-hdfs/
#主机：apollo.dt.com
[[email protected] .ssh]# cd /var/run/ #进入目录/var/run/
[[email protected] run]# mkdir hadoop-hdfs #创建目录hadoop-hdfs
[[email protected] run]# chown -R 775 hadoop-hdfs/ #设置权限
#从机：artemis.dt.com
[[email protected] .ssh]# cd /var/run/ #进入目录/var/run/
[[email protected] run]# mkdir hadoop-hdfs #创建目录hadoop-hdfs
[[email protected] run]# chown -R 775 hadoop-hdfs/ #设置权限
#从机：uranus.dt.com
[[email protected] .ssh]# cd /var/run/ #进入目录/var/run/
[[email protected] run]# mkdir hadoop-hdfs #创建目录hadoop-hdfs
[[email protected] run]# chown -R 775 hadoop-hdfs/ #设置权限
8. 权限配置（所有机器都应当配置）
如果想要 impala 和 YARN 合作，需要把 impala 用户加入 hdfs 组，相关的可以了解的是Llama项目.
impala 在执行 DROP TABLE 操作时，需要把文件移到到 hdfs 的回收站，所以你需要创建一个 hdfs 的目录 /user/impala，并将其设置为impala 用户可写。同样的，impala 需要读取 hive 数据仓库下的数据，故需要把 impala 用户加入 hive 组。
#主机：apollo.dt.com
[[email protected] hadoop]# usermod -G hdfs,hadoop impala
[[email protected] hadoop]# groups impala
impala : impala hdfs hadoop
#从机：artemis.dt.com
[[email protected] hadoop]# usermod -G hdfs,hadoop impala
[[email protected] hadoop]# groups impala
impala : impala hdfs hadoop
#从机：uranus.dt.com
[[email protected] hadoop]# usermod -G hdfs,hadoop impala
[[email protected] hadoop]# groups impala
impala : impala hdfs hadoop
Extension: 创建impala在hdfs上的目录并设置权限
[[email protected] sbin]$ hadoop fs -mkdir -p /user/impala
[[email protected] sbin]$ hadoop fs -chown impala /user/impala
8.启动impala
#主机apollo.dt.com
[[email protected] impala]# service impala-state-store restart --kudu_master_hosts=apollo.dt.com:7051
[[email protected] impala]# service impala-catalog restart --kudu_master_hosts=apollo.dt.com:7051
[[email protected] impala]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051
#从机：artemis.dt.com
[[email protected] .ssh]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051
#从机：uranus.dt.com
[[email protected] .ssh]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051
9. 登录impala-shell
[[email protected] impala]# impala-shell
Starting Impala Shell without Kerberos authentication
Connected to apollo.dt.com:21000
Server version: impalad version 2.7.0-cdh5-IMPALA_KUDU-cdh5 RELEASE (build 48f1ad385382cd90dbaed53b174965251d91d088)
***********************************************************************************
Welcome to the Impala shell. Copyright (c) 2015 Cloudera, Inc. All rights reserved.
(Impala Shell v2.7.0-cdh5-IMPALA_KUDU-cdh5 (48f1ad3) built on Thu Aug 18 12:15:44 PDT 2016)
Run the PROFILE command after a query has finished to see a comprehensive summary
of all the performance and diagnostic information that Impala gathered for that
query. Be warned, it can be very long!
***********************************************************************************
[apollo.dt.com:21000] >
10.在impala-shell端创建表
[apollo.dt.com:21000] > CREATE TABLE dept (
id BIGINT,
name STRING
) distribute by range(name) split rows(('it'),('op'),('hr'))
TBLPROPERTIES(
'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler',
'kudu.table_name' = 'dept',
'kudu.master_addresses' = 'apollo.dt.com:7051',
'kudu.key_columns' = 'id,name'
);
11.在kudu控制台查看impala-shell创建的表