天天看点

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

1.以前我搭建的事0.7.3对应的事spark2.1,现在spark-2.3以后zeppelin-0.7.3不兼容,所以使用0.8不过也折腾了半天jar包不匹配

2.修改配置文件

cd  /zeppelin-0.8.0-bin-all/conf

cp  zeppelin-env.sh.template zeppelin-env.sh

cp zeppelin-site.xml.template zeppelin-site.xml

vim zeppelin-env.sh
           

 配置文件修改

export HIVE_HOME=/data/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive
export JAVA_HOME=/usr/java/jdk1.8.0_144
export MASTER=yarn-client
export ZEPPELIN_JAVA_OPTS="-Dmaster=yarn-client -Dspark.yarn.jar=/data/zeppelin/zeppelin-0.8.0-bin-all/interpreter/spark/spark-interpreter-0.8.0.jar"
#export SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0"
export DEFAULT_HADOOP_HOME=/data/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop
export SPARK_HOME=/data/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2
#export SPARK_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark
export HADOOP_HOME=${HADOOP_HOME:-$DEFAULT_HADOOP_HOME}
echo $HADOOP_HOME
echo $SPARK_HOME
if [ -n "$HADOOP_HOME" ]; then
        export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
fi
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}


export ZEPPELIN_LOG_DIR=/var/log/zeppelin
export ZEPPELIN_PID_DIR=/var/run/zeppelin
export ZEPPELIN_WAR_TEMPDIR=/var/tmp/zeppelin
           

3.配置这些其实已经足够了,还要把hive-site.xml拷贝到zeppelin的conf目录。

在启动

./zeppelin-daemon.sh start
           

4.在界面上配置就可以使用了hive的配置这个

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

以下的lib都是zeppelin-0.8.0-bin-all/lib这个目录

复制spark2.3的/data/cloudera/parcels/SPARK2/lib/spark2/jars 到zeppelin-0.8.0-bin-all/interpreter/spark/dep/ 

把以前的dep目录命名为dep2

hadoop的包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

 json的包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

impala包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

netty包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

impala的插件配置

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

可以进行spark查询了

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

直接写sql也行

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

继续阅读