天天看點

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

1.以前我搭建的事0.7.3對應的事spark2.1,現在spark-2.3以後zeppelin-0.7.3不相容,是以使用0.8不過也折騰了半天jar包不比對

2.修改配置檔案

cd  /zeppelin-0.8.0-bin-all/conf

cp  zeppelin-env.sh.template zeppelin-env.sh

cp zeppelin-site.xml.template zeppelin-site.xml

vim zeppelin-env.sh
           

 配置檔案修改

export HIVE_HOME=/data/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive
export JAVA_HOME=/usr/java/jdk1.8.0_144
export MASTER=yarn-client
export ZEPPELIN_JAVA_OPTS="-Dmaster=yarn-client -Dspark.yarn.jar=/data/zeppelin/zeppelin-0.8.0-bin-all/interpreter/spark/spark-interpreter-0.8.0.jar"
#export SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0"
export DEFAULT_HADOOP_HOME=/data/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop
export SPARK_HOME=/data/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2
#export SPARK_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark
export HADOOP_HOME=${HADOOP_HOME:-$DEFAULT_HADOOP_HOME}
echo $HADOOP_HOME
echo $SPARK_HOME
if [ -n "$HADOOP_HOME" ]; then
        export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
fi
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}


export ZEPPELIN_LOG_DIR=/var/log/zeppelin
export ZEPPELIN_PID_DIR=/var/run/zeppelin
export ZEPPELIN_WAR_TEMPDIR=/var/tmp/zeppelin
           

3.配置這些其實已經足夠了,還要把hive-site.xml拷貝到zeppelin的conf目錄。

在啟動

./zeppelin-daemon.sh start
           

4.在界面上配置就可以使用了hive的配置這個

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

以下的lib都是zeppelin-0.8.0-bin-all/lib這個目錄

複制spark2.3的/data/cloudera/parcels/SPARK2/lib/spark2/jars 到zeppelin-0.8.0-bin-all/interpreter/spark/dep/ 

把以前的dep目錄命名為dep2

hadoop的包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

 json的包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

impala包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

netty包

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

impala的插件配置

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

可以進行spark查詢了

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

直接寫sql也行

SPARK2-2.3.0.cloudera3-1.cdh5.13+zeppelin0.8+impala

繼續閱讀