maven:3.3.9
jdk:java version "1.8.0_51"
spark:spark-1.6.1.tgz
scala:2.11.7
如果scala版本是2.11.x,執行如下腳本
./dev/change-scala-version.sh 2.11
spark預設情況下用scala的2.10.5編譯
編譯指令如下:
mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -Dscala-2.11 -DskipTests clean package
紅色部分為spark-sql連結hive的所需要的依賴,以及指定的scala的版本
注意:hive-site.xml檔案需要放到$SPARK_HOME/conf目錄下,否則找不到表
運用spark-sql通路hive
package com.infra.codelab.spark.hive
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object HiveTest {
val conf = new SparkConf()
val sc = new SparkContext(conf)
def main(args: Array[String]): Unit = {
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.sql("SELECT line FROM filecontent ").collect().foreach(println)
}
}
送出任務:
spark-submit --class com.infra.codelab.spark.hive.HiveTest --master spark://localhost:7077 /home/xiaobin/test/spark/wordcount-0.0.1-SNAPSHOT.jar
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/xiaobin/soft/apache-hive-0.14.0-bin/lib/mysql-connector-java-5.1.35.jar
spark-sql --master spark://xiaobin:7077
spark-sql> select count(*) from filecontent;
483
Time taken: 3.628 seconds, Fetched 1 row(s)