01 第一個 Spark 程式：WordCount

第一個 Spark 程式：WordCount

1. 使用 Spark-shell

1. 準備資料：建立檔案夾

   input

   ，以及

   Words.txt

   檔案

   [[email protected] spark-2.1.1]$ mkdir input
   [[email protected] input]$ vim Words.txt
              

   在檔案中輸入資料：

   hello spark
   hello scala
   hello world
              

2. 進入

   spark-shell

   [[email protected] spark-2.1.1]$ bin/spark-shell 
              

1. 編寫

   WordCount

   程式并運作

   scala> sc.textFile("input/").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect
   res0: Array[(String, Int)] = Array((scala,1), (hello,3), (world,1), (spark,1))  
              

   

2. 使用開發工具 IDEA

1. 建立

   Maven

   項目，并導入如下依賴

   <dependencies>
     <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-core_2.11</artifactId>
       <version>2.1.1</version>
     </dependency>
   </dependencies>
   
   <build>
     <plugins>
       <!-- 打包插件, 否則 scala 類不會編譯并打包進去 -->
       <plugin>
         <groupId>net.alchim31.maven</groupId>
         <artifactId>scala-maven-plugin</artifactId>
         <version>3.4.6</version>
         <executions>
           <execution>
             <goals>
               <goal>compile</goal>
               <goal>testCompile</goal>
             </goals>
           </execution>
         </executions>
       </plugin>
     </plugins>
   </build>
              

2. 建立

   WordCount.scala

   檔案，實作以下代碼

   package com.guli
   
   import org.apache.spark.{SparkConf, SparkContext}
   
   object WordCount {
     def main(args: Array[String]): Unit = {
       val conf: SparkConf = new SparkConf().setAppName("WorldCount").setMaster("local[*]")
       val sc = new SparkContext(conf)
       val wcArray: Array[(String, Int)] = sc.textFile("/Users/zgl/Desktop/input").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect()
       wcArray.foreach(println)
       sc.stop()
     }
   }
              

3. 運作結果

   (scala,1)
   (hello,3)
   (world,1)
   (spark,1)