第二章 spark2.3 mllib机器学习-数据操作

*本系列”spark mllib机器学习”,均以最新spark2.3.0版本为蓝本进行编写,参考书籍<< spark mllib机器学习 >>黄美灵版,转载请注明出处

GitHub地址:https://github.com/future-fu/learnsparkmllib

明细

1.部分spark rdd操作

选取了几个不太常用Rdd算子,但是又比较有意思的,代码如下

package ai.lifecode.datahandle

import ai.lifecode.utils.ShowRddDetails
import org.apache.spark.sql.SparkSession

/**
  * Created by future_fu on 2018/8/31 16:05.
  */
object RDDHandle {
  def main(args: Array[String]): Unit = {
    val spark=SparkSession.builder()
      .master("local[3]")
      .appName("rdd handle")
      .getOrCreate()
    val sc=spark.sparkContext
    //zip 1对1合并操作
    val zipRdd1=sc.parallelize(Array(,,,),)
    val zipRdd2=sc.parallelize(Array("a","b","c","d"),)
    val zipRdd=zipRdd1.zip(zipRdd2)
    println()
   ShowRddDetails.showRDD(zipRdd,"zipRdd",true)

    //subtract减法操作
    val subRdd1=sc.parallelize( to ,)
    val subRdd2=sc.parallelize( to ,)
    val subRdd=subRdd1.subtract(subRdd2)
    ShowRddDetails.showRDD(subRdd,"subRdd",true)

    //randomSplit 根据权重分割
    val randomSplitRdd1=sc.parallelize( to  ,)
    val randomSplitRdd=randomSplitRdd1.randomSplit(Array(,,,))
    ShowRddDetails.showRDD(randomSplitRdd(),"randomSplitRdd 0",true)
    ShowRddDetails.showRDD(randomSplitRdd(),"randomSplitRdd 1",true)
    ShowRddDetails.showRDD(randomSplitRdd(),"randomSplitRdd 2",true)
    ShowRddDetails.showRDD(randomSplitRdd(),"randomSplitRdd 3",true)  

  }

}

运行结果:

=== [class ai.lifecode.utils.ShowRddDetails]: zipRdd ===
(1,a)(2,b)(3,c)(4,d)
=== [class ai.lifecode.utils.ShowRddDetails]: subRdd ===
694758
=== [class ai.lifecode.utils.ShowRddDetails]: randomSplitRdd 0 ===
8
=== [class ai.lifecode.utils.ShowRddDetails]: randomSplitRdd 1 ===
1237
=== [class ai.lifecode.utils.ShowRddDetails]: randomSplitRdd 2 ===
4
=== [class ai.lifecode.utils.ShowRddDetails]: randomSplitRdd 3 ===
569

2.mllib Statistics统计操作

代码如下:

数据源 sample_stat.txt

1   2   3   4   5
6   7   1   5   9
3   5   6   3   1
3   1   1   5   6

package ai.lifecode.datahandle

import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.sql.SparkSession

/**
  * Created by future_fu on 2018/8/31 16:52.
  */
object StatisticsData {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .master("local[3]")
      .appName("rdd handle")
      .getOrCreate()
    val sc = spark.sparkContext
    val path = "data/sample_stat.txt"
    val data = sc.textFile(path)
      .map(_.split("\t"))
      .map(f => f.map(f => f.toDouble))
      .map(f => Vectors.dense(f))
    //计算每列最大值,最小值,平均值,方差值,L1范数,L2范数
    val stat1 = Statistics.colStats(data)
    println("☆┈━═┈━═┈━═┈━═┈━═☆常用计算☆┈━═┈━═┈━═┈━═┈━═☆")
    println(stat1.max)
    println(stat1.min)
    println(stat1.mean)
    println(stat1.variance)
    println(stat1.normL1)
    println(stat1.normL2)

    //相关系数,取值范围[-,] 不相关 [-,]负相关 [,]正相关
    val corr1=Statistics.corr(data,"pearson")
    val corr2=Statistics.corr(data,"spearman")
    val x1=sc.parallelize(Array(,,,))
    val x2=sc.parallelize(Array(,,,))
    val corr3=Statistics.corr(x1,x2,"pearson")
    println("☆┈━═┈━═┈━═┈━═┈━═☆相关系数☆┈━═┈━═┈━═┈━═┈━═☆")
    println(corr1)
    println(corr2)
    println(corr3)

    //假设检验,mllib支持用于判断拟合度或者独立性的Pearson检验.
    // 不同的输入类型决定了是做拟合度检验还是独立性检验.
    // 拟合度输入Vector,独立性输入Matrix

    val v1=Vectors.dense(,)
    val v2=Vectors.dense(,)
    val c1=Statistics.chiSqTest(v1,v2)
    println("☆┈━═┈━═┈━═┈━═┈━═☆假设检验☆┈━═┈━═┈━═┈━═┈━═☆")
    println(v1)
    println(v2)
    println(c1)


  }
}

结果如下:

☆┈━═┈━═┈━═┈━═┈━═☆常用计算☆┈━═┈━═┈━═┈━═┈━═☆
[,,,,]
[,,,,]
[,,,,]
[,,,,]
[,,,,]
[,,,,]
☆┈━═┈━═┈━═┈━═┈━═☆相关系数☆┈━═┈━═┈━═┈━═┈━═☆
                       -  ... ( total)
                          ...
-                        ...
    -  -   ...
       -   ...
                       -  ... ( total)
                         ...
-                       ...
   -  -  ...
      -  ...

☆┈━═┈━═┈━═┈━═┈━═☆假设检验☆┈━═┈━═┈━═┈━═┈━═☆
[,]
[,]
Chi squared test summary:
method: pearson
degrees of freedom =  
statistic =  
pValue =  
Strong presumption against null hypothesis: observed follows the same distribution as expected..

3.mllib的数据格式

util包中包含了如下object

第二章 spark2.3 mllib机器学习-数据操作

MLUtils针对MLlib相关算法进行ETL操作,提供数据.主要方法有:loadLibSVMFile,saveAsLibSVMFile,appendBias,fastSquaredDistance
KMeansDataGenerator 是生成k-means样本模型,方法只有:generateKMeansRDD
LogisticRegressionDataGenerator 逻辑回归的模型,方法只有:generateLogisticRDD
SVMDataGenerator 封装了SVM样本生成方法
MFDataGenerator 封装了MFD样本生成方法

3.1 loadLibSVMFile 加载指定LIBSVM格式文件的方法,根据源码的注释可以了解到

Loads labeled data in the LIBSVM format into an RDD[LabeledPoint].//数据源转化成RDD
The LIBSVM format is a text-based format used by LIBSVM and LIBLINEAR.//基于文本格式
Each line represents a labeled sparse feature vector using the following format:{{{label index1:value1 index2:value2 …}}}, where the indices are one-based and in ascending order.//升序
This method parses each line into a [[org.apache.spark.mllib.regression.LabeledPoint]],where the feature indices are converted to zero-based.//索引从0开始

package ai.lifecode.datahandle

import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.sql.SparkSession

/**
  * Created by future_fu on 2018/9/3 11:42.
  */
object DataFormat {
  def main(args: Array[String]): Unit = {
    val spark=SparkSession.builder()
      .master("local[3]")
      .appName("rdd handle")
      .getOrCreate()
    val sc=spark.sparkContext

    //loadLibSVMFile 加载指定LIBSVM格式文件方法
    val path="data/sample_libsvm_data.txt"
    val data=MLUtils.loadLibSVMFile(sc,path)
    println(data.collect().length)
    println(data.take().toList)

  }
}

结果如下

100
List((0.0,(692,[127,128,129,130,131,154,155,156,157,158,159,181,182,183,184,185,186,187,188,189,207,208,209,210,211,212,213,214,215,216,217,235,236,237,238,239,240,241,242,243,244,245,262,263,264,265,266,267,268,269,270,271,272,273,289,290,291,292,293,294,295,296,297,300,301,302,316,317,318,319,320,321,328,329,330,343,344,345,346,347,348,349,356,357,358,371,372,373,374,384,385,386,399,400,401,412,413,414,426,427,428,429,440,441,442,454,455,456,457,466,467,468,469,470,482,483,484,493,494,495,496,497,510,511,512,520,521,522,523,538,539,540,547,548,549,550,566,567,568,569,570,571,572,573,574,575,576,577,578,594,595,596,597,598,599,600,601,602,603,604,622,623,624,625,626,627,628,629,630,651,652,653,654,655,656,657],[51.0,159.0,253.0,159.0,50.0,48.0,238.0,252.0,252.0,252.0,237.0,54.0,227.0,253.0,252.0,239.0,233.0,252.0,57.0,6.0,10.0,60.0,224.0,252.0,253.0,252.0,202.0,84.0,252.0,253.0,122.0,163.0,252.0,252.0,252.0,253.0,252.0,252.0,96.0,189.0,253.0,167.0,51.0,238.0,253.0,253.0,190.0,114.0,253.0,228.0,47.0,79.0,255.0,168.0,48.0,238.0,252.0,252.0,179.0,12.0,75.0,121.0,21.0,253.0,243.0,50.0,38.0,165.0,253.0,233.0,208.0,84.0,253.0,252.0,165.0,7.0,178.0,252.0,240.0,71.0,19.0,28.0,253.0,252.0,195.0,57.0,252.0,252.0,63.0,253.0,252.0,195.0,198.0,253.0,190.0,255.0,253.0,196.0,76.0,246.0,252.0,112.0,253.0,252.0,148.0,85.0,252.0,230.0,25.0,7.0,135.0,253.0,186.0,12.0,85.0,252.0,223.0,7.0,131.0,252.0,225.0,71.0,85.0,252.0,145.0,48.0,165.0,252.0,173.0,86.0,253.0,225.0,114.0,238.0,253.0,162.0,85.0,252.0,249.0,146.0,48.0,29.0,85.0,178.0,225.0,253.0,223.0,167.0,56.0,85.0,252.0,252.0,252.0,229.0,215.0,252.0,252.0,252.0,196.0,130.0,28.0,199.0,252.0,252.0,253.0,252.0,252.0,233.0,145.0,25.0,128.0,252.0,253.0,252.0,141.0,37.0])), (1.0,(692,[158,159,160,161,185,186,187,188,189,213,214,215,216,217,240,241,242,243,244,245,267,268,269,270,271,295,296,297,298,322,323,324,325,326,349,350,351,352,353,377,378,379,380,381,404,405,406,407,408,431,432,433,434,435,459,460,461,462,463,486,487,488,489,490,514,515,516,517,518,542,543,544,545,569,570,571,572,573,596,597,598,599,600,601,624,625,626,627,652,653,654,655,680,681,682,683],[124.0,253.0,255.0,63.0,96.0,244.0,251.0,253.0,62.0,127.0,251.0,251.0,253.0,62.0,68.0,236.0,251.0,211.0,31.0,8.0,60.0,228.0,251.0,251.0,94.0,155.0,253.0,253.0,189.0,20.0,253.0,251.0,235.0,66.0,32.0,205.0,253.0,251.0,126.0,104.0,251.0,253.0,184.0,15.0,80.0,240.0,251.0,193.0,23.0,32.0,253.0,253.0,253.0,159.0,151.0,251.0,251.0,251.0,39.0,48.0,221.0,251.0,251.0,172.0,234.0,251.0,251.0,196.0,12.0,253.0,251.0,251.0,89.0,159.0,255.0,253.0,253.0,31.0,48.0,228.0,253.0,247.0,140.0,8.0,64.0,251.0,253.0,220.0,64.0,251.0,253.0,220.0,24.0,193.0,253.0,220.0])))

3.2 saveAsLibSVMFile

将LIBSVM格式的数据保存到指定文件中

3.3 appendBias

对向量增加偏置项,用于回归和分类的算法计算中

3.4 fastSquaredDistance

快速计算距离,返回欧式距离,用于k-means

☆┈━═┈━═┈━═┈━═┈━═☆因为您的赞赏,您会读到更多优质文章☆┈━═┈━═┈━═┈━═┈━═☆

第二章 spark2.3 mllib机器学习-数据操作

第二章 spark2.3 mllib机器学习-数据操作

目录

明细

继续阅读

【51CTO学院三周年】自学路上的伴侣

在线教育巨头多邻国Duolingo入华一周年，中国市场马力全开

【分类算法】什么是分类算法定义分类与聚类分类过程方法

申请评分模型拒绝推断（RI）方法申请评分模型拒绝推断（RI）方法

Sql优化一：sql语句优化

Nacos 2.0 升级前后性能对比压测

尚硅谷—韩顺平—图解 Java设计模式（结构型）（55～）

Storm编译打包过程中遇到的一些问题及解决方法

MapReduce的几个企业级经典面试案例MapReduce的几个企业级经典面试案例

9.spark Core 进阶2--Cashe

大数据排错SparkSpark集群启动时候，JAVA_HOME is not sethadoop集群，某台服务器jps无任何输出IDEAkafkahadoopspark sqlfile permissionsIDEA本地测试 - OutOfMemoryError: GC overhead limit exceededhdfs负载均衡

浅谈企业活动中进行数据分析的重要性

Ambari介绍和架构原理

spark/scala关于【资源文件】加载方法概述外部文件加载方案测试资源文件打包入jar包中小结

NOSQL安全攻击

win10本地scala和spark安装安装scala安装spark