Spark ML 特征转换及处理算子实战技巧-Spark商业ML实战

本套技术专栏是作者(秦凯新)平时工作的总结和升华,通过从真实商业环境抽取案例进行总结和分享,并给出商业应用的调优建议和集群环境容量规划等内容,请持续关注本套博客。版权声明:禁止转载,欢迎学习。QQ邮箱地址:[email protected],如有任何商业交流,可随时联系。

1 燃烧吧特征转换

1.1 Tokenization 分词器技术(RegexTokenizer)

Tokenization是将文本(例如句子)分割成单词,默认是空格分割
RegexTokenizer是基于正则表达式进行单词分割,默认打分割方式是’\s+’,

import org.apache.spark.ml.feature.{RegexTokenizer, Tokenizer}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val sentenceDataFrame = spark.createDataFrame(Seq(
  (0, "Hi I heard about Spark"),
  (1, "I wish Java could use case classes"),
  (2, "Logistic,regression,models,are,neat")
)).toDF("id", "sentence")


=>  默认匹配的是空格
val tokenizer = new Tokenizer().setInputCol("sentence").setOutputCol("words")
val tokenized = tokenizer.transform(sentenceDataFrame)

val countTokens = udf { (words: Seq[String]) => words.length }
tokenized.select("sentence", "words").withColumn("tokens", countTokens(col("words"))).show(false)
    
 +-----------------------------------+------------------------------------------+------+
|sentence                           |words                                     |tokens|
+-----------------------------------+------------------------------------------+------+
|Hi I heard about Spark             |[hi, i, heard, about, spark]              |5     |
|I wish Java could use case classes |[i, wish, java, could, use, case, classes]|7     |
|Logistic,regression,models,are,neat|[logistic,regression,models,are,neat]     |1     |
+-----------------------------------+------------------------------------------+------+   
 
    
=> W 表示匹配出的是除单词之外的任何分隔符
val regexTokenizer = new
RegexTokenizer().setInputCol("sentence").setOutputCol("words").setPattern("\\W")       
  
val regexTokenized = regexTokenizer.transform(sentenceDataFrame)
regexTokenized.select("sentence", "words").withColumn("tokens", countTokens(col("words"))).show(false)

scala> regexTokenized.select("sentence", "words").withColumn("tokens", countTokens(col("words"))).show(false)
+-----------------------------------+------------------------------------------+------+
|sentence                           |words                                     |tokens|
+-----------------------------------+------------------------------------------+------+
|Hi I heard about Spark             |[hi, i, heard, about, spark]              |5     |
|I wish Java could use case classes |[i, wish, java, could, use, case, classes]|7     |
|Logistic,regression,models,are,neat|[logistic, regression, models, are, neat] |5     |
+-----------------------------------+------------------------------------------+------+



=>  w 表示为 [a-zA-Z0-9],setGaps(false)表示匹配的不是空格,而是直接匹配出单词
val regexTokenizer = new RegexTokenizer().setInputCol("sentence").setOutputCol("words").setPattern("\\w+").setGaps(false) 
val regexTokenized = regexTokenizer.transform(sentenceDataFrame)

regexTokenized.select("sentence", "words").withColumn("tokens", countTokens(col("words"))).show(false)

 regexTokenized.select("sentence", "words").withColumn("tokens", countTokens(col("words"))).show(false)
 +-----------------------------------+------------------------------------------+------+
|sentence                           |words                                     |tokens|
+-----------------------------------+------------------------------------------+------+
|Hi I heard about Spark             |[hi, i, heard, about, spark]              |5     |
|I wish Java could use case classes |[i, wish, java, could, use, case, classes]|7     |
|Logistic,regression,models,are,neat|[logistic, regression, models, are, neat] |5     |
+-----------------------------------+------------------------------------------+------+

1.2 移除停用词

发现 I ,and ,had , a 被移除

import org.apache.spark.ml.feature.StopWordsRemover

val remover = new StopWordsRemover().setInputCol("raw").setOutputCol("filtered")

val dataSet = spark.createDataFrame(Seq(
  (0, Seq("I", "saw", "the", "red", "balloon")),
  (1, Seq("Mary", "had", "a", "little", "lamb"))
)).toDF("id", "raw")

remover.transform(dataSet).show(false)

scala> remover.transform(dataSet).show(false)
+---+----------------------------+--------------------+
|id |raw                         |filtered            |
+---+----------------------------+--------------------+
|0  |[I, saw, the, red, balloon] |[saw, red, balloon] |
|1  |[Mary, had, a, little, lamb]|[Mary, little, lamb]|
+---+----------------------------+--------------------+

## 要求的默认停用词列表
requirement failed: US is not in the supported language list: french, spanish, german, finnish, turkish, english, russian, norwegian, dutch, danish, hungarian, italian, swedish, portuguese.

scala> StopWordsRemover.loadDefaultStopWords("english")

res17: Array[String] = Array(i, me, my, myself, we, our, ours, ourselves, you, your, yours, yourself, yourselves, he, him, his, himself, she, her, hers, herself, it, its, itself, they, them, their, theirs, themselves, what, which, who, whom, this, that, these, those, am, is, are, was, were, be, been, being, have, has, had, having, do, does, did, doing, a, an, the, and, but, if, or, because, as, until, while, of, at, by, for, with, about, against, between, into, through, during, before, after, above, below, to, from, up, down, in, out, on, off, over, under, again, further, then, once, here, there, when, where, why, how, all, any, both, each, few, more, most, other, some, such, no, nor, not, only, own, same, so, than, too, very, s, t, can, will, just, don, should, now, i'll, you'll, he'll...

1.3 n-gram (得到组合序列)

每个n-gram由n个连续字的空格分隔的字符串表示。 如果输入序列包含少于n个字符串,则不会生成输出。
import org.apache.spark.ml.feature.NGram

    val wordDataFrame = spark.createDataFrame(Seq(
      (0, Array("Hi", "I", "heard", "about", "Spark")),
      (1, Array("I", "wish", "Java", "could", "use", "case", "classes")),
      (2, Array("Logistic", "regression", "models", "are", "neat"))
    )).toDF("id", "words")

    val ngram = new NGram().setN(3).setInputCol("words").setOutputCol("ngrams")

    val ngramDataFrame = ngram.transform(wordDataFrame)
    ngramDataFrame.select("ngrams").show(false)

    +--------------------------------------------------------------------------------+
    |ngrams                                                                          |
    +--------------------------------------------------------------------------------+
    |[Hi I heard, I heard about, heard about Spark]                                  |
    |[I wish Java, wish Java could, Java could use, could use case, use case classes]|
    |[Logistic regression models, regression models are, models are neat]            |
    +--------------------------------------------------------------------------------+

1.4 二值化(还是比较厉害的)

二值化是将数字特征阈值为二进制(0/1)特征的过程。
Binarizer接受通用参数inputCol和outputCol以及二进制阈值。 大于阈值的特征值被二进制化为1.0; 等于或小于阈值的值被二值化为0.0。 inputCol支持Vector和Double类型。

import org.apache.spark.ml.feature.Binarizer

val data = Array((0, 0.1), (1, 0.8), (2, 0.2))
val dataFrame = spark.createDataFrame(data).toDF("id", "feature")

val binarizer: Binarizer = new Binarizer().setInputCol("feature").setOutputCol("binarized_feature").setThreshold(0.5)

val binarizedDataFrame = binarizer.transform(dataFrame)

scala> println(s"Binarizer output with Threshold = ${binarizer.getThreshold}")
Binarizer output with Threshold = 0.5

scala>binarizedDataFrame.show()
+---+-------+-----------------+
| id|feature|binarized_feature|
+---+-------+-----------------+
|  0|    0.1|              0.0|
|  1|    0.8|              1.0|
|  2|    0.2|              0.0|
+---+-------+-----------------+

1.5 规范化(StandardScaler)

StandardScaler处理的对象是每一列,也就是每一维特征,将特征标准化为单位标准差或是0均值,或是0均值单位标准差。

StandardScaler=(x-u)/标准差Sn


主要有两个参数可以设置:

  • withStd: 默认为真。将数据标准化到单位标准差。

  • withMean: 默认为假。是否变换为0均值。 (此种方法将产出一个稠密输出,所以不适用于稀疏输入。)

  • StandardScaler需要fit数据,获取每一维的均值和标准差,来缩放每一维特征。
    StandardScaler是一个Estimator,它可以fit数据集产生一个StandardScalerModel,用来计算汇总统计。

  • 然后产生的模可以用来转换向量至统一的标准差以及(或者)零均值特征。注意如果特征的标准差为零,则该特征在向量中返回的默认值为0.0。

      import org.apache.spark.ml.linalg.Vectors
       
      val dataFrame = spark.createDataFrame(Seq(
      (0, Vectors.dense(1.0, 0.5, -1.0)),
      (1, Vectors.dense(2.0, 1.0, 1.0)),
      (2, Vectors.dense(4.0, 10.0, 2.0))
      )).toDF("id", "features")
       
      dataFrame.show
      
      +---+--------------+
      | id|      features|
      +---+--------------+
      |  0|[1.0,0.5,-1.0]|
      |  1| [2.0,1.0,1.0]|
      |  2|[4.0,10.0,2.0]|
      +---+--------------+
      
      val scaler = new StandardScaler().setInputCol("features").setOutputCol("scaledFeatures").setWithStd(true).setWithMean(false)
      val scalerModel = scaler.fit(dataFrame)
      val scaledData = scalerModel.transform(dataFrame)
      scaledData.show
      
      +---+--------------+--------------------+
      | id|      features|      scaledFeatures|
      +---+--------------+--------------------+
      |  0|[1.0,0.5,-1.0]|[0.65465367070797...|
      |  1| [2.0,1.0,1.0]|[1.30930734141595...|
      |  2|[4.0,10.0,2.0]|[2.61861468283190...|
      +---+--------------+--------------------+
      
      scala>     scaledData.rdd.foreach(println)
      [0,[1.0,0.5,-1.0],[0.6546536707079772,0.09352195295828246,-0.6546536707079771]]
      [1,[2.0,1.0,1.0],[1.3093073414159544,0.18704390591656492,0.6546536707079771]]
      [2,[4.0,10.0,2.0],[2.618614682831909,1.8704390591656492,1.3093073414159542]]
      
      val scaler = new StandardScaler().setInputCol("features").setOutputCol("scaledFeatures").setWithStd(true).setWithMean(true)
      val scalerModel = scaler.fit(dataFrame)
      val scaledData = scalerModel.transform(dataFrame)
      scaledData.show
      
      scala>     scaledData.rdd.foreach(println)
      [0,[1.0,0.5,-1.0],[-0.8728715609439697,-0.6234796863885498,-1.0910894511799618]]
      [1,[2.0,1.0,1.0],[-0.2182178902359925,-0.5299577334302673,0.2182178902359924]]
      [2,[4.0,10.0,2.0],[1.0910894511799618,1.1534374198188169,0.8728715609439696]
    

    1.6 正则化(Normalizer) --面向行

    范数是一种强化了的距离概念,它在定义上比距离多了一条数乘的运算法则。有时候为了便于理解,我们可以把范数当作距离来理解。

  • L1范数是我们经常见到的一种范数,它的定义如下:

  • L2范数是我们最常见最常用的范数了,我们用的最多的度量距离欧氏距离就是一种L2范数,它的定义如下:

  • 是L无穷范数,它主要被用来度量向量元素的最大值,与L0一样,通常情况下表示为:

      import org.apache.spark.ml.feature.Normalizer
       
      正则化每个向量到1阶范数
      将每一行的规整为1阶范数为1的向量,1阶范数即所有值绝对值之和
      val normalizer = new Normalizer().setInputCol("features") .setOutputCol("normFeatures").setP(1.0)
      l1NormData.show()
    
      +---+--------------+------------------+
      | id| features| normFeatures|
      +---+--------------+------------------+
      | 0|[1.0,0.5,-1.0]| [0.4,0.2,-0.4]|
      | 1| [2.0,1.0,1.0]| [0.5,0.25,0.25]|
      | 2|[4.0,10.0,2.0]|[0.25,0.625,0.125]|
      +---+--------------+------------------+
       
      正则化每个向量到无穷阶范数,向量的无穷阶范数即向量中所有值中的最大值
      val lInfNormData = normalizer.transform(dataFrame, normalizer.p -> Double.PositiveInfinity)
      lInfNormData.show()
       
      +---+--------------+--------------+
      | id| features| normFeatures|
      +---+--------------+--------------+
      | 0|[1.0,0.5,-1.0]|[1.0,0.5,-1.0]|
      | 1| [2.0,1.0,1.0]| [1.0,0.5,0.5]|
      | 2|[4.0,10.0,2.0]| [0.4,1.0,0.2]|
      +---+--------------+--------------+
    

    1.7 最大最小值缩放 MinMaxScaler --面向列 (value-Emin/(Emax-Emin))*[max-min]+min

      Emin最小值为每一列最小值
      Emax最小值为每一列最大值
      MinMaxScaler作用同样是每一列,即每一维特征。将每一维特征线性地映射到指定的区间,通常是[0, 1]。
      
      MinMaxScaler计算数据集的汇总统计量,并产生一个MinMaxScalerModel。
      
      注意因为零值转换后可能变为非零值,所以即便为稀疏输入,输出也可能为稠密向量。
      
      该模型可以将独立的特征的值转换到指定的范围内。
      它也有两个参数可以设置:
      
      min: 默认为0。指定区间的下限。
      max: 默认为1。指定区间的上限。
      import org.apache.spark.ml.feature.MinMaxScaler
      
      val dataFrame = spark.createDataFrame(Seq(
      (0, Vectors.dense(1.0, 0.5, -1.0)),
      (1, Vectors.dense(2.0, 1.0, 1.0)),
      (2, Vectors.dense(4.0, 10.0, 2.0))
      )).toDF("id", "features")
       
      val scaler = new MinMaxScaler() .setInputCol("features").setOutputCol("scaledFeatures")
       
      // Compute summary statistics and generate MinMaxScalerModel
      val scalerModel = scaler.fit(dataFrame)
       
      // rescale each feature to range [min, max].
      val scaledData = scalerModel.transform(dataFrame)
      scaledData.select("features", "scaledFeatures").show
       
       每维特征线性地映射,最小值映射到0,最大值映射到1。
      +--------------+-----------------------------------------------------------+
      |features |scaledFeatures |
      +--------------+-----------------------------------------------------------+
      |[1.0,0.5,-1.0]|[0.0,0.0,0.0] |
      |[2.0,1.0,1.0] |[0.3333333333333333,0.05263157894736842,0.6666666666666666]|
      |[4.0,10.0,2.0]|[1.0,1.0,1.0] |
      +--------------+-----------------------------------------------------------+
    

    1.8 最大值-绝对值缩放MaxAbsScaler(面向列-value除以绝对值最大值)

    MaxAbsScaler将每一维的特征变换到[-1,1]闭区间上,通过除以每一维特征上的最大的绝对值,它不会平移整个分布,也不会破坏原来每一个特征向量的稀疏性。因为它不会转移/集中数据,所以不会破坏数据的稀疏性。

      import org.apache.spark.ml.feature.MaxAbsScaler
       
      val scaler = new MaxAbsScaler()
      .setInputCol("features")
      .setOutputCol("scaledFeatures")
       
      val scalerModel = scaler.fit(dataFrame)
       1]
      val scaledData = scalerModel.transform(dataFrame)
      scaledData.select("features", "scaledFeatures").show()
       
      // 每一维的绝对值的最大值为[4, 10, 2]
      +--------------+----------------+
      | features| scaledFeatures|
      +--------------+----------------+
      |[1.0,0.5,-1.0]|[0.25,0.05,-0.5]|
      | [2.0,1.0,1.0]| [0.5,0.1,0.5]|
      |[4.0,10.0,2.0]| [1.0,1.0,1.0]|
      +--------------+----------------+
    

1.9 独热编码(OneHotEncoderEstimator)- 面向列

把每一列的所有可能编码成向量形式:如: 0被编码为:(1,0) 也即一个值变成一个向量。

import org.apache.spark.ml.feature.OneHotEncoderEstimator
val df = spark.createDataFrame(Seq(
  (0.0, 1.0),
  (1.0, 0.0),
  (2.0, 1.0),
  (0.0, 2.0),
  (0.0, 1.0),
  (2.0, 0.0)
)).toDF("categoryIndex1", "categoryIndex2")

val encoder = new OneHotEncoderEstimator().setInputCols(Array("categoryIndex1", "categoryIndex2")) .setOutputCols(Array("categoryVec1", "categoryVec2"))
val model = encoder.fit(df)

val encoded = model.transform(df)
encoded.show()

最终结果:
+--------------+--------------+-------------+-------------+
|categoryIndex1|categoryIndex2| categoryVec1| categoryVec2|
+--------------+--------------+-------------+-------------+
|           0.0|           1.0|(2,[0],[1.0])|(2,[1],[1.0])|
|           1.0|           0.0|(2,[1],[1.0])|(2,[0],[1.0])|
|           2.0|           1.0|    (2,[],[])|(2,[1],[1.0])|
|           0.0|           2.0|(2,[0],[1.0])|    (2,[],[])|
|           0.0|           1.0|(2,[0],[1.0])|(2,[1],[1.0])|
|           2.0|           0.0|    (2,[],[])|(2,[0],[1.0])|
+--------------+--------------+-------------+-------------+


把每一列的所有可能编码成向量形式:如:第一行0被编码为:(1,0,0)
val encoder = new OneHotEncoderEstimator().setInputCols(Array("categoryIndex1", "categoryIndex2")) .setOutputCols(Array("categoryVec1", "categoryVec2")).setDropLast(false)
val model = encoder.fit(df)

val encoded = model.transform(df)
encoded.show()

+--------------+--------------+-------------+-------------+
|categoryIndex1|categoryIndex2| categoryVec1| categoryVec2|
+--------------+--------------+-------------+-------------+
|           0.0|           1.0|(3,[0],[1.0])|(3,[1],[1.0])|
|           1.0|           0.0|(3,[1],[1.0])|(3,[0],[1.0])|
|           2.0|           1.0|(3,[2],[1.0])|(3,[1],[1.0])|
|           0.0|           2.0|(3,[0],[1.0])|(3,[2],[1.0])|
|           0.0|           1.0|(3,[0],[1.0])|(3,[1],[1.0])|
|           2.0|           0.0|(3,[2],[1.0])|(3,[0],[1.0])|
+--------------+--------------+-------------+-------------+

1.9 字符串-索引变换

根据字符串出现的频率来定义标签列,出现频率最高者为0,因此可以看到0,1,2…,但是若出现训练的模型中没有测试的值,将会报错,此时需要setHandleInvalid,跳过即可。

import org.apache.spark.ml.feature.StringIndexer

val df = spark.createDataFrame(
  Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c"))
).toDF("id", "category")

val indexer = new StringIndexer().setInputCol("category").setOutputCol("categoryIndex")
val model =indexer.fit(df)

val indexed = model.transform(df)
indexed.show()
scala> indexed.show
+---+--------+-------------+
| id|category|categoryIndex|
+---+--------+-------------+
|  0|       a|          0.0|
|  1|       b|          2.0|
|  2|       c|          1.0|
|  3|       a|          0.0|
|  4|       a|          0.0|
|  5|       c|          1.0|
+---+--------+-------------+


训练模型中没有d时,会报错,所以需要设置setHandleInvalid
 val df_test = spark.createDataFrame(
  Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c"),(5, "d"))
).toDF("id", "category")

 val indexer_test = new StringIndexer().setInputCol("category").setOutputCol("categoryIndex").setHandleInvalid("skip")
 val model =indexer_test.fit(df)
 
 val indexed_test = model.transform(df_test)

结语

今天是我的CSDN技术专栏突破50篇的日子,值得庆祝!
Spark ML 特征转换及处理算子实战技巧-Spark商业ML实战_第1张图片
秦凯新 于深圳 2018 11 18 1:47

你可能感兴趣的:(大数据,spark,机器学习)