Spark CountVectorizer处理文本特征

声明：版权所有，转载请联系作者并注明出处  http://blog.csdn.net/u013719780?viewmode=contents

博主简介：风雪夜归子（Allen），机器学习算法攻城狮，喜爱钻研Meachine Learning的黑科技，对Deep Learning和Artificial Intelligence充满兴趣，经常关注Kaggle数据挖掘竞赛平台，对数据、Machine Learning和Artificial Intelligence有兴趣的童鞋可以一起探讨哦，个人CSDN博客：http://blog.csdn.net/u013719780?viewmode=contents

CountVectorizer算法是将文本向量转换成稀疏表示打数值向量（字符频率向量）。该数值向量可以传递给其他算法，譬如LDA 。在fitting过程中，CountVectorizer将会把频率高的单词排在前面。可选参数minDF表示文本中必须出现的次数。下面看一个具体的例子。

from pyspark.ml.feature import CountVectorizer

# Input data: Each row is a bag of words with a ID.
df = sqlContext.createDataFrame([
    (0, "a b c".split(" ")),
    (1, "a b b c a".split(" "))
], ["id", "words"])

# fit a CountVectorizerModel from the corpus.
cv = CountVectorizer(inputCol="words", outputCol="features", vocabSize=3, minDF=2.0)
model = cv.fit(df)
result = model.transform(df)
result.show()

+---+---------------+--------------------+
| id|          words|            features|
+---+---------------+--------------------+
|  0|      [a, b, c]|(3,[0,1,2],[1.0,1...|
|  1|[a, b, b, c, a]|(3,[0,1,2],[2.0,2...|
+---+---------------+--------------------+

 

from pyspark.ml.feature import CountVectorizer

# Input data: Each row is a bag of words with a ID.
df = sqlContext.createDataFrame([
    (0, "a b c".split(" ")),
    (1, "a b b c a".split(" "))
], ["id", "words"])

# fit a CountVectorizerModel from the corpus.
cv = CountVectorizer(inputCol="words", outputCol="features", vocabSize=3, minDF=2.0)
model = cv.fit(df)
result = model.transform(df)
result.show()

+---+---------------+--------------------+
| id|          words|            features|
+---+---------------+--------------------+
|  0|      [a, b, c]|(3,[0,1,2],[1.0,1...|
|  1|[a, b, b, c, a]|(3,[0,1,2],[2.0,2...|
+---+---------------+--------------------+