Pyspark机器学习之Word2Vec（推荐系统内容相似）

Word2Vec简介

Word2Vec是一个词嵌入方法，可以计算每个单词在给定的语料库环境下的分布式向量，如果两个单词的语义相近，那么词向量在向量空间中也相互接近，判断向量空间的接近程度来判断来两个单词是否相似。

Word2Vec数学原理

首先导入Word2Vec所需要的包，并创建可以代表文档的词语序列

from pyspark.ml.feature import Word2Vec

from pyspark.sql import SparkSession

spark= SparkSession\
                .builder \
                .appName("dataFrame") \
                .getOrCreate()

# Input data: Each row is a bag of words from a sentence or document.
documentDF = spark.createDataFrame([
    ("Hi I heard about Spark".split(" "), ),
    ("I wish Java could use case classes".split(" "), ),
    ("Logistic regression models are neat".split(" "), )
], ["text"])

# Learn a mapping from words to Vectors.
word2Vec = Word2Vec(vectorSize=3, minCount=0, inputCol="text", outputCol="result")
model = word2Vec.fit(documentDF)

result = model.transform(documentDF)
for row in result.collect():
    text, vector = row
    print("Text: [%s] => \nVector: %s\n" % (", ".join(text), str(vector)))

输出：

Text: [Hi, I, heard, about, Spark] => 
Vector: [-0.05760560743510723,-0.03687768429517746,0.053699607402086263]

Text: [I, wish, Java, could, use, case, classes] => 
Vector: [-0.06942265214664595,-0.07444838913423674,0.029864686142121042]

Text: [Logistic, regression, models, are, neat] => 
Vector: [0.025776204053545373,0.06013465970754624,-0.0191410340834409]

可以看到，文档被转变为了一个3维的特征向量，这些特征向量就可以被应用到相关的机器学习方法中。