Mahout中的一些相似度算法实现解读

    Mahout中实现的推荐算法是协同过滤，而无论是UserCF还是ItemCF都依赖于user相似度或item相似度。本文是对mahout中的一些相似度算法的解读。

Mahout相似度相关类关系如下：

有点乱(^.^)

    从上图可看出，Mahout主要针对用户相似度和物品相似度的计算，并且除了HybridSimilarity之外全都能够用于计算user和item两者的相似度，只有HybridSimilarity只能计算item相似度。接来下分三部分进行分析：继承AbstractSimilarity的，没有集成AbstractSimilarity但是也可以用于user、item相似度计算的，只继承AbstractItemSimilarity的。

继承AbstractSimilarity的实现类

    以下这些类用于无偏好值的用户行为数据。
    在介绍EuclideanDistanceSimilarity、PearsonCorrelationSimilarity、UncenteredCosineSimilarity之前，让我们先了解一下AbstractSimilarity。其实AbstractSimilarity才是这一部分的核心，三个重要方法：userSimilarity()、itemSimilarity()、computeResult()，其中computeResult()是抽象方法，是供给子类去实现的，userSimilarity()、itemSimilarity()中会在计算好相应变量之后调用子类实现的computeResult()，得到的结果在经过处理之后就是相似度。
    在userSimilarity()中，会先计算
user1对物品X偏好值的和sumX以及平方和sumX2、user2对物品Y的偏好值的和sumY以及平方和sumY2、user1与user2的物品的偏好值的乘积sumXY、user1(item1)与user2(item2)的偏好值的平方差sumXYdiff2。代码如下：

sumXY += x * y;
    sumX += x;
    sumX2 += x * x;
    sumY += y;
    sumY2 += y * y;
    double diff = x - y;
    sumXYdiff2 += diff * diff;
    count++;

    其中如果一个user没有对某个item的偏好而另一个user有对这个item的偏好，那么会使用PreferenceInferrer实现类对进行推测，代码如下：

// Only one user expressed a preference, but infer the other one's preference and tally
    // as if the other user expressed that preference
    if (compare < 0) {
    // X has a value; infer Y's
        x = hasPrefTransform
            ? prefTransform.getTransformedValue(xPrefs.get(xPrefIndex))
            : xPrefs.getValue(xPrefIndex);
        y = inferrer.inferPreference(userID2, xIndex);
     } else {
      // compare > 0
        // Y has a value; infer X's
        x = inferrer.inferPreference(userID1, yIndex);
        y = hasPrefTransform
            ? prefTransform.getTransformedValue(yPrefs.get(yPrefIndex))
            : yPrefs.getValue(yPrefIndex);
     }

    然后会调用computeResult()方法，代码如下：

// "Center" the data. If my math is correct, this'll do it.
    double result;
    if (centerData) {
      double meanX = sumX / count;
      double meanY = sumY / count;
      // double centeredSumXY = sumXY - meanY * sumX - meanX * sumY + n * meanX * meanY;
      double centeredSumXY = sumXY - meanY * sumX;
      // double centeredSumX2 = sumX2 - 2.0 * meanX * sumX + n * meanX * meanX;
      double centeredSumX2 = sumX2 - meanX * sumX;
      // double centeredSumY2 = sumY2 - 2.0 * meanY * sumY + n * meanY * meanY;
      double centeredSumY2 = sumY2 - meanY * sumY;
      result = computeResult(count, centeredSumXY, centeredSumX2, centeredSumY2, sumXYdiff2);
    } else {
      result = computeResult(count, sumXY, sumX2, sumY2, sumXYdiff2);
    }

    最后，在经过SimilarityTransform和normalizeWeightResult()两步处理，result就是求得的user1与user2的相似度数值。代码如下：

if (similarityTransform != null) {
      result = similarityTransform.transformSimilarity(itemID1, itemID2, result);
    }

    if (!Double.isNaN(result)) {
      result = normalizeWeightResult(result, count, cachedNumUsers);
    }

    1.PearsonCorrelationSimilarity
    基于皮尔森相关性的相似度计算：sumXY / sqrt(sumX2 * sumY2)，代码如下：

@Override
double computeResult(int n, double sumXY, double sumX2, double sumY2, double sumXYdiff2) {
    if (n == 0) {
      return Double.NaN;
    }
    // Note that sum of X and sum of Y don't appear here since they are assumed to be 0;
    // the data is assumed to be centered.
    double denominator = Math.sqrt(sumX2) * Math.sqrt(sumY2);
    if (denominator == 0.0) {
      // One or both parties has -all- the same ratings;
      // can't really say much similarity under this measure
      return Double.NaN;
    }
    return sumXY / denominator;
}

    2.EuclideanDistanceSimilarity
    基于欧几里德距离的相似度计算：1 / (1 + distance)，代码如下：

@Override
double computeResult(int n, double sumXY, double sumX2, double sumY2, double sumXYdiff2) {
    return 1.0 / (1.0 + Math.sqrt(sumXYdiff2) / Math.sqrt(n));
}

    3.UncenteredCosineSimilarity
    基于余弦的相似度计算，代码如下：

@Override
double computeResult(int n, double sumXY, double sumX2, double sumY2, double sumXYdiff2) {
    if (n == 0) {
      return Double.NaN;
    }
    double denominator = Math.sqrt(sumX2) * Math.sqrt(sumY2);
    if (denominator == 0.0) {
      // One or both parties has -all- the same ratings;
      // can't really say much similarity under this measure
      return Double.NaN;
    }
    return sumXY / denominator;
}

继承AbstractItemSimilarity并实现AbstractItemSimilarity的类

    以下这些类用于无偏好值的用户行为数据。
    1.TanimotoCoefficientSimilarity
    基于谷本系数的相似度：userID1的偏好物品与userID2的偏好物品的交集size/并集size（item与其类似），代码如下：

public double userSimilarity(long userID1, long userID2) throws TasteException {

    DataModel dataModel = getDataModel();
    FastIDSet xPrefs = dataModel.getItemIDsFromUser(userID1);
    FastIDSet yPrefs = dataModel.getItemIDsFromUser(userID2);

    int xPrefsSize = xPrefs.size();
    int yPrefsSize = yPrefs.size();
    if (xPrefsSize == 0 && yPrefsSize == 0) {
      return Double.NaN;
    }
    if (xPrefsSize == 0 || yPrefsSize == 0) {
      return 0.0;
    }

    //计算交集的size
    int intersectionSize =
        xPrefsSize < yPrefsSize ? yPrefs.intersectionSize(xPrefs) : xPrefs.intersectionSize(yPrefs);
    if (intersectionSize == 0) {
      return Double.NaN;
    }
    //计算并集size
    int unionSize = xPrefsSize + yPrefsSize - intersectionSize;

    return (double) intersectionSize / (double) unionSize;
  }

    2.LogLikelihoodSimilarity
    基于对数似然的相似度：基于谷本系数的相似度的升级版，代码如下：

public double userSimilarity(long userID1, long userID2) throws TasteException {
    DataModel dataModel = getDataModel();
    FastIDSet prefs1 = dataModel.getItemIDsFromUser(userID1);
    FastIDSet prefs2 = dataModel.getItemIDsFromUser(userID2);

    long prefs1Size = prefs1.size();
    long prefs2Size = prefs2.size();
    //计算交集size 
    long intersectionSize =
        prefs1Size < prefs2Size ? prefs2.intersectionSize(prefs1) : prefs1.intersectionSize(prefs2);
    if (intersectionSize == 0) {
      return Double.NaN;
    }
    long numItems = dataModel.getNumItems();
    double logLikelihood =
        LogLikelihood.logLikelihoodRatio(intersectionSize,
                                         prefs2Size - intersectionSize,
                                         prefs1Size - intersectionSize,
                                         numItems - prefs1Size - prefs2Size + intersectionSize);
    return 1.0 - 1.0 / (1.0 + logLikelihood);
  }

    3.CityBlockSimilarity
    基于街区距离的相似度，代码如下：

public double userSimilarity(long userID1, long userID2) throws TasteException {
    DataModel dataModel = getDataModel();
    FastIDSet prefs1 = dataModel.getItemIDsFromUser(userID1);
    FastIDSet prefs2 = dataModel.getItemIDsFromUser(userID2);
    int prefs1Size = prefs1.size();
    int prefs2Size = prefs2.size();
    int intersectionSize = prefs1Size < prefs2Size ? prefs2.intersectionSize(prefs1) : prefs1.intersectionSize(prefs2);
    return doSimilarity(prefs1Size, prefs2Size, intersectionSize);
  }
//Calculate City Block Distance from total non-zero values and intersections and map to a similarity value.
private static double doSimilarity(int pref1, int pref2, int intersection) {
    int distance = pref1 + pref2 - 2 * intersection;
    return 1.0 / (1.0 + distance);
}

只继承AbstractItemSimilarity的实现类

    1.TrackItemSimilarity
    代码如下：

public double itemSimilarity(long itemID1, long itemID2) {
    if (itemID1 == itemID2) {
      return 1.0;
    }
    TrackData data1 = trackData.get(itemID1);
    TrackData data2 = trackData.get(itemID2);
    if (data1 == null || data2 == null) {
      return 0.0;
    }

    // Arbitrarily decide that same album means "very similar"
    if (data1.getAlbumID() != TrackData.NO_VALUE_ID && data1.getAlbumID() == data2.getAlbumID()) {
      return 0.9;
    }
    // ... and same artist means "fairly similar"
    if (data1.getArtistID() != TrackData.NO_VALUE_ID && data1.getArtistID() == data2.getArtistID()) {
      return 0.7;
    }

    // Tanimoto coefficient similarity based on genre, but maximum value of 0.25
    FastIDSet genres1 = data1.getGenreIDs();
    FastIDSet genres2 = data2.getGenreIDs();
    if (genres1 == null || genres2 == null) {
      return 0.0;
    }
    int intersectionSize = genres1.intersectionSize(genres2);
    if (intersectionSize == 0) {
      return 0.0;
    }
    int unionSize = genres1.size() + genres2.size() - intersectionSize;
    return (double) intersectionSize / (4.0 * unionSize);
}

    2.HybridSimilarity
    基于混合的相似度计算：LogLikelihoodSimilarity*TrackItemSimilarity。代码如下：

HybridSimilarity(DataModel dataModel, File dataFileDirectory) throws IOException {
    super(dataModel);
    cfSimilarity = new LogLikelihoodSimilarity(dataModel);
    contentSimilarity = new TrackItemSimilarity(dataFileDirectory);
  }

public double itemSimilarity(long itemID1, long itemID2) throws TasteException {
    return contentSimilarity.itemSimilarity(itemID1, itemID2) * cfSimilarity.itemSimilarity(itemID1, itemID2);
}