Fuzzy c-mean 数学推导以及python代码实现

博主比较懒，直入主题。

目标函数：

$$\begin{array}{rl}& \min \sum _{i=1}^C\sum _{j=1}^N{\mu }_{ij}^m\Vert {\mathbf{x}}_j-{\mathbf{c}}_i{\Vert }_2^2\\ & s.t.\sum _{i=1}^C{\mu }_{ij}=1\end{array}$$
在这里，${\mu }_{ij}$为每个样本$j$所属类$i$的隶属度，$m$为模糊指数，${\mathbf{c}}_i$为第$j$个类的类心。
首先我们采用拉格朗日乘子法将约束项合并进目标函数：
$$L=\sum _{i=1}^C\sum _{j=1}^N{\mu }_{ij}^m\Vert {\mathbf{x}}_j-{\mathbf{c}}_i{\Vert }_2^2+\sum _{j=1}^N{\lambda }_j(\sum _{i=1}^C{\mu }_{ij}-1)$$
随后对$\mu$，$\mathbf{c}$，$\lambda$求导：
$$\begin{array}{rl}& \frac{\partial L}{\partial {\mu }_{ij}}=m{\mu }_{ij}^{m-1}\Vert {\mathbf{x}}_j-{\mathbf{c}}_i{\Vert }_2^2+{\lambda }_j=0\\ & {\mu }_{ij}=(\frac{-{\lambda }_j}{m\Vert {\mathbf{x}}_j-{\mathbf{c}}_i{\Vert }_2^2}{)}^{\frac{1}{m-1}}\\ & \because \sum _{i=1}^C{\mu }_{ij}=1\\ & \therefore {\mu }_{ij}=\frac{(\frac{-{\lambda }_j}{m\Vert {\mathbf{x}}_j-{\mathbf{c}}_i{\Vert }_2^2}{)}^{\frac{1}{m-1}}}{{\sum }_{k=1}^C(\frac{-{\lambda }_j}{m\Vert {\mathbf{x}}_j-{\mathbf{c}}_k{\Vert }_2^2}{)}^{\frac{1}{m-1}}}=\frac{\Vert {\mathbf{x}}_j-{\mathbf{c}}_i{\Vert }_2^{\frac{2}{1-m}}}{{\sum }_{k=1}^C\Vert {\mathbf{x}}_j-{\mathbf{c}}_k{\Vert }_2^{\frac{2}{1-m}}}\\ & \frac{\partial L}{\partial {\mathbf{c}}_i}=2\sum _{i=1}^C{\mu }_{ij}^m({\mathbf{x}}_j-{\mathbf{c}}_i)=0\\ & {\mathbf{c}}_i=\frac{{\sum }_{i=1}^C{\mu }_{ij}^m{\mathbf{x}}_j}{{\sum }_{i=1}^C{\mu }_{ij}^m}\end{array}$$
所以FCM的求解算法为：

1. 随机初始化隶属度矩阵$U$
2. 根据$U$计算$\mathbf{c}$
3. 根据$\mathbf{c}$计算$U$
4. 判断是否收敛，是则结束，否则重复步骤2，3

Python代码

import numpy as np


class BaseCluster():
    def __init__(self):
        pass

    def _euclidean_distance(self, X, Y=None):
        """
        return the element-wise euclidean distance between X and Y
        :param X: [n_samples_X, n_features]
        :param Y: if None, return the element-wise distance between X and X, 
        		  else [n_samples_Y, n_features]
        :return: [n_samples_X, n_samples_Y]
        """
        if Y is None:
            Y = X.copy()
        Y = np.expand_dims(np.transpose(Y), 0)
        X = np.expand_dims(X, 2)
        D = np.sum((X - Y)**2, axis=1)
        return np.sqrt(D)

    def normalize_column(self, U):
        return U/np.sum(U, axis=0, keepdims=True)


class FuzzyCMeans(BaseCluster):
    def __init__(self, n_cluster, m, error=1e-5, tol_iter=200, verbose=0):
        """
        Implantation of fuzzy c-means
        :param n_cluster: number of clusters
        :param m: fuzzy index
        :param error: max error for u_old - u_new to break the iteration
        :param tol_iter: total iteration number
        :param verbose: whether to print loss infomation during iteration
        """
        self.n_cluster = n_cluster
        self.m = m
        self.error = error
        self.tol_iter = tol_iter
        self.n_dim = None
        self.verbose = verbose
        self.fitted = False

        super(FuzzyCMeans, self).__init__()

    def fit(self, x, y=None):
        N = x.shape[0]
        self.n_dim = x.shape[1]

        # init U
        U = np.random.rand(self.n_cluster, N)

        self.loss_hist = []
        for t in range(self.tol_iter):
            U, V, loss, signal = self._cmean_update(x, U)
            self.loss_hist.append(loss)
            if self.verbose > 0:
                print('[FCM Iter {}] Loss: {:.4f}'.format(t, loss))
            if signal:
                break
        self.fitted = True
        self.center_ = V
        self.train_u = U
        self.variance_ = np.zeros(self.center_.shape)  # n_clusters * n_dim

        for i in range(self.n_dim):
            self.variance_[:, i] = np.sum(
                U * ((x[:, i][:, np.newaxis] - self.center_[:, i].transpose())**2).T, axis=1
            ) / np.sum(U, axis=1)

        return self

    def _cmean_update(self, x, U):
        old_U = U.copy()
        old_U = np.fmax(old_U, np.finfo(np.float64).eps)
        old_U_unexp = old_U.copy()
        old_U = self.normalize_column(old_U)**self.m

        # compute V
        V = np.dot(old_U, x) / old_U.sum(axis=1, keepdims=True)

        # compute U
        dist = self._euclidean_distance(x, V).T  # n_clusters * n_samples
        dist = np.fmax(dist, np.finfo(np.float64).eps)

        loss = (old_U * dist ** 2).sum()
        dist = dist ** (2/(1-self.m))
        U = self.normalize_column(dist)
        if np.linalg.norm(U - old_U_unexp) < self.error:
            signal = True
        else:
            signal = False
        return U, V, loss, signal

    def predict(self, x):
        """
        predict membership for each element
        :param x:
        :return:
        """
        dist = self._euclidean_distance(x, self.center_).T  # n_clusters * n_samples
        dist = np.fmax(dist, np.finfo(np.float64).eps)

        dist = dist ** (2 / (1 - self.m))
        U = self.normalize_column(dist)
        return U.T