Filters in Feature Selection

Filters are a kind of feature selection algorithms that choose part of the whole feature by score each individual feature and then pick top ranked ones. In Filters, the most important thing is to find a way to score different feature separately. The scores should follow a principle that the more related， the higher it should be. This apparent principle is not easy to implement because that it is difficult to define the word “related” precisely. In fact,the final result may be determined by several different features and the way these features cooperate with each other can be very complicated and non-linear, so it may be not enough to only consider the function of the features “individually”, which is the main shortage for the Filters.

As far as I know, there are two kinds of filters so far. The first kind of filters is build based on the topology of the data set and the other is based on the statistical property of the data, such as mutual information.For the topological filters, each instance of the data is considered to be a point in the Euclidean Space and the researchers try to find out some pattern of the distribution of all the points in space. For example, Relieff compares each instance with its nearest neighbor both with the same label and opposite label, and then increase the score of a feature if the value of this feature in the instance is different from that in the opposite-nearest neighbor, and decrease the score of the value of this feature in the instance is different from that in the same nearest neighbor. For the Mutual Information filters, all the values of a feature are considered to be a sequential of signal and researchers try to rank the features according to the mutual information between the feature signal and the label signal. Mutual information is a concept in information theory and is used to measure the difference between two sequences of data. The definition of mutual information is