数据挖掘学习札记：ID3算法(一)

参考：

1. Wiki上的 ID3 algorithm

2. 百度文库里的一个PPT，有算例， 决策树ID3算法  
3. 百度文库，PPT，很多算例，开始有信息理论，极力推荐阅读，ID3算法
4. 用Python实现ID3和C4.5 决策树ID3和C4.5算法Python实现源码

下面是整理的学习笔记。

用途：

The ID3 algorithm is used by training on a dataset to produce adecision tree which is stored in memory. At runtime, this decision tree is used to classify new unseen test cases by working down the decision tree using the values of this test case to arrive at a terminal node that tells you what class this test case belongs to.

基本思想：

角度1.  越是小型的决策树越优于大的决策树（尽管如此该算法也不是总是生成最小的树形结构）

角度2. 引入信息论中互信息（信息增益），作为判别因素的度量，即：以信息熵的下降速度作为选取测试属性的标准，所选的测试属性是从根到当前节点的路径上尚未被考虑的具有最高信息增益的属性

算法描述：

ID3 (Examples, Target_Attribute, Attributes)
    Create a root node for the tree
    If all examples are positive, Return the single-node tree Root, with label = +.
    If all examples are negative, Return the single-node tree Root, with label = -.
    If number of predicting attributes is empty, then Return the single node tree Root,
    with label = most common value of the target attribute in the examples.
    Otherwise Begin
        A ← The Attribute that best classifies examples.
        Decision Tree attribute for Root = A.
        For each possible value, v_i, of A,
            Add a new tree branch below Root, corresponding to the test A = v_i.
            Let Examples(v_i) be the subset of examples that have the value v_i for A
            If Examples(v_i) is empty
                Then below this new branch add a leaf node with label = most common target value in the examples
            Else below this new branch add the subtree ID3 (Examples(v_i), Target_Attribute, Attributes – {A})
    End
    Return Root

算例：

If S is a collection of 14 examples with 9 YES and 5 NO examples then

Entropy(S) = - (9/14) Log2 (9/14) - (5/14) Log2 (5/14)= 0.940

Notice entropy is 0 if all members of S belong to the same class(the data is perfectly classified). The range of entropy is 0("perfectly classified") to 1 ("totally random").

Gain(S, A) is information gain of example set S on attribute A is defined as

Gain(S, A) = Entropy(S) - S((|S _v| / |S|) * Entropy(S _v))

Where:

S is each value v of all possible values of attribute A

S_v = subset of S for which attribute A has value v

|S_v| = number of elements in S_v

|S| = number of elements in S

Suppose S is a set of 14 examples in which one of the attributes is wind speed. The values of Wind can be Weak or Strong.The classification of these 14 examples are 9 YES and 5 NO. For attribute Wind, suppose there are 8 occurrences of Wind = Weak and 6 occurrences of Wind = Strong. For Wind = Weak, 6 of the examples are YES and 2 are NO. For Wind = Strong, 3 are YES and3 are NO. Therefore

Gain(S,Wind)=Entropy(S)-(8/14)*Entropy(S _weak)-(6/14)*Entropy(S _strong)

= 0.940 - (8/14)*0.811 - (6/14)*1.00

= 0.048

Entropy(S_weak) = - (6/8)*log2(6/8) - (2/8)*log2(2/8)= 0.811

Entropy(S_strong) = - (3/6)*log2(3/6) - (3/6)*log2(3/6)= 1.00

For each attribute, the gain is calculated and the highest gain is used in the decision node.

Suppose we want ID3 to decide whether the weather is amenable to playing baseball. Over the course of 2 weeks, data is collected to help ID3 build a decision tree (see table 1).

The target classification is "should we play baseball?"which can be yes or no.

The weather attributes are outlook, temperature, humidity, and wind speed. They can have the following values:

outlook = { sunny, overcast, rain }

temperature = {hot, mild, cool }

humidity = { high, normal }

wind = {weak, strong }

Examples of set S are:

Day	Outlook	Temperature	Humidity	Wind	Play ball
D1	Sunny	Hot	High	Weak	No
D2	Sunny	Hot	High	Strong	No
D3	Overcast	Hot	High	Weak	Yes
D4	Rain	Mild	High	Weak	Yes
D5	Rain	Cool	Normal	Weak	Yes
D6	Rain	Cool	Normal	Strong	No
D7	Overcast	Cool	Normal	Strong	Yes
D8	Sunny	Mild	High	Weak	No
D9	Sunny	Cool	Normal	Weak	Yes
D10	Rain	Mild	Normal	Weak	Yes
D11	Sunny	Mild	Normal	Strong	Yes
D12	Overcast	Mild	High	Strong	Yes
D13	Overcast	Hot	Normal	Weak	Yes
D14	Rain	Mild	High	Strong	No

Table 1

We need to find which attribute will be the root node in our decision tree. The gain is calculated for all four attributes:

Gain(S, Outlook) = 0.246

Gain(S, Temperature) = 0.029

Gain(S, Humidity) = 0.151

Gain(S, Wind) = 0.048 (calculated above)

Outlook attribute has the highest gain, therefore it is used as the decision attribute in the root node.

Since Outlook has three possible values, the root node has three branches (sunny, overcast, rain). The next question is "what attribute should be tested at the Sunny branch node?" Since we‘ve used Outlook at the root, we only decide on the remaining three attributes: Humidity, Temperature, or Wind.

S_sunny = {D1, D2, D8, D9, D11} = 5 examples from table1 with outlook = sunny

Gain(S_sunny, Humidity) = 0.970

Gain(S_sunny, Temperature) = 0.570

Gain(S_sunny, Wind) = 0.019

Humidity has the highest gain; therefore, it is used as the decision node. This process goes on until all data is classified perfectly or we run out of attributes.

The final decision = tree

The decision tree can also be expressed in rule format:

IF outlook = sunny AND humidity = high THEN playball = no

IF outlook = rain AND humidity = high THEN playball = no

IF outlook = rain AND wind = strong THEN playball = yes

IF outlook = overcast THEN playball = yes

IF outlook = rain AND wind = weak THEN playball = yes

ID3 has been incorporated in a number of commercial rule-induction packages. Some specific applications include medical diagnosis,credit risk assessment of loan applications, equipment malfunctions by their cause, classification of soybean diseases, and web search classification.