cpu 负载平衡 之 计算运行队列和cpu的load

在每个运行队列struct rq里，load代表当前运行队列的负载，同时还有一个cpu_load[]这样的数组，它是一个分级别的代表当前运行队列负载的“替身”。在多cpu调度时，会计算不同的cpu domain的负载，根据不同的index, 会选取相应的cpu_load[]作为当前运行队列的负载返回。

在每个tick处理函数scheduler_tick()中，会调用update_cpu_load(),来更新当前运行队列的负载，便于后面cpu调度调度平衡时选取最忙的cpu,比如调度函数find_busiest_group() ,它的工作是：find_busiest_group finds and returns the busiest CPU group within the domain. 它会通过两 个函数source_load（）和 target_load（）来计算并返回当前运行队列的负载。这两 个函数会有个参数index,它的目的就是选取相应的cpu_load[]。

在系统初始化过程中，会把cpu_load默认为0.  有兴趣的朋友可以参考sched.c  sched_init()函数，linux 2.6.28 line:8286
for (j = 0; j < CPU_LOAD_IDX_MAX; j++)
        rq->cpu_load[j] = 0;

那么后面这个数组是怎么变化的呢？它的变化趋势是什么样子的呢？
我觉得update_cpu_load()比较有意思，来一起分析下。

/*
* Update rq->cpu_load[] statistics. This function is usually called every
* scheduler tick (TICK_NSEC).
*/
static void update_cpu_load(struct rq *this_rq)
{
        unsigned long this_load = this_rq->load.weight;
                //当前运行队列的负载值
        int i, scale;

        this_rq->nr_load_updates++;         
              //代表load的更新次数    每一个tick都会加1 ，真够忙的。

        /* Update our load: */   
               //CPU_LOAD_IDX_MAX在运行队列结构体中=5,这儿对5个等级的cpu_load[]进行更新。
        for (i = 0, scale = 1; i < CPU_LOAD_IDX_MAX; i++, scale += scale) {
                unsigned long old_load, new_load;

                /* scale is effectively 1 << i now, and >> i divides by scale */  
                            //请注意，这个scale就是2 的i 次幂。它的值 分别是  1 2 4 8 16

                old_load = this_rq->cpu_load;  当前cpu_load[]数组里面的值
                new_load = this_load;  当前运行队列的负载值
                /*
                 * Round up the averaging division if load is increasing. This
                 * prevents us from getting stuck on 9 if the load is 10, for
                 * example.
                 */  
                              // round up：目的   如果load是在增长的，不要把余数别浪费了  
                if (new_load > old_load)
                        new_load += scale-1;
                this_rq->cpu_load = (old_load*(scale-1) + new_load) >> i;  
                                //这个公式，我会下面分析。它的意思就是根据当前运行队列的负载以及上次cpu_load[]的数值，计算出当前cpu_load[]应该的变化。
        }
}

现在分析下这个公式;
假设级别为i(0~4),运行队列的load值M,cpu_load=mi,则每次的计算遵循如何规律：
(M-mi)/2^i + mi

i=0:   M - mi + mi
i=1: (M - mi)/2 + mi
i=2: (M-mi)/4 + mi
i=3: (M-mi)/8 + mi
i=4: (M-mi)/16 + mi

由此可见，在M遵循上面的变化趋势下，等级为1的变化最为剧列。。。
另外，如果运行队列的load值比当前cpu_load[]的值大，会对M的值有个补偿：举这样一个例子，
假如M-mi = 17,对于计算i=4，来说，17/16 = 1,余数为1 ，这样太浪费了，，我要把余数也计算进来，所以我要在M-mi处理时，
加上（16-1），这样，就不会浪费余数了。


具体在系统中这个函数是如何变化的呢/    可以在pc terminal看下: cat /proc/sched_debu，可以看到不同cpu的相应数值

通过加一些调试信息看下：

/*
* Update rq->cpu_load[] statistics. This function is usually called
every
* scheduler tick (TICK_NSEC).
*/
static void update_cpu_load(struct rq *this_rq)
{
        unsigned long this_load = this_rq->load.weight;
        int i, scale;

        this_rq->nr_load_updates++;
        printk("Before calculate: %d/n", this_load);
        /* Update our load: */
        for (i = 0, scale = 1; i < CPU_LOAD_IDX_MAX; i++, scale += scale) {
                unsigned long old_load, new_load;

                /* scale is effectively 1 << i now, and >> i divides by scale */

                old_load = this_rq->cpu_load;
                new_load = this_load;
                /*
                 * Round up the averaging division if load is increasing. This
                 * prevents us from getting stuck on 9 if the load is 10, for
                 * example.
                 */
                if (new_load > old_load)
                        new_load += scale-1;
                this_rq->cpu_load = (old_load*(scale-1) + new_load) >> i;
                printk(KERN_INFO "old_load = %d,this_rq->cpu_load[%d] = %d, new_load
= %d/n",old_load, i, this_rq->cpu_load, new_load);
        }

}

debug  information:

Uncompressing
Linux................................................................
[    0.000000] clockevents_register_notifier: How many notifier
register on clocke?
[    0.000000] Linux version 2.6.25 (XXXXXXXX-desktop) (gcc
version 4.2.1) #19
[    0.000000] Linux comes here.....,it will setup timer in pxa3xx...
.....
Total pages4
[    0.007812] Kernel command line: console=ttyS0,115200 init=/init
android comm_v0
[    0.007812] PID hash table entries: 512 (order: 9, 2048 bytes)
[    0.007812] start_kernel:Before arch-init?????
[    0.007812] init_timers:when pxatimer?
[    0.007812] init_timers:Here///
[    0.007812] start_kernel:After initiate cpu-specific tiemrs...
[    0.007812] clock's name: jiffies
[    0.007812] start_kernelxa timer routine is here????
[    0.007812] pxa_timer_init: it's so so so late.........
[    0.007812] clockevents_register_device: Only call when add pxa
timer????
[    0.007812] clockevents_do_notify: I want to know how many times
call
[    0.007812] Console: colour dummy device 80x30
[    0.007812] Dentry cache hash table entries: 16384 (order: 4, 65536
bytes)
[    0.007812] Inode-cache hash table entries: 8192 (order: 3, 32768
bytes)
[    0.007812] Before calculate: 0
[    0.007812] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.007812] old_load = 0,this_rq->cpu_load[1] = 0, new_load = 0
[    0.007812] old_load = 0,this_rq->cpu_load[2] = 0, new_load = 0
[    0.007812] old_load = 0,this_rq->cpu_load[3] = 0, new_load = 0
[    0.007812] old_load = 0,this_rq->cpu_load[4] = 0, new_load = 0
[    0.007812] I think it only execute several times................
[    0.007812] Memory: 116MB = 116MB total
[    0.007812] Memory: 112640KB available (3548K code, 1080K data,
148K init)
[    0.007812] SLUB: Genslabs=12, HWalign=32, Order=0-1, MinObjects=4,
CPUs=1, Nod4
[    0.007812] Calibrating delay loop (skipped)... 622.58 BogoMIPS
preset
[    0.007812] Mount-cache hash table entries: 512
[    0.007812] CPU: Testing write buffer coherency: ok
[    0.007812] net_namespace: 160 bytes
[    0.015625] Before calculate: 7266
[    0.015625] old_load = 0,this_rq->cpu_load[0] = 7266, new_load =
7266
[    0.015625] old_load = 0,this_rq->cpu_load[1] = 3633, new_load =
7267
[    0.015625] old_load = 0,this_rq->cpu_load[2] = 1817, new_load =
7269
[    0.015625] old_load = 0,this_rq->cpu_load[3] = 909, new_load =
7273
[    0.015625] old_load = 0,this_rq->cpu_load[4] = 455, new_load =
7281
[    0.015625] I think it only execute several times................
...
[    0.015625] <MAX9724> GPIOs setup done
[    0.023437] Before calculate: 177522
[    0.023437] old_load = 7266,this_rq->cpu_load[0] = 177522, new_load
= 177522
[    0.023437] old_load = 3633,this_rq->cpu_load[1] = 90578, new_load
= 177523
[    0.023437] old_load = 1817,this_rq->cpu_load[2] = 45744, new_load
= 177525
[    0.023437] old_load = 909,this_rq->cpu_load[3] = 22986, new_load =
177529
[    0.023437] old_load = 455,this_rq->cpu_load[4] = 11522, new_load =
177537
[    0.023437] I think it only execute several times................
...A) and 174 (SCL)
[    0.023437] I2C: i2c-0: PXA I2C adapter
[    0.031250] Before calculate: 0
[    0.031250] old_load = 177522,this_rq->cpu_load[0] = 0, new_load =
0
[    0.031250] old_load = 90578,this_rq->cpu_load[1] = 45289, new_load
= 0
[    0.031250] old_load = 45744,this_rq->cpu_load[2] = 34308, new_load
= 0
[    0.031250] old_load = 22986,this_rq->cpu_load[3] = 20112, new_load
= 0
[    0.031250] old_load = 11522,this_rq->cpu_load[4] = 10801, new_load
= 0
[    0.031250] I think it only execute several times................
[    0.031250] micco(chip id:0x31) detected.
[    0.031250] GPIO-83 autorequested
[    0.039062] Before calculate: 0
[    0.039062] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.039062] old_load = 45289,this_rq->cpu_load[1] = 22644, new_load
= 0
[    0.039062] old_load = 34308,this_rq->cpu_load[2] = 25731, new_load
= 0
[    0.039062] old_load = 20112,this_rq->cpu_load[3] = 17598, new_load
= 0
[    0.039062] old_load = 10801,this_rq->cpu_load[4] = 10125, new_load
= 0
...
[    0.046875] Before calculate: 355044
[    0.046875] old_load = 0,this_rq->cpu_load[0] = 355044, new_load =
355044
[    0.046875] old_load = 22644,this_rq->cpu_load[1] = 188844,
new_load = 355045
[    0.046875] old_load = 25731,this_rq->cpu_load[2] = 108060,
new_load = 355047
[    0.046875] old_load = 17598,this_rq->cpu_load[3] = 59779, new_load
= 355051
[    0.046875] old_load = 10125,this_rq->cpu_load[4] = 31683, new_load
= 355059
[    0.046875] I think it only execute several times................
...
[    0.054687] Before calculate: 0
[    0.054687] old_load = 355044,this_rq->cpu_load[0] = 0, new_load =
0
[    0.054687] old_load = 188844,this_rq->cpu_load[1] = 94422,
new_load = 0
[    0.054687] old_load = 108060,this_rq->cpu_load[2] = 81045,
new_load = 0
[    0.054687] old_load = 59779,this_rq->cpu_load[3] = 52306, new_load
= 0
[    0.054687] old_load = 31683,this_rq->cpu_load[4] = 29702, new_load
= 0
[    0.054687] I think it only execute several times................
[    0.062500] Before calculate: 0
[    0.062500] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.062500] old_load = 94422,this_rq->cpu_load[1] = 47211, new_load
= 0
[    0.062500] old_load = 81045,this_rq->cpu_load[2] = 60783, new_load
= 0
[    0.062500] old_load = 52306,this_rq->cpu_load[3] = 45767, new_load
= 0
[    0.062500] old_load = 29702,this_rq->cpu_load[4] = 27845, new_load
= 0
[    0.062500] I think it only execute several times................
[    0.070312] Before calculate: 0
[    0.070312] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.070312] old_load = 47211,this_rq->cpu_load[1] = 23605, new_load
= 0
[    0.070312] old_load = 60783,this_rq->cpu_load[2] = 45587, new_load
= 0
[    0.070312] old_load = 45767,this_rq->cpu_load[3] = 40046, new_load
= 0
[    0.070312] old_load = 27845,this_rq->cpu_load[4] = 26104, new_load
= 0
[    0.070312] I think it only execute several times................
[    0.078125] Before calculate: 0
[    0.078125] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.078125] old_load = 23605,this_rq->cpu_load[1] = 11802, new_load
= 0
[    0.078125] old_load = 45587,this_rq->cpu_load[2] = 34190, new_load
= 0
[    0.078125] old_load = 40046,this_rq->cpu_load[3] = 35040, new_load
= 0
[    0.078125] old_load = 26104,this_rq->cpu_load[4] = 24472, new_load
= 0
[    0.078125] I think it only execute several times................
...
[    0.085937] Before calculate: 0
[    0.085937] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.085937] old_load = 11802,this_rq->cpu_load[1] = 5901, new_load
= 0
[    0.085937] old_load = 34190,this_rq->cpu_load[2] = 25642, new_load
= 0
[    0.085937] old_load = 35040,this_rq->cpu_load[3] = 30660, new_load
= 0
[    0.085937] old_load = 24472,this_rq->cpu_load[4] = 22942, new_load
= 0
[    0.085937] I think it only execute several times................
[    0.085937] Switched to NOHz mode on CPU #0
[    0.093750] Before calculate: 0
[    0.093750] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.093750] old_load = 5901,this_rq->cpu_load[1] = 2950, new_load =
0
[    0.093750] old_load = 25642,this_rq->cpu_load[2] = 19231, new_load
= 0
[    0.093750] old_load = 30660,this_rq->cpu_load[3] = 26827, new_load
= 0
[    0.093750] old_load = 22942,this_rq->cpu_load[4] = 21508, new_load
= 0
[    0.101562] Before calculate: 0
[    0.101562] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.101562] old_load = 2950,this_rq->cpu_load[1] = 1475, new_load =
0
[    0.101562] old_load = 19231,this_rq->cpu_load[2] = 14423, new_load
= 0
[    0.101562] old_load = 26827,this_rq->cpu_load[3] = 23473, new_load
= 0
[    0.101562] old_load = 21508,this_rq->cpu_load[4] = 20163, new_load
= 0
[    0.109375] Before calculate: 0
[    0.109375] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.109375] old_load = 1475,this_rq->cpu_load[1] = 737, new_load =
0
[    0.109375] old_load = 14423,this_rq->cpu_load[2] = 10817, new_load
= 0
[    0.109375] old_load = 23473,this_rq->cpu_load[3] = 20538, new_load
= 0
[    0.109375] old_load = 20163,this_rq->cpu_load[4] = 18902, new_load
= 0
[    0.117187] Before calculate: 0
[    0.117187] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.117187] old_load = 737,this_rq->cpu_load[1] = 368, new_load = 0
[    0.117187] old_load = 10817,this_rq->cpu_load[2] = 8112, new_load
= 0
[    0.117187] old_load = 20538,this_rq->cpu_load[3] = 17970, new_load
= 0
[    0.117187] old_load = 18902,this_rq->cpu_load[4] = 17720, new_load
= 0
[    0.125000] Before calculate: 0
[    0.125000] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.125000] old_load = 368,this_rq->cpu_load[1] = 184, new_load = 0
[    0.125000] old_load = 8112,this_rq->cpu_load[2] = 6084, new_load =
0
[    0.125000] old_load = 17970,this_rq->cpu_load[3] = 15723, new_load
= 0
[    0.125000] old_load = 17720,this_rq->cpu_load[4] = 16612, new_load
= 0
[    0.132812] Before calculate: 0
[    0.132812] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.132812] old_load = 184,this_rq->cpu_load[1] = 92, new_load = 0
[    0.132812] old_load = 6084,this_rq->cpu_load[2] = 4563, new_load =
0
[    0.132812] old_load = 15723,this_rq->cpu_load[3] = 13757, new_load
= 0
[    0.132812] old_load = 16612,this_rq->cpu_load[4] = 15573, new_load
= 0
[    0.140625] Before calculate: 0
[    0.140625] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.140625] old_load = 92,this_rq->cpu_load[1] = 46, new_load = 0
[    0.140625] old_load = 4563,this_rq->cpu_load[2] = 3422, new_load =
0
[    0.140625] old_load = 13757,this_rq->cpu_load[3] = 12037, new_load
= 0
[    0.140625] old_load = 15573,this_rq->cpu_load[4] = 14599, new_load
= 0
[    0.148437] Before calculate: 0
[    0.148437] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.148437] old_load = 46,this_rq->cpu_load[1] = 23, new_load = 0
[    0.148437] old_load = 3422,this_rq->cpu_load[2] = 2566, new_load =
0
[    0.148437] old_load = 12037,this_rq->cpu_load[3] = 10532, new_load
= 0
[    0.148437] old_load = 14599,this_rq->cpu_load[4] = 13686, new_load
= 0
[...
[    0.156250] TCP reno registered
[    0.156250] Before calculate: 0
[    0.156250] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.156250] old_load = 23,this_rq->cpu_load[1] = 11, new_load = 0
[    0.156250] old_load = 2566,this_rq->cpu_load[2] = 1924, new_load =
0
[    0.156250] old_load = 10532,this_rq->cpu_load[3] = 9215, new_load
= 0
[    0.156250] old_load = 13686,this_rq->cpu_load[4] = 12830, new_load
= 0
[    0.164062] Before calculate: 0
[    0.164062] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.164062] old_load = 11,this_rq->cpu_load[1] = 5, new_load = 0
[    0.164062] old_load = 1924,this_rq->cpu_load[2] = 1443, new_load =
0
[    0.164062] old_load = 9215,this_rq->cpu_load[3] = 8063, new_load =
0
[    0.164062] old_load = 12830,this_rq->cpu_load[4] = 12028, new_load
= 0
[    0.171875] Before calculate: 0
[    0.171875] old_load = 0,this_rq->cpu_load[0] = 0, new_load = 0
[    0.171875] old_load = 5,this_rq->cpu_load[1] = 2, new_load = 0
[    0.171875] old_load = 1443,this_rq->cpu_load[2] = 1082, new_load =
0
[    0.171875] old_load = 8063,this_rq->cpu_load[3] = 7055, new_load =
0
[    0.171875] old_load = 12028,this_rq->cpu_load[4] = 11276, new_load
= 0
[    0.179687] initcall 0xc0022f2c ran for 93 msecs: inet_init
+0x0/0x2e8()
[    0.179687] Unpacking initramfs...Before calculate: 1024
[    0.179687] old_load = 0,this_rq->cpu_load[0] = 1024, new_load =
1024
[    0.179687] old_load = 2,this_rq->cpu_load[1] = 513, new_load =
1025
[    0.179687] old_load = 1082,this_rq->cpu_load[2] = 1067, new_load =
1024
[    0.179687] old_load = 7055,this_rq->cpu_load[3] = 6301, new_load =
1024
[    0.179687] old_load = 11276,this_rq->cpu_load[4] = 10635, new_load
= 1024
[    0.187500] Before calculate: 1024
[    0.187500] old_load = 1024,this_rq->cpu_load[0] = 1024, new_load =
1024
[    0.187500] old_load = 513,this_rq->cpu_load[1] = 769, new_load =
1025
[    0.187500] old_load = 1067,this_rq->cpu_load[2] = 1056, new_load =
1024
[    0.187500] old_load = 6301,this_rq->cpu_load[3] = 5641, new_load =
1024
[    0.187500] old_load = 10635,this_rq->cpu_load[4] = 10034, new_load
= 1024
[    0.195312] Before calculate: 1024
[    0.195312] old_load = 1024,this_rq->cpu_load[0] = 1024, new_load =
1024
[    0.195312] old_load = 769,this_rq->cpu_load[1] = 897, new_load =
1025
[    0.195312] old_load = 1056,this_rq->cpu_load[2] = 1048, new_load =
1024
[    0.195312] old_load = 5641,this_rq->cpu_load[3] = 5063, new_load =
1024
[    0.195312] old_load = 10034,this_rq->cpu_load[4] = 9470, new_load
= 1024
[    0.203125]  done
.....
[    0.203125] ashmem: initialized
[    0.203125] Before calculate: 7266
[    0.203125] old_load = 1024,this_rq->cpu_load[0] = 7266, new_load =
7266
[    0.203125] old_load = 897,this_rq->cpu_load[1] = 4082, new_load =
7267
[    0.203125] old_load = 1048,
........

.......

[   79.617187] Before calculate: 360213
[   79.617187] old_load = 360213,this_rq->cpu_load[0] = 360213, new_load = 360213
[   79.617187] old_load = 360213,this_rq->cpu_load[1] = 360213, new_load = 360213
[   79.617187] old_load = 360213,this_rq->cpu_load[2] = 360213, new_load = 360213
[   79.617187] old_load = 360213,this_rq->cpu_load[3] = 360213, new_load = 360213
[   79.617187] old_load = 360213,this_rq->cpu_load[4] = 360213, new_load = 360213
[   79.656250] Before calculate: 360213
[   79.656250] old_load = 360213,this_rq->cpu_load[0] = 360213, new_load = 360213
[   79.656250] old_load = 360213,this_rq->cpu_load[1] = 360213, new_load = 360213
[   79.656250] old_load = 360213,this_rq->cpu_load[2] = 360213, new_load = 360213
[   79.656250] old_load = 360213,this_rq->cpu_load[3] = 360213, new_load = 360213
[   79.656250] old_load = 360213,this_rq->cpu_load[4] = 360213, new_load = 360213
[   79.695312] Before calculate: 360213
.......

为了大家看得方便，我把无关 的启动信息用...代替了。
大家只需要看我添加的打印信息即可。

可以明显看到,运行队列的负载在变化 。
this_rq->load.weight的变化规律：
初始化0 -->突然变得很大 -->0---->？


通过后面的调试发现，cpu_load[]数组的值，总是在尽量地接近运行队列的load值，，随着scale增大，接进得越慢。。。cpu_load[0]总是与运行队列的load值相同。