Linux调度器性能分析 - 2
This article was firstly published from http://oliveryang.net. The content reuse need include the original link.
SCHEDSTATS Perf Counters - Overview
1. What is the SCHEDSTATS?
SCHEDSTATS is a kernel debug feature which allows scheduler exports its pre-defined performance counters to user space. We can do following things by collecting and analyzing these perf counters,
- Debug or tune scheduler
- Debug or tune specific application or benchmark from scheduling perspective
2. How could we access SCHEDSTATS counters?
When SCHEDSTATS is enabled, scheduler statistics could be accessed by following ways,
Three proc files exported by SCHEDSTATS code
/proc/schedstat, /proc/[pid]/schedstat, /proc/[pid]/sched
Documentation/scheduler/sched-stats.txt file has the full description for file format. We can write user space tools to read and process the proc files.
pre-defined kernel trace points
Kernel trace points could be used by dynamic tracing tools, such as systemtap, perf. So far, in Linux 4.1, there are 4 sched_stat_* trace points defined by SCHEDSTATS code, there are 4 sched_stat_* trace points defined by SCHEDSTATS code.
# perf list | grep sched_stat_ sched:sched_stat_wait [Tracepoint event] sched:sched_stat_sleep [Tracepoint event] sched:sched_stat_iowait [Tracepoint event] sched:sched_stat_blocked [Tracepoint event] sched:sched_stat_runtime [Tracepoint event] >>>> Not a SCHEDSTAT trace pointLinux perf tool, record, report, script sub-commands could be used for getting system wide or per-task statistics.
sleep profiler when SCHEDSTATS is enabled
This needs readprofile command installed in user space. The usage of readprofile could be found from Documentation/basic_profiling.txt. To enable kernel profiler, please refer to Documentation/kernel-parameters.txt. This way is a legacy way and could be replaced by following trace point in latest kernel,
# perf list | grep sched_stat_blocked sched:sched_stat_blocked [Tracepoint event] # perf record -e sched:sched_stat_blocked -a -g sleep 5 # perf script
3. SCHEDSTATS proc files use cases
3.1 System wide statistic
This includes per-cpu(run queue) or per-sched-domain statistics.
**/proc/schedstat**
Implements in scheduler core, which is the common layer for all scheduling classes.
The CPU statistics in /proc/schedstat file is defined as members of struct rq in kernel/sched.c,
struct rq {
[...snipped...]
#ifdef CONFIG_SCHEDSTATS
/* latency stats */
struct sched_info rq_sched_info;
unsigned long long rq_cpu_time;
/* could above be rq->cfs_rq.exec_clock + rq->rt_rq.rt_runtime ? */
/* sys_sched_yield() stats */
unsigned int yld_count;
/* schedule() stats */
unsigned int sched_switch;
unsigned int sched_count;
unsigned int sched_goidle;
/* try_to_wake_up() stats */
unsigned int ttwu_count;
unsigned int ttwu_local;
#endif
[...snipped...]
};
The Domain statistics in /proc/schedstat file is defined as members of struct sched_domain
in include/linux/sched.h,
struct sched_domain {
[...snipped...]
#ifdef CONFIG_SCHEDSTATS
/* load_balance() stats */
unsigned int lb_count[CPU_MAX_IDLE_TYPES];
unsigned int lb_failed[CPU_MAX_IDLE_TYPES];
unsigned int lb_balanced[CPU_MAX_IDLE_TYPES];
unsigned int lb_imbalance[CPU_MAX_IDLE_TYPES];
unsigned int lb_gained[CPU_MAX_IDLE_TYPES];
unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES];
unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES];
unsigned int lb_nobusyq[CPU_MAX_IDLE_TYPES];
/* Active load balancing */
unsigned int alb_count;
unsigned int alb_failed;
unsigned int alb_pushed;
/* SD_BALANCE_EXEC stats */
unsigned int sbe_count;
unsigned int sbe_balanced;
unsigned int sbe_pushed;
/* SD_BALANCE_FORK stats */
unsigned int sbf_count;
unsigned int sbf_balanced;
unsigned int sbf_pushed;
/* try_to_wake_up() stats */
unsigned int ttwu_wake_remote;
unsigned int ttwu_move_affine;
unsigned int ttwu_move_balance;
#endif
[...snipped...]
};
3.2 Per task statistic
/proc/[pid]/schedstat
Common for all scheduling classes.
The statistics for /proc/[pid]/schedstat is defined as member of struct task_struct in include/linux/sched.h,
#if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
struct sched_info {
/* cumulative counters */
unsigned long pcount; /* # of times run on this cpu */
unsigned long long run_delay; /* time spent waiting on a run queue */
/* timestamps */
unsigned long long last_arrival,/* when we last ran on a cpu */
last_queued; /* when we were last queued to run */
};
#endif /* defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT) */
struct task_struct {
[...snipped...]
#if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
struct sched_info sched_info;
#endif
[...snipped...]
};
/proc/[pid]/sched
Only available for CFS tasks. Need enable SCHED_DEBUG as well.
The se statistics for /proc/[pid]/sched is defined as member of struct task_struct in include/linux/sched.h,
#ifdef CONFIG_SCHEDSTATS
struct sched_statistics {
u64 wait_start;
u64 wait_max;
u64 wait_count;
u64 wait_sum;
u64 iowait_count;
u64 iowait_sum;
u64 sleep_start;
u64 sleep_max;
s64 sum_sleep_runtime;
u64 block_start;
u64 block_max;
u64 exec_max;
u64 slice_max;
u64 nr_migrations_cold;
u64 nr_failed_migrations_affine;
u64 nr_failed_migrations_running;
u64 nr_failed_migrations_hot;
u64 nr_forced_migrations;
u64 nr_wakeups;
u64 nr_wakeups_sync;
u64 nr_wakeups_migrate;
u64 nr_wakeups_local;
u64 nr_wakeups_remote;
u64 nr_wakeups_affine;
u64 nr_wakeups_affine_attempts;
u64 nr_wakeups_passive;
u64 nr_wakeups_idle;
};
#endif
struct sched_entity {
[...snipped...]
#ifdef CONFIG_SCHEDSTATS
struct sched_statistics statistics;
#endif
[...snipped...]
};
struct task_struct {
[...snipped...]
struct sched_entity se;
[...snipped...]
};
4. SCHEDSTATS source files
To use SCHEDSTATS, need to enable kernel config SCHEDSTATS. All related code is protected by CONFIG_SCHEDSTATS.
As far as we know, Linux kernel scheduler defined two layers,
4.1 The upper layer is scheduler core which is common layer for all scheduling class.
In Linux 3.2.x, The SCHEDSTATS source files in scheduler common layer are,
include/linux/sched.h
Per-sched-domain and per-task perf counters definitions.
kernel/sched_stats.h
/proc/schestat proc file implementation
fs/proc/base.c
/proc/[pid]/schedstat proc file implementation
kernel/sched.c
Per-runqueue perf counters definitions.
Per-runqueue, per-sched-domain, per-task perf counters implementation, for example, ttwu_stat
kernel/profile.c
The legacy code, profiling code for /proc/profile support, readprofile(1) could read it.
kernel/sched_debug.c
SCHEDSTATS in /proc/sched_debug and /proc/[pid]/sched proc files implementation.
Need enable SCHED_DEBUG at same time.
4.2 The underlying layer is per scheduling class source code.
In Linux 3.2.x, only the CFS scheduling class code has the SCHEDSTATS implementation.
kernel/sched_fair.c
SCHEDSTATS in /proc/[pid]/sched. Need enable SCHED_DEBUG at same time.
/proc/schedstat counters for load balance.
Kernel Trace points for wait, sleep, iowait, blocked(not in 3.2.x) events. See section 3 in this blog.