SQL Server 2008性能故障排查（三）——I/O

接着上一章：CPU瓶颈

I/O瓶颈（I/O Bottlenecks）:

SQLServer的性能严重依赖I/O子系统。除非你的数据库完全加载到物理内存中，否则SQLServer会不断地把数据库文件从缓存池中搬进搬出，这会引起大量的I/O传输。同样地，日志记录在事务被声明为已提交前必须写入磁盘。最后，SQLServer基于许多原因使用tempdb，比如存储临时结果、排序和保持行版本。所以一个好的I/O子系统是SQLServer性能关键。
除非数据文件包括tempdb需要回滚事务，否则日志文件是顺序访问的。而数据文件和tempdb是随机访问的。所以作为常规规则，你应该把日志文件与数据文件分离到独立的磁盘中。本文不是关注于如何配置你的I/O设备，但关注于如何识别你的系统是否有I/O瓶颈。在I/O瓶颈被识别之后，你应该重新配置你的I/O子系统。
如果你的I/O子系统很慢，你的用户将体验得到性能问题，响应时间过慢和因为超时而导致任务失败。
可以使用以下的性能计数器去识别I/O瓶颈。但是要注意，如果你的收集间隔过短，那么平均值会趋向倾斜于低值那段。比如，很难说明为什么I/O会每60秒涨跌。同时，你也不能仅仅根据一个计数器的值来确定是否有瓶颈。需要通过多个值来反复验证你的想法：
PhysicalDisk Object:Avg.Disk Queue：物理读写请求锁等待的平均队列值。当该值长期超过2时，你的系统可能存在I/O瓶颈了。
Avg.Disk Sec/Read：是一个平均秒数，是每秒从磁盘上读取数据的次数，下面是值及其代表意思：
• 小于10ms ——非常好
• 10~20ms——OK
• 20~50ms——慢，需要重视
• 大于50ms——严重的I/O瓶颈。
Avg.Disk Sec/Write：与Avg.Disk Sec/Read相对应。
Physical Disk:%Disk Time：是针对被选定的磁盘忙于读写请求所运行时间的百分数。一般的指标线是大于50%就意味着有I/O瓶颈。
Avg.Disk Reads/Sec：是读操作在磁盘上的频率。确保这个频率低于磁盘极限的85%，当超过了85%后，访问时间就会以指数速度增加。
Avg.Disk Writes/Sec：于Avg.Disk Reads/Sec相对应。
当你使用这些计数器时，你需要对于RAID作出调整，可以使用以下公式：
 Raid 0 -- I/Os per disk = (reads + writes) / number of disks
 Raid 1 -- I/Os per disk = [reads + (2 * writes)] / 2
 Raid 5 -- I/Os per disk = [reads + (4 * writes)] / number of disks
 Raid 10 -- I/Os per disk = [reads + (2 * writes)] / number of disks
比如，如果你有一个RAID-1，使用两个物理磁盘，计数器值为：
Disk Reads/sec 80
Disk Writes/sec 70
Avg.Disk Queue length 5
这样通过公式计算：(80 + (2 * 70))/2 = 110 I/Os 每个磁盘，而你的磁盘等待队列长度等于5/2=2.5。意味着已经到达了I/O瓶颈边界。
你也可以检查lacth等待来识别I/O瓶颈。这种等待说明当一些页面用于读或者写访问时，同时这些页面在缓冲池中不可用（或者不存在）而造成的物理I/O等待。当页面在缓冲池中找不到时。就会产生一个异步的I/O，然后检查这个I/O的状态。当I/O状态已经被标注为已完成时，此时工作负载趋于平衡。否则，将会等待PAGEIOLATCH_EX 或者PAGEIOLATCH_SH，这根据请求类型而定。可以使用一下DMV来发现I/O闩锁的等待统计信息：

Select  wait_type, 
        waiting_tasks_count, 
        wait_time_ms
from	sys.dm_os_wait_stats  
where	wait_type like 'PAGEIOLATCH%'  
order by wait_type

下面是结果的例子：
wait_type       waiting_tasks_count  wait_time_ms   signal_wait_time_ms
-----------------------------------------------------------------------
PAGEIOLATCH_DT  0                    0                    0
PAGEIOLATCH_EX  1230                 791                  11
PAGEIOLATCH_KP  0                    0                    0
PAGEIOLATCH_NL  0                    0                    0
PAGEIOLATCH_SH  13756                7241                 180
PAGEIOLATCH_UP  80                   66                   0
当I/O完成时，工作线程将被至于可运行队列。I/O完成到工作线程确实被排程的时间在signal_wait_time_ms列中可以看到，如果你的waiting_task_counts和wait_time_ms有偏离常值，证明有I/O问题。对于这种情况，有必要在SQLServer运行正常时，建立性能基线和关键的DMV查询输出。这些等待类型能显示出你的I/O子系统是否有严重的瓶颈。但它们不提供任何可见的物理磁盘问题
你可以通过下面的DMV查询来找到目前正在挂起的I/O请求。你可以定期执行下面语句来检查I/O子系统的健康情况和隔离那些有I/O瓶颈的物理磁盘：

select 
    database_id, 
    file_id, 
    io_stall,
    io_pending_ms_ticks,
    scheduler_address 
from	sys.dm_io_virtual_file_stats(NULL, NULL)t1,
        sys.dm_io_pending_io_requests as t2
where	t1.file_handle = t2.io_handle

下面是一个输出例子，是对特定的数据库输出，在运行查询的时刻，有3个被挂起的I/O请求。你可以使用database_id 和file_id列来查找文件所映射的物理磁盘。Io_pending_ms_ticks值表示单个I/O在挂起队列中等待的总时间。


Database_id File_Id io_stall io_pending_ms_ticks scheduler_address
-------------------------------------------------------------
6 1 10804 78 0x0227A040
6 1 10804 78 0x0227A040
6 2 101451 31 0x02720040




解决方案：
当你发现有I/O瓶颈时，你第一本能反应可能是升级I/O子系统，以应对目前的工作负载。这种方式当然有效，但是在此之前，你要考虑在硬件投入上的开销，要检查I/O瓶颈是否因为不正确的配置和/或查询计划导致的。我们建议你根据以下步骤去检查：
1、 配置（Configuration）：检查SQLServer的内存配置。如果SQLServer配置中存在内存不足的问题，这会引起更多I/O开销。你可以检查下面的计数器来识别是否存在内存压力：
 Buffer Cache hit ratio
 Page Life Expectancy
 Checkpoint Pages/sec
 Lazywrites/sec
关于内存压力将在内存篇详细说明
2、 查询计划：检查执行计划和识别哪步导致了更多的I/O消耗。尽可能选择更好的方法比如索引来最小化I/O。如果存在丢失索引，可以使用DTA来找到。
下面的DMV查询可以用于发现批处理或者请求产生最多的I/O的查询。注意这里不统计物理写，如果你懂得数据库是如何运作的，就会知道为什么。在同一个请求中DML和DDL语句，不是直接把数据页写入磁盘，而只有已经提交的事务才会被写入磁盘。通常物理写只在checkpoint或者lazywriter发生时才出现。可以使用下面的DMV来查找产生最多I/O的5个查询，优化这些查询以便实现更少的逻辑读，并进一步缓解缓存池的压力。这样你能提高其他请求在缓存池中直接找到数据的机会（特别在重复执行时），从而替代物理I/O的性能。因此，这个系统的性能都能得到改进。
下面是通过hash join来做两表关联的例子：

create table t1 (c1 int primary key, c2 int, c3 char(8000))
   create table t2  (C4 int, c5 char(8000))
go


   --load the data
declare @i int
select @i = 0
while (@i < 6000) 
begin
    insert into t1 values (@i, @i + 1000, 'hello')
   insert into t2 values (@i,'there')
   set @i = @i + 1
end
--now run the following query
select c1, c5
from t1 INNER HASH JOIN t2 ON t1.c1 = t2.c4
order by c2 


Run another query so that there are two queries to look at for I/O stats


select SUM(c1) from t1

这两个查询在一个批处理中运行，接下来。使用下面的DMV来检查查询引起的I/O：

SELECT TOP 5 
    (total_logical_reads/execution_count) AS avg_logical_reads,
    (total_logical_writes/execution_count) AS avg_logical_writes,
    (total_physical_reads/execution_count) AS avg_phys_reads,
    execution_count, 
    statement_start_offset as stmt_start_offset, 
    (SELECT SUBSTRING(text, statement_start_offset/2 + 1,
        (CASE WHEN statement_end_offset = -1 
            THEN LEN(CONVERT(nvarchar(MAX),text)) * 2 
                ELSE statement_end_offset 
            END - statement_start_offset)/2)
     FROM sys.dm_exec_sql_text(sql_handle)) AS query_text, 
      (SELECT query_plan from sys.dm_exec_query_plan(plan_handle)) as query_plan
FROM sys.dm_exec_query_stats  
ORDER BY (total_logical_reads + total_logical_writes)/execution_count DESC

你当然可以通过改变查询语句来获得不同的数据显示。比如你可以按(total_logical_reads + total_logical_writes)/execution_count 来排序。作为选择，你可能想去按物理I/O来排序等，但是，逻辑读写书对判断是否有I/O问题是很有用的。输出类似这样：
avg_logical_reads    avg_logical_writes   avg_phys_reads       
-----------------    ------------------   ---------------
16639 10   1098
6023 0   0
execution_count      stmt_start_offset
---------------      -----------------
1 0
1 154


Query_text      Query_plan                        
-----------------------------------          -----------
select c1, c5  from t1 INNER HASH JOIN …    
select SUM(c1) from t1                      


这些输出告诉你一些重要的信息，第一，显示最多的I/O。你也可以通过SQL Text列来查看是否可以通过重写语句来降低I/O。验证这些执行计划是否已经最佳的。比如，一个新的索引可能有帮助。第二、第二个批处理不引起任何物理I/O因为所有需要的表的页面已经缓存到缓冲区。第三、执行次数能用于识别是否它是一个一次性查询或者它是否频繁执行，因此需要对此详细考量。
3、 数据压缩：从2008开始，你能使用数据压缩来降低表和索引的体积。压缩程度完全取决于架构和数据分布。一般情况下，可以达到50~60%的压缩率。一些特殊情况下可以达到90%。意味着当你能压缩到50%时，你已经比较有效地降低了I/O。数据压缩会引起CPU增加。这里有一些策略：
为什么不把整个数据库压缩？对此，给出一个极端的例子：如果你有一个大表，叫做T，有10页，而整个数据库有1000万页。压缩T没有多大好处。即使SQLServer能把10页压缩到1页，你努力减少数据库的大小，但你可能会造成CPU的负担增加。在现实的工作负载中，不能很明显地作出选择。但是这个例子只是你在压缩前要考虑的情况而已。我们的建议是：在你压缩一个对象之前，使用sp_estimate_data_compression_savings存储过程来评估它的大小、利用情况和预估压缩等信息。注意以下信息：
 对象的大小是否比数据库总体大小小很多，这样的情况不会给你带来太多好处。
 如果对象经常被用于DML或者SELECT操作，你将面临比较大的CPU消耗。特别是在CPU已经存在瓶颈时，你可以使用sys.dm_db_index_operational_stats去发现对象使用情况来判断表、索引、分区等等的命中情况。
 压缩预留情况是基于架构和基于数据的。实际上，一些对象，压缩后可能会更大。或者节省的空间会微不足道。
如果你有一个分区表，且某些分区的数据不经常访问。你可以使用页压缩来压缩分区和重组索引。这适用在不长使用的分区上。相信信息可以看：(http://blogs.msdn.com/sqlserverstorageengine/archive/tags/Data+Compression/default.aspx)
4、 升级I/O子系统：如果你确保SQLServer的配置合理，并且检查执行计划后仍然存在I/O瓶颈，最后的选择就只能升级I/O带宽了：
 增加更多的物理驱动或者更换更快的磁盘。
 增加更快的I/O控制器。

下一章：tempdb

原文：

I/O Bottlenecks
SQL Server performance depends heavily on the I/O subsystem. Unless your database fits into physical memory, SQL Server constantly brings database pages in and out of the buffer pool. This generates substantial I/O traffic. Similarly, the log records need to be flushed to the disk before a transaction can be declared committed. And finally, SQL Server uses tempdb for various purposes such as storing intermediate results, sorting, and keeping row versions. So a good I/O subsystem is critical to the performance of SQL Server. 
Access to log files is sequential except when a transaction needs to be rolled back while data files, including tempdb, are randomly accessed. So as a general rule, you should have log files on a physical disk that is separate from the data files for better performance. The focus of this paper is not how to configure your I/O devices but to describe ways to identify whether you have I/O bottleneck. After an I/O bottleneck is identified, you may need to reconfigure your I/O subsystem.
If you have a slow I/O subsystem, your users may experience performance problems such as slow response times and tasks that do not complete due to time-outs. 
You can use the following performance counters to identify I/O bottlenecks. Note that these AVG values tend to be skewed (to the low side) if you have an infrequent collection interval. For example, it is hard to tell the nature of an I/O spike with 60-second snapshots. Also, you should not rely on one counter to determine a bottleneck; look for multiple counters to cross-check the validity of your findings.
PhysicalDisk Object: Avg. Disk Queue Length represents the average number of physical read and write requests that were queued on the selected physical disk during the sampling period. If your I/O system is overloaded, more read/write operations will be waiting. If your disk queue length frequently exceeds a value of 2 during peak usage of SQL Server, you might have an I/O bottleneck. 
Avg. Disk Sec/Read is the average time, in seconds, of a read of data from the disk. The following list shows ranges of possible values and what the ranges mean:
• Less than 10 ms - very good
• Between 10 - 20 ms - okay
• Between 20 - 50 ms - slow, needs attention
• Greater than 50 ms – Serious I/O bottleneck
Avg. Disk Sec/Write is the average time, in seconds, of a write of data to the disk. The guidelines for the Avg. Disk Sec/Read values apply here.
Physical Disk: %Disk Time is the percentage of elapsed time that the selected disk drive was busy servicing read or write requests. A general guideline is that if this value is greater than 50 percent, there is an I/O bottleneck. 
Avg. Disk Reads/Sec is the rate of read operations on the disk. Ensure that this number is less than 85 percent of the disk capacity. The disk access time increases exponentially beyond 85 percent capacity. 
Avg. Disk Writes/Sec is the rate of write operations on the disk. Ensure that this number is less than 85 percent of the disk capacity. The disk access time increases exponentially beyond 85 percent capacity.
When you use these counters, you may need to adjust the values for RAID configurations using the following formulas:
• Raid 0 -- I/Os per disk = (reads + writes) / number of disks
• Raid 1 -- I/Os per disk = [reads + (2 * writes)] / 2
• Raid 5 -- I/Os per disk = [reads + (4 * writes)] / number of disks
• Raid 10 -- I/Os per disk = [reads + (2 * writes)] / number of disks
For example, you might have a RAID-1 system with two physical disks with the following values of the counters.


Disk Reads/sec            80
Disk Writes/sec           70
Avg. Disk Queue Length    5


In that case, you are encountering (80 + (2 * 70))/2 = 110 I/Os per disk and your disk queue length = 5/2 = 2.5, which indicates a borderline I/O bottleneck. 
You can also identify I/O bottlenecks by examining the latch waits. These latch waits account for the physical I/O waits when a page is accessed for reading or writing and the page is not available in the buffer pool. When the page is not found in the buffer pool, an asynchronous I/O is posted and then the status of the I/O is checked. If the I/O has already completed, the worker proceeds normally. Otherwise, it waits on PAGEIOLATCH_EX or PAGEIOLATCH_SH, depending upon the type of request. You can use the following DMV query to find I/O latch wait statistics.


Select  wait_type, 
        waiting_tasks_count, 
        wait_time_ms
from sys.dm_os_wait_stats  
where wait_type like 'PAGEIOLATCH%'  
order by wait_type


A sample output follows.


wait_type       waiting_tasks_count  wait_time_ms   signal_wait_time_ms
-----------------------------------------------------------------------
PAGEIOLATCH_DT  0                    0                    0
PAGEIOLATCH_EX  1230                 791                  11
PAGEIOLATCH_KP  0                    0                    0
PAGEIOLATCH_NL  0                    0                    0
PAGEIOLATCH_SH  13756                7241                 180
PAGEIOLATCH_UP  80                   66                   0


When the I/O completes, the worker is placed in the runnable queue. The time between I/O completions until the time the worker is actually scheduled is accounted under the signal_wait_time_ms column. You can identify an I/O problem if your waiting_task_counts and wait_time_ms deviate significantly from what you see normally. For this, it is important to get a baseline of performance counters and key DMV query outputs when SQL Server is running smoothly. These wait_types can indicate whether your I/O subsystem is experiencing a bottleneck, but they do not provide any visibility on the physical disk(s) that are experiencing the problem. 
You can use the following DMV query to find currently pending I/O requests. You can execute this query periodically to check the health of I/O subsystem and to isolate physical disk(s) that are involved in the I/O bottlenecks.


select 
    database_id, 
    file_id, 
    io_stall,
    io_pending_ms_ticks,
    scheduler_address 
from sys.dm_io_virtual_file_stats(NULL, NULL)t1,
        sys.dm_io_pending_io_requests as t2
where t1.file_handle = t2.io_handle


A sample output follows. It shows that on a given database, there are three pending I/Os at this moment. You can use the database_id and file_id columns to find the physical disk the files are mapped to. The io_pending_ms_ticks values represent the total time individual I/Os are waiting in the pending queue. 


Database_id File_Id io_stallio_pending_ms_ticksscheduler_address
----------------------------------------------------------------------
6 1 10804780x0227A040
6 1 10804780x0227A040
6 2 101451310x02720040
 
Resolution
When you see an I/O bottleneck, your first instinct might be to upgrade the I/O subsystem to meet the workload requirements. This will definitely help, but before you go out and invest money in hardware, examine the I/O bottleneck to see whether it is the result of poor configuration and/or query plans. We recommend you to follow the steps below in strict order.
1. Configuration: Check the memory configuration of SQL Server. If SQL Server has been configured with insufficient memory, it will incur more I/O overhead. You can examine the following counters to identify memory pressure:
• Buffer Cache hit ratio
• Page Life Expectancy
• Checkpoint pages/sec
• Lazywrites/sec
For more information about memory pressure, see Memory Bottlenecks earlier in this paper.
2. Query Plans: Examine execution plans and see which plans lead to more I/O being consumed. It is possible that a better plan (for example, index) can minimize I/O. If there are missing indexes, you may want to run Database Engine Tuning Advisor to find missing indexes.
The following DMV query can be used to find which batches or requests are generating the most I/O. Note that we are not accounting for physical writes. This is okay if you consider how databases work. The DML and DDL statements within a request do not directly write data pages to disk. Instead, the physical writes of pages to disks is triggered by statements only by committing transactions. Usually physical writes are done either by checkpoint or by the SQL Server lazy writer. You can use a DMV query like the following to find the five requests that generate the most I/Os. Tuning those queries so that they perform fewer logical reads can relieve pressure on the buffer pool. This enables other requests to find the necessary data in the buffer pool in repeated executions (instead of performing physical I/O). Hence, overall system performance is improved. 
Here is an example of a query that joins two tables with a hash join.


create table t1 (c1 int primary key, c2 int, c3 char(8000))
   create table t2  (C4 int, c5 char(8000))
go


   --load the data
declare @i int
select @i = 0
while (@i < 6000) 
begin
    insert into t1 values (@i, @i + 1000, 'hello')
   insert into t2 values (@i,'there')
   set @i = @i + 1
end
--now run the following query
select c1, c5
from t1 INNER HASH JOIN t2 ON t1.c1 = t2.c4
order by c2 


Run another query so that there are two queries to look at for I/O stats


select SUM(c1) from t1


These two queries are run in the single batch. Next, use the following DMV query to examine the queries that generate the most I/Os 


SELECT TOP 5 
    (total_logical_reads/execution_count) AS avg_logical_reads,
    (total_logical_writes/execution_count) AS avg_logical_writes,
    (total_physical_reads/execution_count) AS avg_phys_reads,
    execution_count, 
    statement_start_offset as stmt_start_offset, 
    (SELECT SUBSTRING(text, statement_start_offset/2 + 1,
        (CASE WHEN statement_end_offset = -1 
            THEN LEN(CONVERT(nvarchar(MAX),text)) * 2 
                ELSE statement_end_offset 
            END - statement_start_offset)/2)
     FROM sys.dm_exec_sql_text(sql_handle)) AS query_text, 
      (SELECT query_plan from sys.dm_exec_query_plan(plan_handle)) as query_plan
FROM sys.dm_exec_query_stats  
ORDER BY (total_logical_reads + total_logical_writes)/execution_count DESC


You can, of course, change this query to get different views on the data. For example, to generate the five requests that generate the most I/Os in single execution, you can order by:
(total_logical_reads + total_logical_writes)/execution_count
Alternatively, you may want to order by physical I/Os and so on. However, logical read/write numbers are very helpful in determining whether or not the plan chosen by the query is optimal. The output of the query is as follows.


avg_logical_reads    avg_logical_writes   avg_phys_reads       
-----------------    ------------------   ---------------
16639 10   1098
6023 0   0
execution_count      stmt_start_offset
---------------      -----------------
1 0
1 154


Query_text      Query_plan                        
-----------------------------------          -----------
select c1, c5  from t1 INNER HASH JOIN …    
select SUM(c1) from t1                      


The output tells you several important things. First, it identifies the queries that are generating the most I/Os. You can also look at the SQL text to see whether the query needs to be re-examined to reduce I/Os. Verify that the query plan is optimal. For example, a new index might be helpful. Second, the second query in the batch does not incur any physical I/Os because all the pages needed for table t1 are already in the buffer pool. Third, the execution count can be used to identify whether it is a one-off query or the one that is executed frequently and therefore needs to be looked into carefully.
3. Data Compression: Starting with SQL Server 2008, you can use the data compression feature to reduce the size of tables and indexes, thereby reducing the size of the whole database. The compression achieved depends on the schema and the data distribution. Typically, you can achieve 50-60% compression. We have seen up to 90% compression in some cases. What it means to you is that if you are able to compress you active data 50%, you have in effect reduced your I/O requirements by half. Data compression comes at the cost of additional CPU, which needs to be weighed in for your workload. Here are some general strategies.
First, why isn’t compressing the whole database blindly such a good idea? Well, to give you an extreme example, if you have a heavily used table T with 10 pages in a database with millions of pages, there is no benefit in compressing T. Even if SQL Server could compress 10 pages to 1 page, you hardly made a dent in the size of the database, but you did add some CPU overhead instead. In a real-life workload, the choices are not this obvious, but this example shows that you must look before you compress. Our recommendation is this: Before you compress an object (for example, a table index or a partition), look at its size, usage, and estimated compression savings by using the sp_estimate_data_compression_savings stored procedure. 
Let us look at each of these in some detail:
• If the size of the object is much smaller than the overall size of the database, it does not buy you much.
• If the object is used heavily both for DML and SELECT operations, you will incur additional CPU overhead that can impact your workload, especially if it makes it CPU bound. You can use sys.dm_db_index _operational_stats to find the usage pattern of objects to identify which tables, indexes, and partitions are being hit the most.
• The compression savings are schema-dependent and data-dependent, and in fact, for some objects, the size after compression can be larger than before, or the space savings can be insignificant. 
If you have a partitioned table where data in some partitions is accessed infrequently, you may want to compress those partitions and associated indexes with page compression. This is a common scenario with partitioned tables where older partitions are referenced infrequently. For example, you might have a table in which sales data is partitioned by quarters across many years. Commonly the queries are run on the current quarter; data from other quarters is not referenced as frequently. So when the current quarter ends, you can change the compression setting for that quarter’s partition. 
For more information about data compression, see the SQL Server Storage Engine Blog (http://blogs.msdn.com/sqlserverstorageengine/archive/tags/Data+Compression/default.aspx) and SQL Server 2008 Books Online.
4. Upgrading the I/O Subsystem: If you have confirmed that SQL Server is configured correctly and examined the query plans and you are still experiencing I/O bottlenecks, the last option left is to upgrade your I/O subsystem to increase I/O bandwidth:
• Add more physical drives to the current disk arrays and/or replace your current disks with faster drives. This helps to boost both read and write access times. But don't add more drives to the array than your I/O controller can support.
• Add faster or additional I/O controllers. Consider adding more cache (if possible) to your current controllers.