SQL Server has encountered 1 occurrence(s) of I/O requests taking longer tha

 	
SQLServerCentral Register

Home»SQL Server 2014»Administration - SQL Server 2014»SQL Server has encountered 1 occurrence(s) of...
SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds

Post reply Like 315  Add to Briefcase 12Next 
SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds ViewOptions
Author	Message
arrjay	Posted 4 years ago #1633869
arrjay SSCrazy
SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)
Group: General Forum Members
Points: 2319 Visits: 600
Hi there - Last night our application experienced a brief outage due to the following error which I found in the error logs; 

SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [O:\logs\DATABASE.ldf] in database [DATABASE] (12). The OS file handle is 0x0000000000000D58. The offset of the latest long I/O is: 0x00000013f80000

Does anyone have any idea what this means? And what may have caused it? Thanks 

--------------------------------------------

Laughing in the face of contention...
 38
Quote
Jack Corbett	Posted 4 years ago #1633909
  Jack Corbett SSC Guru
SSC Guru (126K reputation)SSC Guru (126K reputation)SSC Guru (126K reputation)SSC Guru (126K reputation)SSC Guru (126K reputation)SSC Guru (126K reputation)SSC Guru (126K reputation)SSC Guru (126K reputation)
Group: General Forum Members
Points: 126284 Visits: 15068
It means you probably have an issue somewhere in your storage subsystem. Either disk issues or you somehow saturated it, which is a problem as well.

You can learn more about it here, http://support.microsoft.com/kb/2137408 


Jack Corbett 
Consultant Straight Path Solutions 
Dont let the good be the enemy of the best. -- Paul Fleming
At best you can say that one job may be more secure than another, but total job security is an illusion. -- Rod at work
Check out these links on how to get faster and more accurate answers: 
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a QuestionHow to Post Performance Problems
Crosstabs and Pivots or How to turn rows into columns Part 1
Crosstabs and Pivots or How to turn rows into columns Part 2
 45
Quote
Summer90	Posted 4 years ago #1634573
Summer90 SSC-Insane
SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)
Group: General Forum Members
Points: 21333 Visits: 4110
Is there any disk scans, defrags, Anti Virus software running? This could cause it as well.

The message means that SQL Server sent a request to your storage for data and it took more than 15 seconds to return the results... it should be less than 15 milliseconds.

I have battled these in the past and it is VERY difficult to determine the cause. I even had Microsoft looking into this and all they came back with is saying it is your storage. For one incident of this we DID find it was the storage. Other times we never determined the cause. 
 29
Quote
Jeff Moden	Posted 4 years ago #1634700
Jeff Moden SSC Guru
SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)
Group: General Forum Members
Points: 627097 Visits: 45308
Part of the reason why this problem is so difficult to nail down is because, in my humble opinion, people don't look in the right spot. Instead, they go through the throws of proving the disk system doesn't actually have a problem and that there's usually nothing (such as defrags, backup exec, etc, etc) that's the problem. That's because the error message is really just a symptom of a more insidious problem than hardware or the normal things that you would expect SQL Server to do.

This isn't a rare problem, either. I've found this problem on virtually every system that I've been assigned to work on and few are willing to fix the real cause because it's "too expensive". So they throw more hardware at it and the problem might go away for awhile but it almost always comes back to haunt you because you haven't fixed the real cause of the problem.

How can you find the real cause?

Open SSMS and, if it isn't already open, open the "explorer" by pressing the {f8} key. Right click on the instance and select {Reports}, {Standard Reports}, {Performance - Top Queries by Total I/O}. That will show you the top 10 problems that are the usual cause of this problem. You need to bite the bullet and fix those problems. Once you fix those, fix the next 10 and then the 10 after that (although the first 10 will usually give you the biggest bang for the buck).

After that, you can probably disconnect the garden hose from the CPU cooling and turn off the industrial fan you're using to cool the SAN and the apps will start running just fine with no more such outages. 

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code: 
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
If you think its expensive to hire a professional to do the job, wait until you hire an amateur. -- Red Adair 

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
 16
Quote
Summer90	Posted 4 years ago #1634704
Summer90 SSC-Insane
SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)SSC-Insane (21K reputation)
Group: General Forum Members
Points: 21333 Visits: 4110
Jeff Moden (11/14/2014)
Part of the reason why this problem is so difficult to nail down is because, in my humble opinion, people don't look in the right spot. Instead, they go through the throws of proving the disk system doesn't actually have a problem and that there's usually nothing (such as defrags, backup exec, etc, etc) that's the problem. That's because the error message is really just a symptom of a more insidious problem than hardware or the normal things that you would expect SQL Server to do.

This isn't a rare problem, either. I've found this problem on virtually every system that I've been assigned to work on and few are willing to fix the real cause because it's "too expensive". So they throw more hardware at it and the problem might go away for awhile but it almost always comes back to haunt you because you haven't fixed the real cause of the problem.

How can you find the real cause?

Open SSMS and, if it isn't already open, open the "explorer" by pressing the {f8} key. Right click on the instance and select {Reports}, {Standard Reports}, {Performance - Top Queries by Total I/O}. That will show you the top 10 problems that are the usual cause of this problem. You need to bite the bullet and fix those problems. Once you fix those, fix the next 10 and then the 10 after that (although the first 10 will usually give you the biggest bang for the buck).

After that, you can probably disconnect the garden hose from the CPU cooling and turn off the industrial fan you're using to cool the SAN and the apps will start running just fine with no more such outages.


The latest one that is causing us to see slow I/Os is only a distribution sql server. I have no control over how Microsoft has setup replication 
 29
Quote
Jeff Moden	Posted 4 years ago #1634717
Jeff Moden SSC Guru
SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)SSC Guru (627K reputation)
Group: General Forum Members
Points: 627097 Visits: 45308
Understood and appreciated. There are sometimes those problems. Just to be sure, though, none of the systems that I've encountered this on had replication or anything similar running. 

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code: 
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
If you think its expensive to hire a professional to do the job, wait until you hire an amateur. -- Red Adair 

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
 32
Quote
Perry Whittle	Posted 4 years ago #1634772
Perry Whittle SSC Guru
SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)
Group: General Forum Members
Points: 170792 Visits: 18516
SQL server doesn't tell porky pies, when you get this message it's because an I\O request has taken longer than 15ms. Common with SAN attached disks is storage saturation, but you wont generally see this at the array, level that may be showing no issues. Don't forget an I\O on a SAN has to traverse all the FC switches to the storage processor and then come back. When the request leaves the SP it could take forever due to network flooding. A FC network is similar to a TCP\IP network in that a large number of requests active on the network can cause response issues. 

One thing to check is the queue depth length on the HBAs, if its too high it could cause issues. 

-----------------------------------------------------------------------------------------------------------

"Ya can't make an omelette without breaking just a few eggs" ;-)
 33
Quote
SQL_Baby	Posted 4 years ago #1635529
SQL_Baby SSChasing Mays
SSChasing Mays (639 reputation)SSChasing Mays (639 reputation)SSChasing Mays (639 reputation)SSChasing Mays (639 reputation)SSChasing Mays (639 reputation)SSChasing Mays (639 reputation)SSChasing Mays (639 reputation)SSChasing Mays (639 reputation)
Group: General Forum Members
Points: 639 Visits: 181
Hi,

What some of the other guys here have mentioned are very valid as it could be so many issues. Here's a few extra things to check.

Is your server a VM? Then check your underlying disk subsystem to see whether any other servers are sharing with it. It could be these are very busy and overloading the underlying disk.

It could be a basic configuration issue. On your first post you mentioned Log file. How big is it? Is there many VLFs? Do all your Data and Log files reside on the same disk? What is the RAID level on the disk? 

At the time of the alerts are there any DBCCs or index rebuilds running? 

Regarding the replicated server, if you could describe the setup maybe I can give you some pointers as to what to check there. i.e is it both Publisher and Distributor. How many Subscribers and are they Pull or Push Subscription? What type of Replication is it? Are you getting the Alerts on the Distribution database or the Published database?

Hope this helps... 
 27
Quote
arrjay	Posted 4 years ago #1636698
arrjay SSCrazy
SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)SSCrazy (2.3K reputation)
Group: General Forum Members
Points: 2319 Visits: 600
Hi there - thanks to everyone for their input on this. 

After numerous calls with our hosting partners I can confirm that it was in fact a problem with our network. 

As our database is highly transactional and does not contain any high I/O bound queries therefore I knew it was an issue with the subsystem. We performed several tests on the disk and found that it was performing as expected. We then monitgored the time it took for information to be pass between SQL and the disk, we then found that the network was faulty and not handling the requests as expected, so we changed the network to a higher tunnel link

Thanks again. 

--------------------------------------------

Laughing in the face of contention...
 37
Quote
Perry Whittle	Posted 4 years ago #1636726
Perry Whittle SSC Guru
SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)SSC Guru (170K reputation)
Group: General Forum Members
Points: 170792 Visits: 18516
arrjay (11/20/2014)
We performed several tests on the disk and found that it was performing as expected. We then monitgored the time it took for information to be pass between SQL and the disk, we then found that the network was faulty and not handling the requests as expected, so we changed the network to a higher tunnel link

Classic case scenario, thanks for posting back 

-----------------------------------------------------------------------------------------------------------

"Ya can't make an omelette without breaking just a few eggs" ;-)
 16
Quote
  Go

Post reply Like 315  Add to Briefcase 12Next 

Permissions
You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.	You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Powered by InstantForum 2016-2 Final © 2018
Execution: 0.000. 2 queries. Compression enabled. SQLServerCentral
Register Search

Search discussions

 
 	
Copyright © 2002-2018 Redgate. All Rights Reserved. Privacy Policy. Terms of Use. Report Abuse.