Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

SQL Server has encountered occurrence(s) of I/O requests taking longer than 15 seconds to complete Expand / Collapse
Author
Message
Posted Sunday, September 30, 2012 3:47 PM


Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Thursday, April 17, 2014 8:43 AM
Points: 1,201, Visits: 2,661
I have a few SQL Server 2008R2 SP1 and SP2 as well as a couple SQL Server 2008 SP3 getting this error when I run Update Stats or DBCC CHECKDB. This started a few weeks ago and is getting worse and worse. Before the issue started we did not have any large dbs on this one specific SQL Server and now that we do we are getting these. I know in my research that this can be a very difficult issue to pinpoint what is causing this. I know it isn't AV as the SQL Server files are excluded and we also stopped the service on two of the servers. Anyone ever dealt with this issue? I am going to open a case with Microsoft in the morning.

SQL Server has encountered 22953 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [H:\MSSQL10_50.MSSQLSERVER\MSSQL\Data\xxxxxxxxx.MDF] in database [CORETABLES_PROD] (6). The OS file handle is 0x0000000000000DE4. The offset of the latest long I/O is: 0x000000d152a000



Post #1366297
Posted Sunday, September 30, 2012 9:38 PM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Tuesday, April 01, 2014 3:26 PM
Points: 316, Visits: 1,497
checkdb is very IO intensive, and your IO subsystem isn't keeping up.

What are you using for storage?
Post #1366309
Posted Sunday, September 30, 2012 9:45 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Monday, February 18, 2013 1:47 AM
Points: 196, Visits: 87
SQL Server has encountered 22953 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [H:\MSSQL10_50.MSSQLSERVER\MSSQL\Data\xxxxxxxxx.MDF] in database [CORETABLES_PROD] (6). The OS file handle is 0x0000000000000DE4. The offset of the latest long I/O is: 0x000000d152a000



22953 code is for "Could not remove Change Data Capture metadata for capture instance ‘%s’ and source table ‘%s.%s’. Refer to previous errors in the current session to identify the cause and correct any associated problems."

for more detail please check below mentioned url:

http://sqlserverlearner.com/sql-server-error-list/sql-server-error-messages-list-22000-to-39000

Regards,
Post #1366310
Posted Sunday, September 30, 2012 9:48 PM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Tuesday, April 01, 2014 3:26 PM
Points: 316, Visits: 1,497
vivekkumar341 (9/30/2012)
SQL Server has encountered 22953 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [H:\MSSQL10_50.MSSQLSERVER\MSSQL\Data\xxxxxxxxx.MDF] in database [CORETABLES_PROD] (6). The OS file handle is 0x0000000000000DE4. The offset of the latest long I/O is: 0x000000d152a000



22953 code is for "Could not remove Change Data Capture metadata for capture instance ‘%s’ and source table ‘%s.%s’. Refer to previous errors in the current session to identify the cause and correct any associated problems."

for more detail please check below mentioned url:

http://sqlserverlearner.com/sql-server-error-list/sql-server-error-messages-list-22000-to-39000

Regards,


22953 isn't the error code, it's the number of times the error occured...
Post #1366311
Posted Sunday, September 30, 2012 9:51 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Monday, February 18, 2013 1:47 AM
Points: 196, Visits: 87
Please go through the url and my mistake its error code.
Post #1366312
Posted Sunday, September 30, 2012 10:09 PM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Tuesday, April 01, 2014 3:26 PM
Points: 316, Visits: 1,497
vivekkumar341 (9/30/2012)
Please go through the url and my mistake its error code.


No thanks, I prefer this url: http://support.microsoft.com/kb/2137408
SQL Server logs "Msg 833" when I/O delay problems occur
Post #1366313
Posted Monday, October 01, 2012 3:22 AM
SSC Journeyman

SSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC Journeyman

Group: General Forum Members
Last Login: Wednesday, April 09, 2014 6:57 PM
Points: 81, Visits: 333
I also had the same issue.
It was due to the storage which was used for replication. During the storage replication
we used to get lots I/O messages,even sql services went down.

My windows and storage team had resolved this issue somehow.
Post #1366414
Posted Monday, October 01, 2012 5:10 AM


Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Thursday, April 17, 2014 8:43 AM
Points: 1,201, Visits: 2,661
Thanks for the link SpringTown... I hadn't found that article. I have opened an incident with MSFT and have yet another meeting with the storage gang today. The pressure is on ME to find what is causing this since users come to me complaining about performance issues.


Post #1366485
Posted Monday, October 01, 2012 8:18 AM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Tuesday, April 01, 2014 3:26 PM
Points: 316, Visits: 1,497
Markus (10/1/2012)
Thanks for the link SpringTown... I hadn't found that article. I have opened an incident with MSFT and have yet another meeting with the storage gang today. The pressure is on ME to find what is causing this since users come to me complaining about performance issues.


Ah, the joys of dealing with storage people....

I highly recommend Brent Ozar's webcast: "How to Prove It's a SAN Problem" http://www.brentozar.com/archive/2011/08/how-prove-its-san-problem-webcast-video/

Also, I'd recommend capturing some windows perfmon counters while the problem is occuring.

This is my list for troubleshooting IO issues:

(N: and P: are the SAN volumes that host DB files)
\LogicalDisk(N:)\*
\LogicalDisk(P:)\*
\SQLServer:Buffer Manager\Buffer cache hit ratio
\SQLServer:Buffer Manager\Checkpoint pages/sec
\SQLServer:Buffer Manager\Database pages
\SQLServer:Buffer Manager\Free list stalls/sec
\SQLServer:Buffer Manager\Free pages
\SQLServer:Buffer Manager\Lazy writes/sec
\SQLServer:Buffer Manager\Page life expectancy
\SQLServer:Buffer Manager\Page lookups/sec
\SQLServer:Buffer Manager\Page reads/sec
\SQLServer:Buffer Manager\Page writes/sec
\SQLServer:Buffer Manager\Readahead pages/sec
\SQLServer:Buffer Manager\Reserved pages
\SQLServer:Buffer Manager\Stolen pages
\SQLServer:Buffer Manager\Target pages
\SQLServer:Buffer Manager\Total pages
\Memory\*

Capture at 1 second intervals, (capture only while needed).

Major things to look for, Disk Reads/Sec, Disk Writes/Sec (IOPS) and how that affects latency ( Avg Sec/Write, Avg Sec/Read )
And, what is driving those IOPS (Lazy writes, checkpoints, etc)



Post #1366594
Posted Tuesday, October 02, 2012 10:48 AM


SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Today @ 8:32 AM
Points: 4,128, Visits: 5,839
Markus (10/1/2012)
Thanks for the link SpringTown... I hadn't found that article. I have opened an incident with MSFT and have yet another meeting with the storage gang today. The pressure is on ME to find what is causing this since users come to me complaining about performance issues.


There are a BUNCH of things that can cause IO problems. A ticket with MSFT on this is a non-starter. The problem is your system - period. Hire a good performance consultant and let them efficiently find/fix issues (there are ALWAYS a bunch of low-hanging-fruit issues whenever I do a first review at a new client) and mentor you along the way. Just a few hours or days will likely give you a system that is WAY different from what you are experiencing now!



Best,

Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru at GMail
Post #1367174
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse