Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 123»»»

SQL Server Performance Problems Expand / Collapse
Author
Message
Posted Tuesday, December 18, 2012 10:34 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, January 31, 2013 2:18 AM
Points: 8, Visits: 46
Hi Guys

Occasionally at peak times of the day I get I/O errors on my SQL server (v 2005, 16 core 2.4 ghz, 58 GB ram).

These errors read "SQL server has encountered x occurences of i/o requests taking longer than 15 seconds to complete on ....". The drive is always the data drive but multiple database are affected.

Various indexes and tweaks were put in place over the last year to stabilise things but it's starting to show signs of performance problems again. I've made all the usual changes like splitting data/logs, temp db on own drive, etc. The biggest DB is around 1 TB.

I have installed SQL server 2005 performance dashboard.

What's the best way to use it?

For expensive queries, what am I most interested in from CPU, duration, logical reads, physical reads, logical writes, CLR time? I can feed this back to suppliers and have them tweak their SQL if necessary. I also see various indexes that it suggests I should add.

What else can I check?

Thanks.


Post #1397920
Posted Tuesday, December 18, 2012 10:56 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Wednesday, October 22, 2014 12:34 PM
Points: 31,181, Visits: 15,626
This is really an I/O subsystem issue, and less a query tuning issue. The load from your server exceeds what the I/O system can handle.

http://mssqlwiki.com/2012/08/27/io-requests-taking-longer-than-15-seconds-to-complete-on-file/

Tuning queries to use less reads can help, and indexing better might help, but this is a bit of an art. Really you need more/faster disks in the short term.

In looking at queries, I'd start with a workload trace, then feed that to the DTA and get an idea of what indexes it might recommend.

You can also take your trace and look for those queries with lots of reads in them (or writes) and see if there is something you can do to reduce the reads. Indexing helps here, but potentially rewriting SQL to work on smaller sets of data can help.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1397933
Posted Tuesday, December 18, 2012 11:12 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, January 31, 2013 2:18 AM
Points: 8, Visits: 46
Hi Steve

Thanks for the prompt response. I will read through the article attached.

Given that I see nothing but expansion in the future, it sounds like I can tweak queries and index as much as I like but eventually this problem will surface again. In fact I'd say it is now as this problem had disappeared for a period of time as I changed the block write size to 64k from 8 or whatever the default was.

The cluster nodes appear to have plenty of free CPU and mem but I guess that doesn't rule out the SAN.

I need more evidence in order to request better kit so I will follow the steps in the link and may come back with a question or 2!

Thanks.
Post #1397943
Posted Tuesday, December 18, 2012 11:29 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Wednesday, October 22, 2014 12:34 PM
Points: 31,181, Visits: 15,626
One thing you might examine is your waits for the various files. If you see waits stacking up, then you have IO issues.

A few more references for you:

http://www.simple-talk.com/sql/performance/sql-server-wait-events-taking-the-guesswork-out-of-performance-profiling/
http://www.simple-talk.com/sql/performance/a-performance-troubleshooting-methodology-for-sql-server/
http://www.sqlskills.com/blogs/paul/post/capturing-wait-stats-for-a-single-operation.aspx







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1397957
Posted Wednesday, December 19, 2012 4:37 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, January 31, 2013 2:18 AM
Points: 8, Visits: 46
Regarding the waits, SQL server performance dashboard shows below (taken at 11:20 today)

Wait Category..................Number of Waits..........Wait Time (sec)..........% Wait Time
Parallelism...........................724065872.................6319246.421............... 28.06%
Other ................................1232197629................5075559.493...............22.54%
Sleep.................................425224734..................4091093.732...............18.17%
Latch................................1642991523.................3378139.014...............15.00%
Buffer IO............................148173427...................2212876.593..............9.83%

I am not sure whether that seems good bad or normal! Apologies for the dots....

Post #1398302
Posted Wednesday, December 19, 2012 9:59 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, January 31, 2013 2:18 AM
Points: 8, Visits: 46
The top wait on live always seems to be this:

wait_type wait_time_ms
MSQL_XP 26126343

I believe this is due to SQL waiting on extended stored procs to complete yet I can't find a process relating to this. I use RedGate for SQL backup and the general feedback online is that an update from v5 is required.
Post #1398523
Posted Wednesday, December 19, 2012 11:39 AM


SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Sunday, August 10, 2014 4:15 AM
Points: 293, Visits: 816
It looks like you have got a lot of things going on performance wise. CXPACKET waits (parallelism) should be addressed if > 5%. The IO waiting more that 15 sec indicates an IO subsystem bottleneck and the IO related waits seem to support this. However, that doesn't necessarily mean you need a faster IO subsystem, it could be a number of things including bad indexing, bad maintenance, or a big ad hoc query that needs tuning. Have you run any Performance Monitor Counter logs?

If I were in your shoes I'd start with addressing the CXPACKET waits by modifying Cost threshold for parallelism and Max Degree of Parallelism because that's a relatively easy one. Next I'd check index fragmentation and the last time statistics were updated. If there's no routine maintenance occurring, put it in place.

Next I'd run a PAL analysis, and go from there:
http://pal.codeplex.com/

HTH,
Jon
Post #1398588
Posted Wednesday, December 19, 2012 12:40 PM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, January 31, 2013 2:18 AM
Points: 8, Visits: 46
Thanks Jon. I'll read up on this.

FYI -
max degree of parallelism = 0
Locks = 0
Cost = 5
Query waits = -1

I'm guessing this is the out the box setup as I've never changed it. Does this ring alarm bells or seem perfectly reasonable?

I have 16 processors @ 2.4 GHZ. 58 GB memory with 48 GB allocated to SQL.

Cheers
Post #1398623
Posted Wednesday, December 19, 2012 12:51 PM


SSC-Forever

SSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-Forever

Group: General Forum Members
Last Login: Today @ 5:50 AM
Points: 40,209, Visits: 36,618
Jon.Morisi (12/19/2012)
If I were in your shoes I'd start with addressing the CXPACKET waits by modifying Cost threshold for parallelism and Max Degree of Parallelism because that's a relatively easy one.


Fixing excessive CXPacket waits isn't as simple as that (sure, maxdop 1 will make them go away entirely, but...). Requires investigating whether the CX packet waits are a problem (any time a query runs in parallel, you will get CXPacket waits), what, if anything, are the threads that aren't waiting on CXPacket waiting for, whether there's data skew, etc



Gail Shaw
Microsoft Certified Master: SQL Server 2008, MVP
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass

Post #1398636
Posted Wednesday, December 19, 2012 1:01 PM


Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Thursday, September 25, 2014 9:01 PM
Points: 1,113, Visits: 705
Jon.Morisi (12/19/2012)
CXPACKET waits (parallelism) should be addressed if > 5%.


That's a rather specific pronouncement, especially when you don't know anything about the workload in question. What's your logic there?



--
Adam Machanic
SQL Server MVP
SQLblog.com: THE SQL Server Blog Spot on the Web
Post #1398640
« Prev Topic | Next Topic »

Add to briefcase 123»»»

Permissions Expand / Collapse