SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


SQL Server Performance Problems


SQL Server Performance Problems

Author
Message
kiffab
kiffab
SSC-Enthusiastic
SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)

Group: General Forum Members
Points: 154 Visits: 46
Hi Guys

Occasionally at peak times of the day I get I/O errors on my SQL server (v 2005, 16 core 2.4 ghz, 58 GB ram).

These errors read "SQL server has encountered x occurences of i/o requests taking longer than 15 seconds to complete on ....". The drive is always the data drive but multiple database are affected.

Various indexes and tweaks were put in place over the last year to stabilise things but it's starting to show signs of performance problems again. I've made all the usual changes like splitting data/logs, temp db on own drive, etc. The biggest DB is around 1 TB.

I have installed SQL server 2005 performance dashboard.

What's the best way to use it?

For expensive queries, what am I most interested in from CPU, duration, logical reads, physical reads, logical writes, CLR time? I can feed this back to suppliers and have them tweak their SQL if necessary. I also see various indexes that it suggests I should add.

What else can I check?

Thanks.
Steve Jones
Steve Jones
SSC Guru
SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)

Group: Administrators
Points: 150578 Visits: 19455
This is really an I/O subsystem issue, and less a query tuning issue. The load from your server exceeds what the I/O system can handle.

http://mssqlwiki.com/2012/08/27/io-requests-taking-longer-than-15-seconds-to-complete-on-file/

Tuning queries to use less reads can help, and indexing better might help, but this is a bit of an art. Really you need more/faster disks in the short term.

In looking at queries, I'd start with a workload trace, then feed that to the DTA and get an idea of what indexes it might recommend.

You can also take your trace and look for those queries with lots of reads in them (or writes) and see if there is something you can do to reduce the reads. Indexing helps here, but potentially rewriting SQL to work on smaller sets of data can help.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
kiffab
kiffab
SSC-Enthusiastic
SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)

Group: General Forum Members
Points: 154 Visits: 46
Hi Steve

Thanks for the prompt response. I will read through the article attached.

Given that I see nothing but expansion in the future, it sounds like I can tweak queries and index as much as I like but eventually this problem will surface again. In fact I'd say it is now as this problem had disappeared for a period of time as I changed the block write size to 64k from 8 or whatever the default was.

The cluster nodes appear to have plenty of free CPU and mem but I guess that doesn't rule out the SAN.

I need more evidence in order to request better kit so I will follow the steps in the link and may come back with a question or 2!

Thanks.
Steve Jones
Steve Jones
SSC Guru
SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)SSC Guru (150K reputation)

Group: Administrators
Points: 150578 Visits: 19455
One thing you might examine is your waits for the various files. If you see waits stacking up, then you have IO issues.

A few more references for you:

http://www.simple-talk.com/sql/performance/sql-server-wait-events-taking-the-guesswork-out-of-performance-profiling/
http://www.simple-talk.com/sql/performance/a-performance-troubleshooting-methodology-for-sql-server/
http://www.sqlskills.com/blogs/paul/post/capturing-wait-stats-for-a-single-operation.aspx

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
kiffab
kiffab
SSC-Enthusiastic
SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)

Group: General Forum Members
Points: 154 Visits: 46
Regarding the waits, SQL server performance dashboard shows below (taken at 11:20 today)

Wait Category..................Number of Waits..........Wait Time (sec)..........% Wait Time
Parallelism...........................724065872.................6319246.421............... 28.06%
Other ................................1232197629................5075559.493...............22.54%
Sleep.................................425224734..................4091093.732...............18.17%
Latch................................1642991523.................3378139.014...............15.00%
Buffer IO............................148173427...................2212876.593..............9.83%

I am not sure whether that seems good bad or normal! Apologies for the dots....
kiffab
kiffab
SSC-Enthusiastic
SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)

Group: General Forum Members
Points: 154 Visits: 46
The top wait on live always seems to be this:

wait_type wait_time_ms
MSQL_XP 26126343

I believe this is due to SQL waiting on extended stored procs to complete yet I can't find a process relating to this. I use RedGate for SQL backup and the general feedback online is that an update from v5 is required.
Jon.Morisi
Jon.Morisi
Hall of Fame
Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)

Group: General Forum Members
Points: 3125 Visits: 1142
It looks like you have got a lot of things going on performance wise. CXPACKET waits (parallelism) should be addressed if > 5%. The IO waiting more that 15 sec indicates an IO subsystem bottleneck and the IO related waits seem to support this. However, that doesn't necessarily mean you need a faster IO subsystem, it could be a number of things including bad indexing, bad maintenance, or a big ad hoc query that needs tuning. Have you run any Performance Monitor Counter logs?

If I were in your shoes I'd start with addressing the CXPACKET waits by modifying Cost threshold for parallelism and Max Degree of Parallelism because that's a relatively easy one. Next I'd check index fragmentation and the last time statistics were updated. If there's no routine maintenance occurring, put it in place.

Next I'd run a PAL analysis, and go from there:
http://pal.codeplex.com/

HTH,
Jon
kiffab
kiffab
SSC-Enthusiastic
SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)SSC-Enthusiastic (154 reputation)

Group: General Forum Members
Points: 154 Visits: 46
Thanks Jon. I'll read up on this.

FYI -
max degree of parallelism = 0
Locks = 0
Cost = 5
Query waits = -1

I'm guessing this is the out the box setup as I've never changed it. Does this ring alarm bells or seem perfectly reasonable?

I have 16 processors @ 2.4 GHZ. 58 GB memory with 48 GB allocated to SQL.

Cheers
GilaMonster
GilaMonster
SSC Guru
SSC Guru (232K reputation)SSC Guru (232K reputation)SSC Guru (232K reputation)SSC Guru (232K reputation)SSC Guru (232K reputation)SSC Guru (232K reputation)SSC Guru (232K reputation)SSC Guru (232K reputation)

Group: General Forum Members
Points: 232747 Visits: 46361
Jon.Morisi (12/19/2012)
If I were in your shoes I'd start with addressing the CXPACKET waits by modifying Cost threshold for parallelism and Max Degree of Parallelism because that's a relatively easy one.


Fixing excessive CXPacket waits isn't as simple as that (sure, maxdop 1 will make them go away entirely, but...). Requires investigating whether the CX packet waits are a problem (any time a query runs in parallel, you will get CXPacket waits), what, if anything, are the threads that aren't waiting on CXPacket waiting for, whether there's data skew, etc

Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass


Adam Machanic
Adam Machanic
Hall of Fame
Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)

Group: General Forum Members
Points: 3055 Visits: 714
Jon.Morisi (12/19/2012)
CXPACKET waits (parallelism) should be addressed if > 5%.


That's a rather specific pronouncement, especially when you don't know anything about the workload in question. What's your logic there?

--
Adam Machanic
SQL Server MVP
SQLblog.com: THE SQL Server Blog Spot on the Web
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search