SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


123»»»

Application Slow Expand / Collapse
Author
Message
Posted Saturday, March 08, 2008 8:52 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Wednesday, September 01, 2010 4:59 AM
Points: 20, Visits: 173
Hi,

I have an application in use at a customer site that uses SQL Server 2000 and the customer is complaining about slow performance. The normal issues I receive is that a particular part of the application is running slow. So I request a profiler trace and identify the slow performing query and resolve the problem. This customer says its all slow. I have requested Profiler Traces and examined them and at times I can see queries that normally run in less than 1 second taking 20-30 seconds to complete.

I have also asked the customer for perfmon logs and examined these. The logs show that the database disk (not the log disk) has sometimes queues of 100's of I/O's. The Avg. Disk Sec/Reads show response times for the disk varying from less than 20ms upto 2 seconds.

I believe the issue is with the poor I/O performance, but the customer says its the application.

can anyone advise on how to proceed with an issue like this.

Ronnie
Post #466275
Posted Saturday, March 08, 2008 9:05 AM


SSCrazy Eights

SSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy Eights

Group: General Forum Members
Last Login: Yesterday @ 9:13 PM
Points: 8,605, Visits: 8,787
1. Compile the data to show that there is a problem with disk access.
2. How much memory is on the server? What is the Buffer Cache Hit Ratio? as it seems the server is going to disk too often.

Here is a good article on what to look for and contains some things you do not mention:

http://www.devarticles.com/c/a/SQL-Server/How-to-Perform-a-SQL-Server-Performance-Audit/

Also look at locks and blocks. It is possible that a simple thing like the order of table access in the queries is not consistent.




Jack Corbett

Applications Developer

Don't let the good be the enemy of the best. -- Paul Fleming

Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question
How to Post Performance Problems
Crosstabs and Pivots or How to turn rows into columns Part 1
Crosstabs and Pivots or How to turn rows into columns Part 2
Post #466278
Posted Saturday, March 08, 2008 9:35 AM


SSC-Insane

SSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-Insane

Group: General Forum Members
Last Login: Yesterday @ 10:10 PM
Points: 22,513, Visits: 17,068
I believe the issue is with the poor I/O performance,


I agree with Jack, but I'll say it a differnet way... WHY do you believe that? What proof do you have? In order to convince the customer, you will need to show a comparison between systems under similar load or come up with an irrefutable proof that that is the problem. Until then, you'll be going around in circles with the customer forever.


--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

For better, quicker answers on T-SQL questions, click on the following...
http://www.sqlservercentral.com/Forums/Attachment6921.aspx

For better answers on performance questions, click on the following...
http://www.sqlservercentral.com/articles/SQLServerCentral/66909/
Post #466284
Posted Saturday, March 08, 2008 1:01 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Wednesday, September 01, 2010 4:59 AM
Points: 20, Visits: 173
The profiler trace I have, covering a 90 minute period, shows two queries that on two ocassions took 20+ seconds to run, at separate times. This was out of 250+ runnings of the same queries, the rest of the time the queries completed in less than 2 seconds, returning between 900-1200 rows. The cpu used by the queries is nearly always the same and so are the reads.

How do I find out why the queries have slowed down so dramtically.

I have this application on many other sites and have no problem. The only difference with this customer is the application is on a Cluster with a SAN.

Ronnie
Post #466312
Posted Saturday, March 08, 2008 4:10 PM


SSCrazy Eights

SSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy Eights

Group: General Forum Members
Last Login: Wednesday, September 01, 2010 4:23 PM
Points: 9,341, Visits: 8,799
Although there are a huge number of possible reasons for this behavior, in my experience it is almost always due to one of two probable influences:

1) Irregularities in the optimizer/statistics facilities. SQL Server 2000 had a number of issues in this area which you can research at Microsoft, however, it is usually easy enough to tell if this may be the cause. Compare the query plan of 20 sec responses to the 2 sec instances. If they are significantly different then that is your likely cause. The easiest Quick & Dirty short-term fix is to add query hints to force them to choose the plan you know they should use.

However, the fact that CPU & Read IO's are the same is an indication that this is probably not what is happening. Which leaves you with...

2) Contention. Contention is a broad category that can take many forms, both obvious and subtle. The high queue size on your data disk makes it likely that you are experiencing physical contention for that disk (and not contention for other resources, such as CPU and not logical application contention such as Locking).

At this point, the possible sources of contention break down as follows:

A) Application Internal: In other words, you have too many users trying to do the same thing at the same time. That disk has become the bottleneck that everyone is backing up behind. You need to either put less demand on that disk or give that disk more IO capacity (ie., optimize the App, or the queries/sProcs, tune the tables/indexes, move the tables to less used disks, OR reconfigure the disk (see below)).

B) SQL Server internal: other usages of the SQL server are putting load on that disk that are contending with the app's usage of it. You need to eliminate, reduce or redirect that other SQL usage.

C) Server Internal: non-SQL usages of that disk, but from the same server, are competing for that disk. You need to eliminate, reduce or redirect that other usage.

D) SAN-concealed: SAN's are commonly carved up into separate Logical volumes that are nonetheless competing for the same physical resources. So for instance, your Log disk and your Data disk may actually be sharing the same physical devices (this is a BAD thing, but much more common than you might think). More likely, some completely unrelated server(s) have heavily-used logical volumes that are using the same physical drives as your data disk. Solution: reconfigure your data disk on the SAN.


-- RBarryYoung, (302)375-0451 blog: MovingSQL.com, Twitter: @RBarryYoung
Proactive Performance Solutions, Inc.
"Performance is our middle name."
Post #466321
Posted Wednesday, March 12, 2008 1:38 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Wednesday, September 01, 2010 4:59 AM
Points: 20, Visits: 173
All,

I have had another look through the profiler trace I have to try and get to the bottom of this issue. The worst performing query took 41 seconds to return its data using 0.2 seconds of CPU time. Now my initial theory was that it was an I/O problem, I am not so sure now. During the profile session the query ran 200 plus times with and average run time of 0.5 seconds.

During the 40 seconds that the query took other queries where running and completing in less that 1 second. In fact another user ran the same query with different selection criteria and it returned in less than 1 second. If my I/O theory was right I would have expected other queries to slow down during the 40 seconds period.

When I run the query on the customer site it returns in 1 second or less.

Help

Ronnie
Post #467933
Posted Wednesday, March 12, 2008 2:02 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, September 03, 2010 1:35 AM
Points: 2,127, Visits: 572
Ronnie.Doggart (3/12/2008)
All,

I have had another look through the profiler trace I have to try and get to the bottom of this issue. The worst performing query took 41 seconds to return its data using 0.2 seconds of CPU time. Now my initial theory was that it was an I/O problem, I am not so sure now. During the profile session the query ran 200 plus times with and average run time of 0.5 seconds.

During the 40 seconds that the query took other queries where running and completing in less that 1 second. In fact another user ran the same query with different selection criteria and it returned in less than 1 second. If my I/O theory was right I would have expected other queries to slow down during the 40 seconds period.

When I run the query on the customer site it returns in 1 second or less.

Help

Ronnie


Hi Ronnie,

Ok, let's look at this from a slightly different perspective. Its sounds like you have identified the "slow" query, which runs fine on other sites, but on this particular site runs "slow". Only difference between this site and other sites is that this site has a SAN? Have you isolated the offending query or queries and unit test each one in terms of system resource usage? If not, I would suggest you look into doing this, as this will allow you to understand query behavior. You will need to review the execution plan between where the query runs in 40 seconds vs where the query runs in 1 second. Have you checked for table or index scans? Has the query always run slow on this particular site? Have you checked for blocking during query execution? Have you look at the MAXDOP setting and compared between system(s)?

Thanks,
Phillip Cox
MCITP - DBAdmin
Post #467938
Posted Wednesday, March 12, 2008 4:06 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Wednesday, September 01, 2010 4:59 AM
Points: 20, Visits: 173
All,

I have noticed that on the customer site the following query uses different plans for different data. So as an example:

Table has two indexes:

index1 is on column x
index2 is on columns x and y

select * from table1 where x = 'xyz' and y='ab'

produces a plan that uses index1 and does more reads

select * from table1 where x = 'def' and y='st'

produces a plan that uses index2 and does fewer reads

Why does it not use index2 everytime as its columns are used in the 'where' clause of the select.

Ronnie



Post #467978
Posted Wednesday, March 12, 2008 4:48 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, September 03, 2010 1:35 AM
Points: 2,127, Visits: 572
Ronnie.Doggart (3/12/2008)
All,

I have noticed that on the customer site the following query uses different plans for different data. So as an example:

Table has two indexes:

index1 is on column x
index2 is on columns x and y

select * from table1 where x = 'xyz' and y='ab'

produces a plan that uses index1 and does more reads

select * from table1 where x = 'def' and y='st'

produces a plan that uses index2 and does fewer reads

Why does it not use index2 everytime as its columns are used in the 'where' clause of the select.

Ronnie





Hi Ronnie,

How often are statistics updated? How often are indexes rebuilt or re-organized?
Index2 includes both columns in SARG (e.g. Where clause), thus would be faster in terms of identifying matching rows, as less lookups are required. In comparision, index1 would force a lookup of x first, then lookup of y, so more reads.

Can you past execution plan from both queries?

Thanks,
Phillip Cox
Post #467993
Posted Wednesday, March 12, 2008 5:44 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Wednesday, September 01, 2010 4:59 AM
Points: 20, Visits: 173
All,

Info lgncc_enquiry has 434523 rows and had statistics collected yesterday.

SELECT * FROM LGNCC_ENQUIRY WHERE xref1 = '1001183168' AND objecttype = 'D4'

(971 row(s) affected)

Table 'LGNCC_ENQUIRY'. Scan count 1, logical reads 3060, physical reads 43, read-ahead reads 0.


|--Filter(WHERE[LGNCC_ENQUIRY].[ObjectType]='D4'))
|--Bookmark Lookup(BOOKMARK[Bmk1000]), OBJECT[LGNCC_ENQUIRY]))
|--Index Seek(OBJECT[LGNCC_ENQUIRY].[LGNCC_ENQUIRY_XREF1]), SEEK[LGNCC_ENQUIRY].[xref1]='1001183168') ORDERED FORWARD)



SELECT * FROM LGNCC_ENQUIRY WHERE xref1 = '101000599728' AND objecttype = 'C1'

(1 row(s) affected)

Table 'LGNCC_ENQUIRY'. Scan count 1, logical reads 6, physical reads 0, read-ahead reads 0.


|--Bookmark Lookup(BOOKMARK[Bmk1000]), OBJECT[LGNCC_ENQUIRY]))
|--Index Seek(OBJECT[LGNCC_ENQUIRY].[lgncc_enquiry_t2]), SEEK[LGNCC_ENQUIRY].[xref1]='101000607124' AND [LGNCC_ENQUIRY].[ObjectType]='C1') ORDERED FORWARD)







Ronnie
Post #468015
« Prev Topic | Next Topic »

123»»»

Permissions Expand / Collapse