Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 12»»

Dataware Performance Degradation - Urgent Situation. Expand / Collapse
Author
Message
Posted Wednesday, April 02, 2008 3:14 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, May 16, 2013 3:33 AM
Points: 18, Visits: 299
Thanks in advance !

We are having a situation (PROBLEM) with our DW - SQL SERVER 2000 SP4 ENT X86, it suddenly slowed a lot.

CPU Usage : 20%

IO : AVR DISK QUEUE LENGHT Between 1.5 and 2 (rarelly spikes (20/40)).

MEM : 8 GB, 6GB to SQL SERVER.

All the code that is running against the DB is fined tuned ( Most tables are Index Seeked with Clustered Indexes ), biggest table 450 GB.

No Lock Waits ( Except the main process(spid) that is showed has locked by it's own spid ( latch_xx (sh,ix)).

I guess that the problem is related to storage, because it is only doing 1-3MB sec (sometimes 20MB/sec).

PerfMon ( PROCESS SQLSERVER ( DISK Read Bytes ( between 1-3MB sometimes 20 MB/s), Disk Bytes Write Bytes/Sec same as Reads).

No more sql server instances or other software hogging the server.

I can't understand why the SQL SERVER is never using the IO it is used to use (25MB/S).

I don't see any DISK QUEUE on the Partitions that are used to store the datafiles/logfiles)

No Pages splits in the SQL SERVER ACCESS METHODS.

Sometimes I get about 1 GB in dirty buffers.

98 % Cache Hit Ratio.

ALL points to IO contention, but the counters from windows don't indicate this.

All help is REALLY appreciated.

Best Regards.



Post #478856
Posted Wednesday, April 02, 2008 3:36 PM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, October 12, 2012 7:55 AM
Points: 2,133, Visits: 604
Netic (4/2/2008)
Thanks in advance !

We are having a situation (PROBLEM) with our DW - SQL SERVER 2000 SP4 ENT X86, it suddenly slowed a lot.

CPU Usage : 20%

IO : AVR DISK QUEUE LENGHT Between 1.5 and 2 (rarelly spikes (20/40)).

MEM : 8 GB, 6GB to SQL SERVER.

All the code that is running against the DB is fined tuned ( Most tables are Index Seeked with Clustered Indexes ), biggest table 450 GB.

No Lock Waits ( Except the main process(spid) that is showed has locked by it's own spid ( latch_xx (sh,ix)).

I guess that the problem is related to storage, because it is only doing 1-3MB sec (sometimes 20MB/sec).

PerfMon ( PROCESS SQLSERVER ( DISK Read Bytes ( between 1-3MB sometimes 20 MB/s), Disk Bytes Write Bytes/Sec same as Reads).

No more sql server instances or other software hogging the server.

I can't understand why the SQL SERVER is never using the IO it is used to use (25MB/S).

I don't see any DISK QUEUE on the Partitions that are used to store the datafiles/logfiles)

No Pages splits in the SQL SERVER ACCESS METHODS.

Sometimes I get about 1 GB in dirty buffers.

98 % Cache Hit Ratio.

ALL points to IO contention, but the counters from windows don't indicate this.

All help is REALLY appreciated.

Best Regards.





Hi,

How full are your drives supporting database files? Has anything changed recently? Have you monitored locking? What does memory pressure look like? What does storage system look like (e.g. DAS, Local,NAS or SAN)? Any Anti-virus software installed on server? If so, have SQL Server files been excluded?

Thanks,
Phillip Cox
Post #478866
Posted Wednesday, April 02, 2008 3:43 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, May 16, 2013 3:33 AM
Points: 18, Visits: 299
It's an EMC SAN.

Thanks !
Post #478871
Posted Wednesday, April 02, 2008 3:47 PM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, October 12, 2012 7:55 AM
Points: 2,133, Visits: 604
Any changes to HBA card(s) on server (e.g. driver update)?
What is slow performance based on (e.g. reports taking longer)? How often do you update warehouse?

Thanks,
Phillip
Post #478873
Posted Wednesday, April 02, 2008 3:54 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, May 16, 2013 3:33 AM
Points: 18, Visits: 299
Daily,
Same data volume, but taking 4x the normal time.
I don't have console access to the server...
Complicated situation to expose here...
Need more help please :|

Thanks !
Post #478877
Posted Wednesday, April 02, 2008 3:55 PM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, October 12, 2012 7:55 AM
Points: 2,133, Visits: 604
Netic (4/2/2008)
Daily,
Same data volume, but taking 4x the normal time.
I don't have console access to the server...
Complicated situation to expose here...
Need more help please :|

Thanks !


What is taking 4x longer? Does the SQL Server service have "lock pages in memory" enabled?
Post #478878
Posted Wednesday, April 02, 2008 3:57 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, May 16, 2013 3:33 AM
Points: 18, Visits: 299
The Inserts..
Post #478879
Posted Wednesday, April 02, 2008 3:57 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, May 16, 2013 3:33 AM
Points: 18, Visits: 299
I can't understand why the server is using just 20-30% of Disk time...
Post #478880
Posted Wednesday, April 02, 2008 3:59 PM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, October 12, 2012 7:55 AM
Points: 2,133, Visits: 604
Ok.

Do you drop indexes prior to loading? What recovery mode are you using for target database? Have you checked transaction log usage and disk space where log resides?


Thanks,
Phillip Cox
Post #478882
Posted Wednesday, April 02, 2008 4:07 PM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Yesterday @ 6:49 PM
Points: 6,998, Visits: 13,946
Have your hardware folks taken a look?

Did they start sharing any of the SQL LUN's with other applications? How's the IO looking on the SAN?

We had something like that happen when someone added Exchange and SQl Server to the same arbitrated loop SAN. Let's just say their data access patterns didn't do so well with each other.

Shared LUN's suck for this kind of reason.

Also - are both HBA cards running? Are both channels up?


----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
Post #478885
« Prev Topic | Next Topic »

Add to briefcase 12»»

Permissions Expand / Collapse