• Innerise (7/7/2015)


    Hi All

    We have a 2-node SQL Server 2012 AlwaysOn setup, in synchronous mode with readable replica databases. We have a particular reporting query that we're running against one of the read-only replica databases, which is taking in excess of over 2 hours to complete on the replica node. However, when we run the same query against the primary database on the primary node, it completes in 5 seconds.

    I've attached part of the 2 of the execution plans shown in Redgate SQL Sentry (Primary.png is the quick query, Replica.png is the slow query). These 4 rows are the first stages of the query (clustered index scans), which I believe are the cause of the slowness in the slow-running replica database query.

    You can see that the 'Actual rows', 'Actual Executions' and 'Actual Data Size' are significantly higher in the replica database query. I'm struggling to see how this can be, if the databases are replicas of eachother.

    Both nodes (VMs) are very similar in spec. However, I believe that the underlying storage is slower (SaS as opposed to SSD) on the replica - I'll will confirm this later.

    Any help would be appreciated, as would any pointers as to where I should spend my troubleshooting time.

    Thanks,

    Innerise

    1) Do an IO stall analysis during report runs. That can be a HUGE factor.

    2) Most people don't know that when you make a secondary readable the PRIMARY DATA STRUCTURES are all modified on write to have a 14-byte pointer added so the secondary can do snapshotting to give a consistent read there. This is a horrible thing because it a) fragments pages because everyone has 0 fill factor on all their indexes, b) creates lots more dirty pages and tlog activity that has to be mirrored over to secondary and replayed, c) hits tempdb for snapshotting stuff, d) additional network activity, etc.

    3) Do some binoogling on statistics on readable secondaries.

    4) I didn't review query plans, but did you check compiled/executed values for parameters as well as actual/estimated row counts between the plans?

    Best,
    Kevin G. Boles
    SQL Server Consultant
    SQL MVP 2007-2012
    TheSQLGuru on googles mail service