Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 1234»»»

NOT IN query very expensive, 100% CPU Expand / Collapse
Author
Message
Posted Tuesday, September 11, 2012 12:46 PM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Monday, April 13, 2015 7:42 AM
Points: 65, Visits: 414
Hello all. Hopefully someone will be able to help me out with this query.

The below query takes 42 minutes, returns 1 row and sits at 100% CPU the entire time.
SELECT table1.column1,'99',left(table1.column2,2),max(table1.column3),
'Yes', 'Test'
FROM table1
WHERE ltrim(rtrim(table1.column1))+ltrim(rtrim(left(table1.column2,2)))
NOT IN (SELECT DISTINCT ltrim(rtrim(table2.column1))+ltrim(rtrim(left(table2.column2,2)))
FROM table2)
AND table1.column4 = 'Yes'
GROUP BY table1.column1,left(table1.column2,2)

When I break it into the 2 componants like below, they both finish in < 2 seconds.

Part 1 - 1 seconds, 15,000 rows
SELECT ltrim(rtrim(table1.column1))+ltrim(rtrim(left(table1.column2,2))) as test, table1.column1,'99',left(table1.column2,2),max(table1.column3), 'Yes', 'Test'
FROM table1
WHERE table1.column4 = 'Yes'
GROUP BY table1.column1,left(table1.column2,2)
1 second 15,000 rows

Part 2 - 2 seconds, 19,000 rows
SELECT DISTINCT ltrim(rtrim(table2.column1))+ltrim(rtrim(left(table2.column2,2)))
FROM table2
2 seconds 19,000 rows

What is wrong with the first query that causes it to take so long? It's really just a combination of the bottom 2 queries.

Thanks!
Post #1357636
Posted Tuesday, September 11, 2012 1:20 PM
SSC Journeyman

SSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC Journeyman

Group: General Forum Members
Last Login: Tuesday, August 26, 2014 9:12 AM
Points: 77, Visits: 428
Try using not exists(= boolean) instead of not in(= iteration)



"We never plan to Fail, We just fail to plan":)
Post #1357656
Posted Tuesday, September 11, 2012 1:27 PM


SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Today @ 7:31 AM
Points: 4,690, Visits: 10,788
Another option:
SELECT	table1.column1,
'99',
left(table1.column2,2),
max(table1.column3),
'Yes',
'Test'
FROM table1
LEFT
JOIN table2 ON ltrim(rtrim(table1.column1))+ltrim(rtrim(left(table1.column2,2)))
= ltrim(rtrim(table2.column1))+ltrim(rtrim(left(table2.column2,2)))
WHERE table2.column1 IS NULL
AND table1.column4 = 'Yes'
GROUP BY table1.column1,left(table1.column2,2)




Luis C.
Are you seriously taking the advice and code from someone from the internet without testing it? Do you at least understand it? Or can it easily kill your server?

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1357661
Posted Tuesday, September 11, 2012 2:18 PM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: 2 days ago @ 2:08 PM
Points: 29, Visits: 1,013
SELECT table1.column1,
'99',
left(table1.column2, 2),
max(table1.column3),
'Yes',
'Test'
FROM table1
WHERE NOT EXISTS ( SELECT 1
FROM TABLE2 WHERE ltrim(rtrim(table2.column1)) + ltrim(rtrim(left(table2.column2, 2)))=
ltrim(rtrim(table1.column1)) + ltrim(rtrim(left(table1.column2, 2)))
)

AND table1.column4 = 'Yes'

GROUP BY table1.column1, Left(table1.column2, 2)
Post #1357697
Posted Tuesday, September 11, 2012 2:57 PM


SSChampion

SSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampion

Group: General Forum Members
Last Login: Today @ 7:19 AM
Points: 14,076, Visits: 13,690
carlosaamaral (9/11/2012)
SELECT table1.column1,
'99',
left(table1.column2, 2),
max(table1.column3),
'Yes',
'Test'
FROM table1
WHERE NOT EXISTS ( SELECT 1
FROM TABLE2 WHERE ltrim(rtrim(table2.column1)) + ltrim(rtrim(left(table2.column2, 2)))=
ltrim(rtrim(table1.column1)) + ltrim(rtrim(left(table1.column2, 2)))
)

AND table1.column4 = 'Yes'

GROUP BY table1.column1, Left(table1.column2, 2)


The reason the original (and yours) are slow is because they are not SARGable. The where clause have functions in them which renders the indexes unusable and causes a table/index scan. The left join solution that Luis presented should prove to perform far better.


_______________________________________________________________

Need help? Help us help you.

Read the article at http://www.sqlservercentral.com/articles/Best+Practices/61537/ for best practices on asking questions.

Need to split a string? Try Jeff Moden's splitter.

Cross Tabs and Pivots, Part 1 – Converting Rows to Columns
Cross Tabs and Pivots, Part 2 - Dynamic Cross Tabs
Understanding and Using APPLY (Part 1)
Understanding and Using APPLY (Part 2)
Post #1357711
Posted Tuesday, September 11, 2012 4:29 PM


SSC-Forever

SSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-Forever

Group: General Forum Members
Last Login: Today @ 8:05 AM
Points: 41,856, Visits: 38,505
Is the ltrim and rtrim necessary? SQL ignores trailing spaces when comparing string columns, if you have leading spaces it's probably better to clean up the data if possible.

If you can get rid of the functions (they're often used where not necessary), then use not exists or not in (the left join is slightly slower usually). Also consider that there's a behavioural difference between not in and not exists when nulls are involved (on non-nullable columns they perform identically)

p.s. You don't need a DISTINCT in an IN or NOT IN subquery.



Gail Shaw
Microsoft Certified Master: SQL Server, MVP
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass

Post #1357742
Posted Tuesday, September 11, 2012 8:35 PM
SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Wednesday, May 20, 2015 3:50 PM
Points: 4,625, Visits: 8,504
carlosaamaral (9/11/2012)
SELECT table1.column1,
'99',
left(table1.column2, 2),
max(table1.column3),
'Yes',
'Test'
FROM table1
WHERE NOT EXISTS ( SELECT 1
FROM TABLE2 WHERE ltrim(rtrim(table2.column1)) + ltrim(rtrim(left(table2.column2, 2)))=
ltrim(rtrim(table1.column1)) + ltrim(rtrim(left(table1.column2, 2)))
)

AND table1.column4 = 'Yes'

GROUP BY table1.column1, Left(table1.column2, 2)


I believe the only reason to use concatenation was IN construction.
NOT EXISTS does not need that:
SELECT table1.column1,
'99',
left(table1.column2, 2),
max(table1.column3),
'Yes',
'Test'
FROM table1
WHERE NOT EXISTS ( SELECT *
FROM TABLE2
WHERE table2.column1 = table1.column1
and left(table2.column2, 2) = left(table1.column2, 2)
)
AND table1.column4 = 'Yes'
GROUP BY table1.column1, Left(table1.column2, 2)

I removed trims as most likely they are not required.
RTRIM is not needed for sure, and LTRIM looks like just a sign of a bad habit.

Please check your data to make sure my assumption is correct.
Post #1357768
Posted Wednesday, September 12, 2012 9:28 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Monday, April 13, 2015 7:42 AM
Points: 65, Visits: 414
Thanks everyone. I learned a lot here. I went with the NOT EXISTS and it finishes in 2 seconds now. Very nice!
Post #1358059
Posted Wednesday, September 12, 2012 2:29 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Today @ 7:30 AM
Points: 37,455, Visits: 34,310
scogeb (9/12/2012)
Thanks everyone. I learned a lot here. I went with the NOT EXISTS and it finishes in 2 seconds now. Very nice!


Have you tried the NOT IN without the DISTINCT?

Also, what's with all the LTRIM/RTRIM stuff? Data should be properly stored without leading spaces and trailing spaces hardly ever matter except when determining the actual number of bytes a strinng may contain.


--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #1358211
Posted Thursday, September 13, 2012 7:19 AM
Mr or Mrs. 500

Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500

Group: General Forum Members
Last Login: Thursday, May 14, 2015 10:03 AM
Points: 504, Visits: 766
@Celko

You see that bit of text on the left, under the username - It says Forum Newbie.

stop shouting at people who are inexperienced and making them feel like idiots. If you did it in the office you would be up on a harrassment charge - so consider yourself on one here as well.

There is a big difference between passing on knowlege and showing off. Your posts indicate the latter and no one is impressed.

I am bored of reading your rants at badly formed posts from new forum members and the spewing of technical answers from books which bear no relation to the question being asked.

Please moderate your language or I will ask the board moderators to do it for you. I am sure your knowlege will be useful here, but not your attitude.

Aaron
Post #1358537
« Prev Topic | Next Topic »

Add to briefcase 1234»»»

Permissions Expand / Collapse