Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 


 
     
Recent Posts
   
Popular Topics
Home   
Search
    Members    Calendar    Who's On

Home » SQL Server 2008 » T-SQL (SS2K8) » NOT IN query very expensive, 100% CPU
Add to briefcase
45 posts, Page 1 of 5
12345»»»

NOT IN query very expensive, 100% CPU Expand / Collapse
Rate Topic
Display Mode
Topic Options
Author
Message
scogeb
Posted Tuesday, September 11, 2012 12:46 PM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Yesterday @ 8:48 AM
Points: 64, Visits: 317
Hello all. Hopefully someone will be able to help me out with this query.

The below query takes 42 minutes, returns 1 row and sits at 100% CPU the entire time.
SELECT table1.column1,'99',left(table1.column2,2),max(table1.column3),
'Yes', 'Test'
FROM table1
WHERE ltrim(rtrim(table1.column1))+ltrim(rtrim(left(table1.column2,2)))
NOT IN (SELECT DISTINCT ltrim(rtrim(table2.column1))+ltrim(rtrim(left(table2.column2,2)))
FROM table2)
AND table1.column4 = 'Yes'
GROUP BY table1.column1,left(table1.column2,2)

When I break it into the 2 componants like below, they both finish in < 2 seconds.

Part 1 - 1 seconds, 15,000 rows
SELECT ltrim(rtrim(table1.column1))+ltrim(rtrim(left(table1.column2,2))) as test, table1.column1,'99',left(table1.column2,2),max(table1.column3), 'Yes', 'Test'
FROM table1
WHERE table1.column4 = 'Yes'
GROUP BY table1.column1,left(table1.column2,2)
1 second 15,000 rows

Part 2 - 2 seconds, 19,000 rows
SELECT DISTINCT ltrim(rtrim(table2.column1))+ltrim(rtrim(left(table2.column2,2)))
FROM table2
2 seconds 19,000 rows

What is wrong with the first query that causes it to take so long? It's really just a combination of the bottom 2 queries.

Thanks!
Post #1357636
lrosales
Posted Tuesday, September 11, 2012 1:20 PM
SSC Journeyman

SSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC Journeyman

Group: General Forum Members
Last Login: Wednesday, May 22, 2013 2:00 PM
Points: 77, Visits: 399
Try using not exists(= boolean) instead of not in(= iteration)



"We never plan to Fail, We just fail to plan":)
Post #1357656
Luis Cazares
Posted Tuesday, September 11, 2012 1:27 PM


SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Yesterday @ 4:59 PM
Points: 960, Visits: 1,924
Another option:
SELECT	table1.column1,
	'99',
	left(table1.column2,2),
	max(table1.column3),
	'Yes', 
	'Test' 
FROM	table1 
LEFT
JOIN	table2 ON ltrim(rtrim(table1.column1))+ltrim(rtrim(left(table1.column2,2))) 
	= ltrim(rtrim(table2.column1))+ltrim(rtrim(left(table2.column2,2))) 
WHERE table2.column1 IS NULL
AND table1.column4 = 'Yes'
GROUP BY table1.column1,left(table1.column2,2)

Luis C.
Please don't trust me, test the solutions I give you before using them.
Forum Etiquette: How to post data/code on a forum to get the best help
Post #1357661
carlosaamaral
Posted Tuesday, September 11, 2012 2:18 PM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Yesterday @ 1:53 PM
Points: 28, Visits: 867
SELECT table1.column1,
'99',
left(table1.column2, 2),
max(table1.column3),
'Yes',
'Test'
FROM table1
WHERE NOT EXISTS ( SELECT 1
FROM TABLE2 WHERE ltrim(rtrim(table2.column1)) + ltrim(rtrim(left(table2.column2, 2)))=
ltrim(rtrim(table1.column1)) + ltrim(rtrim(left(table1.column2, 2)))
)

AND table1.column4 = 'Yes'

GROUP BY table1.column1, Left(table1.column2, 2)
Post #1357697
Sean Lange
Posted Tuesday, September 11, 2012 2:57 PM


SSCrazy Eights

SSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy Eights

Group: General Forum Members
Last Login: Yesterday @ 1:17 PM
Points: 8,641, Visits: 8,273
carlosaamaral (9/11/2012)
SELECT table1.column1,
'99',
left(table1.column2, 2),
max(table1.column3),
'Yes',
'Test'
FROM table1
WHERE NOT EXISTS ( SELECT 1
FROM TABLE2 WHERE ltrim(rtrim(table2.column1)) + ltrim(rtrim(left(table2.column2, 2)))=
ltrim(rtrim(table1.column1)) + ltrim(rtrim(left(table1.column2, 2)))
)

AND table1.column4 = 'Yes'

GROUP BY table1.column1, Left(table1.column2, 2)


The reason the original (and yours) are slow is because they are not SARGable. The where clause have functions in them which renders the indexes unusable and causes a table/index scan. The left join solution that Luis presented should prove to perform far better.

_______________________________________________________________

Need help? Help us help you.

Read the article at http://www.sqlservercentral.com/articles/Best+Practices/61537/ for best practices on asking questions.

Need to split a string? Try Jeff Moden's splitter.

Cross Tabs and Pivots, Part 1 – Converting Rows to Columns
Cross Tabs and Pivots, Part 2 - Dynamic Cross Tabs
Post #1357711
GilaMonster
Posted Tuesday, September 11, 2012 4:29 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Today @ 11:45 AM
Points: 37,744, Visits: 30,023
Is the ltrim and rtrim necessary? SQL ignores trailing spaces when comparing string columns, if you have leading spaces it's probably better to clean up the data if possible.

If you can get rid of the functions (they're often used where not necessary), then use not exists or not in (the left join is slightly slower usually). Also consider that there's a behavioural difference between not in and not exists when nulls are involved (on non-nullable columns they perform identically)

p.s. You don't need a DISTINCT in an IN or NOT IN subquery.

Gail Shaw
Microsoft Certified Master: SQL Server 2008, MVP
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass
Post #1357742
Sergiy
Posted Tuesday, September 11, 2012 8:35 PM
SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Yesterday @ 6:59 PM
Points: 4,557, Visits: 8,215
carlosaamaral (9/11/2012)
SELECT table1.column1,
'99',
left(table1.column2, 2),
max(table1.column3),
'Yes',
'Test'
FROM table1
WHERE NOT EXISTS ( SELECT 1
FROM TABLE2 WHERE ltrim(rtrim(table2.column1)) + ltrim(rtrim(left(table2.column2, 2)))=
ltrim(rtrim(table1.column1)) + ltrim(rtrim(left(table1.column2, 2)))
)

AND table1.column4 = 'Yes'

GROUP BY table1.column1, Left(table1.column2, 2)


I believe the only reason to use concatenation was IN construction.
NOT EXISTS does not need that:
SELECT table1.column1,
	'99',
	left(table1.column2, 2),
	max(table1.column3),
	'Yes',
	'Test'
FROM table1  
WHERE NOT EXISTS (  SELECT * 
                FROM TABLE2 
		WHERE  table2.column1 = table1.column1
			and left(table2.column2, 2) = left(table1.column2, 2)
		)
	AND table1.column4 = 'Yes'
GROUP BY table1.column1, Left(table1.column2, 2)

I removed trims as most likely they are not required.
RTRIM is not needed for sure, and LTRIM looks like just a sign of a bad habit.

Please check your data to make sure my assumption is correct.
Post #1357768
scogeb
Posted Wednesday, September 12, 2012 9:28 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Yesterday @ 8:48 AM
Points: 64, Visits: 317
Thanks everyone. I learned a lot here. I went with the NOT EXISTS and it finishes in 2 seconds now. Very nice!
Post #1358059
Jeff Moden
Posted Wednesday, September 12, 2012 2:29 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Yesterday @ 4:51 PM
Points: 32,923, Visits: 26,811
scogeb (9/12/2012)
Thanks everyone. I learned a lot here. I went with the NOT EXISTS and it finishes in 2 seconds now. Very nice!


Have you tried the NOT IN without the DISTINCT?

Also, what's with all the LTRIM/RTRIM stuff? Data should be properly stored without leading spaces and trailing spaces hardly ever matter except when determining the actual number of bytes a strinng may contain.

--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

For better, quicker answers on T-SQL questions, click on the following...
http://www.sqlservercentral.com/articles/Best+Practices/61537/

For better answers on performance questions, click on the following...
http://www.sqlservercentral.com/articles/SQLServerCentral/66909/
Post #1358211
CELKO
Posted Wednesday, September 12, 2012 7:42 PM


SSCommitted

SSCommittedSSCommittedSSCommittedSSCommittedSSCommittedSSCommittedSSCommittedSSCommitted

Group: General Forum Members
Last Login: Tuesday, January 15, 2013 11:11 AM
Points: 1,945, Visits: 2,782
Where is the DDL?

In SQL, each column of a row is a scalar value, representing an attribute. Back in COBOL and assembly language, we had to “cut out” fields from records. Just like you are doing with LEFT(), TRIM() and concatenation. Those functions prevent the optimizer from using indexes or doing much of anything with the data.

Using “IN (SELECT DISTINCT ..)” is not needed; that is one thing the optimizer will handle. We do not put constants in a SELECT list without naming their column. And flags are another left-over from COBOL and assembly language.

Too bad we have no specs, no DDL and have to guess at everything. Her is my guess, if you clean up the DDL and get the tables in First Normal Form (1NF).

SELECT A.column1, A.column2, MAX(A.column3)
FROM Alpha AS A
WHERE NOT EXISTS
(SELECT *
FROM Beta AS B
WHERE A.column1 = B.column1
AND A.column2 = B.column2)
FROM Beta)
AND A.column4 = 'Yes'
GROUP BY A.column1, A.column2;

Books in Celko Series for Morgan-Kaufmann Publishing
Analytics and OLAP in SQL
Data and Databases: Concepts in Practice
Data, Measurements and Standards in SQL
SQL for Smarties
SQL Programming Style
SQL Puzzles and Answers
Thinking in Sets
Trees and Hierarchies in SQL
Post #1358290
« Prev Topic | Next Topic »

Add to briefcase
45 posts, Page 1 of 5
12345»»»

Permissions Expand / Collapse

 
 
Copyright © 2002-2013 Simple Talk Publishing. All Rights Reserved. Privacy Policy. Terms of Use. Report Abuse.