Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


NOT IN query very expensive, 100% CPU


NOT IN query very expensive, 100% CPU

Author
Message
aaron.reese
aaron.reese
Mr or Mrs. 500
Mr or Mrs. 500 (589 reputation)Mr or Mrs. 500 (589 reputation)Mr or Mrs. 500 (589 reputation)Mr or Mrs. 500 (589 reputation)Mr or Mrs. 500 (589 reputation)Mr or Mrs. 500 (589 reputation)Mr or Mrs. 500 (589 reputation)Mr or Mrs. 500 (589 reputation)

Group: General Forum Members
Points: 589 Visits: 898
@Celko

You see that bit of text on the left, under the username - It says Forum Newbie.

stop shouting at people who are inexperienced and making them feel like idiots. If you did it in the office you would be up on a harrassment charge - so consider yourself on one here as well.

There is a big difference between passing on knowlege and showing off. Your posts indicate the latter and no one is impressed.

I am bored of reading your rants at badly formed posts from new forum members and the spewing of technical answers from books which bear no relation to the question being asked.

Please moderate your language or I will ask the board moderators to do it for you. I am sure your knowlege will be useful here, but not your attitude.

Aaron
Eugene Elutin
Eugene Elutin
Hall of Fame
Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)

Group: General Forum Members
Points: 3042 Visits: 5478

...
If you did it in the office...


Looks like he is not working in the "office". He is a TEACHER!
I just can only image that his students love him to the bits, especially when they start to behave as their "teacher" and expose his bright ideas when trying to get a job - they must have huge success.

If some one would tell me on T-SQL interview that "We, SQL developers, do not use IDENTITY as it's just MS SQL Server proprietary feature and it's completely crap candidate for PK", I would just ask one more question (just for having some fun): "What other crap and proprietary features of SQL Server you don't use?" then thanked candidate for attendance and get him out of office.

At the same time, may be his students are trained to be trainers? That is fine...

_____________________________________________
"The only true wisdom is in knowing you know nothing"
"O skol'ko nam otkrytiy chudnyh prevnosit microsofta duh!":-D
(So many miracle inventions provided by MS to us...)

How to post your question to get the best and quick help
scogeb
scogeb
SSC Journeyman
SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)

Group: General Forum Members
Points: 78 Visits: 420
Jeff Moden (9/12/2012)

Have you tried the NOT IN without the DISTINCT?

Also, what's with all the LTRIM/RTRIM stuff? Data should be properly stored without leading spaces and trailing spaces hardly ever matter except when determining the actual number of bytes a strinng may contain.


Yep, same deal with NOT IN without the DISTINCT, 100% CPU for way too long.

Who knows with the LTRIM/RTRIM. This was set up before my time here. We're trying to clean it up though. Trust me, I know it's a mess. Smile Take for instance the left(table2.column2,2), if you're going to use the left 2 spaces for stuff all the time, why not create another column and index it, right? I'm not sure what the original developers were thinking. Probably just wanted to get it done and go home/didn't care.

Thanks for the help!
Lynn Pettis
Lynn Pettis
SSC-Insane
SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)

Group: General Forum Members
Points: 24171 Visits: 37936
scogeb (9/13/2012)
Jeff Moden (9/12/2012)

Have you tried the NOT IN without the DISTINCT?

Also, what's with all the LTRIM/RTRIM stuff? Data should be properly stored without leading spaces and trailing spaces hardly ever matter except when determining the actual number of bytes a strinng may contain.


Yep, same deal with NOT IN without the DISTINCT, 100% CPU for way too long.

Who knows with the LTRIM/RTRIM. This was set up before my time here. We're trying to clean it up though. Trust me, I know it's a mess. Smile Take for instance the left(table2.column2,2), if you're going to use the left 2 spaces for stuff all the time, why not create another column and index it, right? I'm not sure what the original developers were thinking. Probably just wanted to get it done and go home/didn't care.

Thanks for the help!


What would really help us help you is if you would provide the DDL (CREATE TABLE statement) for the tables involved including the indexes currently defined on the tables. Some sample data (as INSERT INTO statements) for the tables, and the actual execution plan for the query (if possible) as a .sqlplan file.

Right now, you are getting some help, but basically just shots in the dark.

Cool
Lynn Pettis

For better assistance in answering your questions, click here
For tips to get better help with Performance Problems, click here
For Running Totals and its variations, click here or when working with partitioned tables
For more about Tally Tables, click here
For more about Cross Tabs and Pivots, click here and here
Managing Transaction Logs

SQL Musings from the Desert Fountain Valley SQL (My Mirror Blog)
scogeb
scogeb
SSC Journeyman
SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)

Group: General Forum Members
Points: 78 Visits: 420
Well I did some more digging and it looks like the ltrim/rtrim is doing something because I get different results if I take them out. From what I gathered so far table1.column1 has trailing spaces in some cases and it looks like that causes the differences. Although there are quit a few nulls in column1 and column2 in both tables as well. It really is a mess. I question the data validation on this as well as I think the program wasn't written very well and now we have to deal with it.
scogeb
scogeb
SSC Journeyman
SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)SSC Journeyman (78 reputation)

Group: General Forum Members
Points: 78 Visits: 420
Lynn Pettis (9/13/2012)

What would really help us help you is if you would provide the DDL (CREATE TABLE statement) for the tables involved including the indexes currently defined on the tables. Some sample data (as INSERT INTO statements) for the tables, and the actual execution plan for the query (if possible) as a .sqlplan file.

Right now, you are getting some help, but basically just shots in the dark.


Your shots in the dark have helped greatly! I think I'm good on this one. Thanks a ton!
GilaMonster
GilaMonster
SSC-Forever
SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)

Group: General Forum Members
Points: 47189 Visits: 44356
scogeb (9/13/2012)
Jeff Moden (9/12/2012)

Have you tried the NOT IN without the DISTINCT?

Also, what's with all the LTRIM/RTRIM stuff? Data should be properly stored without leading spaces and trailing spaces hardly ever matter except when determining the actual number of bytes a strinng may contain.


Yep, same deal with NOT IN without the DISTINCT, 100% CPU for way too long.


Yeah, it will be. SQL ignores DISTINCT when specified in an IN/NOT IN subquery.


Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass


Eugene Elutin
Eugene Elutin
Hall of Fame
Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)Hall of Fame (3K reputation)

Group: General Forum Members
Points: 3042 Visits: 5478
Can you try this:



SELECT column1,'99' [a],left(column2,2) [b], max(column3) [c], 'Yes' [d], 'Test' [e]
,ltrim(rtrim(column1))+ltrim(rtrim(left(column2,2))) as test
INTO #pre1
FROM table1
WHERE column4 = 'Yes'
GROUP BY column1, left(column2,2)

SELECT DISTINCT ltrim(rtrim(column1))+ltrim(rtrim(left(column2,2))) as test
INTO #pre2
FROM table2

SELECT p1.column1, p1.a, p1.b, p1.c, p1.d, p1.e
FROM #pre1 p1
WHERE NOT EXISTS(SELECT 1 FROM #pre2 p2 WHERE p2.test = p1.test)




You may also try creating unique clustered index on #pre2 (test)

_____________________________________________
"The only true wisdom is in knowing you know nothing"
"O skol'ko nam otkrytiy chudnyh prevnosit microsofta duh!":-D
(So many miracle inventions provided by MS to us...)

How to post your question to get the best and quick help
Jeff Moden
Jeff Moden
SSC-Forever
SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)

Group: General Forum Members
Points: 44994 Visits: 39880
GilaMonster (9/13/2012)
scogeb (9/13/2012)
Jeff Moden (9/12/2012)

Have you tried the NOT IN without the DISTINCT?

Also, what's with all the LTRIM/RTRIM stuff? Data should be properly stored without leading spaces and trailing spaces hardly ever matter except when determining the actual number of bytes a strinng may contain.


Yep, same deal with NOT IN without the DISTINCT, 100% CPU for way too long.


Yeah, it will be. SQL ignores DISTINCT when specified in an IN/NOT IN subquery.


Thanks, Gail. I learned something new today.

Just to verify, I tried it a couple of different ways. On larger tables, it'll sometimes add a "Sort (Distinct Sort)" to the execution plan if the optimizer thinks it will help but I couldn't get it to add such a sort using DISTINCT on smaller tables.

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
     Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
Although they tell us that they want it real bad, our primary goal is to ensure that we dont actually give it to them that way.
Although change is inevitable, change for the better is not.
Just because you can do something in PowerShell, doesnt mean you should. Wink

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
GilaMonster
GilaMonster
SSC-Forever
SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)SSC-Forever (47K reputation)

Group: General Forum Members
Points: 47189 Visits: 44356
http://sqlinthewild.co.za/index.php/2011/01/18/distincting-an-in-subquery/


Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass


Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search