<a href="http://g.adspeed.net/ad.php?do=clk&zid=15220&wd=468&ht=60&pair=as" target="_blank"><img style="border:0px;" src="http://g.adspeed.net/ad.php?do=img&zid=15220&wd=468&ht=60&pair=as" alt="i" width="468" height="60"/></a>
Log in
::
Register
::
Not logged in
Home
Tags
Articles
Editorials
Stairways
Forums
Scripts
Videos
Blogs
QotD
Books
Ask SSC
SQL Jobs
Training
Authors
About us
Contact us
Newsletters
Write for us
<a href="http://g.adspeed.net/ad.php?do=clk&zid=15491&wd=120&ht=300&pair=as" target="_top"><img style="border:0px;" src="http://g.adspeed.net/ad.php?do=img&zid=15491&wd=120&ht=300&pair=as" alt="i" width="120" height="300"/></a>
Recent Posts
Recent Posts
Popular Topics
Popular Topics
Home
Search
Members
Calendar
Who's On
Home
»
Data Warehousing
»
Integration Services
»
Restricting Data before using OLE DB...
Restricting Data before using OLE DB Destination
Rate Topic
Display Mode
Topic Options
Author
Message
Jacob Pressures
Jacob Pressures
Posted Tuesday, February 19, 2013 4:01 PM
SSC Veteran
Group: General Forum Members
Last Login: Friday, March 29, 2013 9:29 AM
Points: 289,
Visits: 302
I just used the unpivot tool. So now I have more columns than I need. I know how to map the columns. If I need to use the SQL command, i don't know how I'd use it. I know the above is an either/or situation.
Since my data has been unpivoted in an effort to normalize it, I need to make sure that the information i'm inputting into the destination is DISTINCT. What tool do i need to get DISTINCT data into my destination?
Thanks!
Post #1421881
Evil Kraig F
Evil Kraig F
Posted Tuesday, February 19, 2013 4:07 PM
SSCertifiable
Group: General Forum Members
Last Login: Today @ 4:36 AM
Points: 5,678,
Visits: 6,126
The aggregate tool. Just group by everything.
Be aware, that's a stream-stopper. It forces all rows into it before it releases any because the data is unsorted. So any following constructs in the stream won't start until the aggregate completes.
Under most circumstances I tend to do aggregations at the db tier as it's better able to handle the workload. If you're dropping this to a staging table immediately after you'll be better off (unless the volume difference is drastic) dumping everything and then SELECT DISTINCT'ing into your real table.
If it's for a flatfile or something it's just price of doing business, just try to make sure it's late in the stream.
- Craig Farrell
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions
|
Forum Netiquette
For index/tuning help, follow these directions.
|
Tally Tables
Twitter: @AnyWayDBA
Post #1421883
Phil Parkin
Phil Parkin
Posted Wednesday, February 20, 2013 3:16 AM
SSCarpal Tunnel
Group: General Forum Members
Last Login: Today @ 7:56 AM
Points: 4,242,
Visits: 9,493
Jacob Pressures (2/19/2013)
I just used the unpivot tool. So now I have more columns than I need. I know how to map the columns. If I need to use the SQL command, i don't know how I'd use it. I know the above is an either/or situation.
Since my data has been unpivoted in an effort to normalize it, I need to make sure that the information i'm inputting into the destination is DISTINCT. What tool do i need to get DISTINCT data into my destination?
Thanks!
Quick note: if your source is an RDBMS, it would be faster (probably much faster) to do all of this using SQL rather than SSIS.
____________________________________________________________________________________________
Help us to help you. For better, quicker and more focused answers to your questions, consider following the advice in this link:
http://www.sqlservercentral.com/articles/Best+Practices/61537/
If you are asking for help and your post does not contain a question, you should expect responses which do not contain any answers. Put a question mark in there somewhere - it's not rocket science.
Post #1422002
Jacob Pressures
Jacob Pressures
Posted Wednesday, February 20, 2013 7:32 AM
SSC Veteran
Group: General Forum Members
Last Login: Friday, March 29, 2013 9:29 AM
Points: 289,
Visits: 302
Thanks guys! It sounds like doing the staging table is the easiest. This is just an exercise for my internship to help me understand how to use SSIS. I guess i'll do it both ways for practice.
If i use a staging table, I'm assuming I'd have to end the data flow add another data flow to the Control flow and link the two. I'd then pull the distinct data out of the staging tables and place it in the real destination tables.
This is my understanding. Any alternatives? Once I put something into a destination table can i take it out in the same data flow? This is why i'm thinking I'd need two data flows.
Thanks!
Post #1422083
Phil Parkin
Phil Parkin
Posted Wednesday, February 20, 2013 7:47 AM
SSCarpal Tunnel
Group: General Forum Members
Last Login: Today @ 7:56 AM
Points: 4,242,
Visits: 9,493
There are almost always alternatives, and knowing the pros and cons of each is a really useful thing. In this case, once data is in a staging table I would probably run a T-SQL Merge to get the data to its destination, unless it's all INSERTS, in which case your suggested method will work fine.
____________________________________________________________________________________________
Help us to help you. For better, quicker and more focused answers to your questions, consider following the advice in this link:
http://www.sqlservercentral.com/articles/Best+Practices/61537/
If you are asking for help and your post does not contain a question, you should expect responses which do not contain any answers. Put a question mark in there somewhere - it's not rocket science.
Post #1422091
Jacob Pressures
Jacob Pressures
Posted Wednesday, February 20, 2013 9:56 AM
SSC Veteran
Group: General Forum Members
Last Login: Friday, March 29, 2013 9:29 AM
Points: 289,
Visits: 302
Thanks very much!
Post #1422166
« Prev Topic
|
Next Topic »
Permissions
You
cannot
post new topics.
You
cannot
post topic replies.
You
cannot
post new polls.
You
cannot
post replies to polls.
You
cannot
edit your own topics.
You
cannot
delete your own topics.
You
cannot
edit other topics.
You
cannot
delete other topics.
You
cannot
edit your own posts.
You
cannot
edit other posts.
You
cannot
delete your own posts.
You
cannot
delete other posts.
You
cannot
post events.
You
cannot
edit your own events.
You
cannot
edit other events.
You
cannot
delete your own events.
You
cannot
delete other events.
You
cannot
send private messages.
You
cannot
send emails.
You
may
read topics.
You
cannot
rate topics.
You
cannot
vote within polls.
You
cannot
upload attachments.
You
may
download attachments.
You
cannot
post HTML code.
You
cannot
edit HTML code.
You
cannot
post IFCode.
You
cannot
post JavaScript.
You
cannot
post EmotIcons.
You
cannot
post or upload images.
Copyright © 2002-2013 Simple Talk Publishing. All Rights Reserved.
Privacy Policy.
Terms of Use.
Report Abuse.
