Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


SSIS 101: Isolating Data in a Data Flow


SSIS 101: Isolating Data in a Data Flow

Author
Message
Andy Leonard
Andy Leonard
Say Hey Kid
Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)

Group: General Forum Members
Points: 667 Visits: 1093
Comments posted to this topic are about the item SSIS 101: Isolating Data in a Data Flow

Andy Leonard
Data Philosopher, Enterprise Data & Analytics
Manie Verster
Manie Verster
Ten Centuries
Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)

Group: General Forum Members
Points: 1405 Visits: 1022
Hi Andy,

Nice article but afraid I lost you along the way. You know, where you started with the Union All's and the rest, I still have the red error mark on my ContactStage but don't see where you handled that part. Excuse me if I am silly or stupid but can you help me out here please. I am busy training myself in SSIS and your aticle came on the right time.HeheHehe

:-PManie Verster
Developer
Johannesburg
South Africa

I can do all things through Christ who strengthens me. - Holy Bible
I am a man of fixed and unbending principles, the first of which is to be flexible at all times. - Everett Mckinley Dirkson (Well, I am trying. - Manie Verster)
Jack Corbett
  Jack Corbett
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: General Forum Members
Points: 12334 Visits: 14863
Nice article Andy. I've never used the multicast task and didn't have an idea why I'd use it, but this is a good way use for it in my situation. I've always edited my source query with a where clause to check out problematic or specific rows.

This is another tool in the toolbox. Thanks.



Jack Corbett

Applications Developer

Don't let the good be the enemy of the best. -- Paul Fleming
At best you can say that one job may be more secure than another, but total job security is an illusion. -- Rod at work

Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question
How to Post Performance Problems
Crosstabs and Pivots or How to turn rows into columns Part 1
Crosstabs and Pivots or How to turn rows into columns Part 2
KEVIN.L.STEVENS
KEVIN.L.STEVENS
Forum Newbie
Forum Newbie (4 reputation)Forum Newbie (4 reputation)Forum Newbie (4 reputation)Forum Newbie (4 reputation)Forum Newbie (4 reputation)Forum Newbie (4 reputation)Forum Newbie (4 reputation)Forum Newbie (4 reputation)

Group: General Forum Members
Points: 4 Visits: 9
Here is where Andy tells how to correct the target information:


This scenario occurs with enough frequency to justify the Restore Invalid Column References Editor, shown above. The ContactID column with Lineage ID 32 is shown as invalid, but the References Editor located (and suggests) a column with the same name and data type: the ContactID column with Lineage ID 333. In fact, it’s located new columns for all the invalid references (although the error did not report these additional columns).

Click OK to replace the invalid references with the suggested columns

You have several options in the Restore Invalid Column References Editor. One of them is to let the editor try to restore the invalid links but I usually highlight all of the rows listed and click on the delete invalid rows option and hit "Apply" (there is no OK button). This will close the editor and remove the invalid references. When you reopen the target again everything should fix itself and you are on your way.
DCPeterson
DCPeterson
Ten Centuries
Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)

Group: General Forum Members
Points: 1131 Visits: 432
Nice article Andy, I never really understood what the data viewers were. I sure wish I'd had this information a year ago...

/*****************

If most people are not willing to see the difficulty, this is mainly because, consciously or unconsciously, they assume that it will be they who will settle these questions for the others, and because they are convinced of their own capacity to do this. -Friedrich August von Hayek



*****************/
StarNamer
StarNamer
Ten Centuries
Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)

Group: General Forum Members
Points: 1334 Visits: 1992
Great article. I too am having to do more and more work with SSIS and, naturally, don't have time/budget to go on training Sad hence learning about the tools described here is very helpful.

Thanks.

Derek
bc_
bc_
Ten Centuries
Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)Ten Centuries (1.4K reputation)

Group: General Forum Members
Points: 1404 Visits: 7312
I like the Data Probe idea. That will come in handy for sure.

Thanks,
Burke

bc
mcloney
mcloney
SSC Veteran
SSC Veteran (225 reputation)SSC Veteran (225 reputation)SSC Veteran (225 reputation)SSC Veteran (225 reputation)SSC Veteran (225 reputation)SSC Veteran (225 reputation)SSC Veteran (225 reputation)SSC Veteran (225 reputation)

Group: General Forum Members
Points: 225 Visits: 83
Hi Andy,

Great intro article on SSIS! I've been working with it for over two years, but had never tried anything similar to your "data probe" technique. It makes perfect sense.

One thing I saw in this article that's a little different is the Multicast as sort of a dummy endpoint. I've often seen a Rowcount as a dummy endpoint but never a multicast. Does the endpoint actually do anything in either case in this type of scenario?

Cheers,

-Matthew



Jack Corbett
  Jack Corbett
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: General Forum Members
Points: 12334 Visits: 14863
mcloney (2/11/2009)
Hi Andy,

Great intro article on SSIS! I've been working with it for over two years, but had never tried anything similar to your "data probe" technique. It makes perfect sense.

One thing I saw in this article that's a little different is the Multicast as sort of a dummy endpoint. I've often seen a Rowcount as a dummy endpoint but never a multicast. Does the endpoint actually do anything in either case in this type of scenario?

Cheers,

-Matthew


I actually use the konesans trash destination when I need a dummy end point.



Jack Corbett

Applications Developer

Don't let the good be the enemy of the best. -- Paul Fleming
At best you can say that one job may be more secure than another, but total job security is an illusion. -- Rod at work

Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question
How to Post Performance Problems
Crosstabs and Pivots or How to turn rows into columns Part 1
Crosstabs and Pivots or How to turn rows into columns Part 2
Andy Leonard
Andy Leonard
Say Hey Kid
Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)Say Hey Kid (667 reputation)

Group: General Forum Members
Points: 667 Visits: 1093
Thanks for all the comments!

Matthew - Like Jack, I usually use the Konesans Trash Destination for an endpoint as well. I didn't mention it here. I also use the Script Component and the Derived Column transformations as dummy endpoints for data flow paths - basically anything that will succeed without configuration will do.

:{> Andy

Andy Leonard
Data Philosopher, Enterprise Data & Analytics
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search