SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Aggregation


Aggregation

Author
Message
BSavoie
BSavoie
SSChasing Mays
SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)

Group: General Forum Members
Points: 655 Visits: 795
I'm confused about how to aggregate a value in SSIS. Let's say I have a task that simply exports the employees from the DB to a flat file. I also have to keep track of MAX(EmpoyeeId) so that the next time I do this, I start with all the new employees that have been added since the last time.

How/where do I handle that aggregate? I just want a way to get that value in a variable. My first instinct was to throw an aggregate on the data flow, but I don't think that's right. That will get executed once for every record in the source. That's not what I want. I could also just drop a SQL Task on the control flow and handle it that way, but technically speaking, that's an entirely different query. It's not actually based on the Employees in the data flow.

Maybe a better way to ask the question is how do I get a MAX value from a data source into a user variable.

.
Phil Parkin
Phil Parkin
One Orange Chip
One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)

Group: General Forum Members
Points: 25514 Visits: 20665
Use a SQL Server statement as you suggest (you can get the result of a single-valued T-SQL query into a variable (say MaxID) directly from the Execute SQL task).

Then use that returned value in your source query for the dataflow (pseudo-code: SELECT f1, f2, ... FROM EMPLOYEE WHERE ID <= User::MaxID). That way you know that things are consistent ...

...but, getting to the crux of the matter, how does knowing the MaxID now (for the current extract) help you when the job next runs?


Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.

Please surround any code or links you post with the appropriate IFCode formatting tags. It helps readability a lot.
BSavoie
BSavoie
SSChasing Mays
SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)

Group: General Forum Members
Points: 655 Visits: 795
Thanks Phil, this is really not about employees at all. I just thought that might make my example a little easier to explain. This is really about exporting data to a vendor once per month. Each month, I have to make sure I don't send anything that was sent the previous month, so I keep track of the MAX record id every month and use that as a starting point the next month.

I'll give your example a try. So anything I put in the data flow gets executed once for each record. Do I understand that correctly?

.
Phil Parkin
Phil Parkin
One Orange Chip
One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)

Group: General Forum Members
Points: 25514 Visits: 20665
Correct,


Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.

Please surround any code or links you post with the appropriate IFCode formatting tags. It helps readability a lot.
Jeffrey Williams 3188
Jeffrey Williams 3188
SSCrazy Eights
SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)

Group: General Forum Members
Points: 9880 Visits: 9994
BSavoie (1/23/2010)
Thanks Phil, this is really not about employees at all. I just thought that might make my example a little easier to explain. This is really about exporting data to a vendor once per month. Each month, I have to make sure I don't send anything that was sent the previous month, so I keep track of the MAX record id every month and use that as a starting point the next month.

I'll give your example a try. So anything I put in the data flow gets executed once for each record. Do I understand that correctly?


I think you need to consider this a bit more. In your example, what happens if you send that data on Employee A last month, that employee is terminated this month - but, since it has already been sent you won't be sending it again?

In other words, what about updates to the system that need to be updated in the downstream systems? How are you going to identify those?

Find the column that identifies the last updated date for that entity (or creation date). Then use that date to filter for any records that have been modified since the last time you extracted the data.

Jeffrey Williams
Problems are opportunities brilliantly disguised as insurmountable obstacles.

How to post questions to get better answers faster
Managing Transaction Logs

BSavoie
BSavoie
SSChasing Mays
SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)SSChasing Mays (655 reputation)

Group: General Forum Members
Points: 655 Visits: 795
Thanks Jeffery. I was just using Employees as an example to make things easier. Actually what I'm really sending is all the Guest Reservations that have stayed at a hotel over the last month. So, it's not really a volitle entity like an employee. That was probably a bad example. Thanks for the feedback!

.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search