Track how often table rows are updated for specific columns for ETL purposes

  • CDC of course, but... that is overkill.

     

    I don't want to audit the history, I want an ETL flag to know I need to reprocess those rows.

    I could use CDC but that has several limitations it places on the database, and the overhead is more than I need as I just need a flip switch of changed or not changed to know what rows to process.

    Is there any other lighter weight approach other than full CDC to accomplish this?

  • Well can you just put a last modified date on the record and use that capture deltas?

     

    Another possible option is to snap shot the entire data set at the start of the ETL, if you aren't using all the columns from the tables you only snapshot what you need, and use that as a comparison for the next ETL run.

    • This reply was modified 4 years, 3 months ago by  ZZartin.
  • A trigger on the table - would that work?

    John

  • It's a vendor app database so I can't modify the table.

     

    Yeah, I think trigger is where I will have to go.

     

  • Maxer wrote:

    It's a vendor app database so I can't modify the table.

    Yeah, I think trigger is where I will have to go.

     

    If it's a vendor database, you may also not be allowed to add a trigger as your changing the databases they provide.  You would need to read into the contract between yourselves very carefully to ensure you can in fact do this.

    What about CDC's lighter cousin Change Tracking?

  • What about using extended events to look for specific queries successfully completing?

    "The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
    - Theodore Roosevelt

    Author of:
    SQL Server Execution Plans
    SQL Server Query Performance Tuning

  • This was removed by the editor as SPAM

Viewing 7 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply