Searching for suggestion on how to expand ETLs skills

  • HildaJ

    SSCarpal Tunnel

    Points: 4453

    Good morning everyone,

    I'm very familiar with SSIS and it's tools.  However, sometimes depending on the project that I'm working on, it seems that my skills are limited on how to make this happens.  In addition to phython, powershell and R, what do you audience suggest getting my hands on and expand my skills?

    Thanks.

  • Martin Schoombee

    SSCoach

    Points: 19025

    That's an excellent question.

    My opinion only, but I'd recommend Power Query. From what I've seen coming from Microsoft lately, I wouldn't be surprised if PQ became the defacto language for data wrangling in the future...even in SSIS.

  • tim.ffitch 25252

    SSCommitted

    Points: 1995

    Adaptris have a very useful product called Interlok. If you have a situation where you must collect data from numerous clients, Interlok handles this very well. It spins the design ethos on its head so that the clients send the data to either a central file server or a Message Queue. This is useful where a single SQL server may be unable to keep up with the workload. With the file server approach you can then use either Interlok or SSIS to process the files, with the Message Queue approach you have a second Interlok process to handle records from the Message Queue.

    It's very useful in this respect where the Hosts of the clients do not want in-bound connections but will allow outbound, tighter security that way.  The other boon is that you can set up a Git repository to store the client configurations, that way you can make changes to certain aspects without having to access numerous clients. If the clients themselves have a range of database systems you create a unique configuration for each system, again saving repetition.

  • Jo Pattyn

    SSC-Dedicated

    Points: 31315

    You could explore the Azure datafactory, flows and databricks

Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply