Connecting ERP system E2 and Extract Data , perform Transformation

  • Hello,

    May someone please help me how can I pull data from ERP named as E2. E2 is using SQL Server.

    Looking for detailed steps of connecting to E2 System.

    I am planning to use python scripting.

    Do I need to Install any driver.

    what all configuration details we have to paas in the scripting.

    Once Data got extracted , I will perform transformation and push to Azure SQL, on top of it Powerbi reports will run.

    The major challenge I am facing how to integrate with External E2 ERP.

    Appreciate your help very much.

    Thanks a lot for kind help

    • This topic was modified 3 months, 4 weeks ago by Rahulmsb5.
    • This topic was modified 3 months, 4 weeks ago by Rahulmsb5.
    • This topic was modified 3 months, 4 weeks ago by Rahulmsb5.
  • My opinion - if E2 is running on a SQL Server backend, why reinvent the wheel? Use SSIS for your ETL.

    If that isn't an option for one reason or another, then I'd recommend ignoring the "E2" part of the equation and focus on SQL Server. Do a quick google for Python SQL Server and you'll get a lot of advice.

    That being said, SSIS will be able to pull the data, transform it, and push it to Azure SQL with minimal issues. Python you will have overhead of python running on a server (or workstation?) that you will need to upgrade as python updates come out and possibly require code changes. Python 2.x to 3.x was a big change and required a lot of people to rewrite their scripts. I recently did an upgrade from SSIS 2012 to 2022 (I know... I should have upgraded sooner, but had no time) and it went off pretty painlessly. The jobs all succeeded after I did all of the appropriate steps to upgrade things.

    I strongly encourage you to pick the right tool for the job (SSIS), and not just the tool you are familiar with (python). Picking the right tool for the job will help ensure you are ready to scale and future proof. Plus, if you ever hire a second Data Analyst for SQL Server, they will be familiar with SSIS.

    The above is all just my opinion on what you should do. 
    As with all advice you find on a random internet forum - you shouldn't blindly follow it.  Always test on a test server to see if there is negative side effects before making changes to live!
    I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system.

  • If E2 is built on SQL Server and you have access to the backend database, SSIS might actually be the simplest and most stable way to pull data out — especially for structured ETL into Azure SQL.

    But just a heads-up — even though the data lives in SQL Server, ERP systems often store things in a pretty cryptic way. Table names might not say “Billing” or “Customer” directly. You may need some business/functional understanding to figure out which tables tie to which ERP modules, and how they’re joined.

    That’s where working with someone who knows the ERP structure (or has access to its internal data catalog or mapping) can really help. Otherwise, it’s easy to extract incomplete or misleading data just by going straight to SQL.

    Python makes sense if you don’t have DB access and need to hit APIs or exports. But if you do have backend access, SSIS can make things a lot easier — plus it's made for these kind of source-to-target loads.

    Let us know what kind of access you have to the ERP — DB only, API, or a mix — and we can help fine-tune the steps.

    -Chandan

  • One thing to keep in mind with the SSIS route (which Brian and Chandan are right is the standard tool here): SSIS works great for the initial build but maintaining those packages over time gets painful. Schema changes in the ERP break packages silently, you need Visual Studio to edit them, and version controlling dtsx files is a nightmare. If this is a one time migration thats fine, but if you need the data flowing continuously from E2 to Azure SQL you'll be babysitting those packages forever.

    Chandan made a really good point about the ERP table structure being cryptic. E2 is no exception there. Before you write any ETL or python scripts make sure you get the E2 data dictionary or at least work with someone on the functional side who knows which tables map to which modules. Going straight to the SQL backend without that context is how you end up with reports built on the wrong data.

    We deal with this exact pattern at [Stacksync](https://www.stacksync.com/) syncing ERP data into SQL Server and Azure SQL in real time without maintaining ETL packages. If the data needs to stay current on an ongoing basis its worth looking at before you invest weeks building SSIS packages. But if this is a periodic batch load SSIS will do the job fine.

Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply