April 21, 2016 at 6:28 am
I'm thinking of how to handle the issue we see when a DW has gone live after lots of blood, sweat & tears, lets say the stack is SSIS, SQL Server, SSAS-Multi-dim + BI
Soon after going live or even during the building of the DW the business introduces changes, could be new data types, change in business rules, new product types etc..
IMOH the Enterprise DW (EDW) is hard to maintain in a way that avoids it turning into a monolithic model with an increasing refresh time. Changes need to be slow and methodical.
This is made worse when developers take ownership of a DW post production.
How have people handled this?
I was thinking of using SSAS Tabular for Proof of Concepts/Changes and then only change the EDW if the POC goes on to turn into an enterprise requirement.
I sure would love to be able to branch the EDW i.e. Production - Experimental. This obliviously has impacts.
Best
Lee
April 21, 2016 at 9:50 am
leehbi (4/21/2016)
I'm thinking of how to handle the issue we see when a DW has gone live after lots of blood, sweat & tears, lets say the stack is SSIS, SQL Server, SSAS-Multi-dim + BISoon after going live or even during the building of the DW the business introduces changes, could be new data types, change in business rules, new product types etc..
IMOH the Enterprise DW (EDW) is hard to maintain in a way that avoids it turning into a monolithic model with an increasing refresh time. Changes need to be slow and methodical.
This is made worse when developers take ownership of a DW post production.
How have people handled this?
I was thinking of using SSAS Tabular for Proof of Concepts/Changes and then only change the EDW if the POC goes on to turn into an enterprise requirement.
I sure would love to be able to branch the EDW i.e. Production - Experimental. This obliviously has impacts.
Best
Lee
This is the unfortunate fate of many DW projects: It cannot keep up with changing businesses and business requirements...and I think a lot of the blame lies with the developers and/or technical teams.
It is true that the larger a Enterprise DW gets the more impact small changes might have, but we as technical practitioners need to do a better job at following agile approaches in design and development to counteract it as much as possible. I still see many DW projects follow a classic waterfall development methodology, but it usually takes too long before the business gets to see the end product and by that time things might have changed. The big-bang approach just doesn't work.
In my opinion using something like SSAS tabular or even Power BI (or Power Query / Power View in Excel) is a great way to get to a visual result very quickly, showing the business folks what they will be getting before committing to the development process. Without seeing something physical, it is very difficult for most end users to visualize and articulate their exact requirements.
April 22, 2016 at 7:50 am
I favour agile approaches to BI. There is no better proof than the pudding 🙂
I still think even with user stories we can still run into problems. I think the challenge lies in SSIS. The ETL process is often too big a process for a single developer to easily comprehend.
I wonder if there is a better way of documenting the process. Could use a dependency tracker to target packages I suppose.
April 22, 2016 at 8:56 am
leehbi (4/22/2016)
I favour agile approaches to BI. There is no better proof than the pudding 🙂I still think even with user stories we can still run into problems. I think the challenge lies in SSIS. The ETL process is often too big a process for a single developer to easily comprehend.
I wonder if there is a better way of documenting the process. Could use a dependency tracker to target packages I suppose.
I find that a good source-to-target mapping document, along with comprehensive technical documentation and well thought-out comments in the code (I prefer stored procs over SSIS components) goes a long way.
May 9, 2016 at 7:14 am
I find that a good source-to-target mapping document, along with comprehensive technical documentation and well thought-out comments in the code (I prefer stored procs over SSIS components) goes a long way.
Hi Martin,
a good source-to-target mapping sounds like that I'm searching for.
Our system runs since 2011. Addional Data Marts/Cubes and have been implemented by different developers. We have measures and attributes in different Data Marts which are logical the same objects but different in datatype, length and naming. I think, the best way to get an overview and document to prevent such differences in future is such like a source-to-target mapping.
But, I don't know how to Start. Do you know a tool or strategie? -> except for scanning all SSIS-packages manually 😉
Thanks & kind Regards
Henning
Viewing 5 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply