Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Escape from ETL Hell Expand / Collapse
Author
Message
Posted Saturday, October 26, 2013 1:52 AM
Mr or Mrs. 500

Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500

Group: Administrators
Last Login: Wednesday, April 16, 2014 9:53 AM
Points: 569, Visits: 985
Comments posted to this topic are about the item Escape from ETL Hell
Post #1508644
Posted Saturday, October 26, 2013 9:10 AM


SSChasing Mays

SSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing Mays

Group: General Forum Members
Last Login: 2 days ago @ 4:30 PM
Points: 601, Visits: 1,741
I worked for ten years at a company that had 27 stove piped systems. I have lived in the ETL nightmare for a long time.

My current position was transformed into the ETL process when our company was bought out by a competitor. I don't actually have to do the Load portion but am responsible for getting the data out and transformed to their load specs. It has been interesting to find out how the end-users have abused the applications and databases.

The current system works so well that five employees can transform up to 80 companies per month. And that is hundreds of medical and financial records per facility.

And really that has happened with every ETL process I have worked.

But sometimes it the programmer that does the RBAR process as oppose to UPDATE statement that causes the slow-down. I ran into a one where there was flag for flood insurance (FI_Req) that was based on two other columns in the table. The programmer was going through 500K+ rows to set the flag one by one.

We changed it to a simple set of update statements, and cut 30 minutes out of the processing time.

So now I always look to see what portion of the process is taking the longest and see if I can cut down the processing time.




----------------
Jim P.

A little bit of this and a little byte of that can cause bloatware.
Post #1508661
Posted Saturday, October 26, 2013 3:19 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Yesterday @ 10:06 PM
Points: 35,978, Visits: 30,269
Sounds simple, right? That's its beauty. Much has been written about the intricacies and difficulties of ETL processes, delving deep into the crevices of tools such as SSIS, in order to wring maximum performance from each part of the process. Occasionally, however, someone presents an idea, a solution, of such elegance and simplicity that it immediately seems like common sense, and that everyone must be doing it this way already, though I suspect they aren't. Am I wrong?


Why whatever could you mean, Tony? Doesn't everyone know and enjoy the simplicity and incredible efficiency of such wonders as EDI, XML, HIPPA, CSV and dozens of other standard formats and the high quality, easy to use, intuative tools to exploit them?

{... wait for it...}



{... wait ... }





{... w-a-i-t ... }




{Ok... I can't hold it back any longer...}

BWWWWAAAAA-HAAAAA-HAAAAA-HAAAAA!


--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

"Change is inevitable. Change for the better is not." -- 04 August 2013
(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #1508677
Posted Saturday, October 26, 2013 7:41 PM


SSChasing Mays

SSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing Mays

Group: General Forum Members
Last Login: 2 days ago @ 4:30 PM
Points: 601, Visits: 1,741
Jeff Moden (10/26/2013)
{Ok... I can't hold it back any longer...}

BWWWWAAAAA-HAAAAA-HAAAAA-HAAAAA!


Oh I know. My company shutdown a software line several years ago that was not profitable. They gave the customers a choice: go back to older software that worked or we'll give you your data in CSV/XML and the customer can walk away.

I built the CSV/XML export solution using a combo of MS Access/VBA/stored procs. The data was sensible, useable and appreciated by the former customers.

The SSIS packages to convert the data built by the former developers was fraught with issues, could only be run by them, and took days to process.

I look at it is the use of an off-character (pipe, tilde, backslash) from a text file is the best way to tranfer data.

And then storing XML data is like storing a binary file in a database. Never to be done and a waste of space.




----------------
Jim P.

A little bit of this and a little byte of that can cause bloatware.
Post #1508684
Posted Saturday, October 26, 2013 10:08 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Yesterday @ 10:06 PM
Points: 35,978, Visits: 30,269
Just to be sure, Jim, I wasn't laughing at you. It sounds like you made moves for the better. I was laughing at my own sarcasm on the subject of "improvements" in the world of data transfer.

--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

"Change is inevitable. Change for the better is not." -- 04 August 2013
(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #1508685
Posted Saturday, October 26, 2013 10:57 PM


SSChasing Mays

SSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing Mays

Group: General Forum Members
Last Login: 2 days ago @ 4:30 PM
Points: 601, Visits: 1,741
Jeff Moden (10/26/2013)
Just to be sure, Jim, I wasn't laughing at you. It sounds like you made moves for the better. I was laughing at my own sarcasm on the subject of "improvements" in the world of data transfer.


I quite understood. I'm agreeing with you completely.

I know I have been through ETL hell and I think you have probably been there too. DTS sucked. SSIS is a total f---ing joke. Trying to troubleshoot either is a nightmare. The Oracle built in tool sucks as well. And then I worked with the Great Plains import tool when we went from Peoplesoft Accounting at my last job. It took an employee a month of 40 hour weeks to get three years of data into the GP, because it had to be done day by day. And that was using three desktops that the only input was the date that was needed to be imported.

Anytime someone says that the ETL is process is too slow but a different department is at fault I'm going to tell that department "let me in" or "this is the standard time you have to meet". I'm not going to mess around with it. I know what I have to do. The other people need to meet what I have to do.




----------------
Jim P.

A little bit of this and a little byte of that can cause bloatware.
Post #1508687
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse