Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Problems In Building a Data Warehouse Expand / Collapse
Author
Message
Posted Saturday, January 27, 2007 12:12 PM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Thursday, October 9, 2014 1:15 PM
Points: 2,834, Visits: 3,071
Comments posted here are about the content posted at http://www.sqlservercentral.com/columnists/jchan/2832.asp
Post #340323
Posted Monday, March 5, 2007 2:54 AM
SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Thursday, April 17, 2014 3:29 AM
Points: 229, Visits: 421

You got that right.

I've built a data warehouse (after a fashion). It's brilliant. I can predict stock costs for a year to within 1.8% of actuals.

Unfortunately the users don't really care.

As I see it the problem is largely the 80:20 rule. Top management are really only interested in the first 80% of the implementation. Once your sales orders are running through the system and you're controlling inventory OK (ish) they just don't care. So the onus for the last 20% is between IT and the business units themselves.

What I've found is that actually the people I am trying to work with to resolve data quality issues just don't care. They spend two hours a day fiddling with spreadsheets and take great pride in their accomplishments. These spreadsheets are used for top management reporting, and despite the fact that they aren't accurate another lesson I've learned is that top management DON'T ACTUALLY CARE because information at this level isn't reconciled to anything by anyone except me.

The amount of bull information flowing around our organisation is frightening, but as long as it is delivered people think they've done their job. 

So when I give the user a report reconciling stock reports with sales (which show the flaws in our stock and shop floor reporting procedures) the user just ignores it.

The fact I can prove a 10% deviation between our production, sales, stock take and finance systems is irrelevant because nobody wants to know.

So, in reply to your article, here's what I learned :

1) Very few users can be trusted to tell you what their information requirements are because they see it as infringing on their job, which they perceive to be wasting hours creating bull spreadsheets.

2) It is extremely difficult for IT to resolve data problems themselves, if you try and take on this function (as I did) you end up doing everyone else's job for them, I actually did the stock take myself for a couple of months.

3) ANY manual spreadsheet of any size is AT LEAST 10% inaccurate, but is never checked. Manual errors tend to be small but cumulative.

4) The nature of computerised errors is different - they tend to be generally very accurate but miss the odd bit of info off. This is very easy for users to spot, in my case my reports are 95% accurate but as the users spot the errors quickly they don't trust them.

5) It may just be my users (they are pretty dumb) but they refuse to download a 95% correct report into a spreadsheet and correct it, they would rather create a manual spreadsheet every day. Despite the fact that it is inaccurate and takes 90 minutes longer.

I suspect they just like looking busy.

I give up. I am buying a goat farm in the Hebrides.

Post #349050
Posted Monday, March 5, 2007 6:51 AM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Yesterday @ 11:12 AM
Points: 818, Visits: 2,064
This article doesn't even begin to address what IT staff, the presumed readership of the article, can do to address the situation presented.  Nothing here about determining measures, level of granularity, etc to help management and IT get on the same page.  All we see is the problem.  We all know that there's often a problem.  What we need are recommended solutions and/or courses of action.


Post #349083
Posted Monday, March 5, 2007 7:06 AM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Sunday, September 18, 2011 6:11 PM
Points: 355, Visits: 299
the title is Problems In Building a Data Warehouse.
Post #349087
Posted Monday, March 5, 2007 9:22 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, September 26, 2012 12:03 PM
Points: 28, Visits: 38
Good article on some of the things DBA's should look out for when management approaches them. I think the main point here is that often, DBA's don't fully understand what data warehousing entails. As a result they often find themselves in this position of letting management set false expectations without being able to educate them as to the steps and participation required from the business side.
Post #349171
Posted Monday, March 5, 2007 12:00 PM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Friday, January 24, 2014 12:16 PM
Points: 3, Visits: 28

the most important word was "incrementally".

start small.  simple procedure to keep a modest snowflake going (sales, custs, prods, time  ... 4 tables).  do NOT jump right into Analysis Serv. and all that.

then, start showing some users something.  you will sometimes amaze them with just the basics.  i find that a nice little ad hoc qry tool that they can pound on the snowflake with reaps great rewards.  little by little the light bulbs start going on.  then you can begin the great adventure

and about the 80/20 rule...it's about ROI.  20% of the effort gets 80% of the reward.  so do some little things first.  just to get users to open their eyes.

and i don't mean to belittle them.  but they have so few tools at their disposal in most companies outside of the ERP.

 

 

Post #349219
Posted Monday, March 5, 2007 1:57 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, October 15, 2014 7:10 PM
Points: 192, Visits: 135

Hello Janet Wong:

This is my first time to read your articles. However, after reading it, I begin to read all your previous publishings. They are full with passion from reality and I can not agree with any more....

Keep posting your great work!

thanks

David




Post #349244
Posted Monday, March 5, 2007 3:27 PM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Wednesday, October 15, 2014 7:55 AM
Points: 934, Visits: 1,007

Very good, to-the-point article.  I'm working on a DW project now and it's good to be reminded of some of the issues we will be facing or already have faced.

 

One thing to point out is that a DW project is reverse of a normal project.  Basically we need to deliver something first, then go back and figure out all of the requirements... very incremental process as Joseph pointed out.  This is because the users will not know their requirements when you start... they may think they do but once you deliver some data for them to work with they will always change their minds. 

 

David




Post #349261
Posted Monday, March 19, 2007 10:36 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, August 21, 2008 3:30 PM
Points: 24, Visits: 6

Richard's reponse is kind of funny to me. My organization is the exact opposite. EVERYONE except IT is constantly worrying about data inaccuracy and trying to remove it. For anyone inside IT, unless the final revenue number is wrong, they are loath to change things. It's gotten so bad that our finance department has to build the data warehouse. Yes, I know it is crazy to have a full fledge developer in the IT department, but sometimes you need to do crazy things to get people's attention. (FYI, we are working on plans for moving the development into IT.)

Anyways, Richard, let me know if you are looking for a new job in the next month or two and are interested in taking on a significant role in a DW project. Our company needs IT people with your mindset!

Post #352551
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse