• You got that right.

    I've built a data warehouse (after a fashion). It's brilliant. I can predict stock costs for a year to within 1.8% of actuals.

    Unfortunately the users don't really care.

    As I see it the problem is largely the 80:20 rule. Top management are really only interested in the first 80% of the implementation. Once your sales orders are running through the system and you're controlling inventory OK (ish) they just don't care. So the onus for the last 20% is between IT and the business units themselves.

    What I've found is that actually the people I am trying to work with to resolve data quality issues just don't care. They spend two hours a day fiddling with spreadsheets and take great pride in their accomplishments. These spreadsheets are used for top management reporting, and despite the fact that they aren't accurate another lesson I've learned is that top management DON'T ACTUALLY CARE because information at this level isn't reconciled to anything by anyone except me.

    The amount of bull information flowing around our organisation is frightening, but as long as it is delivered people think they've done their job. 

    So when I give the user a report reconciling stock reports with sales (which show the flaws in our stock and shop floor reporting procedures) the user just ignores it.

    The fact I can prove a 10% deviation between our production, sales, stock take and finance systems is irrelevant because nobody wants to know.

    So, in reply to your article, here's what I learned :

    1) Very few users can be trusted to tell you what their information requirements are because they see it as infringing on their job, which they perceive to be wasting hours creating bull spreadsheets.

    2) It is extremely difficult for IT to resolve data problems themselves, if you try and take on this function (as I did) you end up doing everyone else's job for them, I actually did the stock take myself for a couple of months.

    3) ANY manual spreadsheet of any size is AT LEAST 10% inaccurate, but is never checked. Manual errors tend to be small but cumulative.

    4) The nature of computerised errors is different - they tend to be generally very accurate but miss the odd bit of info off. This is very easy for users to spot, in my case my reports are 95% accurate but as the users spot the errors quickly they don't trust them.

    5) It may just be my users (they are pretty dumb) but they refuse to download a 95% correct report into a spreadsheet and correct it, they would rather create a manual spreadsheet every day. Despite the fact that it is inaccurate and takes 90 minutes longer.

    I suspect they just like looking busy.

    I give up. I am buying a goat farm in the Hebrides.