I had a similiar experience.

A little explaination. In some states there are regulated electric suppliers (big utilities) where energy rates are set by the a state commission (not just the energy but also transmission and distribution costs). These states allow unregulated electric suppliers to offer energy (electricity) on different plans (some bargains, some a trap). The regulated utility still recouped the distribution costs and read the meter.

I worked for one of the unregulated electric suppliers. Started out in one market and expanded.

Now to get to the non-boring part I hope. The regulated utility would send us the meter readings in EDI in a "Standard" format. The company I worked at developed a forecasting method based on the EDI that came from regulated utility. Worked pretty well.

But as they expanded markets they interacted with more regulated utilities. The forecasting there stunk.

When I was contracted it was figure out what the heck is going on.

Turns out the EDI messages were standard, but it was up to each utility to interpret what went where. Given the regulated utilities were not especially fond of the unregulated suppliers they all sent the data out differently.

Moral of the story - the data is there, you just have to find it.

Off topic - Bad data

One of the utilities had one customer had used 1 gigawatts of data for 6 straight days. Right - we had to put data filtering in for instances like plus under reporting of data.

I'm glad to say I did manage to get all data cleanup, EDI processing and forecasting done while I worked myself out of a job.

Phil, I read your "Confessions of an IT Manager" e-Book last week. I felt so guilty about you not getting compensated I made a donation in your name. The establishment was so happy they gave me a pint of Sierra Nevada :-D.