As data professionals, our jobs deal with ensuring that there is data available, accurate, and relevant to the organizations in which we work. However we often go beyond the simple maintenance, gathering, and manipulation of data to help our clients and customers understand the information contained in our systems. In many of my positions, whether as developer or DBA, I've been tasked with working closely with business people to make decisions based on data.

I've always been of the opinion that more data is better, though I certainly understand the problems of Big Data and the potential to reveal false patterns or overwhelm systems with conflicting information. The better I understand the domain, and the better educated the business analysts are, the better we can work to extract information from databases.

However that's not always enough to help an organization act. I read an interesting piece this week on data in sports, which talks about one of the NHL team's managers using data to make decisions on how to restructure his team. It's interesting to see that additional data gives the general manager more insight, and leads him towards a decision, but the rest of the organization can't follow through. In this case it's a matter of money, but resources constrain our efforts in many decisions. I think this is a place where more flexible, and perhaps more in-depth, BI-type analysis of what-if scenarios can be more helpful.

The other piece that caught my eye had to do with real estate, and housing prices, with the author bemoaning the lack of data in determining the value of a house. It's interesting, but to me, it's flawed. More data won't help because houses aren't like many other commodities. One house is not fungible with another one, and the market is both fast moving, and inherently full of friction. We rarely buy a house without visiting it, a task that consumes time, and slows the movement of information through the system. I'm as frustrated as the author, but I don't know that more data would help in this case.

Data is important, and it should be a part of our decisions, but we should remember that data isn't necessarily going to make the decision for us. We need to be ready to incorporate our own knowledge and judgment into a data set to help us decide on a course of action.