This editorial was originally released on Jan 9, 2013. It is being republished as Steve is out of the office.
There's a lot of hype around "big data", a term being thrown around so much in the media that I'm not really sure what it means anymore. Is 1TB "big data"? Is 10,000 transactions/sec big data? Or does it mean that you have more data than your systems can handle, causing queries and reports to run slow?
I almost hope it's the latter. I hope that our managers start to think that when our systems run slower that we're dealing with big data, and we need more resources. The whole big data phenomenon could be a way for data professionals to start a new hardware renaissance, where hardware budgets grow and we begin to replace our current systems with bigger, faster servers.
Or perhaps it's a way to offload some of the system administration for individual servers and move to cloud services. I don't think that's necessarily a bad move for many DBAs as it would allow them to focus on data management, and information extraction rather than dealing with storage and hardware management. Those infrastructure jobs are not going to be the ones you want in the future.
Big Data is in the news, and it's being used by vendors to sell new products and services. From Hadoop to new SANs to BI interfaces, there's no shortage of places where the term "Big Data" might be used to try and shorten the sales cycle. As a data professional, it's important that you understand what your needs really are, and if the term is being in a way that actually provides some value to your company for the money spent. If it is, then use it to upgrade your systems. If it's not, point that out to your boss.