March 20, 2012 at 10:00 pm
Comments posted to this topic are about the item Data Growth
March 20, 2012 at 10:02 pm
[font="Verdana"]Great Pick![/font]
March 21, 2012 at 12:12 am
Good one Steve, Agree with you 🙂
M&M
March 21, 2012 at 2:54 am
Data requirements have increased but apart from storage what cost is there to IT? It is not necessarily IT that provides the front end development for the business. For me I haven't been part of IT for the last 6 years but work in the BI part that moulds the data from source systems to a presentation layer.
So while my job is technical and would fit into the nice IT schema it just isn't part of IT anymore but is paid by Finance. In my previous job the same happened that IT was just the place to keep the servers for the data running and it was Finance and Logistics who actually had the technical skills to work with the data (and footed the bill).
In my position now IT acts as a guardian of the systems I work with but has no input how the systems work together on the front end. In effect we have 2 IT people covering the data reporting systems but 6 people outside of IT actually doing the work with these data reporting systems.
Yes, IT needs adequate funding to provide the services but IT is not "THE" department to provide users with all the reporting requirements that come with data. It becomes more and more a BI team outside of IT and within Finance / Logistics that actually drives turning data into information.
March 21, 2012 at 7:33 am
This is a great topic, although there’s a cost aspect I think may be overlooked. It is my read that IT organizations are mitigating growing costs by hiring less knowledgeable, less experienced, therefore less expensive data professionals. I see the consequences of that choice to be far reaching, but probably less evident initially than they will inevitably be.
March 21, 2012 at 7:46 am
In my experience affordable storage capacity has kept abreast or slightly ahead of storage requirements until quite recently.
What I am seeing now is storage requirements outstripping the capacity/performance available.
A typical data analyst response to a request for a concrete set of requirements is "give me everything for ever". Up to a point (now past) this was feasible. We have reached the point now where there has to be a business case made for storing data.
There are technology solutions to handle larger data volumes but at web scale there is only so much that can be done.
The cloud as it is at present isn't the solution. It may be able to store exobytes and beyond but try accessing that amount of data. I've been told that the realistic IOPS for cloud storage is around 300 (compared to 75 IOPS for a 1TB disk). That isn't a lot.
There is also the problem of recruiting people capable of exploiting "Big Data". You are talking about people who do quite a bit more than SUM, COUNT, GROUP BY.
March 21, 2012 at 8:06 am
One reason why a 40% increase in data growth translates into only a 5% increase in IT budget is that our hardware and database server platforms are designed to handle larger amounts of data. SQL Server 2008 running on a 64bit server with 64GB of RAM simply handles an order of magnitude more data than SQL Server 2000 running on a 32bit 16GB server. Both servers require only a single DBA earning roughly the same salary.
As another example, over the past 20 years, the disk storage size in my personal laptop has increased 1000x (from 200MB to 200GB), but my budget for purchasing a new laptop is about the same: $1,200.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
March 21, 2012 at 8:30 am
Excellent Steve. I've been preaching this reality for a few years now but you summed it up quite nicely. I'm forwarding your column to a few people in this organization in the hopes that my message is reinforced.
Thanks for bringing this up, because all too often the strategic decision makers do not understand the concept you just so eloquently articulated.
The probability of survival is inversely proportional to the angle of arrival.
March 21, 2012 at 9:05 am
David.Poole (3/21/2012)
The cloud as it is at present isn't the solution. It may be able to store exobytes and beyond but try accessing that amount of data. I've been told that the realistic IOPS for cloud storage is around 300 (compared to 75 IOPS for a 1TB disk). That isn't a lot.
March 21, 2012 at 11:44 am
The capabilities of a single server have indeed increased over the past many years, but I'm sure not doing the same job that I once was.
15 years ago, my DR strategy was backups. And, maybe logshipping if we had the budget for a 2nd server. So, pretty easy. Then, clustering comes into the picture. And with it, SAN. My skill level had to go up about 4 notches. In 2005, mirroring was introduced and we started using that technology as well. On top of that, we helped the devs recode applications to take advantage of a mirroring pair with automatic failover. And now, AlwaysOn is coming out. Virtualization? Oh yeah, more to learn. Etc.
So while a single server may still require only a single DBA, managing the larger environment is a lot more than just 5% more complex these days. I might even say it's 40% more complex 😛
Oh, and Steve - I clicked on the little graph above your post expecting to see some relevant data. Took me a minute to realize the time frame was 1991 to 1997. Feel like I got duped!
March 21, 2012 at 2:01 pm
Randy Rabin (3/21/2012)
Oh, and Steve - I clicked on the little graph above your post expecting to see some relevant data. Took me a minute to realize the time frame was 1991 to 1997. Feel like I got duped!
sorry, pressed for time and couldn't find a better one. I suspect a more modern one would have a steeper slope.
Viewing 11 posts - 1 through 10 (of 10 total)
You must be logged in to reply to this topic. Login to reply