Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
Log in  ::  Register  ::  Not logged in

Are you looking to Hadoop?

By Steve Jones,

HadoopI hadn't even heard of Hadoop before, but there was a Hadoop World conference recently and it came to my attention on Twitter. I saw a quote that said "JP Morgan Chase is counting on an order of magnitude savings on data warehousing. " Since it's primarily a Linux based system and only set up for development, not production, on Win32 systems, perhaps that's not surprising.

I tried to read through the quickstart on Apache's site for the common core installation and walk through a few examples, but it's a little hard to tell what exactly the buzz is about. Wikipedia was more help, pointing me to the MapReduce papers that Google published. I'll see if I can work through them  at some point. Hadoop is available under a free license and the list of companies using it for large data set processing is impressive: Yahoo!, Amazon, Facebook, and more.

So what's the purpose? Hadoop appears to allow clusters of servers to perform data processing very efficiently. It's built on it's own distributed file system that scales to handle petabytes of data. That might seem like more data than you and I will ever need to work with, but I remember when it was a challenge to get enough disk drives together to assemble a terabyte in a server. Now I have 1.5TB in my desktop, with room for more.

It's an interesting project, and with data volumes constantly growing, I wonder when we'll see a similar technology in Microsoft's data processing platform. They already purchased a search technology company based on Hadoop, and we might see this used in Bing.

I expect this type of processing, and others like the StreamInsight features in SQL Server 2008 R2, to complement, rather than supplant the traditional SQL database engine.

Steve Jones

The Voice of the DBA Podcasts

Everyday Jones

The podcast feeds are available at Comments are definitely appreciated and wanted, and you can get feeds from there.

You can also follow Steve Jones on Twitter:

Overall RSS Feed: or now on iTunes!

Today's podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at

I really appreciate and value feedback on the podcasts. Let us know what you like, don't like, or even send in ideas for the show. If you'd like to comment, post something here. The boss will be sure to read it.

Total article views: 1301 | Views in the last 30 days: 6
Related Articles

Introduction to Hadoop

Hadoop was created by the Apache foundation as an open-source software framework capable of processi...


Good Intro Podcast on Hadoop

Have you heard about Hadoop but don't know much about it? What about "big data?" Would you like an i...


Hadoop and SQL Server

Hadoop is a technology that's getting quite a bit of attention in the last few years, including inte...


Navigating Hadoop Resources

Learn where to get the latest installation and learning resources for the ever-evolving components o...


Test Hadoop cluster on vmware

SQL Server MVP Jeremiah Peschka posted 2 articles about Hadoop, which makes me be interested on the ...


Join the most active online SQL Server Community

SQL knowledge, delivered daily, free:

Email address:  

You make SSC a better place

As a member of SQLServerCentral, you get free access to loads of fresh content: thousands of articles and SQL scripts, a library of free eBooks, a weekly database news roundup, a great Q & A platform… And it’s our huge, buzzing community of SQL Server Professionals that makes it such a success.

Join us!

Steve Jones

Already a member? Jump in:

Email address:   Password:   Remember me: Forgotten your password?
Steve Jones