Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 

Small Data

By Steve Jones,

Big data is a hot topic these days. Like the "cloud" terminology we've been hearing about for the last few years, there isn't a good definition of what "big data" really is. The best one I've seen so far is data that "doesn't fit in Excel," which I like. So many people perform their analysis on a spreadsheet of sorts, that if the data doesn't fit inside their edition of Excel, they'd probably consider it big.

The problem with big data, however, is that it while it contains more information, it can also contain more irrelevant information. That's noted in this piece on small data (from Brent Ozar, PLF), where the author states the signal to noise ration may be decreased when you examine very large data sets. You may find that there are correlations that appear to causations. With enough data, with enough things to examine, you can often start seeing patterns that aren't really there. These ghost patterns can lead you to draw incorrect, or at least less correct, conclusions if you do not investigate further and test your ideas on portions of your data set.

Some of you might have noticed fractal patterns like this:

This is a well known Mandelbrot Set pattern. However if we were to zoom in on this picture, we'd find that the patterns repeat over and over again. What holds true for the largest image we have holds true inside smaller sections. The pattern repeats. 

The same thing can happen with patterns in business. We may see a pattern in a large set of data, but we should verify that it also holds true for subsections of the same data set before we make a decision based on that pattern.

Steve Jones


The Voice of the DBA Podcasts

We publish three versions of the podcast each day for you to enjoy.

Everyday Jones

The podcast feeds are available at sqlservercentral.mevio.com. Comments are definitely appreciated and wanted, and you can get feeds from there. Overall RSS Feed: or now on iTunes!

Today's podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

You can also follow Steve Jones on Twitter:

Total article views: 175 | Views in the last 30 days: 2
 
Related Articles
BLOG

Podcasting

A new video setup is on the way!!!! Actually I'll do a couple podcasts on podcasting over the hol...

ARTICLE

Podcast Announcements

Podcast Feeds

FORUM

Podcast Problem

Podcast Problem Blocked by group policy

BLOG

Podcast Upgrades

A minor change for the podcasts next week. I got my wireless microphone, and I'm working with it a b...

BLOG

Technical Podcasts I Listen To

There are a few podcasts I tend to listen to as I have time. Since I work with a wide...

Tags
editorial    
 
Contribute

Join the most active online SQL Server Community

SQL knowledge, delivered daily, free:

Email address:  

You make SSC a better place

As a member of SQLServerCentral, you get free access to loads of fresh content: thousands of articles and SQL scripts, a library of free eBooks, a weekly database news roundup, a great Q & A platform… And it’s our huge, buzzing community of SQL Server Professionals that makes it such a success.

Join us!

Steve Jones
Editor, SQLServerCentral.com

Already a member? Jump in:

Email address:   Password:   Remember me: Forgotten your password?
Steve Jones