Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
Log in  ::  Register  ::  Not logged in
Home       Members    Calendar    Who's On

Add to briefcase

Big Data Expand / Collapse
Posted Sunday, April 15, 2012 9:45 PM



Group: General Forum Members
Last Login: Friday, October 9, 2015 12:21 AM
Points: 1,882, Visits: 2,965
Comments posted to this topic are about the item Big Data

Msg 8134, Level 16, State 1, Line 1
Divide by zero error encountered.
Post #1283862
Posted Monday, April 16, 2012 5:23 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Tuesday, November 4, 2014 4:37 PM
Points: 33, Visits: 418
It'll go the route every other new shiny thing in tech goes, I would imagine:

1) There'll be a proliferation of home-rolled cobbled together codes to perform big data gathering, storage, access, and analysis.

2) Some bright CS/Math Ph.D. student and/or prof will prove some basic theorems along these lines regarding efficiency, integrity, and the like.

3) The programming world will coalesce around that foundational work.

4) Frameworks will begin to appear based that work, resulting in standardization of the topic.

4a) There will be non-stop religious wars of the form "zomg dotnet suxxors for this! clearly *nix is the only reasonable platform to do this on!".

5) People will say it's no big deal, and start looking for the next new shiny thing.

6) Throughout the entire process, there will be a group that nay-sayers the entire investigation.
Post #1284018
Posted Monday, April 16, 2012 7:41 AM



Group: General Forum Members
Last Login: Today @ 10:14 AM
Points: 2,890, Visits: 6,972
In the past most data orginated in a database because it was deemed important enough by users that they entered it manually (patient charts, student records, inventory, etc.).

Today, the bulk of "big data" is stuff like web click stream analysis, phone call records, highway traffic monitoring, serialized application objects, etc. Most of it (% of total volume) is derivative, duplicated, and mundane. We have to slog through gigabytes and terabytes of digital.. detritus.. to produce little nuggets of relevent business intelligence.

You are standing in an open field west of a white house, with a boarded front door.... Opening the small mailbox reveals a leaflet.

> read leaflet

"ZORK is a game of adventure, danger, and low cunning. In it you will explore some of the most amazing territory ever seen by mortals."
Post #1284111
Posted Monday, April 16, 2012 8:49 AM

SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Monday, November 16, 2015 9:16 AM
Points: 4,786, Visits: 3,926
For better or worse, big data is my daily bread. We are staging over 150 million rows a day, which I think is "big." I have been warned that it will be 250 million a day before the end of this year.

Big data is not the future, it is the present.
Post #1284172
Posted Tuesday, April 17, 2012 4:10 PM
Hall of Fame

Hall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of Fame

Group: General Forum Members
Last Login: Yesterday @ 3:13 PM
Points: 3,070, Visits: 2,240
What I've found so far is that Big Data is a nice vague term so loved by IT vendors rather like "Cloud".

There is Big Data as in vast amount of data (either in number of records or in storage volume) and the challenges that go with that.
There is Big Data as in a big technical challenge to mine because it is not in a form that traditional RDBMS were designed to handle.

The latter required "Big Brains" to figure out how to get meaning from the verbal diarrhea and literary dysentry from social media sites.

then there is the combination of the two.

Some of the web analytics and click-stream data can be huge in terms of records but not necessarily as big as you think it would be in terms of TB.

As ever one of the hardest challenges is to get some meaningful requirements out of customers for the big data. They kind of know what they want but can't really articulate it.

If you have big data in the cloud then what sort of queries are you going to run against it? If it is analyze a humungous amount of data and bring back a simple and small answerset then you are in with a chance.
If it is bring back a large data set then that is going to sting a bit.

To make this work well or indeed at all database guys and storage/infrastructure guys are going to have to work so closely together people are going to think it is possible for Siamese twins to be born in different families!

LinkedIn Profile
Newbie on
Post #1285342
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse