Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Big Data - Cost of acquiring skills


Big Data - Cost of acquiring skills

Author
Message
David.Poole
David.Poole
Hall of Fame
Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)

Group: General Forum Members
Points: 3668 Visits: 3113
Comments posted to this topic are about the item Big Data - Cost of acquiring skills

LinkedIn Profile

Newbie on www.simple-talk.com
Gary Varga
Gary Varga
SSCrazy Eights
SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)

Group: General Forum Members
Points: 8286 Visits: 6133
Most R&D or experimentation can be done on a "no licence cost" basis. The one cost that NEVER seems to be allowed is time. The nearest I have seen is when a project manager allows a little time for a team to get up to speed with a new technology.

One of the thoughts must be given weight when selecting software to use for experimentation, R&D or a pilot is what is the plan if it is successful? Stick with that technology? Migration to a different technology? etc.

It is not necessarily a cheap way of doing it when the management shout "Ship it!!!" and the support team refuse to support it without additional training as it is nothing they have ever supported before.

NOTE: This is not an anti-OSS rant but a piece of cautionary advice.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!
Eric M Russell
Eric M Russell
SSCarpal Tunnel
SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)

Group: General Forum Members
Points: 4580 Visits: 9502
If management allows an hour for lunch and places no restrictions on how early one can enter the office or how late one leaves, then there can always be time for study. Don't ask for time, just find it and take it.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
David.Poole
David.Poole
Hall of Fame
Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)

Group: General Forum Members
Points: 3668 Visits: 3113
Eric, although there is always tme to squeeze something in and learn I think a more formal recognition of the need for experimentation time is needed.

There has to be quite a bit of rigour and due diligence around experimentation because if an experiment is successful then the outcome can represent disruptive change.
When a disruptive change takes place there will be a minority in favour of it, probably 40% who will accept it with a reasonable amount of proof, probably 40% who will only accept it IF there is extensive proof and the rest will out-right reject it.

Normally the resisters are the domain experts who have a clear view of the magnitude of the impacts of the change. I've been on that side of the fence myself and know that going into big organisational change without the due diligence is an extremely painful experience. Shining a harsh light on an experiment can help to thrash out gaps in the methodology and thought processes behind the experiment. Equally, it can also reveal something that ends up being a really good solution to the problem that inspired the experiment in the first place.

LinkedIn Profile

Newbie on www.simple-talk.com
Gary Varga
Gary Varga
SSCrazy Eights
SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)

Group: General Forum Members
Points: 8286 Visits: 6133
Eric M Russell (4/14/2014)
If management allows an hour for lunch and places no restrictions on how early one can enter the office or how late one leaves, then there can always be time for study. Don't ask for time, just find it and take it.


Good point but I was referring to pilot projects rather than learning. Learning is easier to do piecemeal. In fact it has been proven to be a better learning technique with deeper understanding often attained. I mean more of a structured, proven assessment of use.

Big data is as much about systems design as it is data management.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!
Eric M Russell
Eric M Russell
SSCarpal Tunnel
SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)

Group: General Forum Members
Points: 4580 Visits: 9502
I've got half a dozen retired PC boxes stacked in a corner under my desk on which I'm installing a Ubuntu / Hadoop cluster. My intention is to prove whether or my skunkworks cluster can complete 3 concurrent aggregate queries on a TB of data in less time than the production SQL Server Enterprise instance that we currently use for a staging environment. This type of "project" would normally never get approved, which is just as well, because I've had to put it on the back burner for weeks at time while I focus on my real work.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
Gary Varga
Gary Varga
SSCrazy Eights
SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)

Group: General Forum Members
Points: 8286 Visits: 6133
Eric M Russell (4/14/2014)
I've got half a dozen retired PC boxes stacked in a corner under my desk on which I'm installing a Ubuntu / Hadoop cluster. My intention is to prove whether or my skunkworks cluster can complete 3 concurrent aggregate queries on a TB of data in less time than the production SQL Server Enterprise instance that we currently use for a staging environment. This type of "project" would normally never get approved, which is just as well, because I've had to put it on the back burner for weeks at time while I focus on my real work.


Glad that you have that kind of freedom. You do yourself credit using it that way.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!
David.Poole
David.Poole
Hall of Fame
Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)

Group: General Forum Members
Points: 3668 Visits: 3113
Eric M Russell (4/14/2014)
I've got half a dozen retired PC boxes stacked in a corner under my desk on which I'm installing a Ubuntu / Hadoop cluster. My intention is to prove whether or my skunkworks cluster can complete 3 concurrent aggregate queries on a TB of data in less time than the production SQL Server Enterprise instance that we currently use for a staging environment. This type of "project" would normally never get approved, which is just as well, because I've had to put it on the back burner for weeks at time while I focus on my real work.


It probably won't. If you've got an optimised structure recordset, particularly if you've got page compression on, then a scan through it will be fast.
Hadoop really comes into its own when you have much larger data volumes or you have data of a structure that doesn't naturally fit into a straight tabular format.

I was quite disappointed to find that 1 years worth of web site page impressions (roughly 1 billion records) on AWS 3 node EMR took about 3x as long as the same query on SQL Server 2005. The problem is that the type of query, the type of data and the volume of data didn't present a Hadoop shaped problem.

The beauty of your setup is that you will get to play with a genuine Hadoop cluster and learn the tricks and pitfalls.

One thing you learn quite early on is that having a process to merge smaller files up into bigger ones improves performance dramatically.

LinkedIn Profile

Newbie on www.simple-talk.com
Doctor Who 2
Doctor Who 2
SSC Veteran
SSC Veteran (287 reputation)SSC Veteran (287 reputation)SSC Veteran (287 reputation)SSC Veteran (287 reputation)SSC Veteran (287 reputation)SSC Veteran (287 reputation)SSC Veteran (287 reputation)SSC Veteran (287 reputation)

Group: General Forum Members
Points: 287 Visits: 595
I didn't know that SQL Server Developer Edition was something that one could purchase at a reduced rate. You gave the cost in British pounds - what's it cost in US dollars and where do you get it?

Rod
Eric M Russell
Eric M Russell
SSCarpal Tunnel
SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)SSCarpal Tunnel (4.6K reputation)

Group: General Forum Members
Points: 4580 Visits: 9502
Doctor Who 2 (4/18/2014)
I didn't know that SQL Server Developer Edition was something that one could purchase at a reduced rate. You gave the cost in British pounds - what's it cost in US dollars and where do you get it?

It's about US $50, but maybe even lower with academic discount pricing. Sometimes they hand it out as raffle prizes at user group meetings or in swag bags at tech conventions.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search