Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase ««12

Data Decisions or Instinct? Expand / Collapse
Author
Message
Posted Thursday, July 24, 2014 7:47 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Today @ 3:08 PM
Points: 2,533, Visits: 7,103
From a slightly humours point of view, often there is quite a resemblance to Chinese Whispers in Data Collection, Processing and Analysis processes, which in turn makes one at least question any decisions or findings based on the "data".
Post #1595897
Posted Thursday, July 24, 2014 8:04 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: 2 days ago @ 6:11 AM
Points: 25, Visits: 392
Chris Cradock (7/24/2014)

Data is never going to tell you what to do directly. For that you'd need an idea.
Once you have the idea the next part is collecting the evidence to tell you its a good one.
If you have several 'competing' ideas the evidence can then tell you which one is the best of the bunch.



The problem with ideas is that they are going to impose a narrative on the data. If you are not careful you can impose a false narrative by collecting the wrong data, after data collection you also have the same problem in that a false narrative can be imposed by selective interpretation of the data.

It can be the difference between:

'Can we get data to prove X?' and 'What is this data telling us?'

You said: 'Once you have the idea the next part is collecting the evidence to tell you its a good one.'

Often you are better to look for data that will prove your idea is a bad one - try to challenge your hypothesis rather than support it.
Post #1595901
Posted Thursday, July 24, 2014 9:38 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Yesterday @ 3:11 PM
Points: 31,368, Visits: 15,837
OCTom (7/24/2014)
Steve,

Does this really apply to data professionals? I have been a programmer and data provider since 1982 and I have never been asked to interpret the data. It was always someone else's job to do that. I provide the data, the reports, and the Excel spreadsheets. I think that it is dangerous for the same person who is providing the data to be making decisions based on it. I just have to make certain that the data is correct and has not been corrupted, and, that it is reported correctly.


Maybe some, maybe not others. I've often worked with business people to identify patterns and understand data. Sometimes it's a back and forth as I find business people I've worked with often haven't understood how to interpret large sets of data, or how averages, counts, distribution, and other statistical measures impact things. On the other hand, I don't always understand the importance or impact of a particular piece of data, and I can work with them to better understand how to structure things.

As an example, years ago (far too many, like 18), we were trying to determine how to plan for inventory of a few products. Demand can vary, but the salesman suspected some broad patterns. However it wasn't quite apparent from the data. I would look at patterns, go talk with them, get feedback on what they saw based on my patterns, and eventually we could make a decision on how to predict a portion of our demand (70% or so). Then the ordering people would need to apply their knowledge for the rest.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1595942
Posted Thursday, July 24, 2014 9:46 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Yesterday @ 3:11 PM
Points: 31,368, Visits: 15,837
SQLRNNR (7/24/2014)

While stories are great, I think the story should complement the data. People should be able to use their judgement and make informed decisions and many times that may involve relying on the gut feeling.


Yes, though the gut needs some data support. Not 100%, but no 10% either. I'd hope for a preponderance of support, at least 60-70% supported by data.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1595944
Posted Thursday, July 24, 2014 9:55 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: 2 days ago @ 12:50 PM
Points: 2,500, Visits: 1,586
OCTom (7/24/2014)
Steve,

Does this really apply to data professionals? I have been a programmer and data provider since 1982 and I have never been asked to interpret the data. It was always someone else's job to do that. I provide the data, the reports, and the Excel spreadsheets. I think that it is dangerous for the same person who is providing the data to be making decisions based on it.


Tom, I agree that no decision should be made by the one who gathers and supplies the data without the work of others. However, in the decision process the one gathering the data is often crucial to the evaluation of that data. Any filtering, aggregation, normalization, and other valid or required manipulation of the data to become information needs to be explained to those pouring over it.

I have also been in the business for some time, and I have gathered both data and decision packages. I have often been involved in the decisions or at least in the discussions that led to the decisions. That should not be rare, but common.

But I have never been nor will I ever be the only one reviewing the data and making the decision. After decades of being in this and like business I have, like you, considerable knowledge of the processes, practices, validity, and quality of the data. When we were a rookies making a report, we followed certain instructions and did as we were told. Now after the years of experience I would hope that we are exercising the knowledge we gained concerning the business that our contribution is far greater, and our opinion considered. That is what we do.

M.


Not all gray hairs are Dinosaurs!
Post #1595947
Posted Thursday, July 24, 2014 10:13 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: 2 days ago @ 12:50 PM
Points: 2,500, Visits: 1,586
Steve,

In reading both your post and the reference article, I find them both true and accurate. There are those who will look at data as the hallowed scrolls that should never be questioned. I remember years back someone saying "the, data, the whole data, and nothing but the data!" In addition, I think that was followed by "So Help Me Codd." However, that was long ago and about modeling.

However, it is not really just the data. Trends within the data, trends in the metadata concerning use and usability tend to help us think more of the process and what we are doing. Further, there is a knowing that something is not right, or trending in a different direction and things could be made better. This is not always a wild dream as some things are, but it often is some specific out of the box thinking that might just revolutionize what we are looking at.

We have to have the statisticians and gurus to maintain the status quo and advance it as far as it can be. Nevertheless, we must also have those who have the gut feel that it can be made better. Some of the time, you know in your knower that things need to changes, even if the data has not told the story yet. Your knower and experience can forewarn us that change is necessary to better meet the future that is headed your way.


M.


Not all gray hairs are Dinosaurs!
Post #1595953
Posted Thursday, July 24, 2014 11:44 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Yesterday @ 3:11 PM
Points: 31,368, Visits: 15,837
Miles Neale (7/24/2014)
Steve,

In reading both your post and the reference article, I find them both true and accurate.


Thanks


We have to have the statisticians and gurus to maintain the status quo and advance it as far as it can be. Nevertheless, we must also have those who have the gut feel that it can be made better.


Agree. We need some collaboration between multiple disciplines and groups to make better decisions.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1595972
Posted Thursday, July 24, 2014 12:43 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: 2 days ago @ 5:55 AM
Points: 155, Visits: 285
It should also be remembered that the methodology in gathering and analysing the data is as important as the data itself (Google Flu Trends anybody?).

____________
Just my $0.02 from over here in the cheap seats of the peanut gallery - please adjust for inflation and/or your local currency.
Post #1595992
Posted Thursday, July 24, 2014 3:52 PM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Yesterday @ 6:20 AM
Points: 2,923, Visits: 1,874
RP1966 (7/24/2014)
In relation to this question I would recommend everyone read at least the first section of 'Thinking, Fast and Slow' by Daniel Kahneman.


+1

I see people using data to strengthen the case for something they believe in and not using data to as the source of those beliefs.


LinkedIn Profile
Newbie on www.simple-talk.com
Post #1596037
Posted Thursday, July 24, 2014 6:06 PM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Yesterday @ 7:28 AM
Points: 7,179, Visits: 15,775
Miles Neale (7/24/2014)
Steve,

In reading both your post and the reference article, I find them both true and accurate. There are those who will look at data as the hallowed scrolls that should never be questioned. I remember years back someone saying "the, data, the whole data, and nothing but the data!" In addition, I think that was followed by "So Help Me Codd." However, that was long ago and about modeling.

However, it is not really just the data. Trends within the data, trends in the metadata concerning use and usability tend to help us think more of the process and what we are doing. Further, there is a knowing that something is not right, or trending in a different direction and things could be made better. This is not always a wild dream as some things are, but it often is some specific out of the box thinking that might just revolutionize what we are looking at.

We have to have the statisticians and gurus to maintain the status quo and advance it as far as it can be. Nevertheless, we must also have those who have the gut feel that it can be made better. Some of the time, you know in your knower that things need to changes, even if the data has not told the story yet. Your knower and experience can forewarn us that change is necessary to better meet the future that is headed your way.


M.


And that's where the chicken and egg part of this conversation comes in. In many cases the "gut feeling" you're sensing is related to data you've previously digested: your "knower" has detected a much higher level pattern or the fact that the old pattern no longer plays, and the current data you're seeing just hasn't caught up (or isn't designed to pick up on the new pattern).

My first boss was like this: her "gut" was based on 30 years in the trenches, so it was really grounded in a much larger body of facts than I could easily muster. She couldn't necessarily justify how she got to many of her conclusions, but put to the test, they were dead on most times.


----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
Post #1596051
« Prev Topic | Next Topic »

Add to briefcase ««12

Permissions Expand / Collapse