SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Normal Data


Normal Data

Author
Message
Steve Jones
Steve Jones
SSC Guru
SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)

Group: Administrators
Points: 602364 Visits: 21101
Comments posted to this topic are about the item Normal Data

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
brian.fine 41497
brian.fine 41497
SSC Rookie
SSC Rookie (45 reputation)SSC Rookie (45 reputation)SSC Rookie (45 reputation)SSC Rookie (45 reputation)SSC Rookie (45 reputation)SSC Rookie (45 reputation)SSC Rookie (45 reputation)SSC Rookie (45 reputation)

Group: General Forum Members
Points: 45 Visits: 96
Boy, is that ever true about feedback. I've taught computer classes around the country for most of the last 25 years. The negative feedback forms that I've received I could see as I was reading your column.

I must have several thousand positive ones and even a couple I've framed, however, when I close my eyes I can clearly see the negs.
patrickmcginnis59 10839
patrickmcginnis59 10839
SSCoach
SSCoach (19K reputation)SSCoach (19K reputation)SSCoach (19K reputation)SSCoach (19K reputation)SSCoach (19K reputation)SSCoach (19K reputation)SSCoach (19K reputation)SSCoach (19K reputation)

Group: General Forum Members
Points: 19341 Visits: 7991
One of my favorite stories about decisions based on data!

Wald applied his statistical skills in World War II to the problem of bomber losses to enemy fire. A study had been made of the damage to returning aircraft and it had been proposed that armor be added to those areas that showed the most damage. Wald's unique insight was that the holes from flak and bullets on the bombers that did return represented the areas where they were able to take damage. The data showed that there were similar patches on each returning bomber where there was no damage from enemy fire, leading Wald to conclude that these patches were the weak spots that led to the loss of a plane if hit, and that must be reinforced. This is still considered today seminal work in the then-fledgling discipline of operational research.

http://en.wikipedia.org/wiki/Abraham_Wald

to properly post on a forum:http://www.sqlservercentral.com/articles/61537/
Eric M Russell
Eric M Russell
SSC Guru
SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)

Group: General Forum Members
Points: 108453 Visits: 14551
They're somehow assigning more weight to the negative rating then the all the others. There's perhaps some value here in doing so, since there might be a legitimate complaint, but often we need to discard this one form as an outlier.

If an attendee gives a negative rating (one or two on a scale of five), then they should provide related comments. For example, if The speaker didn't present well; then how exactly? Was it the volume of his voice, the quality of the slides, or perhaps the inappropriateness of an opening joke?


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
GoofyGuy
GoofyGuy
SSCertifiable
SSCertifiable (5K reputation)SSCertifiable (5K reputation)SSCertifiable (5K reputation)SSCertifiable (5K reputation)SSCertifiable (5K reputation)SSCertifiable (5K reputation)SSCertifiable (5K reputation)SSCertifiable (5K reputation)

Group: General Forum Members
Points: 5019 Visits: 971
... there might be a legitimate complaint, but often we need to discard this one form as an outlier.

Chances are it seemed legitimate to the person who wrote the complaint.

As an IT manager, I found I dismiss 'outlier complaints' - from my staff and from my customers - at my own risk. There is often wisdom to be had from such complaints.

I'd rather have the one complainer than a thousand yes-men.
Iwas Bornready
Iwas Bornready
SSC Guru
SSC Guru (65K reputation)SSC Guru (65K reputation)SSC Guru (65K reputation)SSC Guru (65K reputation)SSC Guru (65K reputation)SSC Guru (65K reputation)SSC Guru (65K reputation)SSC Guru (65K reputation)

Group: General Forum Members
Points: 65318 Visits: 886
I love looking at data, seeing if I can find patterns, or hunting down the odd duck, and then relating it to the business.
Eric M Russell
Eric M Russell
SSC Guru
SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)

Group: General Forum Members
Points: 108453 Visits: 14551
Our ETL process ingests data from over a hundred external clients, and what is accepted as "normal" is a constant debate. There is a pre-production load step called AutoCertification that performs record count, cardinality, and standard deviation queries before proceeding to load. If a dataset is outside a configured threshold, then it's held back in stage, flagged, and an alert will show in the ETL monitoring dashboard. If the data analyst clears an alert, then at that point the standard, min and max can be edited, thus defining a "new normal" for that specific data source going forward.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
AZJim
AZJim
Ten Centuries
Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)

Group: General Forum Members
Points: 1238 Visits: 308
I think this is an important post for every DBA to read. I have been a DBA for almost 30 years (started way back in the ancient days on the mainframe). DBAs are notorious in thinking that averages are as reliable as moral codes chiseled in stone. One distribution that didn't get mentioned in the article is bimodal. Basically your distribution diagram looks like a two-humped camel. If you have data like this, an average is as meaningful as the meeting minutes from the local Liars' Club.

We should all be grateful that the DBMS vendors have made the Optimizer aware of the statistics meta data found in the system tables. For example, you might think that an index will fix a performance problem when searching on a column that has 100 unique values. The problem is, three of the values make up 95 percent of the data. Statistics will help for such non-uniform distribution. If you are searching for one of the three values, the Optimizer won't use the index (assuming your statistics are current). On the other hand, if you are searching for one of the rarely populated values, you can get enormous amount of filtering/selectivity.

The rule here is to know your data, run reorgs/rebuild regularly on your indexes, and run statistics religiously.
Steve Jones
Steve Jones
SSC Guru
SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)

Group: Administrators
Points: 602364 Visits: 21101
GoofyGuy (1/28/2015)
... there might be a legitimate complaint, but often we need to discard this one form as an outlier.

Chances are it seemed legitimate to the person who wrote the complaint.

As an IT manager, I found I dismiss 'outlier complaints' - from my staff and from my customers - at my own risk. There is often wisdom to be had from such complaints.

I'd rather have the one complainer than a thousand yes-men.


Often, not always.

The complainers do sometimes point out things that need to be handled, but in my experience across hundreds of talks, they usually don't. Often they complain about the agenda they didn't read, or the light in the room, which isn't helpful to me, and often can't be changed. Or the audio when they sat at the front (loud) or back (soft).

I read every comment, and judge them. Often they're discarded because they don't make sense.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Sigerson
Sigerson
Hall of Fame
Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)

Group: General Forum Members
Points: 3109 Visits: 1246
patrickmcginnis59 10839 (1/28/2015)
One of my favorite stories about decisions based on data!

Wald applied his statistical skills in World War II to the problem of bomber losses to enemy fire. A study had been made of the damage to returning aircraft and it had been proposed that armor be added to those areas that showed the most damage. Wald's unique insight was that the holes from flak and bullets on the bombers that did return represented the areas where they were able to take damage. The data showed that there were similar patches on each returning bomber where there was no damage from enemy fire, leading Wald to conclude that these patches were the weak spots that led to the loss of a plane if hit, and that must be reinforced. This is still considered today seminal work in the then-fledgling discipline of operational research.

http://en.wikipedia.org/wiki/Abraham_Wald


I loved this! Thanks for including it. Now, if I could only learn to code more like this.

Sigerson

"No pressure, no diamonds." - Thomas Carlyle
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum








































































































































































SQLServerCentral


Search