Log in
::
Register
::
Not logged in
Home
Tags
Articles
Editorials
Stairways
Forums
Scripts
Videos
Blogs
QotD
Books
Ask SSC
SQL Jobs
Training
Authors
About us
Contact us
Newsletters
Write for us
Recent Posts
Recent Posts
Popular Topics
Popular Topics
Home
Search
Members
Calendar
Who's On
Home
»
SQLServerCentral.com
»
Editorials
»
Too Much Data
Too Much Data
Rate Topic
Display Mode
Topic Options
Author
Message
Steve Jones - SSC Editor
Steve Jones - SSC Editor
Posted Tuesday, November 13, 2012 12:33 AM
SSC-Dedicated
Group: Administrators
Last Login: Today @ 11:20 AM
Points: 31,437,
Visits: 13,752
Comments posted to this topic are about the item
Too Much Data
Follow me on Twitter:
@way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
Post #1383959
Eric M Russell
Eric M Russell
Posted Tuesday, November 13, 2012 8:16 AM
Ten Centuries
Group: General Forum Members
Last Login: Yesterday @ 9:21 AM
Points: 1,164,
Visits: 3,340
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage. For the most part, I think that poor data modeling is the primary problem. Many of the data modeling decisions that developers are making when designing data warehouses actually result in worse (not better) performance.
For example, I've seen 'Person' tables that contain the full address and multiple phone numbers. Do your research before deciding to denormalize a table for performance reasons.
I've seen tables containing various integer columns where the datatypes are all an 8 byte BigInt. For example: Sex BigInt, MaritalStaus BigInt, etc. The guy who did this explained the reasoning as follows: "because SQL Server is running on a 64bit operating system, it's more efficient to use 64bit integers". It was a specious claim that couldn't be proven, and even if it were marginally true, the data pages from this table were still comsuming more I/O and memory.
Also, another big one is date/time values contained in VarChar columns, which not only consumes more resources, but it's problematic in terms of performance and data quality as well.
"Wise people understand the 10,000 things without going to each one.
They know them without having to look at each one,
and they transform all without acting on each one." - The Tao Te Ching: Verse 47
Post #1384122
Abrar Ahmad_
Abrar Ahmad_
Posted Tuesday, November 13, 2012 10:30 PM
SSC Veteran
Group: General Forum Members
Last Login: Saturday, May 18, 2013 5:49 AM
Points: 202,
Visits: 1,043
Eric M Russell (11/13/2012)
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage. For the most part, I think that poor data modeling is the primary problem. Many of the data modeling decisions that developers are making when designing data warehouses actually result in worse (not better) performance.
For example, I've seen 'Person' tables that contain the full address and multiple phone numbers. Do your research before deciding to denormalize a table for performance reasons.
I've seen tables containing various integer columns where the datatypes are all an 8 byte BigInt. For example: Sex BigInt, MaritalStaus BigInt, etc. The guy who did this explained the reasoning as follows: "because SQL Server is running on a 64bit operating system, it's more efficient to use 64bit integers". It was a specious claim that couldn't be proven, and even if it were marginally true, the data pages from this table were still comsuming more I/O and memory.
Also, another big one is date/time values contained in VarChar columns, which not only consumes more resources, but it's problematic in terms of performance and data quality as well.
Noticed same in may environment, even the data type of transactional database column was set to Numeric(11,0) or +! When inquired the answer was, we thought of that our bigint or likewise data types may cannot serve the future load of 1-2 TB of data! Which in-fact not possible but the reason exists
Post #1384389
Jo Pattyn
Jo Pattyn
Posted Wednesday, November 14, 2012 10:27 AM
Ten Centuries
Group: General Forum Members
Last Login: Today @ 3:30 PM
Points: 1,332,
Visits: 4,325
Save the environment and enable free compression in the standard editions.
Post #1384763
Steve Jones - SSC Editor
Steve Jones - SSC Editor
Posted Wednesday, November 14, 2012 1:18 PM
SSC-Dedicated
Group: Administrators
Last Login: Today @ 11:20 AM
Points: 31,437,
Visits: 13,752
Jo Pattyn (11/14/2012)
Save the environment and enable free compression in the standard editions.
I wish. Not sure we'll see this anytime soon.
Follow me on Twitter:
@way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
Post #1384831
Bhuvnesh
Bhuvnesh
Posted Wednesday, December 26, 2012 4:06 AM
SSCrazy
Group: General Forum Members
Last Login: Tuesday, March 26, 2013 8:41 AM
Points: 2,562,
Visits: 3,451
Eric M Russell (11/13/2012)
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage.
on the contrary , In my last company we had started to denormailzed the table . use the flat table concept at some places.. removing the foreign key an d maintain it logically and with application code.Also partitoning was being used extensively. we moved to replication from mirroring environment.
-------Bhuvnesh----------
While 1 = 1 (Learning SQL....)
Click to get fast response of your post
Post #1400204
Eric M Russell
Eric M Russell
Posted Friday, December 28, 2012 7:41 AM
Ten Centuries
Group: General Forum Members
Last Login: Yesterday @ 9:21 AM
Points: 1,164,
Visits: 3,340
Bhuvnesh (12/26/2012)
Eric M Russell (11/13/2012)
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage.
on the contrary , In my last company we had started to denormailzed the table . use the flat table concept at some places.. removing the foreign key an d maintain it logically and with application code.Also partitoning was being used extensively. we moved to replication from mirroring environment.
The problem with enforcing referential integrity at the application layer is the potential for bad data to fall through the cracks. For example, the primary source for data may be the application, but there may also be data originating from ETL processes. That means the data validation logic has to be coded in multiple locations and perhaps even maintained by seperate developers.
"Wise people understand the 10,000 things without going to each one.
They know them without having to look at each one,
and they transform all without acting on each one." - The Tao Te Ching: Verse 47
Post #1400918
Jeff Moden
Jeff Moden
Posted Friday, December 28, 2012 11:34 AM
SSC-Dedicated
Group: General Forum Members
Last Login: Today @ 3:06 PM
Points: 32,930,
Visits: 26,816
From the article
The solutions aren't new and innovative; they're back to basics ideas.
It's funny (and tremedously rewarding in more ways than one) how it almost always comes back to that.
--Jeff Moden
"
RBAR
is pronounced "ree-bar" and is a "Modenism" for "
R
ow-
B
y-
A
gonizing-
R
ow".
First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."
For better, quicker answers on T-SQL questions, click on the following...
http://www.sqlservercentral.com/articles/Best+Practices/61537/
For better answers on performance questions, click on the following...
http://www.sqlservercentral.com/articles/SQLServerCentral/66909/
Post #1400995
« Prev Topic
|
Next Topic »
Permissions
You
cannot
post new topics.
You
cannot
post topic replies.
You
cannot
post new polls.
You
cannot
post replies to polls.
You
cannot
edit your own topics.
You
cannot
delete your own topics.
You
cannot
edit other topics.
You
cannot
delete other topics.
You
cannot
edit your own posts.
You
cannot
edit other posts.
You
cannot
delete your own posts.
You
cannot
delete other posts.
You
cannot
post events.
You
cannot
edit your own events.
You
cannot
edit other events.
You
cannot
delete your own events.
You
cannot
delete other events.
You
cannot
send private messages.
You
cannot
send emails.
You
may
read topics.
You
cannot
rate topics.
You
cannot
vote within polls.
You
cannot
upload attachments.
You
may
download attachments.
You
cannot
post HTML code.
You
cannot
edit HTML code.
You
cannot
post IFCode.
You
cannot
post JavaScript.
You
cannot
post EmotIcons.
You
cannot
post or upload images.
Copyright © 2002-2013 Simple Talk Publishing. All Rights Reserved.
Privacy Policy.
Terms of Use.
Report Abuse.