SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Too Much Data


Too Much Data

Author
Message
Steve Jones
Steve Jones
SSC Guru
SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)

Group: Administrators
Points: 147265 Visits: 19434
Comments posted to this topic are about the item Too Much Data

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Eric M Russell
Eric M Russell
One Orange Chip
One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)

Group: General Forum Members
Points: 29174 Visits: 11514
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage. For the most part, I think that poor data modeling is the primary problem. Many of the data modeling decisions that developers are making when designing data warehouses actually result in worse (not better) performance.
For example, I've seen 'Person' tables that contain the full address and multiple phone numbers. Do your research before deciding to denormalize a table for performance reasons.
I've seen tables containing various integer columns where the datatypes are all an 8 byte BigInt. For example: Sex BigInt, MaritalStaus BigInt, etc. The guy who did this explained the reasoning as follows: "because SQL Server is running on a 64bit operating system, it's more efficient to use 64bit integers". It was a specious claim that couldn't be proven, and even if it were marginally true, the data pages from this table were still comsuming more I/O and memory.
Also, another big one is date/time values contained in VarChar columns, which not only consumes more resources, but it's problematic in terms of performance and data quality as well.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
Abrar Ahmad_
Abrar Ahmad_
SSC Eights!
SSC Eights! (900 reputation)SSC Eights! (900 reputation)SSC Eights! (900 reputation)SSC Eights! (900 reputation)SSC Eights! (900 reputation)SSC Eights! (900 reputation)SSC Eights! (900 reputation)SSC Eights! (900 reputation)

Group: General Forum Members
Points: 900 Visits: 1305
Eric M Russell (11/13/2012)
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage. For the most part, I think that poor data modeling is the primary problem. Many of the data modeling decisions that developers are making when designing data warehouses actually result in worse (not better) performance.
For example, I've seen 'Person' tables that contain the full address and multiple phone numbers. Do your research before deciding to denormalize a table for performance reasons.
I've seen tables containing various integer columns where the datatypes are all an 8 byte BigInt. For example: Sex BigInt, MaritalStaus BigInt, etc. The guy who did this explained the reasoning as follows: "because SQL Server is running on a 64bit operating system, it's more efficient to use 64bit integers". It was a specious claim that couldn't be proven, and even if it were marginally true, the data pages from this table were still comsuming more I/O and memory.
Also, another big one is date/time values contained in VarChar columns, which not only consumes more resources, but it's problematic in terms of performance and data quality as well.


Noticed same in may environment, even the data type of transactional database column was set to Numeric(11,0) or +! When inquired the answer was, we thought of that our bigint or likewise data types may cannot serve the future load of 1-2 TB of data! Which in-fact not possible but the reason exists Crazy
Jo Pattyn
Jo Pattyn
SSCertifiable
SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)

Group: General Forum Members
Points: 7220 Visits: 10018
Save the environment and enable free compression in the standard editions.
Steve Jones
Steve Jones
SSC Guru
SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)SSC Guru (147K reputation)

Group: Administrators
Points: 147265 Visits: 19434
Jo Pattyn (11/14/2012)
Save the environment and enable free compression in the standard editions.


I wish. Not sure we'll see this anytime soon.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Bhuvnesh
Bhuvnesh
SSChampion
SSChampion (13K reputation)SSChampion (13K reputation)SSChampion (13K reputation)SSChampion (13K reputation)SSChampion (13K reputation)SSChampion (13K reputation)SSChampion (13K reputation)SSChampion (13K reputation)

Group: General Forum Members
Points: 13358 Visits: 4077
Eric M Russell (11/13/2012)
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage.
on the contrary , In my last company we had started to denormailzed the table . use the flat table concept at some places.. removing the foreign key an d maintain it logically and with application code.Also partitoning was being used extensively. we moved to replication from mirroring environment.

-------Bhuvnesh----------
I work only to learn Sql Server...though my company pays me for getting their stuff done;-)
Eric M Russell
Eric M Russell
One Orange Chip
One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)One Orange Chip (29K reputation)

Group: General Forum Members
Points: 29174 Visits: 11514
Bhuvnesh (12/26/2012)
Eric M Russell (11/13/2012)
Practically all of the really large databases I've worked with in the past could have benefitted from better normalization and data type usage.
on the contrary , In my last company we had started to denormailzed the table . use the flat table concept at some places.. removing the foreign key an d maintain it logically and with application code.Also partitoning was being used extensively. we moved to replication from mirroring environment.

The problem with enforcing referential integrity at the application layer is the potential for bad data to fall through the cracks. For example, the primary source for data may be the application, but there may also be data originating from ETL processes. That means the data validation logic has to be coded in multiple locations and perhaps even maintained by seperate developers.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
Jeff Moden
Jeff Moden
SSC Guru
SSC Guru (216K reputation)SSC Guru (216K reputation)SSC Guru (216K reputation)SSC Guru (216K reputation)SSC Guru (216K reputation)SSC Guru (216K reputation)SSC Guru (216K reputation)SSC Guru (216K reputation)

Group: General Forum Members
Points: 216390 Visits: 41986
From the article
The solutions aren't new and innovative; they're back to basics ideas.


It's funny (and tremedously rewarding in more ways than one) how it almost always comes back to that.

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
If you think its expensive to hire a professional to do the job, wait until you hire an amateur. -- Red Adair

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search