If you had a blank slate, what would you do...

Question

If you had a blank slate, what would you do...

Not Mine

Valued Member

Points: 64
More actions
April 9, 2007 at 6:30 pm

#95293

Let's say you have 500 GB of data coming in, and don't have any hardware or software installed yet. You have a blank slate. What hardware would you install, and what version of SQL would you install? That's the position I'm in. I don't know what to do.

Viewing 6 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply

Jeff Moden SSC Guru Points: 1004641 More actions · Answer 1

I think you're the same guy from the "other" forum that asked the same question.... if I recall correctly, you will actually receive two 500GB files from your client. That means you need a TB of data to store the initial data file, and a bit more than that to store it in a table or tables. So, you need at least 2 TB of disk space just to store the data. Then, there's indexes, keys, log files, system files, etc, etc. I'd recommend that's probably going to be a bit bigger than the really decent desktop box you spoke of before.

You also mentioned using MySQL in the other forum... I'm thinking that's probably not the best choice for that amount of data but I'm not a MySQL user, so I don't know for sure.

If the data is important, spend some money on storing it and accessing it. I know... not what you wanted to hear...

And, no, SQL Server Enterprise and Standard Editions DON'T run on a desktop... they only run on a Windows Server. You could cheat a bit with Developer Edition, but if you get caught using the Developer Edition in violation of the user license, you will lose your business when they whack you with a $200k S.P.A. fine. It's not worth the risk... find some hardware guys you trust and have them build a server for you. Don't forget the backup device(s) either...

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

David McFarland SSChampion Points: 11815 More actions · Answer 2

Do I have a blank check to go with that blank slate?

I'll second what Jeff mentioned, but it would also help greatly if we knew what you were going to do with the data, how many 9s of uptime you need, how big of a deal it would be if you lost the data, etc. Basically, is the data just going to sit there with a couple of rarely used reports on top of it, or is it going to be heavily taxed every second. Is the data a one time thing, or will there be updates? How frequently? You get the idea. The more info we have, the better we can help.

Not Mine Valued Member Points: 64 More actions · Answer 3

Thanks for your help so far.

This data is the KEY to my business. I will run some SQL script against the data, which will produce a record set. Once I have a record set, the data will get dumped into an Access table for some manual review. The SQL script will reduce the number of rows by at least 90%.

I currently have all of my data in Access, but this new client has an incredible amount of data.

SwePeso SSC-Dedicated Points: 39758 More actions · Answer 4

10% of 500GB is still 50GB, which MS Access is not designed for.

Even only 1% of the records would still be 5 GB, which is still too much for MS Access.

If you have all the data in MS Access right now, I would suspect a COMPACT operation would decrease the size of the database.

N 56°04'39.16"
E 12°55'05.25"

Steve Jones - SSC Editor SSC Guru Points: 738693 More actions · Answer 5

Good suggestions above, I'd be sure you get a server class machine, use RAID, at least RAID 5 for the data storage. If you can afford to add in 2 RAID 1 drives for the logs, I'd do that. You'll get a lot of log growth in adding the data, so you need space for that.

Also if they say 500GB now, it could grow, so the 2TB suggestion seems to make space. Could be 5 drives or so these days with 300-500GB drives. Spend the $5-6k it will require for the machine. The Dell Outlet has some great deals on large machines.