October 1, 2008 at 11:22 am
Seconded that DAS is not a viable solution for what you are trying to accomplish. Performance would be horrendous, and I would hate to think of the number of drives you would be using (and all being at less than optimal performance as their size would need to be extremely large).
Do you have a handle on what kind of hardware that you will be running on? That can make a lot of decisions for you.
I run a multi-terabyte datamart, don't have the best hardware, and have it connected to a SAN. With that I can load over 24 million rows and create a full set of rollup data in under three hours, it's all about design and making the best of what you have. I would try really hard to get on a SAN though. Really really hard.:Whistling:
October 8, 2008 at 11:47 am
Hi
Considering 6TB would be the volume of data/year, i am looking at about 18GB/day (540GB/month) and 6.480TB/year; We are planning to have log shipping; The problem with data loading is that the Transaction log increases in size and (correct me if i am wrong) unless it is backed up it does not release / resize itself; Since there would be log shipping it should release / resize itself; A test load of data revealed (not on the actual hardware but on a different dev box) a rate of 190MB/min i am looking at a load time of approx 97 mins; So scheduling for log shipping every 120 mins should ease it; Even if i schedule it every 10 mins it should still freeze the log and still apply it on the standby. But if there is any error while loading then cleaning up the production data becomes a messy issue. I was reading on the web about someone suggesting to load the data to a temporary table and do a exchange partition (I am brand new to this SQL server partition , primarily a Oracle DBA and i am not in a position to validate the above suggestion since i am still learning) sort of stuff. In the light of this i would like to ask the Terabyte Gurus as to
1. how you people are handling the load with a standby in place
2. how much of daily load you are doing 3. How much does the log grow for the load (see my values below)
3. how you handle the cleanup / rollback process if applicable.
4. What happens to the exchange partition if the partition is already having data or do i need to have subpartitions by day and merge it into the superior partition which might by month or quarter(if that is the correct word)?
5. Does SQL Server provide for SubParitions?
6. How do i select data from a particular partition (In Oracle you can say select .. from table partition (px) where ..).
Sorry for the long posting. I am still reading on this partition stuff. I did create File groups , Partition Scheme ,Partition Function and tables associated with it and loaded approx 600MB of data yesterday. The log file grew to 1.8GB (hmm thrice the size)?
Thanks in Advance
October 8, 2008 at 12:15 pm
Use:
- professional help ( SQLserver and shrink 😉 )
Both should be good
Some more info needed:
- OLTP / OLAP ?
- 15TB plus .... how many processors (cores ?), how many TB RAM ?
- implement ALL best practises for HW configuration as wel as for SQLServer configuration / db design, ..
- decent backup solution ( softwares that compress on the fly)
- application expectations (avg response times, avg number of queries, avg number of connections per minute/hour/day)
- performance monitoring solution (what is going on on the instance, what's the baseline, what's the evolution, .... )
(e.g. Quest foglite)
- Read only partitions ?
- single table (partitioned) filegroups ?
- distribution of filegroups / files per partition ?
- Decent sizing of tempdb (size, number of files, number of disks ( luns ) )
- Can you use SQL 2008 ? Manly because of the management framework !
Johan
Learn to play, play to learn !
Dont drive faster than your guardian angel can fly ...
but keeping both feet on the ground wont get you anywhere :w00t:
- How to post Performance Problems
- How to post data/code to get the best help[/url]
- How to prevent a sore throat after hours of presenting ppt
press F1 for solution, press shift+F1 for urgent solution 😀
Need a bit of Powershell? How about this
Who am I ? Sometimes this is me but most of the time this is me
Viewing 3 posts - 16 through 17 (of 17 total)
You must be logged in to reply to this topic. Login to reply