December 18, 2008 at 10:34 am
nice eitorial today Phil. we recently experienced that very scenario, where the design did not scale up very well. when we were finally called into action to investigate, our reaction was "...surprised that this ever worked on any scale..." needless to say, a little extra unit testing would have been nice.
and the system is (probably) going to be stressed more in the next few months, hopefully someone finds the time to do some testing before they flip another switch.
----------------------
https://thomaslarock.com
December 18, 2008 at 1:37 pm
The statement has been made in essence that one cannot assume to know the real size of an intended data system during dev and test.I must disagree. The business of the client almost always defines the needs, whether those in charge recognize those needs or not. Again, the business, not the representative or owner of the business, defines the needs. In both examples given as examples in the previous post, one could have managed a small development spike (borrowing from Extreme lingo here) to assess an average of at least a percentage of the share involved.For example: I was involved in a 6 month contract to manage the entire data flow from man-hours, to production costs, to delivery bill-of-lading documents, for a specific buy-out and deployment of product to over 300 new locations. The company had no idea even if it would be a profitable venture, let alone how they would manage production output. So we took a sampling from about 10% of the locations, obtained data for those, assesed cost on each basic service, associated man-hours for each service, and was able to report the expected value, cost, and man-hours of the entire job, let alone the expected size of the data project itself. The data applied to both the database and controlled the actual output flow of the work, and gave the client specific estimates on what man-power was required to do the job.One can and must be able to estimate the general size of the project. One may not be able to put a real count on number of rows or relative disk size. However one, with enough experience should very well know the sheer scope of the information, given the client's business size and expected growth. Test beyond that margin. When in doubt, assume it will be larger than anticipated.So, instead of hard count estimates, I tend to lean on about 5 categorical sizes of small, medium, large, very large, and horrendously large. This consideration is given to every table in the test system. Some tables are lookup and as such remain small even in systems that are overall horrendously large. Some tables, such as transactional history on frequently changed data, are expected to be extremely high row count, but relatively narrow, so query time is affected, but inserts are not (especially frequent in many-to-many relationships). And then there are some tables that are so wide that one even considers breaking them into sections.The point is, there is no reason to believe one cannot make at least a very good educated decision about expected size of database, both in count and physical disk space, and add a little (or a lot) extra just to make sure the ironworks float. Know your client. Know the industry. Know generics about business data in general. If one does not know these, one should not be leading a project.The show stopper, more often than not, is that clients tend to fault on the side of too skimpy, especially when it comes to hardware, and then on both great DBA and Admin management. Many clients tend to not want to spend anything until it is failing. I have seen a little too often the lofty goals of a client get mangled by the infrastructure they did not think was critical, thinking they'd just hire someone when they got into a jam. It takes years off their growth margins, and lost opportunity costs.As an aside, in the example I gave above, that client had actually lost a major contract because their internal data guys could not give them the answers that would prove the job manageable, cost-worthy, nor the structure of the management flow of data. They simply could not see outside of the box of their current billing and production systems. A year and a half later, another similar job came up, and one of the owners decided get help. Within two weeks we gave them what they needed to make the necessary decisions, and get the job done. My contract was immediately extended from 2 week assessment to the full 6 months of deployment to manage the data flow. Trucks were loaded and delivered by the new system lading orders. They made millions. We made a year's salary in 6 months. It was win-win... but their internal IT people were never happy about it, but could say nothing.
December 19, 2008 at 5:03 am
I didn't intend to say that you can't get a good estimate of the size of the data/transactions/etc for any projects. Just that, all too often, the estimates will be low.
I've encountered situations where it was categorically stated during design that something was out-of-scope and then, a few months after go-live, management decided to bring it into scope. Also, I've had cases where a detailed analysis and projection of volmes based on existing and anticipated business is done and then, after implementation, unexpected new business turns up which throws out all the projections.
Basic rule seems to be to assume that all volumes are significantly underestimated!
Derek
December 19, 2008 at 6:15 am
Yes. All volumetrics are an educated guess. Well, they were when I did them. Multiply by the contingency factor and you have a size to use all the way from unit testing through to release!
Bless you all for some really fascinating contributions to the discussion. It is a shame we can't pass round a virtual bottle of 'seasonal' port and toast our virtual friendships.
Best wishes,
Phil Factor
December 19, 2008 at 6:38 am
Merry Chistmas to you Phil....can I say that?
--Shaun
Hiding under a desk from SSIS Implemenation Work :crazy:
December 19, 2008 at 6:57 am
Our office will be closed from today until 5th January so I probably wont' be partaking of any more discussions this year, so...
I, too, would like to thank Steve and all the many regular (and irregular) contributors who've given me much food for thought about SQL Server and many other topics.
Merry Christmas (or Seasons Greetings) to all and let's hope 2009 is better year for everyone.
Derek
December 19, 2008 at 7:16 am
December 19, 2008 at 7:27 am
Merry Christmas and peace and hope for the new year!
December 19, 2008 at 7:29 am
Cheers, everyone!
December 19, 2008 at 8:21 am
I'm envious, closed until the 5th?! We were lucky to get Friday the 26th off: City Council screwed up last year when they set the holiday schedule and did not include Friday. Some time during the year they slip-streamed the holiday into the schedule.
Best wishes, everyone, for a Cool Yule! And I heartily second the motion for 2009 being better than 2008. :hehe:
-----
[font="Arial"]Knowledge is of two kinds. We know a subject ourselves or we know where we can find information upon it. --Samuel Johnson[/font]
December 19, 2008 at 8:31 am
Ha, the only days I get off are when I can bribe, err, convince Phil to help out.
Merry Christmas, Happy Holidays, I'll be here and we'll see you on the other side.
December 19, 2008 at 8:35 am
I'm the new kid on the block at my job this year, so I get to "keep the lights on"... Have some 'seasonal Port' for me, would you?
Back to the conversation though. I think there's a bt of a disconnect giong on. While it may be difficult to assess what the final size may look like in some apps, coding to meet or beat certain specific metrics is important. I've been striving to get these very things put into the specs themselves, so that the usually hidden assumption isn't hidden anymore, and you can actually start coding for performance as well as for features.
Ultimately - whether you decide to pick today's size, some estimate of where the app will be in x years, or just pick a number out of a hat, putting it "on paper" at least gives you something to go back to when whatever gets built isn't adequate anymore.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
December 19, 2008 at 8:42 am
Final thought -
In the old days a target was something to shoot for.
Today a target is something that must be achieved, being close is branded as failure.
--Shaun
Hiding under a desk from SSIS Implemenation Work :crazy:
December 19, 2008 at 8:50 am
Matt Miller (12/19/2008)
I'm the new kid on the block at my job this year, so I get to "keep the lights on"...
Bummer. Hope nothing breaks.
This is the first year for a while where I'm not on call over Christmas and New Year.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
December 19, 2008 at 8:59 am
GilaMonster (12/19/2008)
Matt Miller (12/19/2008)
I'm the new kid on the block at my job this year, so I get to "keep the lights on"...Bummer. Hope nothing breaks.
This is the first year for a while where I'm not on call over Christmas and New Year.
Same for me - no more christmas support (though I am in on the 29th, ran out of holidays)
This is the first year in 15 I have used all of my holiday days 😀
Gail - you will probably be on the beach at christmas anyway!
--Shaun
Hiding under a desk from SSIS Implemenation Work :crazy:
Viewing 15 posts - 16 through 30 (of 32 total)
You must be logged in to reply to this topic. Login to reply