January 7, 2012 at 11:19 am
Comments posted to this topic are about the item On Database Migrations and Agility
Best wishes,
Phil Factor
January 7, 2012 at 11:48 am
The not-so-flattering Agile analogy (not invented by me) is like trying to build a house one-room-at-a-time (where software is the "fine carpentry and dry wall" and data structures are the "foundation and roof"). "O, wow, I really like this kitchen" says the customer, "let's add a bedroom". Seems easy if you're doing fine carpentry and drywall. However, how do you do Agile foundation work and roofing? Brings to mind the parable told by the radical rabbi about the foolish man building his house on sand. The wise man built on a solid foundation.
January 8, 2012 at 6:27 pm
It therefore always surprises me when Application programmers tell me that all 'Agile' techniques are directly applicable to database development.
Heh... what did you expect, Phil? Many application programmers and their managers have misinterpreted the Agile Manifesto as meaning there doesn't need to be any documentation or preparation and have also misinterpreted what Knuth said about pre-optimization being the root of all evil. With exceptions, of course, most of the application programmers that I've met and worked with personally all think the same thing... disk space is cheap, machines with more memory are coming, stored procedures are an evil to be avoided, and an RDBMS is just a place to store data.
As a very interesting (to me, anyway) sidebar, I'm currently working with 3 wonderful Application Developers that have seen the proverbial light and actually get it. Like the song says, "I'm in Heavennnnn!" I think that part of the reason why they get it is because they're the ones that have to clean up after their "Agile/No Pre-Optimization/Design-on-the-fly" predecessors.
--Jeff Moden
Change is inevitable... Change for the better is not.
January 9, 2012 at 2:27 am
The Agile stuff ain't new at all. I worked in a CSC team in the Nineties practicing some techniques that were recognizably 'Agile'. The only difference was that we used Big sheets of paper stuck on the walls with Blu-tac rather than post-its and called the meeting 'shirtsleeve meetings'. I was in the Data Architecture team and we insisted on getting the data architecture planned out in its entirety before the coding started. It was the only part of the project that got finished, all beautifully mapped out in ERwin.
Best wishes,
Phil Factor
January 9, 2012 at 9:03 am
Jeff Moden (1/8/2012)
It therefore always surprises me when Application programmers tell me that all 'Agile' techniques are directly applicable to database development.
Heh... what did you expect, Phil? Many application programmers and their managers have misinterpreted the Agile Manifesto as meaning there doesn't need to be any documentation or preparation and have also misinterpreted what Knuth said about pre-optimization being the root of all evil. With exceptions, of course, most of the application programmers that I've met and worked with personally all think the same thing... disk space is cheap, machines with more memory are coming, stored procedures are an evil to be avoided, and an RDBMS is just a place to store data.
As a very interesting (to me, anyway) sidebar, I'm currently working with 3 wonderful Application Developers that have seen the proverbial light and actually get it. Like the song says, "I'm in Heavennnnn!" I think that part of the reason why they get it is because they're the ones that have to clean up after their "Agile/No Pre-Optimization/Design-on-the-fly" predecessors.
Amazing what happens when confronted clean up is involved.
Kind of like when I ground boxes on the floor, and then started to weld.
Every welder should have to grind their own welds.
It tends to improve the process.:hehe:
January 9, 2012 at 10:34 am
The point of Agile is to allow business rules in the software to evolve as business rules in the business evolve.
There are ways to build a database to allow for ease-of-evolution, and hence Agile databases. Ironically, they require even more up-front planning and homework, not less.
But what most developers mean by "Agile" is actually "Cowboy". Database development can be quite Agile. It creates a horrible mess when it's done Cowboy-Coder style.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
January 9, 2012 at 10:00 pm
GSquared (1/9/2012)
But what most developers mean by "Agile" is actually "Cowboy".
Hear here!!! GUS FOR PRESIDENT!!! 😀
--Jeff Moden
Change is inevitable... Change for the better is not.
January 11, 2012 at 11:32 am
We do Agile dev in our shop and spend a decent amount of time planning DB changes up-front to avoid doing it over as we move forward. Sometimes we overplan, sometimes underplan, but it helps to at least attempt to get it right up-front. It helped once the team finally understood that you couldn't just drop tables and create new ones like you can with DLLs and EXEs. It took a while for that to sink in, but they understood it and it helped us handle DB migrations a little better in the future. There are still times we need to do a large data operation the night of a release, but on the whole knowing that helped us make better plans for the future. Finding the correct balance of "just enough" as opposed to "plan it as far out as we can" can still be challenging, but having the team backing up those decisions makes a huge difference.
I actually like the Agile plan when it can work. I've seen too many large DB Projects fail due to using waterfall (think Amazon Falls) methodology and then years of work to deliver the finished project. I'd much rather start small enough to show something and get feedback as we go. That helps us make course corrections and tackle more important needs as they come up instead of delaying them further.
January 21, 2012 at 6:16 pm
ganotedp (1/7/2012)
The not-so-flattering Agile analogy (not invented by me) is like trying to build a house one-room-at-a-time (where software is the "fine carpentry and dry wall" and data structures are the "foundation and roof"). "O, wow, I really like this kitchen" says the customer, "let's add a bedroom". Seems easy if you're doing fine carpentry and drywall. However, how do you do Agile foundation work and roofing? Brings to mind the parable told by the radical rabbi about the foolish man building his house on sand. The wise man built on a solid foundation.
I wanted to write a more lengthy response to this analogy than would be appropriate in this space. You can find it at: http://surroundingthecode.wordpress.com/2012/01/22/agile-development-the-wrong-analogy-again-how-about-this-one
January 22, 2012 at 1:15 am
Peter Schott (1/11/2012)
I actually like the Agile plan when it can work. I've seen too many large DB Projects fail due to using waterfall (think Amazon Falls) methodology and then years of work to deliver the finished project. I'd much rather start small enough to show something and get feedback as we go. That helps us make course corrections and tackle more important needs as they come up instead of delaying them further.
I agree with this (having been wandering amongst a few agile teams of late) but with a caveat. The powers that be must understand that you WILL require entire sprints for re-design of the core module to integrate new work. It's not an optional choice. This will require downtime when the fixes are put into place.
With that caveat, I agree you can modularly build databases within the agile structure. However, I also agree with Gus. I've seen too many shops that got their prod powers removed and suddenly Agile is 'problematic'. It's all to often a Cowboy mentality.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
January 22, 2012 at 11:41 am
I can't really concur with Dreamweaver's analogy with human development. But Evil Craig F (Ferguson?) is spot on with:
The powers that be must understand that you WILL require entire sprints for re-design of the core module to integrate new work. It's not an optional choice. This will require downtime when the fixes are put into place.
Continuing the building analogy, it's like responding (to the customer's request for a bedroom after seeing the dynamite kitchen): "Sure, but we've got to shut down the kitchen, and tear down some of the existing work to rebuild this into a 2 room module. You won't be able to use the kitchen for a while."
Sure, it's not an exact analogy. More like you're building modular housing that you can crane into position. But you still need time to extend or replace the foundation, and move all the owner's stuff (data) from the one room into the new 2 room module, etc. etc.
Of course, I think most people would concur with Peter Schott that:
I'd much rather start small enough to show something and get feedback as we go. That helps us make course corrections and tackle more important needs as they come up instead of delaying them further.
In my more cynical moments, I feel like I'm one of the workers on the (in)famous Winchester Mystery House...
January 22, 2012 at 12:46 pm
ganotedp (1/22/2012)
I can't really concur with Dreamweaver's analogy with human development. But Evil Craig F (Ferguson?) is spot on with:
Craig Farrell. 🙂 Pleasure to meet ya. It's in the sig. The nickname's a joke that I happened to like. 😎
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
January 22, 2012 at 1:20 pm
Good to meet you! Just had this vague notion that Craig Ferguson was more than just a late night TV host... what if he were a database guy by day! 😉
January 22, 2012 at 3:18 pm
Evil Kraig F (1/22/2012)
With that caveat, I agree you can modularly build databases within the agile structure. However, I also agree with Gus. I've seen too many shops that got their prod powers removed and suddenly Agile is 'problematic'. It's all to often a Cowboy mentality.
Along those lines, ones you have a decent Continuous Delivery system in place, this shouldn't be as much of a problem. You have your downstream environment(s) that are exact copies of your production ones, gen up the change script(s) using your automated tools, then just hand them off to the team doing the changes in Prod. They should have sufficient permissions to do the changes needed and ideally you have at least one downstream environment that's a duplicate of your production environment so the testing and notes about other pieces of the equation can be handled. We've been using the VS2010 DB Projects to store and push our changes and are pretty close to having the ability to just build/deploy all the way up the chain.
As for complete rebuilds, ideally the project hasn't changed too much along the way and/or you've done the appropriate research before starting that you have a pretty good idea of where this is going in the long run so total redesigns aren't needed. 🙂 If not then you have a lot of other problems to deal with in data conversions, rollbacks, new structures, and all sorts of other things. It's necessary sometimes, but it can often be avoided with some good discussions and planning up front. (not to the waterfall point, but enough where you've got a good enough design for most of the objects)
January 22, 2012 at 4:50 pm
Peter Schott (1/22/2012)
Along those lines, ones you have a decent Continuous Delivery system in place, this shouldn't be as much of a problem. You have your downstream environment(s) that are exact copies of your production ones, gen up the change script(s) using your automated tools, then just hand them off to the team doing the changes in Prod. They should have sufficient permissions to do the changes needed and ideally you have at least one downstream environment that's a duplicate of your production environment so the testing and notes about other pieces of the equation can be handled. We've been using the VS2010 DB Projects to store and push our changes and are pretty close to having the ability to just build/deploy all the way up the chain.
Build and deployment of new modules aren't really my concern there. The equivalent of adding a scoring system to the existing inventory system isn't where Agile starts to breakdown, at least in my mind, from a database perspective.
However, allow me an actual example that I walked headfirst into a few years ago.
A simple e-sales system had started with a solid core scope and understanding. Products were included, carts were built, etc etc. It worked very well. Attached to this were additional modules. Eventually, a catalog system was built in for custom client pricing. Certain deals were to be offered to certain clients. For example, students at particular schools could order things cheaper by being a student.
So overrides were put in place in certain places, the cart was modified, etc etc. Eventually this cart proc became about 30 pages long, and was so convoluted that processing a cart to completion could take up to 10 seconds. This was unacceptable due to ever increasing volume order.
A full redesign of the structure, incorporating the new modules into the core structure, allowed for a much fuller and less intricate system of maintaining the data. This took 3 months of pure dev work, no new features, no ... you get the drift. It wasn't possible to do this in small doses, we had to redesign basically from scratch for all the new modular components that had been included.
I'm all for agile as long as the time is taken every few years to clean up the mess it leaves behind.
It's not because you can't plan for the scope, it's because the very NATURE of Agile is planned scope creep.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
Viewing 15 posts - 1 through 15 (of 17 total)
You must be logged in to reply to this topic. Login to reply