December 6, 2016 at 2:39 pm
I'm actually in the middle of a systematic nuking of a full-stack disaster, which is, in my opinion, generally the best way to go if the whole system really is a disaster.
In our case, we've got classic ASP sites running off of Access databases, alongside WebForms running off of 2 different SQL Server generations, and a couple of mix-and-match sites with all of the above, none of which was source-controlled until I came along, most of which is not documented in any meaningful way, and 3 of the 5 developers who made the mess aren't here anymore.
Our solution is basically to chunk pieces of the system, carefully map out where data is, what it's used for, and how it interacts with other parts of the whole, then rewrite it from the ground up instead of trying to fix what's already there.
I think the passive approach is more shoved down our throats than anything else; I don't think people get into this field to sit on their hands until they are forced to fix problems.
And I think that actually being able to completely nuke an old system and rewrite it from the ground up without doing the incremental approach is pretty much impossible in a production system. I think I saw a comment to the effect of, "explaining to the higher-ups that the rebuild is going to take 2 years and they're not going to actually see a difference doesn't go over very well"; I don't know that it's possible to get around that problem in real life.
December 6, 2016 at 2:47 pm
reminds me of netscape the browser and company.
https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
December 6, 2016 at 6:20 pm
I would run out of fingers if I tried to count the number of times I have inherited a database mess with a new job. The new employer always promised they were fully committed to the second approach, but they have always taken the first approach. The technical debt continued to grow and the organization continued to suffer. I have never had the opportunity for the third approach, although I have deemed it the most prudent approach in several cases. What do I do about it? What *can* you do when the organization refuses to recognize they are compounding their misery on a daily basis? I start looking for a new opportunity, like I am right now! 😛
Creator of SQLFacts, a free suite of tools for SQL Server database professionals.
December 6, 2016 at 7:10 pm
Wingenious (12/6/2016)
I would run out of fingers if I tried to count the number of times I have inherited a database mess with a new job. The new employer always promised they were fully committed to the second approach, but they have always taken the first approach. The technical debt continued to grow and the organization continued to suffer. I have never had the opportunity for the third approach, although I have deemed it the most prudent approach in several cases. What do I do about it? What *can* you do when the organization refuses to recognize they are compounding their misery on a daily basis? I start looking for a new opportunity, like I am right now! 😛
Wingenious, I hear your feelings and wish you the best in your search for a new opportunity. I think there is often a point at which your contribution has been made and you need to keep your own interest foremost. I did the opposite a couple times in my career and put myself second. It was not appreciated, and I eventually was summarily ushered out. 'What appears to be the light at the end of the tunnel is sometimes another train'. The only good that came of those eleven years of being on call 24/7/365, sacrificing family life and spending nights and weekends on the job at one employer in particular was that the company had a great profit-sharing plan that I got to take with me and formed the original basis of my retirement, which after a 42-year IT career is now very comfortable.
Rick
Disaster Recovery = Backup ( Backup ( Your Backup ) )
December 7, 2016 at 12:48 am
patrickmcginnis59 10839 (12/6/2016)
reminds me of netscape the browser and company.https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
As usual Joel's experience is exactly what I came across in my career.
If I was to choose the nuclear option it would be based on the underlying tech being obsolete, the needs of the end users not being met and possibly impossible to meet.
I could write an article on it.
Personally I believe it should be legal to shoot people who use the phrase "not fit for purpose" without due care and attention
December 7, 2016 at 8:29 am
Wingenious (12/6/2016)
I would run out of fingers if I tried to count the number of times I have inherited a database mess with a new job. The new employer always promised they were fully committed to the second approach, but they have always taken the first approach. The technical debt continued to grow and the organization continued to suffer. I have never had the opportunity for the third approach, although I have deemed it the most prudent approach in several cases. What do I do about it? What *can* you do when the organization refuses to recognize they are compounding their misery on a daily basis? I start looking for a new opportunity, like I am right now! 😛
It amazes me just how much technical debt companies are willing to accrue. I think if they do it long enough they become so accustom to poor databases, applications and user satisfaction that they just know no other way of life.
Kindest Regards, Rod Connect with me on LinkedIn.
December 7, 2016 at 8:44 am
I'm currently in the second condition, where I was hired to refactor pieces of a very large, very poorly performing system. Customers were beginning to bail. Rip and replace was simply out of the question, so adding indexes and optimizing procedures (as well as C# code) has been a huge aspect of my contribution. Fortunately, I've been given a lot of latitude, and I've been able to work on pieces outside the original scope. There's been a lot of noticeable improvement in the areas I've touched, but the culture of this place strikes me as strongly resistant to change. The other developers here seem satisfied with the first condition. Sometimes I feel like someone who's been patching holes in a boat while a few other folks keep shooting in new ones. It's not really quite so bad all the time, but I've been building lots of tools to clue me in when new holes appear. And they do.
December 7, 2016 at 9:09 am
David.Poole (12/7/2016)
If I was to choose the nuclear option it would be based on the underlying tech being obsolete, the needs of the end users not being met and possibly impossible to meet.I could write an article on it.
Personally I believe it should be legal to shoot people who use the phrase "not fit for purpose" without due care and attention
I do not come to the nuclear option conclusion as a reflex, but nor do I reject it as a reflex. I'm sure we could find plenty of examples where rewriting an application did not go very well for an organization. However, I'm sure we could also find plenty of examples where rewriting an application was a brilliant move. I think Microsoft has done pretty well for itself with Windows, and Windows has been rewritten a few times.
I think the success (or failure) of any effort to rebuild an application/database system from scratch has more to do with the people involved than the fact that a crappy system already exists.
Creator of SQLFacts, a free suite of tools for SQL Server database professionals.
December 7, 2016 at 10:29 am
Not sure how many times Windows has been rewritten. I suspect that it is like Grandfather's axe, or in other words it followed type 2 evolution albeit a more aggressive form.
Fully agree that the people make the difference. I have gained a full appreciation for the axiom "people and process over tools and technology"
The nuclear option requires all parties involved to have a good understanding of what is involved, what is being committed to and precisely what their commitment level must be. My experience is that the conjunction of these three things is rarely there.
December 7, 2016 at 10:34 am
I have two favourite experiences of this type of choice:
1) Development team checked everything in and left.
I was called to get a system developed for my client's client both to a known state and maintainable again. The team managing and developing the system had left "overnight" from the management perspective. I have no idea why they left nor was I interested. There was no choice but to maintain the existing source code base. Sometimes it is out of your hands!!!
2) Emergency fixes deemed good enough.
I was brough in to fix a fraud detection system that was bringing down the website of a world leading company whose business is based off of the back of sport events. The website went down each year during one particular sporting event. I estimated that the rewrite would take between 6-9 months. The next annual event was in three months. I did some analysis how this application interacted with SQL Server and calculated that a certain set of fixes could make sure that the event in 3 months would not be catastrophic. It would still bring down the website at the subsequent event in 15 months.
My team implemented the "Phase 0" fixes and the website survived the annual sporting event for the first time in a number of years. So the rewrite was canned. Roll it forward 12 months and I was reliably informed that they had to stop using the fraud detection system during this annual event to avoid the website going down. As I predicted. Using analysis.
BTW The loss to the company if the website went down on this day was a high 7 figure number (GBP) - nearly 8 figures. The ROI would have been made approximately 32 times over within months of project completion.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
December 7, 2016 at 12:03 pm
If we were all deeply pessimistic about redeveloping software then we might still be using VisiCalc.
Sure, redeveloping software involves costs and risks, but so does running a business on a software system that's hard to support, hard to enhance, and fragile.
If redeveloping software (with an existing system as a guide) is such an onerous undertaking then where does anybody find the intestinal fortitude to develop software in the first place?
Creator of SQLFacts, a free suite of tools for SQL Server database professionals.
December 7, 2016 at 1:07 pm
Wingenious (12/7/2016)
If we were all deeply pessimistic about redeveloping software then we might still be using VisiCalc.Sure, redeveloping software involves costs and risks, but so does running a business on a software system that's hard to support, hard to enhance, and fragile.
If redeveloping software (with an existing system as a guide) is such an onerous undertaking then where does anybody find the intestinal fortitude to develop software in the first place?
Well, I guess I was (used to be) one of the optimistic ones. I recall at one point taking on a project that involved moving my company from IBM hardware to Unisys hardware and software, converting a custom-designed interactive order processing package from RPG to Cobol, converting a custom-designed dialup order entry package using Telxon handheld units, customizing a 600k line interactive Cobol inventory, ordering, and account receivable package, moving to the A/R system from manual posting machines and paper ledger cards, all in one move. Add to that the situation in which all of my users were 50+ year old folks who had used their manual systems for many years and had no exposure to online interactive systems. My one side-kick and I kept 24-hour online production going and until the new hardware was delivered commuted 130 miles for testing at a hardware vendor site at night.
Yeah, I guess we were pretty optimistic, but we pulled it off without too much pain and suffering.
Rick
Disaster Recovery = Backup ( Backup ( Your Backup ) )
December 10, 2016 at 12:57 pm
Wingenious (12/7/2016)
If we were all deeply pessimistic about redeveloping software then we might still be using VisiCalc.Sure, redeveloping software involves costs and risks, but so does running a business on a software system that's hard to support, hard to enhance, and fragile.
If redeveloping software (with an existing system as a guide) is such an onerous undertaking then where does anybody find the intestinal fortitude to develop software in the first place?
If we were all deeply pessimisti about redevloping software then we might all be using C++.
Sure, redeveloping software involves costs and risks, but so does running a business on a software system that's hard to support, hard to enhance, and fragile.
Oh - wait a minute - almost everyone still is using C++ (and other awful stuff) - the proportion of stuff written in decent languages (e.g. the variosu ML dialects, Prolog and Parlog, and and Haskell, and even F#) is still ridiculously small (but nothing like as small as the proportion of bugs written in those languages).
Tom
December 11, 2016 at 6:33 am
I'm pessimist when a proposed nuclear option does not offer a significant jump forward or does not form part of a roadmap that offers significant improvements.
I'm an optimist when the change is all to enable a clearly stated goal. A lack of clarity in what you want to achieve is never a good starting point in a radical change programme.
Radical change is always painful, otherwise it wouldn't be radical change. If I'm going to go through such a change I want to know that it is worth while and not CV driven development.
VisiCalc was good but let's be honest, it was a starting point in an industry that wasn't mature. Not everyone had a PC on their desk, even in the IT department!
The situation we are talking about here is a mature and critical business system with all sorts of hidden functions and mystery processes that only become apparent after they are switched off. I've seen huge firefighting effort put into dealing with unknown system features being decommissioned and being discovered to be essential to an obscure but essential business process
December 11, 2016 at 7:49 am
I've seen more so many messes inherited by myself and others that I don't even want to count them. I prefer the systematic approach, but have seen the passive approach taken too many times.
My favorite is a rewrite and reorganization of an overblown, unnecessarily complicated and inefficient process. I recommended a rewrite back in 2012, where a new system would be written from scratch and the UAT would be to run the old and new systems in parallel and compare the results. Unfortunately, nothing was done and the old system continued to be used.
In 2013, the process was looked at again, but nothing was done because it was deemed to be too much work.
By mid-2015, things got bad enough that the rewrite had to proceed. The new system was delivered in late 2015 and UAT started, but UAT was then dropped because comparing the results with the old system was time-consuming.
By mid-2016, it was being crushed under its own weight. Updates were made to the new system to catch up on the 6 months of updates to the old system , but it has yet to be implemented. The performance and scalability problems are absolutely horrible. Maybe I'll have an update to the story in 2017.
Viewing 15 posts - 31 through 45 (of 49 total)
You must be logged in to reply to this topic. Login to reply