February 12, 2013 at 12:09 am
Comments posted to this topic are about the item Practicing Deployments
February 12, 2013 at 2:00 am
One of the key parts of this is automation. Automate what you can to make it repeatable. This reduces the possibilities of human error which is something we endeavour to minimise as much as possible because it is our greatest weakness (as well as being our greatest strength - adaptability being the converse). Remove as much doubt as possible and, yes...
...get build and deployment sorted out early on. This means it can be sorted out when the deployment is at its simplest, when there is the least commercial pressure and when stakeholders have the most goodwill (the hiccups of early deployments will be countered by the continually successful deployments thereafter).
As for early deployment being a mark of a professional, I think that it can be a little harsh. Perhaps the desire for early deployment is the mark of a professional?
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
February 12, 2013 at 11:01 am
I think there's a big caveat when using a model of deploying early and often. This approach can too easily lead to a culture where the early adopters of the "production" software are in reality just beta testers. There are reasons why the belief exists that you don't install new Microsoft products untill after SP1.
In the business world, people are trying to perform tasks, and want a stable environment to do it in. It's a thin tightrope to walk between being agile and responsive to business needs vs. getting something out just to say you've accomplished something. I've seen too many times in both in-house software development and comercial software where short term thinking and a desire to meet artificial dates rushed things into use that weren't ready. I think the real thing to take away from the referenced article is to keep the user community involved more, give people what they really need, and try to remember that quantity does not trump quality.
February 12, 2013 at 11:25 am
Totally agree. Initial deployments should be internal in my opinion.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
February 18, 2013 at 10:37 am
I work for a software consumer, as opposed to a development house. We are not happy with the quality of deployments in the health care industry. Let's skip software quality for now and stick to the point.
Let me start with saying that we have some vendors that do an exceptional job. Some of our vendors do remote installs and upgrades, some come on site, and the best of them tend to do very well with their product. Some of them even have some understanding of third party tools that their application requires. Kudos to those and other companies that take the time to do the best they can.
We also have vendors that absolutely need to find another way to do things. In some cases, everything about their process is flawed. In other cases, what is usually done very well, fails in cases where things are slightly out of norm. The industry has an issue with this, and it needs to be fixed. Below are some examples of things I have seen.
A vendor is doing an upgrade on one of our systems. The old database is SQL 2000. The new database is SQL 2008. They provided instructions for SQL 2005. The requirements they provided for the servers were wrong, and 75% of them had to be rebuilt. They uninstalled SQL without permission and later restored a test database over future production, with the expectation that it will be used going forward as a test database. Guess we won't ever need a production database...
Another vendor typically does excellent work. Previous upgrades have gone very smooth, with any issues that came up being within what I feel are acceptable limits. This last upgrade was one of the worst I have ever experienced. Things that were outside their control probably led to most of the issues, for example the birth of a baby a week before the upgrade, but the lack of processes to cover this type of issue is unacceptable. People get sick, and replacements need to be able to jump in and hit the ground running. Documentation of the customer's system isn't hard to create, and can go a long way to eliminating issues when replacement staff must take over. Some of the other issues were simply due to mistakes being made, but I don't know that they would have occurred had there not been a change in team members so close to go live.
Part of the issue here is that companies are always trying to do more with less. Increasing the profit margin by reducing staff can only go so far before this leads to costs on the other side that end up in a loss of revenue and reputation. Unwillingness to hire people with the knowledge required, and unwillingness to train those that are employed, is leading to an environment where mistakes occur with regularity, and in some cases are extremely costly.
Fixing these issues is larger than a single developer, but we can point out where weaknesses exist, and we can improve our own processes in the hope of reducing their impact on the overall environment.
Dave
February 18, 2013 at 1:40 pm
djackson 22568 (2/18/2013)
Part of the issue here is that companies are always trying to do more with less. Increasing the profit margin by reducing staff can only go so far before this leads to costs on the other side that end up in a loss of revenue and reputation. Unwillingness to hire people with the knowledge required, and unwillingness to train those that are employed, is leading to an environment where mistakes occur with regularity, and in some cases are extremely costly.
Good point, and this environment as you stated so accurately above, usually initiates and results in a never-ending chain of high turnover rate as well. Which just adds to the quality assurance problems they are already experiencing, because of what you stated above. In my experience in the past, a lot of these IT shops that are experiencing this condition, the management is either clueless or just doesn't care.:-D
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
February 18, 2013 at 1:50 pm
TravisDBA (2/18/2013)
...the management is either clueless or just doesn't care.:-D
Not always mutually exclusive. Unfortunately.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
September 7, 2016 at 12:37 am
Nice to see an outdated article 🙂
Maybe the IT-industry here overall has actually grown and improved.
A lot of companies in Sweden and other countries I know speak about and are implementing DevOps. https://en.wikipedia.org/wiki/DevOps
September 7, 2016 at 12:54 am
In a past life I worked in a company that provided web sites using CMSs. At the time these things had quite complicated infrastructure requirements if you wanted to deploy a production grade system with high availability.
Inevitably a deployment would require us to work with the clients other IT providers. We always proposed that we be prime contractor in any such project as this would give us the authority to control the planning, rehearsals (plural) and eventual deployment. We became very good at mitigation planning due to painful experience.
There are big IT vendors out there wh charge premium rates for a level of service that would have to improve exponentially just to reach a standard below abysmal.
Just to give an example, my company planned for myself and a colleague to perform a complete install over 2 days. We reckoned that the 2 of us could do the entire thing by the end of the first afternoon and could give whatever mentoring and training the client asked for as a bonus (the value is always in the 2nd order).
We arrived at the data centre slightly before 8am and after some discussions with security (we weren't on their list) we were shown to a reception area and told to wait.....and wait. Over the next 2 hours we repeatedly asked to speak to the vendors project manager only to be fobbed off. Eventually we were told that he was on two weeks leave and they would have to find someone else to manage the process. Another 2 hours of delays and we decided to have an early lunch while the vendor attempted to find their collective backsides with both hands and a flashlight.
Obviously we had an off premises telephone chat with our line manager to see if he could get the client to lean on the vendor.
We got back to the vendor data centre and are eventually let in to the server room. The clients brand new servers are anything but new. One of them had a red warning light on it. A quick run through our checklist failed 2 of the 5 servers.
We were supposed to install a vendor bought copy of SQLSERVER. Vendor hadn't bought it.
Eventually we get to the point where we can begin installation at 3pm having lost most of the day. At 5pm we are instructed to leave the premises because the vendor only allows 3rd party access between 8am and 5pm.
A night in the worst hotel in Reading and a toxic breakfast didn't set us up for the next day. All 5 servers needed to be able to see each other. They couldn't. They should all have identical base installations. They didn't. Everything the vendor could have cocked up they had done so to Olympic gold standards.
The ufpshot was that any future involvement with this vendor made us quote the client or an extra day of health checks
September 7, 2016 at 1:54 am
IceDread (9/7/2016)
Maybe the IT-industry here overall has actually grown and improved.
A lot of companies in Sweden and other countries I know speak about and are implementing DevOps. https://en.wikipedia.org/wiki/DevOps%5B/quote%5D
...and yet a lot aren't. Still. Or planning to do it instead of doing it and calling it DevOps anyway.
Well that's my experience (both personally and through discussion with others) in the UK.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
September 7, 2016 at 2:07 am
David.Poole (9/7/2016)
...A night in the worst hotel in Reading and a toxic breakfast didn't set us up for the next day...
Reading is a charm.
Did that company go belly up a couple of years ago? About 2012/2013? I think I may have done some work for them.
The company I am thinking of couldn't even manage their own internal network properly. I bought a small switch because they had 4 Ethernet ports for 6 hot desks and I needed 2 for myself. Oh and no one seemed to have cables nor were any available. Shocking.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
September 7, 2016 at 6:51 am
We deploy once a month and are our own customer so have a vested interest in making sure they go smoothly. After a few years of this I think we have it down fairly well.
September 7, 2016 at 7:12 am
Iwas Bornready (9/7/2016)
... are our own customer so have a vested interest in making sure they go smoothly...
If it was for clients then surely it is the same:
not smooth deployments = customer distatisfaction + job insecurity
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
September 7, 2016 at 9:57 am
Gary Varga (2/12/2013)
One of the key parts of this is automation. Automate what you can to make it repeatable. This reduces the possibilities of human error which is something we endeavour to minimise as much as possible because it is our greatest weakness (as well as being our greatest strength - adaptability being the converse). Remove as much doubt as possible and, yes...
Our deployments tend to be early morning (last 2 have required being in the office at 3am!) so everything we can do to eliminate human error, we do.
September 7, 2016 at 10:40 am
Gazareth (9/7/2016)
Gary Varga (2/12/2013)
One of the key parts of this is automation. Automate what you can to make it repeatable. This reduces the possibilities of human error which is something we endeavour to minimise as much as possible because it is our greatest weakness (as well as being our greatest strength - adaptability being the converse). Remove as much doubt as possible and, yes...Our deployments tend to be early morning (last 2 have required being in the office at 3am!) so everything we can do to eliminate human error, we do.
Good man. 😉
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
Viewing 15 posts - 1 through 15 (of 15 total)
You must be logged in to reply to this topic. Login to reply