September 11, 2018 at 4:29 am
Thanks for all the replies everyone! I don't think this should be an issue about trust (though some will make it about that)... it's about focus. The focus of developers is to code and quickly get things deployed to meet ridiculous deadlines. The focus of the DBA is to keep the lights on in production and avoid downtime. It's not about the people involved, it's about the role. When DBAs start out they make mistakes in Prod because they have the access and they are learning the ropes - they soon learn to be cautious. A well trained DBA is one who follows the checks and balances because they've learned the hard way that it's the only way to avoid downtime. They also don't (or shouldn't) have the same time pressures as devs and so learn not to take short cuts (we have time pressures alright but spread out over projects/BAU and not as acutely focused as the devs striving to deliver on one or two projects at a time). Separation of duties means each team have their own focus as it would be difficult to find the time to know it all (and do it all properly).
I am not talking about DBAs doing deployments to Prod - i believe the ideal is automated deployments (initiated by devs) to ensure speed and consistency. But anything that cannot be auto deployed, or any sort of troubleshooting that involves changing settings etc in Prod, should be a DBA, And obviously any release to prod would be tested in the preceding environments, and the change should be scheduled in with time in advance unless an emergency and the DBAs obviously made aware.
I do know some DBAs that work in an environment (a hedge fund) where devs have access to production, but it's very strictly managed with full auditing/reporting and devs can be called out at all times to fix issues. Seems to work for them but i haven't worked in an environment like that myself so can't comment further.
Anyway - these posts have helped me get to an answer re the original question which is not allow anyone to modify jobs (create/alter/drop) in production other than DBAs, but continue to work with devs to improve the automated deployment process so things can happen quickly AND consistently.
Thanks all 🙂
September 11, 2018 at 8:25 am
doodlingdba - Tuesday, September 11, 2018 4:29 AMit's about focus. The focus of developers is to code and quickly get things deployed to meet ridiculous deadlines. The focus of the DBA is to keep the lights on in production and avoid downtime.
Heh... that's why I took the position on all this that I took. Those two items are, many times, mutually exclusive. 😀
Anyway - these posts have helped me get to an answer re the original question which is not allow anyone to modify jobs (create/alter/drop) in production other than DBAs, but continue to work with devs to improve the automated deployment process so things can happen quickly AND consistently.
That, my good Sir, is (IMHO) exactly correct. My only word of caution is that the fastest way to deploy bad code is to automate the deployments. It's not that automatic deployment is bad (it's not, it's an absolute joy especially for the DBA!). It's only bad if the code hasn't followed the logical path from Dev to QA to UAT to Prod. If it goes straight from Dev to Prod, there's going to be some unexpected downtime involved.
I also know that you know that but I had to say it out loud for others that may be reading this, especially new or timid DBAs.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 11, 2018 at 9:10 am
TomThomson - Monday, September 10, 2018 3:05 PMJeff Moden - Sunday, September 9, 2018 10:03 AMI totally agree (except for the Devs having "Write" access to prod). I've not explained our whole "emergency " process that supports even such emergencies. It's the same process that we always use except that it's greatly accelerated by having people in hot standby for the next step. And, yes, I agree that the Devs know the steps and the order, etc, etc. That's true with any deployment we do, which is done 100% by scripts.
...
...
Getting back to the subject of this post, what about "simply" allowing Devs to control jobs? The answer is "No". They're busy enough. They don't have the time to do the research necessary to determine if jobs will conflict with other jobs. They don't have the time to determine if there are "hot spots" in the schedule that are already "full" for CPU and disk usage". And, many of them don't have the understanding that not everything can run at the top of the hour and than not all jobs can run once per minute just because the users want it to.Protect the company by protecting the system, protecting the data, and protecting the Developers because all 3 are of critical importance to protecting the company.
Jeff, I think you've missed the point that it's all about who you can trust. Or perhaps you believe that being trustworthy is something that is inevitably the natural state for producton people while developers are probably not trustworthy. Or maybe you know developers who operate differently from those I know, because you don't operate my rule that some-one who has not spent at least a few months doing customer support (maybe for you I should say "production support") can be anthing, as a developer, other than a very junior trainee.
So I suspect that you've not been faced with situation where the production staff for most of the systems you are responsible for don't work for the same company as you - they work for somone who buys software and services from the company you work for, and are often willing to deliberately deviate from an installation script in order to create down-time which can then be used to demand a reduction in your company's fees. Not surprisingly, the senior management of the company that employs those production staff usually regards that as unacceptable because such delays will affect their customers and reflect badly on them, and so they want us to actively prevent their operations people from doing any non-trivial upgrades in which they might add something to generate downtime . Bizarre, yes - but selling services in India and in the Middle East and some parts of Europe and North America means that at least some of your customers will have junior operations people like that and the senior management will know it but not be able to prove it against individuals - usually, not always. One customer (Middle East) thought they had no problem although every time they escalated a problem up to my level we found that the customer 's network had been carefully revised to prevent our customer support people and our fixit experts (developers, of course) getting access. Dealing with those people was not fun - when every "we have a problem" is immediately followed by "oh, we seem to have lost the ability to connect you to the machines" that's a pretty clear indication that the production team is dishonest. It was still going on when I retired - that customer's IT managers were either extremely dim or part of the problem.
I had some real fun with one very different customer, their IT director was having difficulty with concealing his amusement as one of his people repeatedly tried - after having been given database SA and OS full privilege to allow him to experiment - to install some unauthorised software (ie software that my modified Windows embedded client operating system did not recognise as authorised) on one windows (embedded) based client terminal, using his domain operator full privilege, and discovered that when he tried to use unauthorised software after installing it it didn't work and when he looked for it after the installation had run "successfully" there was no sign of it in the system, nor in the OS directory. That convinced the customer IT director that he could trust our system, and also convinced the guy he had trying to bypass it that when he (the IT director) insisted that servers running the server-side components of our softwares would have no sysop passwords available to his staff that was probably the right thing to do - perhaps because it would stop them wasting their time as he had for a day or two while trying to insert unauthorised software.
Strangely enough, I found most of our customers in the UK had asked us to ensure that privileged passwords were unavailable to their staff too - it turns out that production staff also want to demonstrate how clever they are, and sometimes go beyond what should be their limits. The one customer who insisted on their staff having privileged passwords for machines running our systems had a catastrophic influx of advertisements for pornography into the system, which was a source of complaint from several of their customers and probably the cause of our contract with them not being renewed at next review - it was that incident that triggered me into getting the "only authorised software" checks developed for the client machines - I can't remember whose virus/worm/etc protection software we used on customers' servers, except that it actually worked when it and some MS defense software were both running and that it certainly wasn't Symantec as their stuff was both underperformant and vastly overpriced back in those days (about 13 or 14 years ago). None of our other customers ever had any problems like that, even without that "only authorised software" mechanism on the clients. We gave one other customer (Barbados Sandy Lane Hotel) sysadmin access on all servers running our software - a pity, really, as otherwise some of our staff would have been able to go out there regularly, and they'd have regarded it as a great bonus - but those people really did know what they were doing and really (I think) were honest.
I don't see how working in that environment I could have required the company to say "OPs does all the upgrades". It's just not workable, obviously. And an awful lot of people are in situations like that. It is quite simple a question of who can you trust to do it right - and in your situation at Proctor Financial I would be amazed if you couldn't trust the company's production people - they are part of your people (just as I trusted in-house production although I often didn't trust out-of house production).
And I agree completely about he need to protect the developers from managers. It's something I tried for most of my working life. And I hope (and believe) that I mostly succeeded in doing that.
No... the point wasn't lost on me. But look at all that you wrote. Even supposedly trusted people did things they should not have. I've been through much the same gambit as you in that area.
The sorry fact of the matter is that people in my past have taught me (sometimes the hard way) to trust.... no one. 😉 At least not in an absolute fashion. I've worked with some really cracker-jack teams but even they take inappropriate shortcuts or just flat out miss stuff that leaves you scratching your head because you actually gave them written step by step instructions that they asked for.
There's also human nature to contend with (as you documented above). Some folks will cross the line no matter where the line is either because they think they know better (and won't communicate what they're trying to do because they don't want to hear the word "No") or because they're under some form of pressure from others and don't know how to or simply won't say "No" themselves.
There's also the human nature of some folks to not do things correctly when no one is watching because they either forget or they, once again, think that the "rules" (whatever they are) don't actually need to be followed.
I have lot's of examples but they're similar to the ones you gave above.
Heh... Remembering what Reagan said (the origins of which are actually seem to be Russian), I must have a little Russian in my Heinz 57 blood because my "Trust but verify" nature has prevented a whole lot of crises. And, very much like the code you described that prevented unauthorized software from being deployed, I also setup walls and many of those walls are in the form of denied privilege. We also have the management sanctioned rule that no on deploys their own code. I didn't know it when I made that rule a long time ago but it turns out to be one of the things that the auditors check for (we're are heavily involved in the banking industry and suffer frequent audits where we actually need to prove such things). Even without such audits, I'd still have such a rule to prevent Developers and others from suffering the consequence of having unchecked/untested code make it to production and then have them get fired because of it (lost a few good ones early in my career because of that and don't want to go through that again, either).
--Jeff Moden
Change is inevitable... Change for the better is not.
Viewing 3 posts - 16 through 17 (of 17 total)
You must be logged in to reply to this topic. Login to reply