September 7, 2018 at 3:39 am
Hi all
Trying to get an idea of what people are doing out in the industry around SQL job management. I'm a DBA and the dveelopers where I work do not have sysadmin in Production. They can do their own automated deployments, but they use a deployment tool under a specific login which has the necessary permissions. However, I was approached by a developer yesterday who said it would make like a lot easier for them if they could deploy jobs directly from Visual Studio using teir own id - to allow this I would need to give them SQL Agent Operator role in MSDB which would allow them to fully administer SQL agent jobs in production. I'm uneasy with this as we would have to assign it to all developers (we don't add individual logins) - don't judge me, just trying to keep the light son 🙂 The scenario my developer described is this: if you want to change a package level parameter in an SSIS package (in Visual Studio), this automatically makes a change to the SQL agent job calling the package which means the job needs to be redeployed. So even though they can deploy the package with the new parameters without elevated permissions, they cannot update or deploy the jobs directly in SQL so they will need to use the change control system to re-deploy the job. Appreciate it's annoying for them but also don't like the idea of any dev being able to edit any job... hell, even i don't like to tinker with live jobs (such a mission to restore from MSDB and yes i know we should be scripting out our jobs and storing in TFS or similar but another thing on my to do list that i haven't one yet...)
Interested to hear ideas, thought, concerns from all you DBA/Devs out there!
Doodles
September 7, 2018 at 7:34 am
doodlingdba - Friday, September 7, 2018 3:39 AMHi allTrying to get an idea of what people are doing out in the industry around SQL job management. I'm a DBA and the dveelopers where I work do not have sysadmin in Production. They can do their own automated deployments, but they use a deployment tool under a specific login which has the necessary permissions. However, I was approached by a developer yesterday who said it would make like a lot easier for them if they could deploy jobs directly from Visual Studio using teir own id - to allow this I would need to give them SQL Agent Operator role in MSDB which would allow them to fully administer SQL agent jobs in production. I'm uneasy with this as we would have to assign it to all developers (we don't add individual logins) - don't judge me, just trying to keep the light son 🙂 The scenario my developer described is this: if you want to change a package level parameter in an SSIS package (in Visual Studio), this automatically makes a change to the SQL agent job calling the package which means the job needs to be redeployed. So even though they can deploy the package with the new parameters without elevated permissions, they cannot update or deploy the jobs directly in SQL so they will need to use the change control system to re-deploy the job. Appreciate it's annoying for them but also don't like the idea of any dev being able to edit any job... hell, even i don't like to tinker with live jobs (such a mission to restore from MSDB and yes i know we should be scripting out our jobs and storing in TFS or similar but another thing on my to do list that i haven't one yet...)
Interested to hear ideas, thought, concerns from all you DBA/Devs out there!
Doodles
No... God No! It'll become a classic case of the mouse guarding the cheese. I'll also tell you that you won't be able to pass most of the significant audits that are out that that require that no one should deploy their own code. Even as a DBA and someone that knows a fair bit about SQL Server and T-SQL, I still have someone peer review and test my code and, with only rare exceptions that are authorized in writing by other people, have other people deploy my code. And all of it is carefully tracked as is the testing and approval process.
Every shop I've ever worked in that has had Developers doing deployments to Prod has been a disaster of performance problems and terrible mistakes. Without the checks, balances, and documented testing, it always turns out to be a cowboy shop. Does it make things slower? YES!!!! That's because you're doing more to ensure the quality and accuracy of things that make it to production AS YOU SHOULD BE!
And, to be sure, it's not that Developers or others can't do a good job. We have Front-end Developers in my shop that I'd literally trust with my life, never mind trust with the data. They key is that all people make mistakes. The only way to prevent that is to have a system of checks and balances that provide the opportunity to catch mistakes prior to being deployed to production.
As for the Managers that may want to allow Developers to have deployment rights on Production, tell them no and remind them that "If you want it real bad, that's the way you'll get it".
Protect the data. Protect the system. Protect the Developers and the users.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 7, 2018 at 7:39 am
It really depends on your organization and just how often these kind of situations arise, not to mention what level of accountability you want the developers to have in production. If it's a case of them just not wanting to follow standard procedure once in awhile then it probably doesn't make sense. On the other hand if this is something they're doing routinely and you trust them to do it and are willing to hold them accountable if they mess something up then it might make sense.
September 7, 2018 at 7:46 am
ZZartin - Friday, September 7, 2018 7:39 AMIt really depends on your organization and just how often these kind of situations arise, not to mention what level of accountability you want the developers to have in production. If it's a case of them just not wanting to follow standard procedure once in awhile then it probably doesn't make sense. On the other hand if this is something they're doing routinely and you trust them to do it and are willing to hold them accountable if they mess something up then it might make sense.
I have to disagree. Even the standard procedure should clearly identify any and all exceptions including those that are required for supposed emergencies. And, yes, they should be "exceptions" and they should be rare. You have to protect against the mistakes that you know are going to happen because you've already seen them happen many times, I'm sure. Even if you totally disagree with that, the auditors won't. 😉
--Jeff Moden
Change is inevitable... Change for the better is not.
September 7, 2018 at 8:07 am
Jeff Moden - Friday, September 7, 2018 7:46 AMZZartin - Friday, September 7, 2018 7:39 AMIt really depends on your organization and just how often these kind of situations arise, not to mention what level of accountability you want the developers to have in production. If it's a case of them just not wanting to follow standard procedure once in awhile then it probably doesn't make sense. On the other hand if this is something they're doing routinely and you trust them to do it and are willing to hold them accountable if they mess something up then it might make sense.I have to disagree. Even the standard procedure should clearly identify any and all exceptions including those that are required for supposed emergencies. And, yes, they should be "exceptions" and they should be rare. You have to protect against the mistakes that you know are going to happen because you've already seen them happen many times, I'm sure. Even if you totally disagree with that, the auditors won't. 😉
Well most auditors don't know they're head from a hole in the ground, all you have to do is change the DB developer's title to DBA and they won't know the difference between who should and shouldn't be making production changes 😛
For a lot of places it doesn't make sense to say people who do development should never touch production, that may or may not be the case here.
September 7, 2018 at 10:44 am
ZZartin - Friday, September 7, 2018 8:07 AMJeff Moden - Friday, September 7, 2018 7:46 AMZZartin - Friday, September 7, 2018 7:39 AMIt really depends on your organization and just how often these kind of situations arise, not to mention what level of accountability you want the developers to have in production. If it's a case of them just not wanting to follow standard procedure once in awhile then it probably doesn't make sense. On the other hand if this is something they're doing routinely and you trust them to do it and are willing to hold them accountable if they mess something up then it might make sense.I have to disagree. Even the standard procedure should clearly identify any and all exceptions including those that are required for supposed emergencies. And, yes, they should be "exceptions" and they should be rare. You have to protect against the mistakes that you know are going to happen because you've already seen them happen many times, I'm sure. Even if you totally disagree with that, the auditors won't. 😉
Well most auditors don't know they're head from a hole in the ground, all you have to do is change the DB developer's title to DBA and they won't know the difference between who should and shouldn't be making production changes 😛
For a lot of places it doesn't make sense to say people who do development should never touch production, that may or may not be the case here.
To continue a good discussion, when does it make sense to generally allow Development to write to production outside of authorized emergency actions? The only time I can think of such a thing is (as you partially suggested) is when the DBA and the Developer are the same person in a very small environment.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 7, 2018 at 10:55 am
Jeff Moden - Friday, September 7, 2018 10:44 AMThe only time I can think of such a thing is (as you partially suggested) is when the DBA and the Developer are the same person in a very small environment.
That is me (one of two developers) although I am not a DBA I do, do design, implementation and maintenance but my work is QA tested by Business Analysts and end users with sign off.
Far away is close at hand in the images of elsewhere.
Anon.
September 7, 2018 at 12:21 pm
Jeff Moden - Friday, September 7, 2018 10:44 AMZZartin - Friday, September 7, 2018 8:07 AMJeff Moden - Friday, September 7, 2018 7:46 AMZZartin - Friday, September 7, 2018 7:39 AMIt really depends on your organization and just how often these kind of situations arise, not to mention what level of accountability you want the developers to have in production. If it's a case of them just not wanting to follow standard procedure once in awhile then it probably doesn't make sense. On the other hand if this is something they're doing routinely and you trust them to do it and are willing to hold them accountable if they mess something up then it might make sense.I have to disagree. Even the standard procedure should clearly identify any and all exceptions including those that are required for supposed emergencies. And, yes, they should be "exceptions" and they should be rare. You have to protect against the mistakes that you know are going to happen because you've already seen them happen many times, I'm sure. Even if you totally disagree with that, the auditors won't. 😉
Well most auditors don't know they're head from a hole in the ground, all you have to do is change the DB developer's title to DBA and they won't know the difference between who should and shouldn't be making production changes 😛
For a lot of places it doesn't make sense to say people who do development should never touch production, that may or may not be the case here.
To continue a good discussion, when does it make sense to generally allow Development to write to production outside of authorized emergency actions? The only time I can think of such a thing is (as you partially suggested) is when the DBA and the Developer are the same person in a very small environment.
I think it's not as binary as developers should never touch production or developers should only touch production in an extremely small shop or developers should have full access to production. Other than the small shop with only one or two DB people situation another situation is say a small team is in charge of a single application within a larger organization, they wouldn't need access to do everything everywhere in production but within the realm of that application it would make sense. So that would also add the element of server architecture in as well.
There's also the people element as well, some developers shouldn't be allowed anywhere near a production environment, if you can't handle making changes in dev and following the migration chain you probably shouldn't be in the same room as anything related to production. But some developers can, I also generally feel if you don't trust your developers to at least some degree in production why are you even trusting them to develop for you in the first place(at some point everything they do ends up in production)?
September 7, 2018 at 3:13 pm
If you are making changes in production you should have in place a change management process which generally prohibits ad hoc changes to production systems. One should have at the very least a dev/test and production environments and preferably discreet dev, test and production environments. Changes should be made in development and reviewed before deployment to test. If an existing application or database exists, the test environment(s) should look exactly like production before deployment of changes in dev are deployed to test. After passing muster in test, and after taking a backup, only then should changes be deployed to production. Who carries out these tasks may vary from org to org but these inconvenient hurdles should be maintained to protect your production environment. It's your data and business so don't screw it up.
September 7, 2018 at 5:23 pm
ZZartin - Friday, September 7, 2018 12:21 PMJeff Moden - Friday, September 7, 2018 10:44 AMZZartin - Friday, September 7, 2018 8:07 AMJeff Moden - Friday, September 7, 2018 7:46 AMZZartin - Friday, September 7, 2018 7:39 AMIt really depends on your organization and just how often these kind of situations arise, not to mention what level of accountability you want the developers to have in production. If it's a case of them just not wanting to follow standard procedure once in awhile then it probably doesn't make sense. On the other hand if this is something they're doing routinely and you trust them to do it and are willing to hold them accountable if they mess something up then it might make sense.I have to disagree. Even the standard procedure should clearly identify any and all exceptions including those that are required for supposed emergencies. And, yes, they should be "exceptions" and they should be rare. You have to protect against the mistakes that you know are going to happen because you've already seen them happen many times, I'm sure. Even if you totally disagree with that, the auditors won't. 😉
Well most auditors don't know they're head from a hole in the ground, all you have to do is change the DB developer's title to DBA and they won't know the difference between who should and shouldn't be making production changes 😛
For a lot of places it doesn't make sense to say people who do development should never touch production, that may or may not be the case here.
To continue a good discussion, when does it make sense to generally allow Development to write to production outside of authorized emergency actions? The only time I can think of such a thing is (as you partially suggested) is when the DBA and the Developer are the same person in a very small environment.
I think it's not as binary as developers should never touch production or developers should only touch production in an extremely small shop or developers should have full access to production. Other than the small shop with only one or two DB people situation another situation is say a small team is in charge of a single application within a larger organization, they wouldn't need access to do everything everywhere in production but within the realm of that application it would make sense. So that would also add the element of server architecture in as well.
There's also the people element as well, some developers shouldn't be allowed anywhere near a production environment, if you can't handle making changes in dev and following the migration chain you probably shouldn't be in the same room as anything related to production. But some developers can, I also generally feel if you don't trust your developers to at least some degree in production why are you even trusting them to develop for you in the first place(at some point everything they do ends up in production)?
I'd tend to disagree with that use case as well. Even if you're a small team, unless your department is 100% isolated from all other teams, there would be very few reasons why every application would have willy nilly processes and single points of failure such as what this would invite. You would have to coordinate with others, pre-agreed to downtime windows, etc... and thus a need for this tiny application not to tear a hole in the fabric of space within the organization.
Even in the use case Jeff mentioned, unless there is just one single IT person (i.e. the entire IT and technical staff = 1), I'd make sure my code is reviewed by someone else and deployed only after that has occurred. If it can't be reviewed - there's ALWAYS a heads-up e-mail etc...
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
September 8, 2018 at 12:40 pm
ZZartin - Friday, September 7, 2018 12:21 PMJeff Moden - Friday, September 7, 2018 10:44 AMZZartin - Friday, September 7, 2018 8:07 AMJeff Moden - Friday, September 7, 2018 7:46 AMZZartin - Friday, September 7, 2018 7:39 AMIt really depends on your organization and just how often these kind of situations arise, not to mention what level of accountability you want the developers to have in production. If it's a case of them just not wanting to follow standard procedure once in awhile then it probably doesn't make sense. On the other hand if this is something they're doing routinely and you trust them to do it and are willing to hold them accountable if they mess something up then it might make sense.I have to disagree. Even the standard procedure should clearly identify any and all exceptions including those that are required for supposed emergencies. And, yes, they should be "exceptions" and they should be rare. You have to protect against the mistakes that you know are going to happen because you've already seen them happen many times, I'm sure. Even if you totally disagree with that, the auditors won't. 😉
Well most auditors don't know they're head from a hole in the ground, all you have to do is change the DB developer's title to DBA and they won't know the difference between who should and shouldn't be making production changes 😛
For a lot of places it doesn't make sense to say people who do development should never touch production, that may or may not be the case here.
To continue a good discussion, when does it make sense to generally allow Development to write to production outside of authorized emergency actions? The only time I can think of such a thing is (as you partially suggested) is when the DBA and the Developer are the same person in a very small environment.
I think it's not as binary as developers should never touch production or developers should only touch production in an extremely small shop or developers should have full access to production. Other than the small shop with only one or two DB people situation another situation is say a small team is in charge of a single application within a larger organization, they wouldn't need access to do everything everywhere in production but within the realm of that application it would make sense. So that would also add the element of server architecture in as well.
There's also the people element as well, some developers shouldn't be allowed anywhere near a production environment, if you can't handle making changes in dev and following the migration chain you probably shouldn't be in the same room as anything related to production. But some developers can, I also generally feel if you don't trust your developers to at least some degree in production why are you even trusting them to develop for you in the first place(at some point everything they do ends up in production)?
I'll have to agree to disagree. Unchecked deployment to prod will always cause problems that could be easily avoided. Developing in prod is just "death by development". Note that I'm not saying that you won't ever need prod data to get a particular job done. You just don't need to do the job in prod to get the job done.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 8, 2018 at 7:07 pm
Jeff Moden - Friday, September 7, 2018 10:44 AMTo continue a good discussion, when does it make sense to generally allow Development to write to production outside of authorized emergency actions? The only time I can think of such a thing is (as you partially suggested) is when the DBA and the Developer are the same person in a very small environment.
Well, I think that restricting Dev updates of Prod to emergencies only is the sort of artifical barrier that can quickly degenerate into harmful bureaucracy. An update may need to be applied using some pretty nasty and complex procedure, and in those cases it may be better if development actually carries out that procedure - they will have had far more expeience of doing that sort of thing than the production team, because they regulrly experience working with faster changing versions of the system than production is used to. To get things right it is essential that development is test-oriented at every level of granularity; then someone needs to be responsible for carrying out quality checks on what development have produced; and then someone has to be responsible for determining whether development's proposed method of deployment (which will usually be use a procedure which production is used to following) is acceptable (that requires tests including verification of the duration of deployment to an instance of the system). The production teams have to be able to veto deployment if they think that proposed method is too disruptive (eg will keep the production system out of service for too long while the updates are being deployed). But when it comes to actually updating the production systems, if the deployment method requires people with a clear understanding of how the upgrade works, why things need to be done in a specific order, and what has to be done when idiosyncracies in a particular production system necessitate variations (all of which, including any potential variations in the deployment method, has to be documented and agreed and signed off by production and quality control and production before deployment to production systems begins) it will sometimes be clear that the lowest risk way to deploy the updates is to allow development to do it, or to have development thoroughly monitor and supervise production while production does it.
This examination of what needs to be done, and how (and by whom0 should be carried out for every update, including emergency updates (for emergencies the evaluation will have to be carried out very fast). It should not be restricted to emergency cases. Nor should it be restricted so that in every non-emergency case the deployment should be carried out by production using whatever process they like or so that in emergency cases the development team could update production systems without any evaluation of whether it would be better to allow the production team to handle the update - proper evaluation is essential in all cases.
Probably what I'm describing is less efffective and less desirable when all the production systems running the (software) product use identical hardware and have interfaces to identical other software, and the porduction team and the development team work for the same company, and all the production systems have identical functionality. But if you work for a computing or software business, as opposed to for a company whose only connection with computing is that it has a has one of two computers and has developed some applications that help it run its business, life won't always be like that. For me, between 2002 and 2009 (my last job before I retired) life wasn't like that - and in most of the jobs that I had before 2002 life wasn't like that either, in fact for a total of about 32 years out of the 42 years I worked after finishing my second degree life wasn't like that. So I'm in favour of the idea that development and production often work best together without being stuck with bureaucratic nonsense that tries to make things go better by having a great big barrier between them.
Tom
September 9, 2018 at 10:03 am
TomThomson - Saturday, September 8, 2018 7:07 PMJeff Moden - Friday, September 7, 2018 10:44 AMTo continue a good discussion, when does it make sense to generally allow Development to write to production outside of authorized emergency actions? The only time I can think of such a thing is (as you partially suggested) is when the DBA and the Developer are the same person in a very small environment.Well, I think that restricting Dev updates of Prod to emergencies only is the sort of artifical barrier that can quickly degenerate into harmful bureaucracy. An update may need to be applied using some pretty nasty and complex procedure, and in those cases it may be better if development actually carries out that procedure - they will have had far more expeience of doing that sort of thing than the production team, because they regulrly experience working with faster changing versions of the system than production is used to. To get things right it is essential that development is test-oriented at every level of granularity; then someone needs to be responsible for carrying out quality checks on what development have produced; and then someone has to be responsible for determining whether development's proposed method of deployment (which will usually be use a procedure which production is used to following) is acceptable (that requires tests including verification of the duration of deployment to an instance of the system). The production teams have to be able to veto deployment if they think that proposed method is too disruptive (eg will keep the production system out of service for too long while the updates are being deployed). But when it comes to actually updating the production systems, if the deployment method requires people with a clear understanding of how the upgrade works, why things need to be done in a specific order, and what has to be done when idiosyncracies in a particular production system necessitate variations (all of which, including any potential variations in the deployment method, has to be documented and agreed and signed off by production and quality control and production before deployment to production systems begins) it will sometimes be clear that the lowest risk way to deploy the updates is to allow development to do it, or to have development thoroughly monitor and supervise production while production does it.
This examination of what needs to be done, and how (and by whom0 should be carried out for every update, including emergency updates (for emergencies the evaluation will have to be carried out very fast). It should not be restricted to emergency cases. Nor should it be restricted so that in every non-emergency case the deployment should be carried out by production using whatever process they like or so that in emergency cases the development team could update production systems without any evaluation of whether it would be better to allow the production team to handle the update - proper evaluation is essential in all cases.Probably what I'm describing is less efffective and less desirable when all the production systems running the (software) product use identical hardware and have interfaces to identical other software, and the porduction team and the development team work for the same company, and all the production systems have identical functionality. But if you work for a computing or software business, as opposed to for a company whose only connection with computing is that it has a has one of two computers and has developed some applications that help it run its business, life won't always be like that. For me, between 2002 and 2009 (my last job before I retired) life wasn't like that - and in most of the jobs that I had before 2002 life wasn't like that either, in fact for a total of about 32 years out of the 42 years I worked after finishing my second degree life wasn't like that. So I'm in favour of the idea that development and production often work best together without being stuck with bureaucratic nonsense that tries to make things go better by having a great big barrier between them.
I totally agree (except for the Devs having "Write" access to prod). I've not explained our whole "emergency " process that supports even such emergencies. It's the same process that we always use except that it's greatly accelerated by having people in hot standby for the next step. And, yes, I agree that the Devs know the steps and the order, etc, etc. That's true with any deployment we do, which is done 100% by scripts.
There are cases when the data to be affected is simply not available in lesser environments than prod. That's why we have two things in place.
1. We do a restore from backup of the most important databases every night after backups complete. There are lot's of advantages to doing such a thing other than making sure the restores work and one of them is a place for such emergency development and testing.
2. We have also purchased technology with our SSD system that allows us to flash a copy of the entire production box (several TB) to a special "standby" machine in 10 minutes or less. This is actually our "staging" box where we do a final deployment and test of all scripts that have made it through QA and UAT.
3. We also do "reduced size restores" of obfuscated data from prod to the lesser environments for development on a regular basis.
I agree with your next possible objection to that, being that not everyone can either afford any of that, doesn't know how to do that, or the data is simply too big to do such a thing. I will say that if you don't have a Development Environment that could also double as a test environment (we do have separate test environments), then you're on the road to hell for future unintended failures in prod that will need "emergency action" to repair.
In such cases, you do allow the Devs both "Read" and "View" objects in Production. The DBA can also help by quickly creating reduced copies of certain tables in lesser environments. The Devs, DBA, and the QA people all working in concert isn't anything new and has lately been included in a relatively new term called "DevOps", which is actually a very old and very effective culture that I've always insisted on since I started working with computers in earnest several decades ago.
The bottom line is, there is no reason for Devs to have "Write" access to production data and little reason to have "Read" access. At least not in the shops I've worked in. There's also no reason that code should ever be deployed to production without it going through some decent form of QA. Even better if it also goes through UAT.
And, to be sure, I used to be a Developer and I work very closely with the Developers. Part of my job is to protect the Developers because Management frequently will not. Management (not usually in IT) will frequently insist that they have to have something done this instant and the poor Developers are pressured by artificial schedule and willy nilly demands that make no sense when you shine a light on them. Because the Developers have frequently been made fearful of losing their jobs for being "uncooperative" if the say "No" even for all the proper reasons, they just up and do things. If they are simply denied the privs to deploy anything to prod, that angst is totally removed and I encourage the Devs to say, "I can't because I don't have the privs. You need to talk to the DBA and the Development Manager".
That's also a part of the reason why I sit with the Developers. I'm the junk yard dog with no chain and I will go to the wall for the Developers if anyone tries to compel them to violate the practices and safeguards that we've setup in our well documented standard and emergency operating procedures that have been approved by Mangers both in IT and several other departments, including the CEO.
And, to be sure, I did the same thing when I was "IT" as the sole person in IT. I never deployed code without a shoulder-to-shoulder review/demonstration with the Managers that asked for the functionality.
Does all that slow things down? You bet... especially the frequency of "emergencies". 😀
Getting back to the subject of this post, what about "simply" allowing Devs to control jobs? The answer is "No". They're busy enough. They don't have the time to do the research necessary to determine if jobs will conflict with other jobs. They don't have the time to determine if there are "hot spots" in the schedule that are already "full" for CPU and disk usage". And, many of them don't have the understanding that not everything can run at the top of the hour and than not all jobs can run once per minute just because the users want it to.
Protect the company by protecting the system, protecting the data, and protecting the Developers because all 3 are of critical importance to protecting the company.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 10, 2018 at 8:52 am
We don't let devs modify production code or processes. There is a line between development and implementation. Having said that we've hacked msdb a bit to allow some our devs access to view production jobs and job history, and to re-run jobs when needed. But they can't create or alter them in production.
September 10, 2018 at 3:05 pm
Jeff Moden - Sunday, September 9, 2018 10:03 AMI totally agree (except for the Devs having "Write" access to prod). I've not explained our whole "emergency " process that supports even such emergencies. It's the same process that we always use except that it's greatly accelerated by having people in hot standby for the next step. And, yes, I agree that the Devs know the steps and the order, etc, etc. That's true with any deployment we do, which is done 100% by scripts.
...
...
Getting back to the subject of this post, what about "simply" allowing Devs to control jobs? The answer is "No". They're busy enough. They don't have the time to do the research necessary to determine if jobs will conflict with other jobs. They don't have the time to determine if there are "hot spots" in the schedule that are already "full" for CPU and disk usage". And, many of them don't have the understanding that not everything can run at the top of the hour and than not all jobs can run once per minute just because the users want it to.Protect the company by protecting the system, protecting the data, and protecting the Developers because all 3 are of critical importance to protecting the company.
Jeff, I think you've missed the point that it's all about who you can trust. Or perhaps you believe that being trustworthy is something that is inevitably the natural state for producton people while developers are probably not trustworthy. Or maybe you know developers who operate differently from those I know, because you don't operate my rule that some-one who has not spent at least a few months doing customer support (maybe for you I should say "production support") can be anthing, as a developer, other than a very junior trainee.
So I suspect that you've not been faced with situation where the production staff for most of the systems you are responsible for don't work for the same company as you - they work for somone who buys software and services from the company you work for, and are often willing to deliberately deviate from an installation script in order to create down-time which can then be used to demand a reduction in your company's fees. Not surprisingly, the senior management of the company that employs those production staff usually regards that as unacceptable because such delays will affect their customers and reflect badly on them, and so they want us to actively prevent their operations people from doing any non-trivial upgrades in which they might add something to generate downtime . Bizarre, yes - but selling services in India and in the Middle East and some parts of Europe and North America means that at least some of your customers will have junior operations people like that and the senior management will know it but not be able to prove it against individuals - usually, not always. One customer (Middle East) thought they had no problem although every time they escalated a problem up to my level we found that the customer 's network had been carefully revised to prevent our customer support people and our fixit experts (developers, of course) getting access. Dealing with those people was not fun - when every "we have a problem" is immediately followed by "oh, we seem to have lost the ability to connect you to the machines" that's a pretty clear indication that the production team is dishonest. It was still going on when I retired - that customer's IT managers were either extremely dim or part of the problem.
I had some real fun with one very different customer, their IT director was having difficulty with concealing his amusement as one of his people repeatedly tried - after having been given database SA and OS full privilege to allow him to experiment - to install some unauthorised software (ie software that my modified Windows embedded client operating system did not recognise as authorised) on one windows (embedded) based client terminal, using his domain operator full privilege, and discovered that when he tried to use unauthorised software after installing it it didn't work and when he looked for it after the installation had run "successfully" there was no sign of it in the system, nor in the OS directory. That convinced the customer IT director that he could trust our system, and also convinced the guy he had trying to bypass it that when he (the IT director) insisted that servers running the server-side components of our softwares would have no sysop passwords available to his staff that was probably the right thing to do - perhaps because it would stop them wasting their time as he had for a day or two while trying to insert unauthorised software.
Strangely enough, I found most of our customers in the UK had asked us to ensure that privileged passwords were unavailable to their staff too - it turns out that production staff also want to demonstrate how clever they are, and sometimes go beyond what should be their limits. The one customer who insisted on their staff having privileged passwords for machines running our systems had a catastrophic influx of advertisements for pornography into the system, which was a source of complaint from several of their customers and probably the cause of our contract with them not being renewed at next review - it was that incident that triggered me into getting the "only authorised software" checks developed for the client machines - I can't remember whose virus/worm/etc protection software we used on customers' servers, except that it actually worked when it and some MS defense software were both running and that it certainly wasn't Symantec as their stuff was both underperformant and vastly overpriced back in those days (about 13 or 14 years ago). None of our other customers ever had any problems like that, even without that "only authorised software" mechanism on the clients. We gave one other customer (Barbados Sandy Lane Hotel) sysadmin access on all servers running our software - a pity, really, as otherwise some of our staff would have been able to go out there regularly, and they'd have regarded it as a great bonus - but those people really did know what they were doing and really (I think) were honest.
I don't see how working in that environment I could have required the company to say "OPs does all the upgrades". It's just not workable, obviously. And an awful lot of people are in situations like that. It is quite simple a question of who can you trust to do it right - and in your situation at Proctor Financial I would be amazed if you couldn't trust the company's production people - they are part of your people (just as I trusted in-house production although I often didn't trust out-of house production).
And I agree completely about he need to protect the developers from managers. It's something I tried for most of my working life. And I hope (and believe) that I mostly succeeded in doing that.
Tom
Viewing 15 posts - 1 through 15 (of 17 total)
You must be logged in to reply to this topic. Login to reply