A Bad First Day

  • Comments posted to this topic are about the item A Bad First Day

  • At my last job, on day 1, they wanted me to look at a stored procedure I would be working with daily. I needed to copy and paste a table name in. Right-click, rename, Control C, escape. Only I didn't Control C. I Control X'd. Whoops.

    No one was the wiser. Thank you, clipboard. Thank you.

    --- Remember, if you don't document your work, Apollo 13 doesn't come home.

  • With regard to the Reddit article, what kind of deadbeat, out of his depth CTO fires somebody on their first day for a mistake like that? It's no way to treat a new hire, and they should never have allowed that situation to arise in the first place.

  • Yeah, I saw that article. The guy was a scapegoat, pure and simple. The fact he used the freaking manual to do his set up in any rational company would mean that:

    1) He'd be in a safe playpen, able to screw around to his heart's content and never even see production data, much less have full access to it (especially unawares!).

    2) Following the manual's instructions should have protected him from repercussions, even from such a monumental disaster. The firing was somebody high up covering their behind. Not cool.

    Clearly the manual he was given was intended for production set up, not development, and that was the true trigger of the disaster. Along with a chain of other stupid decisions on the IT architect's part.

    For instance our backup system has evolved  over time into a differential DR system with two geographically isolated redundant (and encrypted) backups for the onsite backup system. All three hold *months* worth of our data, we can restore any file as it existed on any date in the past. Including SQL Server databases.

    That gives me the warm and fuzzies. Triple safety nets are cool. 😎

  • Wow, that Reddit post was submitted only 11 days ago, and already it has 4,512 comments. It's quickly evolved into a full length novel. Based at least on the information provided in the original post, there was no criminal intent or negligence, essentially what happened is that a process failed due to a minor mistake in an environment that was poorly secured and engineered to protect against failure to begin with. It's very unlikely the organization will peruse any legal action against the author, doing so would gain nothing for the organization and would only draw attention to the organization's own incompetence. My suggestion at this point is that the author focus on landing a new job. He could even reach out the CTO and ask for a letter of recommendation, emphasizing the point that it would be best for all parties involved that the author simply move on to another position and the matter be kept private.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • I like your wording.

    There are any number of ways that a person can make a mistake on their first day. To prevent this, a good process should limit the damage that an individual can cause because of ignorance. This might be especially important for anyone that might have access to production data.

    While you make a point about "first day on the job", you didn't limit your statement to newbies.  The fact is that technology workers who have any idea what they are doing tend to recognize that mistakes can happen and process is a key way to minimize those mistakes.  The people on this forum who post frequently give me the impression that they try to find ways to limit their errors.  All of us should try to keep that in mind as we work.

    Dave

  • Actually getting dumped was probably a blessing. Any place run like that has got to be a terrible place to work.

    ...

    -- FORTRAN manual for Xerox Computers --

  • Yeah that's a pretty crummy way to start(and end) your first job.  It was refreshing to remember being that new to the industry, if someone handed me a document and told me to follow the instructions on it with no knowledge of the systems and tried to blame me for trashing a production server my response now would likely a two fingered salute not any fear over having done it.

  • I saw the article too (it's getting a lot of play on techie sites right now,) and the author did nothing incorrect.  Frankly, the blame starts with the person who wrote the directions for putting in / leaving in the information to connect to production, even if the directions spelled out "use the login information the install process will return."  SOMEONE is going to not read / skip / misunderstand / forget about that and use the included information.
    The second person at-fault is whoever allows access to the production server from development machines.  I know sometimes such access is needed, but ideally either the production should've been on a different subnet that it takes some hoop-jumping to get to, or blocked off completely from the dev network, or at least having a different login / password for full access (ie, the login in the document maybe grants read-only.)
    Lastly, any business that basically kills the messenger, is not going to be a good place to work.

    While I know there are mistakes in my documentation that I don't know about, I've always felt the best way to write documentation is to presume that your Mom / Grandma / 5yr old niece or nephew is going to be the one having to follow those directions.  Presume they'll need everything spelled out and anything that can cause a problem needs to be bold, italics, and a 24pt font with multiple warnings.
    And even then they'll manage to break it.

  • Beatrix Kiddo - Wednesday, June 14, 2017 6:18 AM

    With regard to the Reddit article, what kind of deadbeat, out of his depth CTO fires somebody on their first day for a mistake like that? It's no way to treat a new hire, and they should never have allowed that situation to arise in the first place.

    Plenty of them. I've known a few like this that would blame others, no matter what the circumstances.

  • djackson 22568 - Wednesday, June 14, 2017 8:18 AM

    I like your wording.

    There are any number of ways that a person can make a mistake on their first day. To prevent this, a good process should limit the damage that an individual can cause because of ignorance. This might be especially important for anyone that might have access to production data.

    While you make a point about "first day on the job", you didn't limit your statement to newbies.  The fact is that technology workers who have any idea what they are doing tend to recognize that mistakes can happen and process is a key way to minimize those mistakes.  The people on this forum who post frequently give me the impression that they try to find ways to limit their errors.  All of us should try to keep that in mind as we work.

    True, it's first chance to skill/task/process x when you haven't really done it recently. Easy to make those mistakes.

    A good reason  to practice skills and knowledge for DR or crisis situations.

  • If anyone should commit (proverbial) Hara-Kiri over this, it should be the CTO, not the DBA.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • I think the best cautionary tale for the business (not the poor guy who was the scapegoat in this case) was this line: "So i left. I kept an eye on slack, and from what i can tell the backups were not restoring and it seemed like the entire dev team was on full on panic mode."

    Whenever I read a comment like this, now that I have been lucky enough to find the people at SQL Server Central and similar database sites, I know that was a major root problem waiting to rear its ugly head. Having backups that don't restore made a bad problem (the dev manual writing, TBH) much, much worse.

    - webrunner

    -------------------
    A SQL query walks into a bar and sees two tables. He walks up to them and asks, "Can I join you?"
    Ref.: http://tkyte.blogspot.com/2009/02/sql-joke.html

  • jasona.work - Wednesday, June 14, 2017 9:32 AM

    While I know there are mistakes in my documentation that I don't know about, I've always felt the best way to write documentation is to presume that your Mom / Grandma / 5yr old niece or nephew is going to be the one having to follow those directions.  Presume they'll need everything spelled out and anything that can cause a problem needs to be bold, italics, and a 24pt font with multiple warnings.
    And even then they'll manage to break it.

    I generally write that kind of thing as if the first person to use the documentation will be the junior DBA who's on call on Christmas afternoon, and if he can't figure it out he's going to phone me

    Gail Shaw
    Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
    SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

    We walk in the dark places no others will enter
    We stand on the bridge and no one may pass
  • GilaMonster - Wednesday, June 14, 2017 2:11 PM

    I generally write that kind of thing as if the first person to use the documentation will be the junior DBA who's on call on Christmas afternoon, and if he can't figure it out he's going to phone me

    I use the most junior person, or an accidental sysadmin, to test the procedure as well. That will tell me where I'm making assumptions that will cause issues.

Viewing 15 posts - 1 through 15 (of 17 total)

You must be logged in to reply to this topic. Login to reply