July 19, 2010 at 9:40 pm
Comments posted to this topic are about the item Incompetent or Malicious
July 19, 2010 at 11:51 pm
I would say both. They knew what was wanted but gave what they thought was best. I hate it when I want to buy something and people want to tell me it's not good. Good advice is always welcome but I can see when someone is just trying to give you something that is not right for you. If I want to buy a brown bread someone must not try to sell a white bread to me.
:-PManie Verster
Developer
Johannesburg
South Africa
I can do all things through Christ who strengthens me. - Holy Bible
I am a man of fixed and unbending principles, the first of which is to be flexible at all times. - Everett Mckinley Dirkson (Well, I am trying. - Manie Verster)
July 20, 2010 at 1:50 am
I'd have felt a bit better about drawing a conclusion had the blog entry indicated that the guy said "I need a X terabytes more space on a minimum Y disk RAID 0+1 array. Let me know the LUN."
Ask for a terabyte or two in a day and age when multi-terabyte drives are available...
There is also the fact that there is no indication of whether or not the SAN guys knew the average anticipated read/write rate - but there is an appearance of performance warnings which were accepted, leading any normal person to think "Huh...must not matter.".
That latter conclusion could be seen to be a normal one, when the databases of "stuff" that rarely get hit at all has grown in an exponential relationship to Oracle version numbers, just nobody wants to be responsible for archiving them and purging them off.
Of course, I'd have to know that the SAN guys really did know their SCSI, too; might have just been an occasion for simple training....or for the generation of a lovely new form:
1) Space required
2) RAID level
3) Anticipated read/write rate
4) Whose budget
That last...that can have an awful lot to do with what you get.:-D
Jumping to conclusions doesn't work for managers, either - it is just easier to get away with after a certain level.
July 20, 2010 at 2:11 am
If collaboration isn't considered a natural behaviour in an enterprise, it's up to the top level managers to enforce it !
All for the best interest of the company in the long term.
If you discouver games as stated by the blog, untangle them at the level needed to enforce this.
If still people or teams refuse to work together for the companies best interests, it's up to the managers to draw the right conclusion(s). The soft way or the hard way.
As it is a dba's task to ask the same questions time and again, it's up to the san admins to do the same to manage their infrastructures to the optimum.
The worst question, is always the one that hasn't be asked.
Johan
Learn to play, play to learn !
Dont drive faster than your guardian angel can fly ...
but keeping both feet on the ground wont get you anywhere :w00t:
- How to post Performance Problems
- How to post data/code to get the best help[/url]
- How to prevent a sore throat after hours of presenting ppt
press F1 for solution, press shift+F1 for urgent solution 😀
Need a bit of Powershell? How about this
Who am I ? Sometimes this is me but most of the time this is me
July 20, 2010 at 2:11 am
Surely though Stephen, a decent SAN Admin would recognise that (s)he hadn't been given enough information or that the person making the request didn't understand SANs enough, and then ask the appropriate questions.
Are you suggesting that when somebody comes to me and asks for some account sales data, that it's pefectly fine for me to just give them a count of the sales for the current day?
Surely I would be expected to dig into their question a bit and suggest something for them based on what they're trying to achieve?
July 20, 2010 at 3:07 am
Without being there is no way we will ever know how the undocumented conversation actually went down but I can say that something this important should have been recorded in a scheduled meeting with all the players present, so that the original request and all questions and answers as well as concerns could have been put on the record. I have been to many meetings like this in the past where the minutes were actually recorded. Issues this important shouldn't be handed down by the network guys to DBA's to managers by word of mouth. "Well, that was what they told me..." is a dog that won't hunt. However, with that said, I learned long ago that successful managers ask better questions/requests, as a result, they get better answers. A manager once told me "I don't care about learning about technical stuff." My response was "How can you ask intelligent questions if you are ignorant about your technical environment? His response was "that is what I pay you for." My immediate response was "Sure, you do, but how does that alleviate you from asking intelligent questions about an environment that you are ultimately in charge of?" He did not have an answer. The bottom line is this, just because you are a manager does not automatically preclude you from doing research and asking pointed questions. So, do not depend solely on your staff to keep you informed, sometimes they don't. 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
July 20, 2010 at 5:31 am
The answer to your question - from my perspective - lies within the answer to the question: Is the DBA responsible for protecting the data in the database, or does that responsibility fall into the realm of the SAN hardware jockey?
Me, I never left it up to a hardware guy; should a disk or array fail, his or her job ended after 15 minutes work slamming new disks in. It was going to be me who had to spend hours or days recovering what could be recovered from backup; it was me who was going to have to explain the particulars of the failure and what was lost and what was recovered; it was me who was going to be in the position of finger-pointing at the hardware jockey because my data - my responsibility - had disappeared as a result of insufficient communication.
The principle of CYA would seem to dictate inquiring as to the physical nature of the storage.
July 20, 2010 at 6:41 am
stephen k. gartrell (7/20/2010)
The answer to your question - from my perspective - lies within the answer to the question: Is the DBA responsible for protecting the data in the database, or does that responsibility fall into the realm of the SAN hardware jockey?Me, I never left it up to a hardware guy; should a disk or array fail, his or her job ended after 15 minutes work slamming new disks in. It was going to be me who had to spend hours or days recovering what could be recovered from backup; it was me who was going to have to explain the particulars of the failure and what was lost and what was recovered; it was me who was going to be in the position of finger-pointing at the hardware jockey because my data - my responsibility - had disappeared as a result of insufficient communication.
The principle of CYA would seem to dictate inquiring as to the physical nature of the storage.
Ok, whose responsibility is it when a disk drive fails and you lose your database because the drives it sits on overheated and failed because the air conditioning in the server room stop performing and the tempature in the server room went up to 100 degrees? Is that the DBA's fault? Is that the network admin's fault? Or the air conditioning guy that hasn't serviced the air conditioning in a year? Does it really matter at that point? My point is finger pointing after the fact is really irrelevant many times and does not really matter anyway. Your database is gone. You cover yourself by exploring as many high availability options as possible BEFORE things happen.,But even then Murphy's Law can still get you but it is better than just taking somebody's word for it, and the only way to do that is to be armed with ALL of the options and the only way anyone including managers, can do this adequatelty is to ask pointed questions and do their research. Managers should not automatically get a pass on doing this just because they have staff under them to tell them this IMHO 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
July 20, 2010 at 7:24 am
"The LUN was a single disk instead of an array, and used this to prove they should be in charge of disk design when performance problems ensued."
My comment is this. I find it unclear exactly what was asked for, but that after the fact there where issues due to only one disk or spindle being included in the LUN.
There is no information on why this happened. It seems some type of blamestorm occurred supposing this was the SAN admin or hardware guys fault.
WHY?
If you don't listen to the mechanic, should you discipline or fire him when the car stops performing as needed?
Is that a good management trait in your world?
I have been told several times this when asking HAL<> data centers for a new LUN:
"Do to our data center requirements and limitation of hardware that will support a LUN of that size we can either put a single new disk in an existing cabinet for your account, or add a new cabinet <$> 4 new disks <$> new controller cards <$> and implement your new LUN in that. However that will add to your monthly footprint costs <$>... etc.. etc...
I would bet that a similar conversation happened with the people involved in this instance.
July 20, 2010 at 7:53 am
The DBA, San admin and manager "share blame", but as a DBA, our first job, IMHO, is to protect the data. That means being involved in every aspect of the process of managing the data which includes calling out specs or at least knowing the storage specs and verifying that they deliver the performance and redundancy that we ALWAYS use (don't we, in a SAN?) At least, I have always assumed that anything I touched was my responsibility, including any knowledge of things I cannot or do not PHYSICALLY touch. Yes, I know, any one getting paid to be a SAN admin should know better and he deserves to be called on the carpet, or maybe the guy who was supervising him if he was inexperienced (why would he be making new LUNs again?).
Bottom line, plenty of blame to share, but you know that YOU, as the DBA, will be the one that everyone expected to take care of it. So be the one and watch the value of your stock as an employee continue to rise. A smart manager once told me, "Anyone can find problems. Smart people find solutions." Every day that we are not swamped with problems (does that exist? 😉 ) is a day to make something better. I guess having been a manager once gives me a different perspective. I try to be an employee everyday that I would want working for me. It makes me a better (and happier) employee and makes my company love me ("They like me, they REALLY like me!")
KILL 51
Peter Trast
Microsoft Certified ...(insert many literal strings here)
Microsoft Design Architect with Alexander Open Systems
July 20, 2010 at 8:01 am
There is enough blame to go around on all sides.
1) The DBA made a request for disk space, but assumed that 1TB would be allocated across multiple spindles without specifying what it is he expected (i.e. multiple spindles).
2) The SAN Admin upon receiving the request granted the 1TB of space, taking the easiest route to store the information by selecting a single spindle. Request filled.
3) Just because the request came from a DBA (or DBA group) does not automatically mean that they need 1TB of high-performance disk space. How is the SAN Admin supposed to know if this is just space to store backups, other files, or as a script repository if no one provides the parameters?
The moral of the story here is pretty clear to me. When you request SAN space it is ESSENTIAL that the parameters be specified.
A conversation in the Datacenter below the Emerald City, Land of Oz:
DBA: "I need 1TB of space for a highly transactional database. Ideally I need this spread across 10 spindles. Can you create a LUN like that for me?"
SAN Admin: "No, I can't create something exactly like that, but I can make sure that you have top performance. Does the 1TB need to be contiguous space or is the 1TB the total of space you need for multiple files/stores"
Yes, I live in an idealistic world where administrators get along. I realize that exchanges as shown above are rarely this civil, but why shouldn't they be?
We expect children to get along on the playground, why should we expect less of ourselves as Technical Administrators?
Regards, Irish
July 20, 2010 at 8:09 am
Having been in the position of knowing that my managers are asking the same question about me, I have some sympathy for the SAN guy. In my case, I was put in charge of hardware although my training is primarily software, so from my perspective, I was in over my head, or if you want to be crass, incompetent.
It is up to management to know what their people are capable of and to listen to their feedback so they know when that team meeting is needed. The article says, "the san guy warned us", so something was said, and it appears to me it was not heard. MiketheGeek's comments on the article are very good, and Sean's response clears some of this up.
FYI, I never did anything malicious. In all cases, I would find ways to present the cost/benefit analysis of what I was doing or what I saw happening. I have been amazed at how short sighted management usually is about that sort of analysis. The difficult cost to demonstrate is the cost of making last minute decision with little thought, or consistently asking people to do things in a shorter time than they say they can.
July 20, 2010 at 8:27 am
Sounds like no one read his follow-up post on the blog:
"...However, in this case we had been on a couple calls with EMC about this and even had their top engineers on the phone with us. I had been telling them for 2yrs that we could have LUNs larger than 68GB but my SAN guys kept saying that it was impossible w/o killing performance. Then EMC agreed with me and we all agreed that doing this wouldn’t be a huge issue as long as we did it right. This is a company with gazillions of dollars to throw at hardware and they have a HUGE warehouse… so money isn’t the issue. When all was said and done, the SAN guy decided he didn’t like the DBA showing him up in front of his main vendor so he put a single drive in there instead of the 20 we had agreed on. On both calls with EMC we all agreed that having plenty of disks was essential. So not only was there no miscommunication, there was no middleman. There were only 2 SAN guys and I submitted the request myself directly to them. I get that you’re trying to smooth things over and give alternative reasons why this happened, but I just don’t see much wiggle room in this case."
I usually tend towards incompetent as the most likely explanation but malicious is possible, although in this case, I would call it childish or petty. If the situation was really as he described, I would say firing the SAN guy is justified.
July 20, 2010 at 8:31 am
That follow-up came in while I was posting. So any post before that would have valid questions. Although I see no reason to question his judgement, we are left with the choice of accepting his interpretation or not.
July 20, 2010 at 8:34 am
Michael,
Thanks for that note. I had read that comment, and didn't get the point across in the editorial. The SAN guys deliberately put less disk behind the LUN. It definitely is petty, and if it had been done before, I might be looking for a new SAN guy.
Viewing 15 posts - 1 through 15 (of 62 total)
You must be logged in to reply to this topic. Login to reply