Sometimes It IS The Network

Andy Warren, 2013-07-31

Over the years when a performance problem comes up there is always some speculation that it’s a network issue and not the database (can’t be us!). I always ask a few quick questions to see if I can see a reason to pursue the network angle:

  • Is the problem affecting multiple database servers?
  • Is the problem affecting multiple databases and/or multiple applications?
  • Is the problem experienced in a particular geographic location?

I won’t say it’s never the network, but usually when it’s the network everything is slow or down. Check, ask them to check, but assume it’s a database or application issue is my rule of thumb.


I worked with a client where everything was running fine. The server in question had the IP changed to meet some security requirements, it came back up fine and all seemed ok, except that jobs were taking 3x to 5x as long. Nothing changed but the IP. How could that cause a problem? Seems like network doesn’t it?

Network team swore it wasn’t them. No way could changing an IP affect performance. A more likely culprit given the reason for the change was the firewall. Firewall team swore it was not them. Database team goes back to look again, sees nothing wrong on the server. Changes IP back to old IP, performance is fine.

So what do you do now? No one sees a cause, but clearly something is wrong.

They flipped the IP back again, performance drops immediately. I still thought the firewall had to be the problem. I’m not a firewall guy so I’m pushing for details, what kind of rules are running, which rules are getting hit, etc, looking for clues. Finally with some arm twisting I have the firewall taken offline, removing it from the equation. Performance still bad. Firewall team mad. And yes, the database team was still sad.

Now we go back to the network team. We’ve proved that it’s running fine with old IP, miserable with the new IP. After agreeing to look again, reluctantly, because it makes no sense, they find the problem. The new segment had packet inspection enabled, the old segment did not. The high amount of data being transferred was maxing out the switch and that was the bottleneck. Turned it off, presto, all was well again.

So for once it was the network. I’ll probably never see that root cause again, but now I know to ask about it, just in case.





Related content

Database Mirroring FAQ: Can a 2008 SQL instance be used as the witness for a 2005 database mirroring setup?

Question: Can a 2008 SQL instance be used as the witness for a 2005 database mirroring setup? This question was sent to me via email. My reply follows. Can a 2008 SQL instance be used as the witness for a 2005 database mirroring setup? Databases to be mirrored are currently running on 2005 SQL instances but will be upgraded to 2008 SQL in the near future.

Robert Davis


1,567 reads

Networking – Part 4

You may want to read Part 1 , Part 2 , and Part 3 before continuing. This time around I’d like to talk about social networking. We’ll start with social networking. Facebook, MySpace, and Twitter are all good examples of using technology to let…

Andy Warren


1,530 reads

Speaking at Community Events – More Thoughts

Last week I posted Speaking at Community Events – Time to Raise the Bar?, a first cut at talking about to what degree we should require experience for speakers at events like SQLSaturday as well as when it might be appropriate to add additional focus/limitations on the presentations that are accepted. I’ve got a few more thoughts on the topic this week, and I look forward to your comments.

Andy Warren


360 reads