We have a couple of boxes at a local Denver co-location facility for the training business, SQL Share. Our firewall died a few weeks ago, and so I called a friend that I've typically used for network stuff. He found us one on eBay for $200 and last night was the time to install it. We'd been trying to coordinate things, and last night worked.

I'd given him the IPs and setup, and some preliminary work had been done yesterday, but there are things that you can't test until you get the real network set up. I should have known things were bad when we arrived and I couldn't get Jordan in at first. The security is tight and I hadn't said "2 people" so we needed to wait and get authorization from the company I rent space from. Once that was done, we got in and needed to get cage nuts into the rack for the firewall.

There's a shelf right above us and we had little space. I managed to use a screwdriver to get the top two nuts in. Then I went to get the last one in and couldn't. So I tried to squeeze it with finger and slipped, cutting below the thumbnail, and starting to bleed a little. While I sucked on it to get the bleeding stopped, Jordan told me I didn't need to mess with it that way and then proceeded to do the same thing.

A nice delay while both of us stopped the bleeding. Then we mounted things, got it plugged in, and started to configure it. Jordan did the work while I stood around, coughing in the extremely dry air of the colo. Fortunately we were in the hot aisle, and not the cold one, but it was still hard.

We had issues getting the firewall to first allow things out, and then to allow things back in. It's a little flaky to work with IOS, and you have to go slow. We kept having issues and things went slow. Our expected 30-60 minutes turned into 120 almost. That wasn't the way I wanted to spend a Saturday night, but it wasn't all bad. I saw a guy from the Denver User group there who was rebuilding a few boxes for his company and expected to be there for hours.

We finally narrowed it down to the ARP cache in the ISP switches. At the lowest level of networking, the switches are often set up with specific groups of ports working together. To speed the movement of packets, the switches cache the MAC address of your network card. That's the address burned into the card. It's similar to the IP address you have and the DNS caching your machine does. In this case it means when we unplugged the server and plugged that cord into the firewall right away, the switch thought it was a momentary interruption and didn't reset the ARP cache with the MAC addresses in it's memory tables.

Once we realized that our rules were correct, we went to the Network Operations Center for the colo and had them clear their switch's cache. That cleared things up and once that was done, things looked OK. Jordan drove to Barnes and Noble to check things from the outside while I hit the bank for funds.

We met up, thinks were working, and we called it a night.