The 2014 Apple keynote was today, with the announcement of the iPhone 6 and iWatch taking center stage. I was excited, with an aging iPhone 4S that needs to be replaced. I tuned in, expecting to watch the announcements while taking care of some busywork on the other monitor.

At first I saw this screen (though at about 20 minutes beforehand)

About 10 minutes before the event, the countdown switched, and noted that my browser (Firefox) wasn’t supported. I almost switched to Chrome, but then saw the fine print. I wasn’t thinking and didn’t take a picture as I ran to grab my iPad, but Karen Lopez posted an “altered” version of the Apple page.

Fair enough. Apple doesn’t have to broadcast to everyone and I get that they want to let their iPhone/iPad/iTouch/Mac users see it live. The keynote was available about 30 minutes after the end for everyone, so no big deal.

In fact, in the scheme of my life, not being able to watch live wasn’t a big deal. I can catch the “showmanship” an hour later.

However I was surprised when my iPad kept showing a test screen from the broadcast truck. When it connected, it was sputtering and halting, even when a speed test showed me getting 20+MBps at the device.

I did manage to watch a little, with a Chinese translation overlaid on the sound, at a higher volume than the speaker for me. Slightly annoying, but I could deal with it.

About 15 minutes in, things started crashing, with Safari on my iOS 7 iPad and iPhone crashing. The iPhone would start Safari and connect to the Apple home page and show this:

The iPad connected a few times and showed this:

I gave up, going on with my day, knowing I could watch it later. However that brings up the question.

Did Apple test in Production?

By this I mean, did they actually test the broadcast of this event today, with the live event, or did they test things last week on a copy of what this environment would look like? Did they actually run through the same settings for broadcast that they would do today?

I do suspect they did some testing, and certainly it’s possible that many, many more people than they expected tuned in. They may have been overwhelmed, and that’s understandable. I doubt anyone has any good way to determine what capacity to plan for, other than to just guess. This event might have been larger than the Super Bowl, World Cup, etc.

Or maybe Apple didn’t want to bother scaling to a capacity beyond some xx% of Apple device owners.

Any of those is fine with me, and I can understand someone deciding to spend $yyy on this, though I’d argue it was a mistake since this is a great chance for an amazing amount of publicity and hype for their products.

However there are certainly other problems that I question the level and detail devoted to testing. The overlay of foreign languages means that someone hadn’t configured their audio correctly, and more importantly, no one was monitoring to stop the problems quickly. The problems with the stream meeting devices might have been capacity, but crashing the browser? Either there’s a major QuickTime issue, or the encoding was broken in some way (perhaps again, by capacity).

This is a shot at Microsoft as well, but I thought Tim Ford’s summary was spot on:

We talk about thorough testing on a copy of production. A real, scale, same sized version. Few of us do it, but certainly some do, especially those with big budgets. Apple has big budgets, so I suspect that their testing was cut short, or someone assumed things would work the same way they had in the last keynote, which went off without a hitch for me.

Complacency, coarse attention to detail, poor quality control and monitoring seemed to be in play today. None of which as a good thing.

Perhaps Apple was testing in production.

Filed under: Blog Tagged: software development, syndicated