I've been out of pocket and off the forums for the last few weeks as

we've had several major tasks where I work with respect to our Active

Directory migration. This past weekend involved application migrations.

Because of the nature of the migration, we provided test environments

prior to "flipping the switch" this weekend. Some application team seem

to have taken great advantage of those environments. Those applications

came across with no real issues. Others, however, weren't so

successful. It wasn't until we flipped the switch in the production

environment that they realized parts of their application broke. This

caused us to go back and ask a very simple question, "Did you test this

(whatever it was that was broken) in the test environments?" The

typical response we received was, "We didn't think to do so." That

brought up the next question, "These weren't in your test plan?" And we

got the expected response, "Unfortunately, no."

Now before anyone goes throwing shoes, fruit, or anything else at these

application teams, the fact of the matter is I've seen this before in

every other place I've worked. When it comes to testing a system or

application, writing a thorough test plan isn't an easy task. It sounds

easy enough to do but when you start to write one, especially for a

complex application, the reality is that a comprehensive test plan is

extremely difficult to create. Sure, the concepts are easy: find out

what key business processes the application provides and ensure those

work properly. But what is meant by work properly? And how many

different cases do you have to test per business process? Do you test

them all? Do you test just the major ones? What level is considered

satisfactory?

Testing isn't easy. It's time consuming, it's resource draining, and

it's very hard to define, at least in the software world, what is a

successful test and what isn't. And the problem just isn't limited to

the software world. For instance, how do you test a bridge's maximum

weight without destroying it? How about an elevator? We see the little

sign that says maximum capacity, but how do we know for sure that

elevator can take that maximum weight level? Sure, there may be a

yearly review by an inspector, but does that maximum capacity get

looked at every single time? No, it does not.

With all that said, testing isn't something we scrap because it's

difficult. It does require time and resources and any successful

project should include time to develop and review the test plans as

well as the time it takes to actually use them. Sometimes in the

process of testing we determine we forgot things. This was the case

when we were looking at our test and implementation plans for the start

of the Active Directory migration, when we were tackling the underlying

infrastructure. Before we did step 1 in production, we ran through our

implementation and test plans in a stand-alone environment. We had a

simple philosophy: if we had to make a change to either an

implementation or a test plan, we'd run through it in the test

environment from scratch again. Because of that design philosophy we

had zero issues when we migrated over WINS, DNS, DHCP, and stoop up the

Active Directory domains. We had the luxury of doing this because of

the nature of our project: Active Directory. You don't want to get that

wrong when it comes to infrastructure. Most projects don't have that

sort of time.

However, I think an important principle to take from our initial

testing is we were ready to modify our test plans as we discovered

gaps. We didn't sit on them for a month or two until the implementation

was done. We reviewed and changed them immediately while the gaps were

still fresh in our minds. This ensured that even though we may not have

gotten everything right in the initial test plans or subsequent

revisions, we'd have that issue resolved for the next time.

Test plans... no one loves them but most folks understand the need for them.