I've been out of pocket and off the forums for the last few weeks as we've had several major tasks where I work with respect to our Active Directory migration. This past weekend involved application migrations. Because of the nature of the migration, we provided test environments prior to "flipping the switch" this weekend. Some application team seem to have taken great advantage of those environments. Those applications came across with no real issues. Others, however, weren't so successful. It wasn't until we flipped the switch in the production environment that they realized parts of their application broke. This caused us to go back and ask a very simple question, "Did you test this (whatever it was that was broken) in the test environments?" The typical response we received was, "We didn't think to do so." That brought up the next question, "These weren't in your test plan?" And we got the expected response, "Unfortunately, no."
Now before anyone goes throwing shoes, fruit, or anything else at these
application teams, the fact of the matter is I've seen this before in
every other place I've worked. When it comes to testing a system or
application, writing a thorough test plan isn't an easy task. It sounds
easy enough to do but when you start to write one, especially for a
complex application, the reality is that a comprehensive test plan is
extremely difficult to create. Sure, the concepts are easy: find out
what key business processes the application provides and ensure those
work properly. But what is meant by work properly? And how many
different cases do you have to test per business process? Do you test
them all? Do you test just the major ones? What level is considered
Testing isn't easy. It's time consuming, it's resource draining, and
it's very hard to define, at least in the software world, what is a
successful test and what isn't. And the problem just isn't limited to
the software world. For instance, how do you test a bridge's maximum
weight without destroying it? How about an elevator? We see the little
sign that says maximum capacity, but how do we know for sure that
elevator can take that maximum weight level? Sure, there may be a
yearly review by an inspector, but does that maximum capacity get
looked at every single time? No, it does not.
With all that said, testing isn't something we scrap because it's
difficult. It does require time and resources and any successful
project should include time to develop and review the test plans as
well as the time it takes to actually use them. Sometimes in the
process of testing we determine we forgot things. This was the case
when we were looking at our test and implementation plans for the start
of the Active Directory migration, when we were tackling the underlying
infrastructure. Before we did step 1 in production, we ran through our
implementation and test plans in a stand-alone environment. We had a
simple philosophy: if we had to make a change to either an
implementation or a test plan, we'd run through it in the test
environment from scratch again. Because of that design philosophy we
had zero issues when we migrated over WINS, DNS, DHCP, and stoop up the
Active Directory domains. We had the luxury of doing this because of
the nature of our project: Active Directory. You don't want to get that
wrong when it comes to infrastructure. Most projects don't have that
sort of time.
However, I think an important principle to take from our initial
testing is we were ready to modify our test plans as we discovered
gaps. We didn't sit on them for a month or two until the implementation
was done. We reviewed and changed them immediately while the gaps were
still fresh in our minds. This ensured that even though we may not have
gotten everything right in the initial test plans or subsequent
revisions, we'd have that issue resolved for the next time.
Test plans... no one loves them but most folks understand the need for them.
IT Security, MySQL, Perl, SQL Server, and Windows technologies.