Having perused the responses, I find my fingers itching to send my own little opinion. 🙂

I live in a different world (non-IT) and have been investigating for a couple of years what exactly is the problem with software development. I knew I didn't have what I needed, but was curious as to why. Books don't have the answers. So I launched into some mini-development projects just to find answers. I didn't care so much whether they failed or were successful, but rather wanted to understand clearly what were the factors in play.

What I think I discovered is a HUGE opportunity. To oversimplify, failure is largely due to lack of preparation or resources (a subset of preparation). It doesn't matter who should have or didn't prepare. Blame is useless. But analysis can be very productive.

Management often does not understand the needs of technical people. Good technology lives in the realm of mind-numbing detail, with no shortcuts. So I see opportunity for management-type people to spend enough time and do enough homework to truly understand techies and speak their language. To fail to do that costs millions. To not understand that it costs millions and not discern the need for thorough, non-glamorous preparation is to identify oneself as unqualified for the position.

Technical people, on the other hand, often underestimate the complexity of what is needed (in my observations). They think that a good design can be derived from a board meeting where domain experts tell them what they need. That's like telling an aeronautical engineer that you need an airplane that can take off, fly, and then land. What kind of mess would you get with that kind of detail? So the developers go merrily off, make something that takes off, flies, and lands. But the user is wholly unimpressed.

This is where scope creep and change requests come from. I would argue that the user, including domain experts, are entirely unable to tell the developers what they need in the same way a pilot can't tell an engineer how to make a plane. The pilot knows when he sees what he wants and has what he wants, but is unable to articulate that in sufficient detail prior to starting work on the plane.

So what's the solution? Tiny, incremental, functioning prototypes. I know the industry mantra is to make prototypes as quickly as possible with no real code behind them. I understand the expense of putting a lot of time there. But being a "domain expert" in my own little world, I can tell you that from the time I designed a screenshot that I thought was "just right" until I really had a screenshot that WAS just right, was anywhere from 10 to 50 revisions average. Reason: you can't tell if it's "just right" until you actually use it in the real environment where it's proposed to live. It's surprising what surfaces that is good or bad about a screenshot when you're actually using it in real life. This means it connects to real data.

This would likely apply more to very complex work processes, not necessarily applicable in every situation. But most of the easy stuff is already done, leaving the more complex processes still to do. So much for my thoughts!!

Sam