When the census was last taken in the US, it was still mostly a paper affair. My wife and I sat down with the form, filled it out, and mailed it back to the US government. While much of the data is publicly available, the process of gathering the data was primarily an analog process. That was in 2010, and governments are trying to do things differently. I suspect our 2020 census in the US will be mostly a digital effort, though I hope the companies setting up the applications learn from other countries' issues.

Australia had their census this year, 2016, and they planned on having most citizens complete the form online, with a requirement to opt-in to get the paper version. However, things didn't go smoothly. There was a DDOS attack on the night the census was set to collect data. There were also concerns from many people over data privacy. All in all, not a great showing for the Australian Bureau of Statistics and their IT contractors. Things got worse in the weekend after the census was scheduled when a few university students built a website designed to handle a larger load than the government over a weekend for substantially less money.

To be fair, the university students' site wouldn't necessarily meet all the requirements for privacy and data security. It also doesn't seem to have the validation and verification that a website needs. Not to mention a database setup to keep the data for later analysis. Those things would certainly need to be added and tested, but this does show that sometimes there are better ways to develop software than the way that has been done in the past. Large government projects get bogged down, perhaps worse than many corporate projects, but I'm not sure. Perhaps we just need people that code better, and certainly, people that manage developers better.

Certainly the cost of projects for the things like the census might not go down much. After all, I'm sure a fair amount of the cost charged by IBM goes back to the salespeople and engineers responsible for landing the contract. However, I also think that far too often there is too much overhead in large projects starts to obscure the most important part of any software system: it needs to work. I would prefer that even long term, large scale projects be developed in an agile fashion, and take on beta testers throughout (public or private). Let actual users play with the site in advance, report bugs, simulate a load, even watch the site develop in real time.

And when you're ready to deploy, from experience, I can tell you it's not too hard to delete all that transactional data before you go live.