I'm part of a small group presently in the task of developing a database and application which has the ability to become and is projected to become a VLDB. At some points, we have seen growth per day of around 2 to 10GB or more of data.

This challenge is ongoing and has been for the past 7 years, using all sorts of software and several core changes, each with their own advantages and disadvantages. Overall, it all comes down to how you plan to use the data.

In our particualr system, we have found that there is around 40gb of data that's constantly changing and updating. The rest is simply colleting and being stored for later use or archiving. Archiving is easy as you can distribute this data across multiple machines as necessary, for example, one of our developmental archives is currently using around 100 DB servers each with MySQL (cost biased for this quantity) holding around 70Gb of archive data each.

We have found that through parallell access and indexing, we have achieved our desired result, but this more comes from how it's programmed. We knew beforehand how the data is going to be used and have developed out scale-out based upon that. Without knowing how the data will be used (to a large extent), it would be almost impossible to handle a database over 100Gb.

All in all, it really comes down to the developer(s) behind the infrastructure. Time is money, but a wrongly developed database or quickly written one can cost far much more in the long run when a query that should take 200ms to run takes 200seconds or 200minutes.

Databases are going to be something that will continue to develop over time and as technology gets better, we will find better ways of handling things. At the present time, I/O handlers and companies are holding database development back as this is the core part of a database, without I/O there is no DB, conversly if your I/O is slow, it doesn't matter how good the database is or how well it's indexed it'll be slow too.

I think in the scale of VLDBs you need to have a much greater planning and projecting before implemtation, it takes a much broader thinking and thinking out of the box to achieve success. Always remember the golden rule... "Nothing is impossible."