About a decade ago I did some presentations on Filestream and Filetable in SQL Server. These were technologies that Microsoft added with the idea that lots of data would need to be linked to data in a database, but wasn't necessarily a good fit for a relational table. At the time, I looked at the growth of data, which was incredible back then. Statistics I had from that time were from the invention of the printing press for the next 500 years, we created 1 EB of data. In the 2011/2012 time frame, we created 2EB each day.
That's changed. I saw a blog on the story behind the explosive growth of data, which looked at a recent IDC report that showed we created and replicated (a strange term) about 64ZB of data in 2020. With a ZB being 1000 EB, we can see that a decade has brought us multiple orders of magnitude of data growth.
That doesn't mean we need to store all this data. The report notes that a lot of this is consumption data, which is transferred, but not saved. This means that we do need strong networks, but that most of the time the data is being viewed and not actually stored. That's certainly the case with things I see on Netflix, where I may end up transferring part of a show multiple times to get through it, usually because I fall asleep.
The estimate for the next 5 years, through 2025, does indicate that we expect to need 200% more storage capacity and that a lot of data storage is going to be at the edge, in devices and server machines, as well as in the core, made up of data centers, cloud and private. While plenty of this data will be outside of databases, and plenty will be in non-relational stores, this does tell me that working in the database business as a career is one that's going to be in demand for a while.
I've enjoyed my career working with data, and I expect to keep doing so for the foreseeable future. I am confident there will always be work for those that know how to work with data in a variety of situations. I also expect a very high demand, and high compensation for those that do it well.
