The State of Data Growth

About a decade ago I did some presentations on Filestream and Filetable in SQL Server.  These were technologies that Microsoft added with the idea that lots of data would need to be linked to data in a database, but wasn't necessarily a good fit for a relational table. At the time, I looked at the growth of data, which was incredible back then. Statistics I had from that time were from the invention of the printing press for the next 500 years, we created 1 EB of data. In the 2011/2012 time frame, we created 2EB each day.

That's changed. I saw a blog on the story behind the explosive growth of data, which looked at a recent IDC report that showed we created and replicated (a strange term) about 64ZB of data in 2020. With a ZB being 1000 EB, we can see that a decade has brought us multiple orders of magnitude of data growth.

That doesn't mean we need to store all this data. The report notes that a lot of this is consumption data, which is transferred, but not saved. This means that we do need strong networks, but that most of the time the data is being viewed and not actually stored. That's certainly the case with things I see on Netflix, where I may end up transferring part of a show multiple times to get through it, usually because I fall asleep.

The estimate for the next 5 years, through 2025, does indicate that we expect to need 200% more storage capacity and that a lot of data storage is going to be at the edge, in devices and server machines, as well as in the core, made up of data centers, cloud and private. While plenty of this data will be outside of databases, and plenty will be in non-relational stores, this does tell me that working in the database business as a career is one that's going to be in demand for a while.

I've enjoyed my career working with data, and I expect to keep doing so for the foreseeable future. I am confident there will always be work for those that know how to work with data in a variety of situations. I also expect a very high demand, and high compensation for those that do it well.

Steve Jones - SSC Editor

Join the debate, and respond to the editorial on the forums

Over 2500 global database professionals answered questions on their experiences with database monitoring in our annual survey earlier this year. We’re creating four reports from the findings and in our first, The real-world challenges of growing server estates, you’ll discover both insights and trends along with advice for how to tackle the most common challenges.

The very definition of open-source software means that developers can change the code, which means that exclusive, paid, licensing is not a part of the deal. In short, open-source is free. Free of licensing anyway. So, why might you choose to pay for an open-source software like Flyway? Grant Fritchey explores.

One of the best scenes from Silicon Valley is Jian...

Click to learn more about author Seapee Bajaj. In ...

Query store hints enables changing query plan shap...

In the initial post, we talked about some very basic items to get you up and running with temporal tables. In part 2, I’ll cover a few more things to think about and implement when working with temporal tables.

SQL 2017 introduced the ability to pause and resum...

Trace flags are used to turn certain settings on or off in SQL Server. They can range from suppressing backup logs to affecting query plans. While some can be...

I sit on the porch and discuss your highly voted questions from https://pollgab.com/room/brento. Today’s discussion includes: 00:00 Introductions 00:27 DK: What was your Gordian Knot problem? 03:25 George: How...

In this article we cover how to implement a batch ...

In this article we look at using streams in Snowflake to set up a change data capture scenario where only new or modified rows are processed.

Microsoft provides many ways to run SQL Server in Azure, but which do you choose? In this article, Robert Sheldon explains the Azure SQL options.

This article will explore Azure automation for sca...

Click to learn more about author Ashok Sharma. When it comes to data storage, the temperature must be taken into consideration. The level of layered data storage service, from...

At Data Saturday Oslo 2021, I delivered the keynote called Turning Discomfort into Personal Growth in a Fast-Paced World. The recording is now available! Continue Reading I Can't Keep Up!...

Join Donovan Brown, Partner Program Manager, and Mark Fussell, Lead Program Manager from the Azure CTO Incubations team at Microsoft as they talk all things modern Cloud Native development with Dapr.

Data Platform Virtual Summit 2021 (DPS 2021) is r...

I’m thrilled to announce my preconference traini...

How can I see the current PoSH security ?

Introduction In my previous post, I covered the use of PowerShell Basic Functions. In this post we’ll cover Advanced Functions. Advanced Functions provide several abilities over basic ones. First,...

Welcome to the final installment of our Azure Purview Series, focusing on Purview Data Map. The Purview Data Map is the foundation for data discovery and serves as the...

Take the following sample code.    Four similar ...

“The data in a relational database is stored in form of a table. A table makes the data look organized. Yet in some cases we might face issues while...

There is no excerpt because this is a protected po...

This article explains the SSIS pivot transformation and how it differs from the unpivot transformation. This article is the thirteenth article in the SSIS feature face-to-face series, which aims to...

Inspired by an earlier blog where we looked at ‘How Interchangeable Delta Tables Are Between Databricks and Synapse‘ I decided to do a similar exercise, but this time with...

In this article, we will learn how to use Azure Data Factory to import data from OData APIs on Azure storage repositories. Introduction Data practice has the following major...

It has Android 11 and an SoC that would be at home in a 2020 flagship smartphone.

Earlier this week I did a live stream event.  I put out a poll and asked people to tell me what topic they wanted me to talk about.  The...

NONVISUAL: Mark the filter as NonVisual. https://dax.guide/nonvisual/

This video describes how we answer a question abou...

NATURALINNERJOIN: Joins the Left table with right table using the Inner Join semantics. https://dax.guide/naturalinnerjoin/ NATURALLEFTOUTERJOIN: Joins the Left table with right table using the Left Outer Join semantics. https://dax.guide/naturalleftouterjoin/

It is a quite common need to find the available free space in HDFS. There are a couple of ways to find this information.

Inorganic Lots of people say these are rare — a...

Jam Job Hash spills are, as we’ll see, sometimes identified by a different wait than sort spills. In small quantities, spills are often not worth bothering with. But when they...

In this article, we will explore in detail the factors that cause SQL Server stored procedures to be recompiled. Introduction SQL Server Stored procedures are the make-ready T-SQL codes...

Oracle has built-in functionality to analyze text. Sanil Mhatre demonstrates how to perform a sentiment analysis on data stored in a table using Oracle Text.

Ken Puls and I decided we would do a live stream c...

This article introduces the Power BI User-defined Aggregations feature. It includes a walkthrough example (with video) showing how to configure and test a simple model to use this feature. If you have not used Power BI User-defined Aggregations before, need a refresher or reminder, or are just keen to understand how to configure, this article is for you.

A very common requirement for a Power BI report that I stumble across at almost all of my customers is to automatically show data for the current day/month/year when a report is opened. At first sight this seems like a very trivial problem but once you dig into the problem, you will realize that all of the common solutions out there have some disadvantages and only solve the problem partially.

When is a date not a date? We all use them in our ...

Reading Time: 4 minutes In this post I want to share a nifty Power BI fix in Synapse Studio that I found. So that you can link to...

The integration between Azure Log Analytics and Power BI opens up a lot of new possibilities for Power BI administrators. It’s still in preview (there’s a lot more cool...

This book aims to introduce Azure data services and their use with different types of data and workloads, how does it fare?

Databricks SQL already provides a first-class user experience for BI and SQL directly on the data lake, and today, we are excited to announce another step in making data and AI simple with Databricks Serverless SQL.

Scalar UDF inlining is now available for Azure S...

We recently released the next beta of the Microsoft Drivers for PHP for SQL Server, version 5.10.0-beta1. This beta release has been built with PHP 7.3+ and tested on...

The magrittr package is a part of the extended tidyverse – i.e., not one of the ones normally loaded. It is the one that supplies the pipe operator (%>%), but it turns out that the package actually contains four pipe operators in total. All are intended to streamline and improve the readability of code, though the three non-basic ones are a bit more situational, and I’ve rarely seen them used, so I thought I would go into them a bit.

I’m going to be fairly high level on these becau...

Have you ever come across triggers being used to audit table changes? In this post, we’ll look at a few examples of DML triggers in action. Tracking LastEditDate with...

This article will walk you through Grant, With Grant, Revoke and Deny statements for both on-premises and Azure SQL Database. Principals, Securable and Permissions It is an essential and...

111 years ago almost to the day, a murder was committed which ultimately led to the first criminal trial to use fingerprints as evidence. We've all since watched enough...

We knew the basics of this story, but it’s good to have more detail. Here’s me in 2015 about this Juniper hack. Here’s me in 2007 on the NSA backdoor.

Respawn investigating, does not believe the exploit poses serious risks.

Martin Tingley with Wenjing Zheng, Simon Ejdemyr, ...

Many a time we need copies of an object but with d...

In 2012 SQL Server introduced the SEQUENCE object....

In this article we look at some of the differences...

For years, when it came to getting rid of the spac...

LAG pulls a column from another row without a self...

Itzik Ben-Gan returns to his table expressions series, covering what you need to think about when using a view to modify its underlying table(s). The post Fundamentals of Table Expressions,...

The T-SQL SUM function is one of the fundamental functions. Its purpose is to calculate the sum of all values in the specified column – note that it applies...

In this tip, I discuss one way to help avoid infinite loops in common while loop patterns.

Learn about the various ways to use the GROUP BY command to summarize and rollup query results. We look at the basics with examples as well as using group...

Is SQL DISTINCT good (or bad) when you need to remove duplicates in results? Some say it’s good and add DISTINCT when duplicates appear. Some say it’s bad and suggest using...

Benefit comes as Amazon warehouses face increasing...

Say goodbye to the blinding-white Google.com start...

It’s hard to pinpoint the first use of the phras...

Company says it wants 180 to 200 GWh in battery capacity by 2030.

One of the things I hate most about using remote desktop is when the resolution for the remote server is ... Continue reading