The modern data landscape

Have you ever wondered how much data we generate every single day? Or the effort required to store and access your data on demand? What about the infrastructure or the services required to make all of this happen? Not to mention the engineering effort put in to make all of this happen. If you have, you are in the right place. These questions inspired me to dive deep into the realms of DevOps and SRE and inspired the creation of this book.

Technology impacts almost every aspect of our lives. We are more connected than ever, with access to more information and services than we even realize. It’s not just our computers, phones, or tablets that are connected to the internet, but our cars, cameras, watches, televisions, speakers, and more. The more digital native we become, the bigger our digital footprint grows.

A digital footprint, also known as a digital shadow, is a collection of data that represents an individual’s interactions and activities across digital platforms and the internet. This data can be categorized as either passive, where it’s generated without direct interaction – such as browsing history – or active, resulting from deliberate online actions such as social media posts or emails. Your digital footprint serves as an online record of your digital presence, and it can have lasting implications for your privacy and reputation.

As of 2022, researchers estimate that out of 8 billion people (the world’s population as of 2022), approximately 5 billion utilize the internet daily. Compared to the 2 billion that was measured in 2012, this is a 250% increase over 10 years. This is an incredible increase. See the following figure for reference:

Figure 1.1 – Daily internet users (in billions)

Each person who has a digital presence generates digital footprints in two ways.

The first is actively. When you browse a website, upload a picture, send an email, or make a video call, you generate data that will be utilized and stored for some time. The other, less obvious way is passive data generation. If you, like me, utilize digital services with push notifications on or have GPS enabled on your phone with a timeline, for example, you are generating data every minute of the day – even if you do not use these services actively. Prime examples can be any Internet of Things (IoT) devices, something such as an internet-enabled security camera – even if you are not actively using it, it’s still generating data and constantly uploading it to your service provider for safekeeping. IoT devices are the secondary source of data generators right after us active internet surfers. Researchers estimate that approximately 13 billion IoT devices are being connected and in daily use as of 2022, with the expectation that this figure will become close to 30 billion by the end of 2030. See the following figure for reference:

Figure 1.2 – Connected IoT devices (in the billions)

Combining the 5 billion active internet users with the 13 billion connected IoT devices, it is easy to guess that our combined digital footprint must be ginormous. Yet trying to guess the exact number is much harder than you might think. Give it a try.

As of 2023, it is estimated that we generate approximately 3.5 exabytes of data every single day. This is about 1 exabyte more than what was estimated in 2021. To help visualize how much data we are talking about, let me try to put this into perspective. Let’s say you have a notebook (or one of the latest phones) with 1 TB of storage capacity. If you were to use this 1 TB storage to store all this information, it would be full in less than 0.025 seconds. An alternative way to think about it is that we can fill 3,670,016 devices with 1 TB storage within 24 hours.

How do we generate data today?

Well, for starters, we collectively send approximately 333.2 billion emails per day. This means that more than 3.5 million emails are sent per second. We also make over 0.5 billion hours of video calls, stream more than 200 million hours of media content, and share more than 5 billion videos and photos every single day.

So, yes, that’s a lot of us armed with many devices (on average, one active internet user had about 2.6 IoT devices in 2022) generating an unbelievable amount of data every single day. But the challenge does not stop at the amount of data alone. The speed and reliability of interacting with it are just as important as, if not more important than, the storage itself. Have you ever searched for one of your photos to show someone, but it was slow and took forever to find, so you gave up? We have all been there, but can you remember just how much time after doing this that you decided to abandon your search?

As technology advances, we gain quicker access to information and multitask more efficiently, which may be contributing to a gradual decline in our attention spans. Research shows that in 2000, the average attention span was 12 seconds. Since then, significant technological milestones have occurred: the advent of the iPhone, YouTube, various generations of mobile networks, Wikipedia, and Spotify, to name a few. Internet speed has also soared, moving from an average of 127 kilobits per second in 2000 to 4.4 Mbps by 2010, and hitting an average of 50.8 Mbps by 2020 – with some areas experiencing speeds well over 200 Mbps today.

As the digital landscape accelerates, so do our expectations, resulting in further erosion of our attention spans. By 2015, that 12-second average had fallen to just 8.25 seconds and dropped slightly below 8 seconds by 2022.