The future of data storage (2024)

The data storage industry has evolved significantly over the last 30 years, but there has been a real step change within the last decade.

Not only has the amount of data, especially unstructured data, increased, but cloud-based storage has prompted companies – and individuals – to adopt a ‘store everything’ mindset rather than take the time to consider the true value of that information.

While cloud access to data has been a critical driver for business innovation, many data storage strategies have not kept pace with the diverse needs of digital transformation. How many businesses can confidently point to their five-year strategy and demonstrate not only an ability to scale to meet data growth but also the key availability, performance, sustainability and security features

With questions being asked about the true environmental and financial cost of data storage, Dene Lewis, Chief Technology Officer at CAE Technology Services Ltd, looks at how and why data storage has evolved, what is being done to innovate and solve the potential issues modern data storage presents to people, organizations and the environment – as well as looks to what the future may hold.

Uncapped growth

The data we produce and utilize has grown at a phenomenal pace – by 2025 we will be storing 160 zettabytes a year, much of it in the cloud. The challenge for businesses – and the IT industry – is that the growth in data volumes, especially unstructured data, is not gradual. Each technology innovation, in high-resolution imagery, for example, can double the storage demands overnight.

While the millions, possibly even billions of personal photographs and videos stored and posted to multiple social media locations are, without a doubt, a significant contributing factor, businesses are also struggling with data growth that has fundamentally outstripped expectations.

Just consider the storage implications when an NHS Trust upgrades the CT scanner software, improving the quality and hence the size of high-resolution images two, three, or even four-fold in one moment.

Or the rapid evolution of IoT, which is allowing businesses to generate increasingly high levels of valuable data. Organizations are investigating the use of AI to optimize business processes, which creates an additional data source that ultimately will support business growth, but it will need to be stored and that storage costs money as well as having an environmental impact.

Every aspect of stakeholder engagement and interaction now demands increasing data volumes – all of which need to be stored multiple times to ensure business continuity and support disaster recovery plans. The evolution of storage technology means large quantities of data can be stored in smaller footprints at the edge, the data center, or in the cloud – supporting distributed data growth.

But is this financially, or environmentally sustainable?

Unfortunately, there is not a clear-cut ‘yes’ or ‘no’. Why? because, as much as the individual components of data storage have become super-efficient, the reality is that the power consumption required has exponentially increased. Leaving some real considerations for organizations looking to optimize an existing, or create a new data storage strategy.

Time to take responsibility

The IT industry continues to innovate and address the rising data storage challenge. Storage density improvement has seen the price per gig for storage reduce in recent years, dropping by over 80% from 2009 to 2022, allowing businesses, in theory, to scale up data storage and positively impact the bottom line.

The storage market has moved on from the old days of installing rack upon rack of mechanical spinning disks, which aside from being low on capacity, gave IT Teams a constant headache of ensuring optimum performance.

Those days are behind us, however, there are now different solutions available that bring their own financial, operational, and sustainability challenges to overcome.

It is important to note that storing data in the cloud can be perceived as easy and relatively pain-free to manage, but shifting or migrating data between cloud services, or from on-premises to the cloud is complex. Organizations dealing with high volumes of data at scale need to consider where they want to put the data and make the most appropriate and well-informed decisions about their data storage strategy up-front, rather than getting deep into a project and then making the choice.

Although cost in financial terms is always considered, it may not always be fully understood by businesses comparing different options for hybrid cloud solutions. And increasingly important costs such as environmental and sustainability factors are rarely considered by businesses at the planning stage.

Do we need to store everything?

So, is it viable to retain our ‘store everything’ approach, when data center’s are estimated to be responsible for up to three per cent of global electricity consumption today and projected to touch four per cent by 2030?

And, if so, what is the sustainability compromise?

Businesses face ever more serious demands regarding sustainable operations and reporting – as well as stakeholder expectations. The volume of data stored is not an issue that can be influenced by regulators, as it will differ from business to business, however, governments globally are keen to achieve ESG goals.

I firmly believe that the industry will continue to innovate and build with sustainability at the core of its objectives, to build sustainable ways of storing data and sustainable storage device technology, from hard drives to storage arrays and data services that maximize efficiency and enable every business to reduce the carbon footprint per terabyte of storage.

Strategic planning – the balancing act

This is not an issue for the IT industry alone. The responsibility also sits with businesses to put in place robust data management and storage strategies. However fast the data grows, what are the implications of having a reactive approach to investment in cloud storage without considering the business needs for cost, performance, security and sustainability?

Given the scale of data growth, businesses must plan ahead, having a multi-year approach to data storage.

A comprehensive five-year forward view should encompass both the intended strategy and structural framework, along with a well-defined plan. It should take into account not just the anticipated data expansion and its influencing factors but also utilize this data to evaluate strategic alternatives for data storage that align with the organization's objectives.

Put simply - how and where should data be stored? And for what reason?

Thinking laterally

Data storage cannot be considered in isolation from the rest of the IT and wider business strategy.

We need to be thinking strategically and holistically about data storage – not just picking it off as a tactical issue in an isolated area of the business. It must be considered as a broad strategic approach that spans edge, data center’s, the cloud and everything in between.

Creating the right data strategy is far more nuanced than simply spinning up another cloud storage subscription and, at every step, organizations should be considering the sustainability implications of storage decisions.

Of course, there are difficult questions ahead as the demands for data change again. Generative AI and Machine Learning algorithms will require access to petabytes of data to accurately train AI models, which will impact financial and sustainability factors.

Ultimately, data growth has significantly increased and will continue to do so. What happens when everybody gets online?

There is set to be another massive step change in data volumes and data usage.

The question is how is the world, as both individuals and businesses, going to respond?

We've featured the best green web hosting.