Main content

AI-driven change and data management simplicity — key technology trends for 2024

Richard Timperlake Profile picture for user Richard Timperlake January 10, 2024
We can't predict what 2024 holds in store - but there are trends in data management that organizations should be mindful of. Richard Timperlake of Confluent selects a few to watch.

Digital transformation change management, IoT, new technology bigdata © Anawat_s -
(© Anawat_s -

As interest surrounding artificial intelligence (AI) gathers pace — and organizations begin the task of trying to understand how best to leverage this technology — one thing is becoming increasingly apparent. AI is nothing without access to accurate, real-time data.

Here are the key trends that I expect to see gaining momentum as we head into 2024, from organizations looking to simplify their data management, to businesses introducing new processes around building AI models, as the technology continues to dominate the agenda.

2024 is going to witness an increased uptake of data streaming

Businesses and organizations are only just starting to truly understand the value that data streaming can deliver. For those that are joining the dots, they are on the cusp of something truly transformational.

It reminds me of the emergence of customer relationship management (CRM). Back in the early 2000s nobody had heard of CRM. Yet today, it’s impossible to imagine a world without the likes of Oracle, SAP and Salesforce.

There’s a strong resonance between where CRM was 20 years ago and where data streaming is today. That’s why, when looking ahead to 2024, the future is incredibly bright — and all signs point to a significant acceleration in the take-up of data streaming.

The message I’m hearing from businesses is simple. They’ve tried out data streaming on relatively small use cases. Now they want to make this ‘mission critical’ across multiple lines of business — and even make this the central nervous system across their entire enterprise.

Companies are looking to simplify their data management 

Over the last 20 years or so, IT environments have become increasingly sophisticated and complex — and that’s made tool sprawl a big problem.

Increasingly, we’re seeing companies wanting to simplify their data management — moving away from using specialized tools to manage data, especially those designed to do one particular task within the life cycle of the data.

Whether this growing trend is being adopted to make systems less complex or to cut costs is up for debate. What can’t be ignored is that this rationalization process will gain momentum in 2024 to deliver quicker insights and more useful information to the businesses that take it on board.

Financial services will continue to lead in data streaming

Although there’s an uptick in the number of industries interested in data streaming — including the manufacturing, telcos and government — financial services continue to lead the way.

In the world of payments, for example, we expect to see growth amid pressure to provide robust and secure systems that meet ever-tighter regulations. We have one customer currently processing hundreds of billions of dollars’ worth of payments every year in an environment that is secure, resilient and well governed through Confluent, and we’ll see examples such as these increase further in 2024.

Transformation of regulations and policies will continue to evolve, changing the financial services landscape

In July this year, Consumer Duty Regulation came into force in the UK. It set in motion the Financial Conduct Authority’s three-year strategy of preventing ‘failing firms from causing significant harm to consumers and market participants’ — and committing to a higher standard of consumer protection in financial services.

This regulation will evolve in 2024 and we’ll continue to see transformation when it comes to policies in protecting consumers. This regulation means that for financial services, they must understand the benefits to their customer before offering a service. And this means truly understanding their needs and not going ahead with a ‘one size fits all’ approach.

Next year, organizations will need to have an informed view of their customers' needs in real-time, in order to adhere to the regulations at play.

AI-driven change is unstoppable, but guardrails are essential

And finally, all types of AI — especially generative — will continue to dominate the agenda. In the wider world, it faces legal and regulatory challenges that need to be addressed, not least in areas such as copyright. Businesses need to introduce due diligence before data is used — and some kind of remediation if it’s used incorrectly.

Closer to home, businesses need to develop processes around how they build different models that will deliver value to their organisations and their customers. This can only be successful if the bedrock of source data is secure. After all, you can’t simply build a model and flood it with data without understanding the quality and veracity of the information.

Looking ahead to 2024

I'm positive about my outlook for 2024 and we should build on this wave of opportunity. If nothing else, the twin concerns of inflation and high interest rates are being tamed, and in turn this means organizations and businesses should have more confidence to invest in technology.

Increasingly, that means optimizing the use of data and investing in data-centric systems that help provide even deeper commercial insights and improve customer experience.  Through this lens, and viewing data as a product I expect to see customers driving the monetization of data even more as we go through 2024.

A grey colored placeholder image