Data Streaming requires IT leaders to adopt a new mindset to build digital-first applications

Holger Temme Profile picture for user Holger Temme February 27, 2024
Data streaming can enable businesses to become more responsive. But it isn't a bolt-on. Confluent's Holger Temme considers two companies who have changed the way that they think about their architecture to get ahead.

digital-first mindset business leader © YuriArcurs -
(© YuriArcurs -

Data streaming is powering the modern world, enabling businesses to connect their applications, systems, and software-as-a-service (SaaS) layers in real-time to meet today’s challenges head-on.

Armed with a unified and constantly up-to-date view of their data, data streaming enables businesses to deliver streamlined workflows, greater automation, superior customer experiences and more efficient business operations.

But managing these real-time data streams — which are processed as soon as they are generated — isn’t easy. Enabling the free flow of data can be complex, especially if organizations choose to adopt hybrid architectures that straddle both cloud and on-premise environments.

But organizations must address these issues if they are to succeed in a digital-first world.

Streaming data is a must for modern businesses

In last year’s Data Streaming Report: Moving Up the Maturity Curve, Confluent found that data streaming creates a more responsive business. It said:

With data volume growing at breakneck speed, the ability to harness its power in real-time is the key to driving greater business insight, accelerating innovation, and reducing risks.

The report found that for 89% of respondents, investments in data streaming are important, with 44% citing it as a top strategic priority. And over two-thirds (72%) of IT leaders say they are using data streaming to power critical applications.

But it also found that while real-time data streaming is essential, it’s also challenging. Seven in 10 (74%) of those surveyed cite fragmented projects as a major hurdle, while 72% say a lack of relevant skills, inconsistent use of integration methods and standards, and legacy-related constraints as other challenges to advancing data streaming.

But there is another hurdle that also needs to be addressed. The industry needs to decide whether it intends to stick with the ‘old, tried and tested way of building applications’ or embrace the ‘new way’ built around data in motion.

That’s because data streaming isn’t a bolt-on technology. It fundamentally changes the way applications are engineered.

For those looking to leverage data streaming, it poses a fundamental question. Do they want to build applications using the skills and methodology they learned at university? Or are they going to embrace this totally new paradigm of how to build applications?

In my book, this is relatively simple to answer because of the limits posed by old-world technology. And it seems I’m not alone. It’s an approach already adopted by AO — one of the UK’s leading electrical retailers operating predominantly online for the last 20 years.

Data streaming is the new paradigm

It found that the COVID-19 pandemic caused a dramatic shift in consumer shopping habits, which led to a different approach to its online operation.

Commenting at the time, AO Founder and Chief Executive Officer, John Roberts, said the retailer “saw 10 years’ change in consumer behaviour in just 10 weeks”.

As a result, AO invested in a real-time event streaming platform capable of delivering bespoke online experiences. It did this by combining historical customer data with clickstream data and other real-time digital signals from across the business. Jon Vines, Software Development Team Lead at AO, added:

“Our hyper-personalized approach is delivering measurable results.

In our A/B testing, we’ve seen a significant increase in customer conversion rates. That’s proof that our decision to adopt a real-time event streaming approach was the right one.

And AO is not alone. Many developers use technology rooted in the Apache Software Foundation (ASF), an organization widely recognized and respected within the open-source community and among software developers.

For event-driven architecture to work, businesses need to employ two key elements: storage and processing.

Apache Kafka is a distributed streaming storage platform used for building real-time data pipelines and streaming applications. Apache Flink is an open-source stream processing framework and distributed processing engine.

Together, they have become the de facto chosen architecture by developers for the streaming pipeline giving businesses and organizations a real competitive advantage.

But it’s still early days, with only 5% of new applications being built in this new way using Kafka and Flink architecture. But that figure is growing.

Businesses must build applications differently if they want to stream data 

Thanks to Kafka/Flink, developers can now build applications with completely different characteristics compared to how they built applications in the past 30 years. If you look at any of the digital natives, none are building applications the old way. They know it's simply not doable.

Take Trust Bank, Singapore’s first digitally-native bank. It was launched in 2022 to deliver a world-class customer experience and cutting-edge security built on a foundation of real-time data.

It uses a data streaming platform for its event-driven architecture, enabling different teams to produce, share, and consume self-service data products in the form of real-time streams. According to Rajay Rai, Chief Information Officer at Trust Bank:

Everything we do is in real time because batch processing is an old way of thinking. The longer your data waits, the less value it has. So, as data comes through, you need to be able to act on it, or enrich it quickly

With data volume growing at breakneck speed, the ability to harness its power in real-time is key to accelerating innovation, reducing risk, and creating a more responsive business model.

And while it’s true that some traditional organizations continue to build applications based on ‘old school’ learning, time is running out.

The only way to change this general paradigm is to build applications differently. And the only way to deliver real-world business impact is via stream processing.

A grey colored placeholder image