DataStax Astra gets support for Kafka, RabbitMQ and JMS in bid to capture the ‘full data story’

DataStax is hoping to provide buyers with more flexibility and choice when it comes to building event-driven architectures across their organization, by adding API-level support for a broader range of messaging tools, including Kafka, RabbitMQ and Java Message Service (JMS), to its Apache Pulsar based streaming service, Astra Streaming. 

The announcement comes as DataStax moves out of beta and into general availability for Astra Streaming. Speaking with DataStax CEO Chet Kapoor recently, he said that the company decided to pursue Pulsar to underpin its streaming platform, rather than Kafka (which does have a larger community around it), because of its cost-of-ownership and cloud native capabilities. 

Kapoor said that DataStax’s decision to adopt Pulsar was driven by an understanding that the Cassandra-based database vendor was picking a streaming technology ‘for the next decade’. 

Clearly recognizing that Kafka is currently the most popular streaming technology in the market, DataStax also pointed to research from GigaOm that states Pulsar achieves 35% higher performance and up to 81% lower costs than Kafka (over three years). 

But by integrating with older messaging tools, such as RabbitMQ and JMS, DataStax is also seeking to give customers choice about how they approach their event-driven architecture deployments, meaning that they don’t have to go ‘all in’ or rip and replace their older systems. This could well appeal to the market. 

Chris Latimer, VP of Product Management at DataStax, said: 

We’ve found that while organizations like the Kafka API, they are getting increasingly frustrated by its sprawling architecture and the high licensing costs required to make Kafka enterprise-ready. 

With Astra Streaming, any organization can now leverage their investment in Kafka and get the benefits of the superior performance, elastic scale and compelling economics of the Pulsar-based Astra Streaming - without rewriting their Kafka apps. 

The key components of the DataStax Astra Streaming announcement are that it: 

• Leverages existing messaging/pub sub apps and turns them into streaming apps with a drop-in replacement; Kafka applications require zero rewrites

• Multi-cloud and on prem - supports a unified event fabric across an enterprise's data-in-motion, spread across their entire data estate: on premises, in the cloud and at the edge.

• Is a real-time data ecosystem - through a wide range of connectors, Astra Streaming connects to an enterprise’s data ecosystem, enables real-time data to flow instantly from data sources and applications to streaming analytics and machine learning systems. It’s also integrated with Astra DB, with its CDC capabilities.

The database is just the end of the story

Speaking with Latimer ahead of the announcement, he explained that the key message to take away from the Astra Streaming announcement is that DataStax is seeking to offer buyers a platform that captures the ‘full data story’. 

But in order to get to this fuller picture of data capture across the enterprise, buyers need tools that incorporate data from a variety of sources. He said: 

One of the things that we discovered throughout this process is that there are a lot of organizations that are still struggling with legacy messaging platforms. 

So they have brought in things like JMS, or various MQ-based systems over the last couple of decades, and there's a growing pressure on these organizations to modernize. 

A lot of times these older platforms aren't keeping up anymore and so middleware teams are dealing with outages, dealing with instability, really trying to find ways to improve that, become more cloud native, and to accelerate developer productivity and velocity. 

So one of the things that we have done with our Astra streaming platform is that we've added API support for three additional protocols beyond Pulsar. So we have support for JMS, support for Kafka and support for RabbitMQ. 

Latimer said that whilst DataStax is traditionally a database vendor, based on Apache Cassandra, the data found in a database is just the end of the data story. What DataStax is doing is looking to fill in the gaps. He added:

If you think about the data that resides in your database, like a customer places an order, if you think of the journey that that data went on, as a story, you can think of the data in the database as the end of the story. 

And you can think of event streaming as everything that led up to that conclusion. So basically, the beginning, the middle - and the database tells the end. If you look at something like Kafka, for example, only a tiny sliver of that story is currently captured inside of Kafka. 

Most enterprises, if you look at replaying what led up to that order, for example, the customer could have looked at 10 different things, the customer could have added something into their cart, the customer could have left the site for two days and came back - all of those things are things that event streaming can tell you. 

Latimer explained that a lot of those events are captured in older messaging platforms that are “just ephemeral”, as a result many pieces of the story are being lost. He said that businesses have huge blind spots around their ability to understand the customer, and the journey that their customers go on, which creates a major barrier to being data driven for that organization. Latimer added:

So what we did is we said we want to be able to capture a complete view of that entire journey that the customer went on, or that the data went on. And in order to do that, we realized that we're going to have to support these additional protocols. 

I think it really signals that DataStax has a complete vision for data. Going back to the analogy of ‘your data is telling a story’. You need someplace to put the conclusion, you need something scalable, highly performant - and that's what we have with Astra. 

And then you also want to have a way of capturing all of the events that led up to that final state, and that's really what we're doing on event streaming. 

My take

DataStax clearly understands that the benefits of Pulsar need to be communicated effectively in order build a broader community of support, especially when going up against Kafka. But it also understands that buyers may not want to place all their bets on one streaming technology just yet, when they already have older messaging tools in place. And so by offering an integrated solution that can cater to these - a platform of platforms approach, if you will - DataStax may win favour over other providers that take a more locked-in approach. Time will tell and we look forward to speaking to some of the customers using these solutions, in order to get a deeper understanding of how this plays out.