Apps are from Mars, data is from Venus, can APIs marry the two? An interview with SnapLogic CEO Gaurav Dhillon

Have we got the wrong end of the stick when we think about integration in enterprise IT? Based on the acronyms we use, it's always been about connecting applications — whether you're using legacy EAI tools (as in Enterprise Application Integration) or taking a more modern API-centric approach (as in Application Programming Interface). But more often than not it's the underlying data we're interested in, and the applications are merely incidental. An effective integration strategy therefore is one that connects to both applications and data in a single platform. According to Gaurav Dhillon, CEO of enterprise integration vendor SnapLogic:

We like to say, apps are from Mars and data is from Venus ...

Enterprises have this habit of treating data over here, apps over there. And the truth is, you can't get a digital transformation that way. It'll either be patchy and not useful, or it'll be fast and dumb.

As the co-founder and former CEO of enterprise integration pioneer Informatica, before leaving to found SnapLogic in 2006, Dhillon has strong views on how enterprise integration plays out today. The rise of API-centric integration, microservices architectures, no-code development tools and Robotic Process Automation (RPA) seem to promise new solutions to age-old problems. But in a wide-ranging conversation, Dhillon pointed out the drawbacks for enterprises of embracing these new approaches without a well-defined strategy. He sums up:

The truth is there is no silver bullet. APIs are not a silver bullet in the connected enterprise, and nor is no-code.

Connecting data and applications

With 78% of businesses across the US and UK set to accelerate spending on automation projects in the next 12 months, there's new urgency to make sure this spend achieves the desired goals. The figure comes from a survey carried out in February 2021 by research firm 3GEM on behalf of SnapLogic. It surveyed 400 IT decision makers, half in the US and half in the UK. The most popular goals were cost savings (cited by 63% of respondents), increased customer engagement and satisfaction (60%), employee productivity (59%) and growing topline revenues (63% in the US but bizarrely only 38% among the UK's IT folk, apparently far less confident of their ability to influence business outcomes).

Achieving those goals depends on connecting data as well as applications, argues Dhillon. Doing that successfully depends on creating some consistency in the key data objects such as customer and employee, so that you can then build a set of APIs that connect into the new business models and ways of working you want to create. He explains:

All of these [capabilities] are interacting with the same employees and customers and the assets of the company, being surfaced in new economic models. To do that you need to connect — and you also need the analytics to do it well. I think divorcing application connectivity from data connectivity is just false rhetoric.

One way to achieve data consistency across a family of applications is to run everything on a single vendor's platform, but in most enterprises that's impractical. Instead, Dhillon recommends what he calls a "solar system" approach in which there's a core system such as an ERP platform, with various major planets such as CRM, HCM, collaboration and so on. Many enterprises are finally starting to embrace cloud data stores such as Amazon RedShift and Snowflake for analytics, too. Then add an integration platform such as SnapLogic to provide the "interplanetary transportation" that ferries data between the various destinations. Having a well-designed architecture behind all of this is the foundation of future success, as he explains:

The architecture of your business has to be thought through in a good way. That modeling then lets you make good choices of API's, good choices of connectivity, and good choices of analytics and AI/ML in the future. It all needs to be the same or you have a sub-optimal [transportation] system.

And of course, somebody will come sell you some new technology, but either have dumb integration, or smart but slow analytics. If you want something that is fast and smart, you need to do it in one platform.

Architecture matters

Too many enterprises have fallen for the allure of new technologies without thinking through the architecture. He believes the rise of microservices and the culture of 'an app for that' has led to unmanaged "chaos at scale" in many enterprises. He says:

You can't just have a lot of cute little connections and make the world better. It doesn't work ...

My advice is to have a unified platform, have an architecture — it may be SnapLogic, it may be something else. But that thinking is very important. Because we deal with the consequences of people who didn't do that in the last century. And now when they try to transform themselves, they cannot. And they go through tens of millions of dollars with shiny objects ...

There's no substitute for clear thinking, clear design, of what the main [data] objects inside the business are. And then you can do all sorts of transformations, once you've identified with clarity, what the five to seven really important moving bits are. There's never 500 — the important bits are usually five to seven.

He singles out RPA as another technology that has been oversold, often where it's not needed. He comments:

I think the majority of RPA projects in the enterprise are misapplications of RPA technology ... RPA has a place, which is talking to things that don't have a clean, well-lighted API or data apparatus. But for anybody who does, RPA just seems a complete misfit. It feels like a square peg in a round hole.

Among the challenges to automation projects cited by repondents to the February survey, the top four were legacy technology (55%), a lack of internal skills (40%), a shift to remote working (40%), and compliance issues (37%). Having a single platform that can manage all of the API connections to data and applications helps to overcome some of these challenges, by setting guiderails that developers and users must work within to avoid conflicts or compliance issues. Dhillon comments:

We have to have good neighborhoods — the American saying is, good fences make good neighbors

Augmenting capacity with AI

Dhillon cites the example of one large customer that has replaced several legacy integration solutions with SnapLogic and slashed its six-month backlog. Around 1,800 unique users now log in to the platform each week and self-serve the connections and integrations they need. The vendor also sees many enterprises with a hybrid IT landscape where a well-planned integration architecture allows co-existence of new cloud applications and solutions with older on-premise ERP systems. He says:

If they have something like SnapLogic, they can have their cake and eat it, too. They don't have to spend $10-20 million trying to get an ERP [in the cloud]. They buy the best new cloud stuff, leave what makes sense on premise.

SnapLogic is using AI to make workflow automation and data integration more accessible to regular users. Whereas previous iterations of SnapLogic put integration in the hands of business analysts, this still wasn't reaching the kind of power user who's comfortable working in Excel or PowerBI. Dhillon says:

To do that we've added a perspective of using Natural Language Programming (NLP), using our traditional machine learning strengths, to automatically help you connect things. So we've gone from sort of jigsaw puzzles to almost like magnets that click together ...

If you think of a pyramid of IT people, analysts, and then every employee in the company, [this] lets us go serve the needs of every employee in the company.

At the same time, the Iris AI capability has helped advanced users work faster. He explains:

We thought that machine learning, ... the Iris AI, would be useful to novices. But the experts like it too because it speeds up — it's like Google Waze. Even though you know your way home, sometimes there's a shorter way, based on what's going on around you.

My take

How we've always done things often limits our thinking about how we should do them today. In my view, the world of integration is still far too centered around a mindset in which data is hidden away in application silos. In the past, we had no choice but to go through applications to get at the data. Today that's no longer the case.

An integration strategy — or as I've always preferred to call it, a connection strategy — no longer has to be application-centric. Therefore, when we adopt an API-centric approach, we are also leaving behind the old concept of data and application tiers. Instead, the APIs embody what I call a tierless architecture of serverless resources and headless interfaces.

So far, so blue-sky. A conversation with Dhillon brings the topic back down to the real world of enterprise IT, where all of this change has to be achieved without sacrificing resilience, security and governance. There is still a huge gulf in thinking between those who build cloud-native systems and those who manage long-established enterprise IT estates. Dhillon is eager to see enterprise IT embrace the cloud — and frustrated at how long it's been taking — but he empathizes with their challenges too. We may agree that the shift to tierless, API-centric architectures is for real, but he provides a welcome reality check on the need to carefully plan and manage that journey.