MuleSoft CTO and Sir Tim Berners-Lee want to open up API silos

Integration vendor and now Salesforce subsidiary MuleSoft has announced updates to its Anypoint API platform, including new security and compliance features, multi-cloud container deployment, a built-in performance monitoring function and ML-driven suggestions to speed integration mapping. But what really piqued my interest at its Connect18 conference in San Jose last week were the hints about future directions, not only from MuleSoft CTO Uri Sarid but also from keynote guest and instigator of the WorldWide Web, Sir Tim Berners-Lee.

What MuleSoft is exploring has much in common with the direction Berners-Lee wants the Web to go, and both of them argue for a new openness in the emerging world of microservices based architectures. I think this signals that we are now on the threshold of an important step in the evolution of computing. We have made enough progress that we can go back and undo some of the steps that got us to this point — because a refactoring is needed before we can move forward with what comes next.

So far, so cryptic — let me explain what I mean.

When an API was a black box

When we first started moving to service-oriented and as-a-service architectures around a decade ago, one of the crucial tenets was that, behind its API, a service was a black box. This was because it simply wasn't possible to deliver services at scale unless the provider had absolute freedom to decide how they delivered the service. So long as they complied with their published specifications and their promised service levels, they could implement the service however they liked.

I remember going to great lengths to defend this principle against the conventional wisdom of the time. In the client-server computing era, service consumers felt that it was their prerogative to dictate how the service was built, because they believed that would ensure it was optimized for their needs. It took a long time and much heartache for all concerned to gradually establish that it was better to allow the service provider the freedom to decide how to provide their service. This enabled them to deliver improvements to all of their service consumers that far outweighed the smaller advantages that individual consumers might gain from demanding their own custom design.

Ten years on, what used to be the mainstream view has been turned on its head. There are exceptions — technologists are a stubborn bunch — but in the main it's accepted that everyone gets a much better outcome if service providers have autonomy to decide how they build and operate their services. Service consumers are still in control, because if they don't like what's on offer from one provider, they can always switch to another provider. But they exercise control through making that choice, not by dictating terms to the provider.

What I saw last week is a substantial revision of that black box mantra. Now that we've established the principle of provider autonomy, we can start to open up that black box and allow service consumers some introspection into how the service is built. Not so that they can insist that the provider builds it some other way, but so that they can exercise a more informed choice.

Mapping the API topograph

MuleSoft CTO Uri Sarid, during his keynote technology presentation, introduced the concept of the topograph — a mapping of the API ecosystem as a distributed graph. This provides a structured catalog of the applications that use APIs, the APIs themselves, the policies associated with the APIs, and also looks inside the APIs to record the underlying components. The point is to have enough information to be able to identify potential points of failure before even building a new application or mashup, Sarid explains:

Sometimes it's very important for the enterprise to know what's happening behind the API. [For example] you're using it within the rules of the API, it seems to be OK, but the guy who implemented it didn't realize you would be storing sensitive information. They never imagined, because they just gave you a string, that what you would put in that string is a social security number. It's not something they could have anticipated, but it's the kind of question you can answer with a topograph.

Sometimes it's good to really not pay any attention to what's happening behind the interface. And sometimes it's critical to understand the downstream side effects of actually using that interface. With a topograph, you can do both.

The compliance example is just one of many. In his on-stage demonstration, Sarid showed a complex retail application that connected personalization data into a Google API, where the topograph made it possible to detect a potential point of failure because the application was trying to send a data field that wasn't recognized in the target API. The point is that having this knowledge about the make-up of each API makes it possible to avoid errors, security lapses and performance glitches at the design stage.

Sir Tim Berners-Lee and the case for Solid

Sir Tim Berners-Lee, speaking on the keynote stage the following day, also argued the merits of opening up APIs. Traditionally, the convention in procedural programming has been to hide the inner workings, he explains:

You don't allow me to mess with it because you reserve the right to change the way it works afterwards. You're going to provide a certain functionality ... and all the intermediate values that you arrived at, you keep to yourself so you can change those algorithms afterwards. So information hiding is a big part of the concept of procedural, functional programming, APIs.

But this creates separate silos of information, he argues, which means that individuals often find it hard to share their own data with people they'd like to see it. For example, this is a common frustration wtih social networks:

People ... have friends on Facebook and they have colleagues on LinkedIn and they have photos on Flickr and if they want to share the photos on Flickr with their friends on Facebook, they have to upload the pictures to Facebook. If they want to share that with people on LinkedIn, then they have to use LinkedIn, and so they're actually in these silos.

The same is true of many other types of data, and also in the enterprise, he goes on, where systems that could enable individuals to share information remain closed off.

When we've got these silos, our ability to share information is very limited.

If you're really sure that what you're building is a silo and you want to build an HR system which is really a silo, then fine. But actually it turns out — I think when you look at most organizations internally — there is the desperate lack of value, missing opportunity, from not being able to share.

Berners-Lee is leading a project at MIT called Solid which aims to standardize how personal data is stored so that it's no longer tied to specific applications.

Imagine if all the data that touched me, you could store it in any store, you could use any application to use it. Because in fact instead of having specific APIs, we're saying we'll have one big generic API, which is just the ability to read and write arbitrary structured data. In the Solid system, you could give me your card, I can take your ID up and I can give you access to anything ... We're going to break down those walls between these silos.

My take

APIs have emerged as a fundamental building block of the digital world. But so far, they've emerged in a haphazard way. The proposals from MuleSoft and from Berners-Lee's Solid initiative each provide an opportunity to bring more order to the API ecosystem. While still respecting the autonomy of API providers, greater openness will allow even more efficient reuse of resources, and take APIs to a higher level of performance and reliability.