Chef’s heavy metal approach to biz apps innovation


Chef Software’s CTO and co-founder strikes a musical chord to show how Habitat Builder can help end user businesses innovate the new applications they will need in the near future.

Adam Jacob, the CTO and co-founder of Chef Software, doesn’t see the company’s latest offering – Habitat Builder – as anything but a toolkit for down-the-line applications developers, but he does pitch it as playing an important part in a transition in business services innovation and development.

Habitat, introduced last year, is a tool in which innovations can be captured and exploited. The latest addition, Builder, was formally introduced to the world at Chef’s recent London Summit. This, as the name implies, is the toolset designed to help developers pull those innovations together as working business solutions.

It is a SaaS-based service that provides a way to package apps simply and consistently for deployment and management across flexible cloud-native architectures using tools such as Docker Swarm, Kubernetes and Cloud Foundry, providing both developers and operations teams control over the lifecycle of containerised applications. It does not require a particular export format or runtime in advance as such decisions can be made at the time of deployment.

Habitat can automatically detect what language tooling is being used and build deployable single artifacts which contain the application together with the libraries and dependencies it needs to run in any traditional or cloud-native architecture. The claim is that the developer learning curve is shorter and less steep.

As well as the build capability, it also provides an artifact store and full application supervision covering runtime lifecycle, configuration updates, clustering topologies and update strategies. This is integrated in the deployed artefact to automate application delivery and service provision. It can deploy the application in any format and runtime, suitable for both on-premise and cloud native architectures. Native integration with Github is provided for source code, and with Docker Hub for container format export. More integrations are expected to follow.

An old idea, but something new

Jacob is the first to acknowledge that Habitat is a continuation of an ethos that has its roots in the Parallels Application Packaging Standard (APS) from around a decade ago:

The idea that an application is responsible for its own behavior is not new. The hard part is figuring out how we actually build a system that is capable of being composed into something new; where the units of composition are bigger than what we do now. For example, the systems we’re designing now are still working with the networking designs we came up with in the ‘70s. We’ve been extending it over time but those primitives creep up, but the challenge is what the behaviour is, what do we ask of an application that is separate from what the application does.

One noticeable factor with Chef is the company’s readiness to talk in terms of developing business services rather than applications, for a business service may involve the collaborative use of several applications (which themselves may also be used in collaboration with other applications to build other business services). Jacob says:

If you go to you see an application, but it is enormous and to render it – the last time I looked, took some 300 different service calls to other applications or services. So yes, business services are made up other services, many of which can be quite small domain services.  I think this is the mark of a legitimate philosophical change. We are realising that we need more re-useable, composable building blocks, But what is their shape? That is the question we haven’t really got our head round yet.

Here comes the music

One thing that is interesting here is that Jacob is quite keen on the use of musical analogies to try and describe what is happening within this philosophical change. For example, does he see Habitat as the orchestrator, or the composer that the orchestrator uses, or is it perhaps the conductor?

He sides against the term conductor as that implies a degree of centralisation, where the members of a band, or orchestra, know what to play. They are watching the conductor less for timing and more for the `feel’ and emotion. But Habitat has no central point, instead the applications network themselves together to inform each about what is happening. It allows users to build a model of what a service is, and what sorts of behaviour can be expected from it, and because the services can deploy themselves it becomes possible to build a system that will allow the applications to relate to each other.

The Habitat supervisor will work with the applications to organise themselves to deploy in the most appropriate fashion. He likens this to the way humans would do it, a mixture of ‘who wants to go first?’ and ‘it’s probably better if you go first’. How that happens can take very little co-ordination and does not need a ‘conductor’. So orchestration becomes a distributed, collective process, rather than a prescriptive one.

Taking the music analogy deeper, he moved on to likening the process to the manuscript where people understand the rules of music and writing it, but can then write anything they want – be it weird or wonderful:

When you figure out how to put the elements together that is when you get the good music. For the most part we have just been building iterations on a theme. We’re getting better and better at building new versions of what we had before.

One of the core messages of Chef Software is the notion of continuous automation, which tackles a common misconception that, once an automated system is put in place. then it never needs changing. In Jacob’s view all systems have to change – and be open to change – on a potentially continuous basis. The systems need to evolve, even if only because people evolve:

We have this false belief that automation will solve our problems, and once they are solved we can move on to other things. That’s not really how it works.

Automation is only able to solve some parts of a problem, which does allow moving on – a bit. But things don’t stay static. Automating other bits of a process can require change to the bits already automated, and aspects that have been automated change with market needs. In short, nothing stands still, so automation has to be able to move just as fast as the changes.

There is a corollary here, of course, which is that users may start to be concerned that, with continuous automation automatically re-adjusting and re-optimising itself to meet changing requirements, they will lose control of everything, warns Jacob:

I think users are starting to think like that and starting to understand that. People don’t tend to think like that about either systems or their businesses. They tend to think of them as fixed. There might be an evolution over time, but the pace is slow and its usually very disruptive. And that idea has served us well, but it doesn’t work well enough. For the people inside it there is a huge list of stuff that needs fixing. There are some organisations that have worked out how to do that and those organisations are doing quite well.

This does mean businesses having a clear idea of what they want to achieve with such a process. As an example, he posed the idea of grocery chain Tesco deciding it wanted to change to be just like Amazon. That, he suggested, would be doomed to failure as Amazon would always be able to do `Amazon’ better. But Tesco always has room to be a better Tesco, adapting itself to be a better interpretation of its own brand values.

And here comes the heavy metal

Here, music came back into the discussion as an analogy. Jacob declares:

I like heavy metal music. If you made me play country music I expect I could do it but I’m not going to like it. And I think people are starting to get it that saying, ‘We need to be like that’ [whatever is the perceived business equivalent of country music to him] is not the right answer.

And here, in a way, is a pretty good analogy of the way that the important future innovations will start to emerge from the user community rather than the technologists. Anyone who has ever played in a rock band – even just a jamming blues get together or a folk music session in the back room of a pub – will recognise the following scenario: someone will start noodling away at something – a bit of a guitar riff, the first two lines of a possible song, a drum pattern, anything really. It will usually prompt someone else to join in, adding, say, a bass line. It might die out of its own accord, or it might start to grow and stabilise, with other players suggesting additions, augmentations, harmony lines and the rest. Sometimes the collective team (ie band) will decide it is going nowhere special and drop the idea, but other times, out of that initial noodling, emerges a new song.

That, to the mind of Jacob, is very much the way that new service innovations will now start to come about. And it is the market need that Habitat Builder has been designed to service:

Our current systems don’t work that way. And it is why Habitat has been designed with, as far as is possible, no preconceptions as to how any user might actually use it.

My take

This is yet more evidence that the software vendor community are waking up to the idea that they are no longer the core driver of future innovation, particularly in the business systems and management arena. It is the users that have the need and feel the pain and it will be them, once the right development environments and tools are available, that will know what makes a good solution and pain killer. This certainly looks like a good contribution towards meeting those needs.

Image credit - Che