Delivering devops tools to the tribes and squads at Sky Bet

Online sports betting is a fast-growing, multi-billion market in the UK, with betting on soccer alone generating half a billion pounds ($0.6bn) in annual revenues, closely followed by horse racing at more than a third of a billion ($0.4bn). To compete in this market, businesses have to operate highly scalable, adaptable and resilient digital platforms — which puts the industry on the leading edge of agile development, devops and continuous delivery.

A characteristic of this fast-moving, highly iterative approach to software development is that people often work in small, autonomous teams — which is why there's a need for the kind of teamwork playbook recently launched by software project tools vendor Atlassian. I had a fascinating insight into how such teams work last month when I met Gavin Harris, engineering manager for platform services at Sky Betting & Gaming, one of the UK's leading online sports betting providers, which last year span out of the Rupert Murdoch-owned Sky media and telecoms business.

The Sky Bet applications have to perform under intense load because sports betting activity is concentrated around specific events. On a typical Saturday during the UK soccer season there are 300,000 users getting real-time updates on changing odds, with the peak in match betting coming just before kick-off at 3pm nationwide. Meanwhile there's a horse race taking place somewhere in the UK every 10 minutes between 1pm and 7pm. The biggest peak comes one weekend in April, when many people who don't normally bet on horse races will place a bet on the annual Grand National. This year the site was processing more than 10,000 bets per minute for the last 20 minutes before the race started — at the same time as settling soccer bets as those games came to an end.

The company relies on various tools and services to manage development and operations — AWS for certain applications that take advantage of autoscaling, New Relic to monitor live operations, and various Chef tools to manage development and testing. It's the job of Harris's team to support the Chef tools:

We want our developers, our sysadmins, our testers, all of them collaborating around a shared toolkit. That for us is Chef and the ecosystem that brings.

It's about standardization. We can run workshops and we don't have to tailor it for each of our individual teams.

Development at Sky Betting is organized according to the Spotify-inspired Scaling Agile model in which 'tribes' of small autonomous teams, known as 'squads', each take ownership of specific services.

The small team has got ownership over the entire piece of their service, they're doing the deployments, they're doing the builds, they're making sure that it's compliant and all that — but they're dragging in these shared bits of tooling to make that happen.

Rapid innovation

There may be dozens of releases across all products and components on any given day, with the velocity of change depending on what's going on that particular day. This highly devolved way of working is the key to very rapid product innovation as new technologies or business capabilities arrive in the market, says Harris.

We wanted to increase the velocity of delivery that gives you. We want people to be get early versions of code in front of other developers so they can start collaborating on that stuff earlier. That helps us keep pace with a very fast-paced industry where, you know, it's all about having the latest greatest features. I think a testament to that is some of the features we've released recently. We were the first to market with Apple Pay, for example, in the betting and gaming sector.

Integrating all of that functionality into the app, testing it across all of our various products and releases and environment and integration for the software, that's the bit that Chef helps you with.

Each of the teams has responsibility for a specific component of the Sky Bet website and its related services.

The log in functionality and the SSO [single sign-on] across the different products, that's run by one team. The front end of the Bet website is looked after by another squad. The data that's on the Bet website, that comes from a different squad.

Those three different squads, they're all using Chef to test and build their software and using the tools that my team provides to deploy and release that software. It's all done in a very similar way so that we get a lot of shared efficiency.

These small squads who you've probably one, maybe two guys that have got a significant amount of operational experience, then you've got a number of developers that have got similar. It's about taking advantage of those specialties.

No prescriptions

Harris's team started out five years ago as a project team charged with fixing disaster recovery, and grew out of that into its current role of providing a range of core services such as compliance and version management. But while he sets the framework within which other teams can choose to work, it's a highly consensual culture.

We definitely don't set the rules. We're not prescriptive, we provide the platform and the tools. The squads that are releasing their applications, they can use these tools or they don't have to. They can do something else if they want to. Most of them choose not to do something else, but we're not prescriptive at all.

We provide you with a Chef tool but we don't say what your recipes need to do. We don't say how you need to release your application. We don't say how you manage security patches in your autonomous team. That's up to your autonomous team to decide. It's your application, you own it and you run it. We just provide tools that make it possible to do that ...

It's kind of like being a third party in the internal environment, really. We want our relationship with our internal customers to be very similar to a relationship you'd have with say, Amazon. We don't want people to be having to need to talk to us to be able do something. We want them to have that power and freedom to experiment, to try new things.

There's a constant process of feedback so that Harris's team can plan ahead for capabilities it needs to put on the roadmap.

Each of the tribes — the top level organisations — they have roadmaps of the squads in them. They go through a process to prioritize things within that roadmap.

We try to align the roadmap for my team with what the tribes have got planned for the next six months. We take their requirements and try and turn that into requirements for the tooling of platforms.

We don't try and push things out and dictate 'This is how you're going to do it.' We want to see what they want to be doing and then be proactively creating tooling ready for when they need it.

No more rigid ITIL

It's a completely different model from the classic enterprise development shop, says Harris.

I used to work in large enterprise and it was all very rigid ITIL, a completely different environment than I'm in now, and it's eye-opening really.

Because you've got these people that are looking after this service, they own it and it's theirs. They care about it when it breaks and they want it to run smoothly. They're invested in it so they're much more enthusiastic about it. Rather than being part of a large team that's got this large piece of software that 'I've written a few functions for it, but ...'

This different attitude underlines how distinctive the devops model is in practice, he adds.

Devops, it's easy to try and distill down to a job title and a thing that somebody does but it's not — it's a culture and a way of approaching problems more than anything.

One disadvantage of adopting such a modern way of approaching development is that there's a limited talent pool to draw on, so Sky Betting has been testing out a graduate training program as a way of bringing in fresh blood. The graduate academy puts a couple dozen technology graduates through a three-month training program.

We introduce them to some of the tools and techniques that we use — agile, some of our languages, some of the testing styles and things. Then they're going to break up into some small autonomous teams, they're going to go out into the tribes and then hopefully start to deliver projects in the next couple of weeks when they finish that training program.

Then hopefully next year we can do a similar thing. We did recruitment fairs at universities around the country and a selection process. We can fulfill some of our recruiting needs that way.

My take

A fascinating insight into a real-world example of the pioneering teamwork models being adopted by developers.