Everyone wants a slice of Splunk at Domino's Pizza

In recent years Domino's Pizza has tried to position itself as an e-commerce company, rather than a traditional takeout service – thanks to the top-down

support from CEO J. Patrick Doyle, who has publicly said that he wants to use technology to give the company a competitive advantage.

This is evident from the omni-channel presence that Domino's holds, where customers are able to order food in-store, over the phone, via a website, and through cross-platform mobile applications. Not only this, but the company has introduced a lot of interesting and innovative tools for customers, some of which include voice ordering through the mobile apps and the popular 'pizza tracker.

These tools and platforms support Domino's Pizza's 10,000 stores worldwide and enable $8 billion worth of sales a year.

However, as is the case with any successful omni-channel e-commerce company, when you have a lot of touch points and a number of systems enabling the delivery of your services, you are quickly inundated with a hell of a lot of data that needs to be managed and analysed.

And until recently, managing the company's application and platform data was a headache, with much of its log files in a 'giant mess' – according to site reliability and engineering manager, Russell Turner.

Turner was speaking at Splunk's annual user conference in Las Vegas this week, where he explained that he introduced the operational intelligence tool to replace some APM tools, to not only save costs, but to also search data more quickly, monitor performance and to get better insights into how customers are interacting with Domino's. Turner said:

All this is data keeps feeding into us. Marketing wants to go 8 million miles an hour, development wants to go 10 million miles an hour, it's very challenging.

We did a proof of concept with Splunk for the first time in 2009, but we didn't have a success criteria defined. It was a neat toy but we felt like we didn't need it, we didn't really understand it at the time. But then we started to realise we didn't have a lot of visibility on what our applications were doing and it took a lot of time to find a needle in a haystack. We needed something that would search faster.

We had been using using an APM solution, but the bill was astronomical. I just couldn't go to the business asking for that money. So, an engineer came to me and suggested Splunk – and we ended up rushing it in quickly on old hardware.

Turner explained that Splunk was approximately a quarter of the price of the APM solution that he been being used and provided 80 percent of the functionality. It is being used to correlate all of the data that Domino's has coming in from application and middleware logs – it is not currently being used

at the infrastructure layer, but there are plans for this in the future. Turner said:

Splunk was faster, way cheaper, better at reporting with access logs, was easier to deploy, and had better scalability.

Before, everything was a manual search, it was painful and slow, not very accurate, and we were completely in a reactive mode. Now it is a million times easier to get through the logs, plus all the other data that has been there for years.

This has allowed us to do things like set proactive alarms to alert us to dips in sales and we have been able to retire monitoring tools. We were never able to give you a baseline or trending for performance, now we have those and can do all those KPIs that management wants.

Some of the interesting things that Turner and his team are using Splunk for include:

• An interactive map has been created and is displayed in the Domino's welcome centre, which shows in real-time all of the orders coming in from across the United States. Turner said he didn't realise that this would bring business value, but he explained that employees are “mesmerized” by it and the development team loves working on it.
• Similarly, Turner's team has created a real-time feedback wall, which scrapes the logs for comments from customers. None of the comments are screened and employees can constantly see what Domino's customers are saying about the products being bought. Turner said its useful for employees to see customer affirmation and how the business is being perceived – for better or worse.
• A dashboard has been created to show order counts, as well as error counts. Domino's can see where it should be on these counts, based on performance in previous weeks, and the Splunk dashboard also helps the team to troubleshoot outages.
• Splunk also analyses the Domino's payment processes. Domino's has a variety of ways in which it accepts payments – for example, via gift card, cash, credit card. Not only this, processing of payment is owned by different parties depending on if the payment is in store, or online, etc. Splunk helps the company to see how quickly the payments are being processed, who is processing the fastest, as well as compares the speeds of different payment types.
• Promotional support – this is something that isn't currently live and is still in the pipeline. But Turner believes there is a strong use case for monitoring how offers and coupons are impacting sales in real-time. He hopes that in the future this will allow Domino's to change its promotional offers in real-time if some work better than others. At present, teams have to wait approximately a day to see the impact of any promotional activity.
• Splunk is also being used to monitor the performance of the Domino's in-house developed point of sales systems.

Turner said that the benefits of Splunk are now being recognised outside of the IT department, with other teams increasingly wanting to use the tool to get

better insights out of their data. He said:

I was trying to keep this under wraps, because I didn't want to become the real time reporting team. But that was naïve, because the tool is just so powerful. I was so naïve to even think it could stay that way.

You have to come up with a business case and recognise its effectiveness, because every time we have exposed this technology to another team they think of ways to use it that yields fascinating results. Anyone outside of your immediate group, get them to come and look at it, because I bet there is ways for you to leverage it that you haven't even thought of yet.

However, Turner admits that he rushed in Splunk when he first decided to deploy it, because he didn't realise the scale at which it would be used, and as a result he made a lot of bad decisions and wrote a lot of “bad cheques”. He is now planning to reverse these mistakes and bring in the Splunk team to get it rolled out onto newer hardware and to build a more stable environment for the future. He said:

We cut corners at first thinking it was just going to be us - but now it has blown up into this big huge thing, we have got to make up for those mistakes.

Turner also advised any other company that is considering rolling out Splunk to figure out the use case before they dive in and to prepare to scale it up a lot more quickly than may be anticipated – companies should be prepared to put a lot of data into the tool (Splunk charges based on how much data is used).

We fell down a lot back in 2009. We didn't have a use case we were solving - if we did have a use case in 2009 when we first started using it, we would probably be so far ahead of the game.

Also, what you think you need to index, it's probably double or triple that, because once you start seeing all of that data come in, then you're going to want more and more. It's easier to go to your boss and ask for X amount, than get it wrong and have to go back and ask for more – ask for too much, because you are going to use it. We have had to increase our licence a few times.

However, what this means is that I can now pay guys with big brains to do some super sexy stuff, instead of sifting through logs.