However, this project was a new one for the organisation, in that it hadn’t really developed an e-commerce streaming/download proposition in the past.
Zoe Bolton, head of service operations for BBC Worldwide, took to the stage at SplunkLive! in London this week to explain to delegates how although she didn’t have the technical resources in her team, but that she needed a deep-dive monitoring tool for the new product so that she could improve and optimise the customer journey for viewers.
She opted for the Splunk Cloud, which allowed her to get set up and monitoring in under eight weeks, without having to worry about managing any on-premise infrastructure. Bolton explained:
I was responsible for developing the service operations. The people, the processes and deciding what toolsets we were going to use. It was the complete unknown. - We didn’t know what it was going to look like when it launched, it really was around best judgement in terms of what type of people we needed, how many people we needed, what our support hours were.
We were like a start-up model really, even though we were working for huge organisation. We couldn’t really rob from the existing operations team, because we hadn’t really done anything like this worldwide around e-commerce before and it’s a lot more complicated.
We are responsible for the front end, so tool sets for monitoring the site were extremely important and we had to make decisions about how we were going to do that.
Focusing on the core
Bolton said that she wanted to focus on the core customer journey across the BBC Store, which consists of people signing into the site, browsing the catalogue, making purchases and actually playing content smoothly.
She added that these four components were essential and that if any one of them failed or didn’t work effectively it directly resulted in “brand and revenue impact”.
However, monitoring these services was not straightforward given that each of them was provided by multiple service providers. Bolton explained:
What complicated the matter was that each of the core journeys was provided by different service providers. Login journey was a number of third parties, purchase was managed by someone else, playback was a different supplier. As you can imagine, it’s one customer experience, but we were paying for third parties to provide this for us, so we weren’t getting that end-to-end view of the customer journey. We wanted a tool that could do that and show us how the site was performing overall for the customers.
And for the BBC team, Splunk, or more specifically Splunk Cloud, was the best choice.
That resulted in us choosing Splunk cloud, which allowed us to do that. We were getting quite close to launch and we needed the solution, it was a quick implementation for us. Maybe 8 weeks before launch we wanted to focus on setting up the monitoring, not just setting up the infrastructure.
I didn’t have the technical resource in the team to manage on-premise infrastructure. Generally it enabled us to focus on what we wanted to do for the business. User interface, dashboards and visualisations were a critical success factor for us - i wanted to establish all of them before we even launched.
So we spent a lot of time understanding where all the data sources were coming from the third parties and tools and systems we were using, we mapped what each of them was going to tell us in terms of the customer journey and fed that into Splunk.
Mapping the data
Bolton said that her team then spent a significant amount of time, which she described as a “challenging experience, working with her dev people translating the data received from the third party suppliers into a customer journey. The key question being - what does this code mean with regards to a customer using the site? She said:
We tried our best to translate that data so we could build a picture around customer experience. By the time we launched that’s basically what we did, we translated the data as much as we could.
We started doing performance monitoring - how quickly was the site performing? We broke the journeys down even further than the core journeys, for example if people were accessing their transaction history - how quick was that? How quick was it for people to get access to their programmes? How long does it take people to get access to their online programmes before they start watching content? We alert on all of this, so if it deviates to the norm we receive an alert.
It was so important for us to break down the journey - if we do get an alert it’s very quick for us to spot on the dashboard where the trends are, where the issues are. Having those already established dashboards was critical to us.
Some of the other areas that the BBC team are monitoring include authentication, so gathering operationalintelligence around registrations, how many new customers they are getting, when they are sending out marketing emails, the percentage of sign in successes, how sign ins are performing at peak times.
The team is also looking at the breakdown of payments. Not only the success of payments, but success of payments by payment type. So if there is a problem with Paypal, for example, the BBC team knows that this is going to affect 40% of payments going through the site.
We can then measure the impact of a major incident, so if a major incident occurs at 8pm at night we know exactly what the impact is to our customers. We can also understand if the errors we see are system errors or human errors.
However, the service operations team isn’t keeping this data to itself, it is also sharing it with other departments, such as marketing and e-commerce. Bolton said:
We are not just using and translating this data into IT and operational intelligence, we have also made marketing, e-commerce and finance very aware of the data in Splunk and how we can help them.
It’s not to replace their business analytics, but we have definitely found that there is a use of this data for them and we send them reports on a daily basis, which they can use for a quick indication of what the previous daily sales were. It was quicker than finance could. They also can see the spread of sales across the day. And we can also quickly report the use and success of promo codes.
We’ve got a long way to go, we’ve only been using it six months. The goal is to speed up resolution times and maintain a good service.