You can't run a consumer business effectively these days without being completely data driven.
Mark Holt speaks from experience as CTO of Trainline, whose £1.9 billion ($2.5bn) annual sales of rail tickets make it the UK’s sixth largest online retailer. Constant testing helps it maintain and improve its sales through its web and mobile apps.
We test everything. We have a multi-variate testing tool that works both in the app and in the website. Every customer that comes up at our website or on our app, is seeing a slightly different version of the site. We're constantly testing to see which ones are more effective at getting customers through the experience.
The company even tested the effect of different colors when it rebranded last year to drop the .com from its name.
The difference between white text on a mint background, and blue text on a mint background is millions of pounds in revenue. The sensitivities to this stuff are incredible. It's a huge part of what we do.
The data shows that reducing latency by a third of a second can produce an extra £8 million ($10m) in annual revenue. Among other tools, Trainline — which has just completed upgrading its infrastructure from Oracle Exadata to Amazon Web Services — uses New Relic software to track application performance in real time so that Holt's team can see immediately when behavior is changing.
The real time nature of this is fantastic. It's no good to me to discover a week later that our fulfilment mix has changed. Real time is really important to me.
Customers don't experience averages
Understanding every detail of what's happening is also a crucial element. "Customers don't experience averages," says Holt. If the data's not granular enough, then it may be masking two completely opposite results by averaging across positive and negative impacts. He gave an example from the company's mobile application.
This is where getting deep inside the data is really important. We saw an example in the mobile app, where if customers came in via one route to a screen and they interacted with a new button that we'd put on there, they were 4% less likely to convert than if they came via a different route to it. Whereas, if they came via [another] route, to the same screen, and played with the button, they were 2% more likely to convert.
Now we hide it when they come in via one route, we include it when they come in via the other.
This example demonstrates that it's not purely about response times, though those are important. It's also how people react to what they see on the screen - a phenomenon Holt calls "cognitive drag." One example was when customers bought tickets to London terminals, they were offered a hotel promotion. But this even happened when the ticket was for a short 10-minute journey.
That sort of thing, that's cognitive drag that's stopping people. People are going, 'What, I'm just coming here to buy a train ticket and now you're offering me a hotel? Why? Why am I doing that?'
We want to help people through the experience of actually getting what they want. Some people, that's a value-add and the hotel is useful and the travel insurance is useful and the cancellation protection is useful.
Personalization is a little bit about making sure that we ask the customers for whom it's going to be useful.
Investing in data science
That's why Trainline is increasing its investment in data analysis with the recruitment its first data scientist — who was instantly able to identify patterns that can help reveal when customers will want to buy those value-add products, says Holt.
As part of his interview process he identified with an amazing degree of accuracy, which customers were likely to convert to some of the ancillary products that we have, and which ones weren't.
It was things like, people booking outside of their normal pattern. If they normally book these kind of trains, they're very consistent, and then they change to a different sort of train, with a different sort of booking, that's when they're going to say [I want] cancellation protection.
There's also potential to mine Trainline's historic data to give customers useful information, for example to let them know which are the busiest trains and which are likely to have more space available. Holt can also envisage using the data to offer more airline-style price management, but as Trainline doesn't set prices itself — that's up to the train operators — for now it has to content itself with other aspects of the customer experience.
We can't vary the prices. We can indicate to customers this train is looking really busy, you might want to [book a different one] ... because, again, we're about the journey experience.
Being able to say to a customer, we haven't [only] sold you a ticket, but we've made your trip better, because you're on a train that isn't as busy as the train you otherwise would have been on, actually is really important.
The objective is to enable what Trainline calls 'smarter journeys' for its customers — or as Holt puts it, "to create a magic carpet ride when they get on the train." Not only selling a ticket, but also helping customers complete their travel even when things go awry.
That moment of magic, is exactly what we need to create from a train travel perspective. Your train has been delayed, you're going to get home at this time instead. We've rerouted you, your ticket is valid on these trains, you are all good. Those are the little moments of magic that make all the difference.
But beyond its own use of data, much of what Trainline can achieve depends on interactions with the train operators and how much investment they are willing to put into enabling those more magical journeys. E-ticketing is one example. At the moment it's available on 30% of routes, says Holt. But for a seamless experience, it requires ticket barriers at the station that can read a QR code from a mobile phone. No operator is currently completely e-ticket enabled, and some have barely started, although the government is encouraging more aggressive adoption of the technology.
We know that customers spend up to 20 minutes queueing at ticket machines. The amount of time people are wasting queuing for pieces of dead tree when they could just go [on their phones], 'I've bought it, I'm going to go.'
Buying it on the way to the station would be much better. We've been working with the industry to roll it out and we'll be doing a lot more of that this year.
The notion of train travel becoming a 'magic carpet' experience is one that prompted wry laughs from a London audience. Trainline's impact is limited to how customers buy and use their tickets — the actual journey remains in the hands of the UK's privately owned train operators, some of whom are failing miserably at their remit.
But what Trainline is able to do to by analyzing operational data and fine-tuning the online experience demonstrates how powerful even small design decisions can be. It's a fascinating example of best practice for online businesses that operate at volume.