Trade Me bids for better business insight with Splunk

[sws_grey_box box_size="690"]SUMMARY -  New Zealand’s biggest auction and classifieds site is using the Splunk database to mix machine data with business data to better understand the customer experience.  [/sws_grey_box]

There aren’t many things a Kiwi can’t buy or sell on Trade Me, New Zealand’s biggest internet auction and classifieds site. Back in March, a man was convicted of manufacturing methamphetamine with equipment bought there.

Another used the site to sell his ‘soul’ - in actual fact, a skull-shaped bottle of dirty water - for the handsome sum of NZ$46 (US$37).

And on a more legitimate note, New Zealand’s Ministry of Business, Innovation and Employment plans to use Trade Me to auction millions of dollars’ worth of radio spectrum to local broadcasters.

Whichever way you look at it, Trade Me is, as its head of infrastructure Matt de Deventer puts it:

a big fish in a reasonably small pond.

In a country with a population of just under 4.5 million, it boasts almost 3.5 million registered member accounts (although some belong to sellers outside of New Zealand with customer there). It accounts for around 70% of all traffic flows in New Zealand, as measured independently by Nielsen, and as of August 2014, attracted just shy of 800,000 unique visitors per day.

For the last couple of years, Trade Me’s IT team has been using the Splunk database to capture, store and analyse machine data to spot problems within its IT environment and identify their root cause. More recently, they’ve expanded their use of Splunk, bringing in more contextual business data from other databases, to build real-time views of business operations.

Van Deventer takes up the story:

Initially, Splunk was intended simply as a syslog replacement. It really proved its worth there. Before we had Splunk, when something went wrong, for example, with image serving, it could be tricky to work out where in the stack the problem was, whether it was in the caching layer, the network layer, the storage layer or somewhere else entirely.

Given that Trade Me has around 600 million images on its site and receives around 10,000 image request per second, the work Splunk performed for monitoring and alerting was extremely valuable.

More insight

But once Trade Me had been using Splunk for some time, van Deventer says, he began to realise that there was potential here to get a lot more insight, especially into issues that went above and beyond how IT operations were running and focused more on the customer experience.

The key, he realised, would be Splunk’s DB Connect tool, which allows it to reach into relational databases in the Trade Me estate and pull out structured business data on customers, listings and transactions.

By mixing this information in Splunk with the clickstream and mobile data it already keeps there, Trade Me is able to gain more business-focused insights into user preferences, bidding behaviour and listings popularity.

It’s also able to track the effect of changes to the website and mobile apps on real-time customer experiences. Changes are deployed in twice-daily windows of between five and ten changes each time, he explains:

We can’t afford to screw up. With Splunk, we can see the impact of changes straight away,  from the impact of a tiny textual change at one end of the spectrum to major changes in the way that images are displayed, for example. If a change proves to be a problem for customers on a particular device, for example, we know about it immediately.

New insights have led to the creation of new features, services and campaigns. For example, the company was recently able to demonstrate the value of a television advertising campaign for its Trade Me Jobs website by correlating the times of day the ad was shown with inbound volumes of job applications.

But more importantly, Trade Me is also able to better understand the multi-channel customer experience: which devices customers use to browse the site, which devices they prefer when it comes to making a purchase, what kinds of listings are most likely to attract mobile users and so on. Van Deventer explains:

So now we can see, for example, that I browsed the site on my iPhone, searched for a car, looked at a particular listing and, two days later, logged on via the desktop website and bought it.

As a result, Splunk has been a big hit with business analysts at Trade Me, who have been trained to use the database and to build their own dashboards, he adds:

They now use it daily to generate their own business insight reports out of massive volumes of formerly disparate data sets. What we’re finding, as we go along with this, is that the more we put in, the more we get out. We can throw pretty much any data into Splunk - and as we add more and more information, we’re getting more and more interesting insights.