Companies want to be intelligent with machine data, but most aren't managing it

I want to start this post by saying that I try really hard to stay away from writing about research and reports – particularly those sponsored by vendors – because more often than not they are self serving and there is a commercial agenda at play. However, occasionally the odd one grabs my interest and today I received a report from Splunk (no sniggering from those of you in the UK), which looks at how European businesses are dealing with the challenge of using machine data to become more operationally intelligent. So why the interest?

Well, first of all I think Splunk is a company worth watching – although they cater for a very niche area (machine data and operational

intelligence doesn't exactly get most people's juices flowing), they have an incredible list of customers that swear by the product. In the past I've written about Splunk being deployed at Barclays, WorldPay, e-Travel SA and Dominos Pizza, plus others, and all have been very open about the use of the tool and its benefits. That's not to say that Splunk is perfect - I have also spoken to customers that complain about its user interface and it being an expensive product. However, this is usually coupled with an unequivocal belief that there isn't much else there in the market that can match its capabilities and that it is worth the spend.

Apart from my personal interest in Splunk, I also think that operational intelligence is an area that most digital companies will need to invest in in the future if they want to provide a seamless customer experience that is stable. Although it sounds dry and sometimes the business case can be a little obscure, there are real benefits to be had. See my previous case studies (links above) for some examples. Anyway, enough with the justifications and on to the report...

A desire to improve, but a lack of capability 

The report itself was produced by Bob Tarzey over at Quocirca, a well respected industry analyst, where he surveyed nearly 400 companies across Europe on the topic of machine data and operational intelligence – you can download a copy of the report here, providing you're willing to hand over some personal details to Splunk. The research found that the more transaction intensive businesses – namely financial sector and telcos – were the ones that believe operational intelligence can provide them with business insights that they can't get from elsewhere. However, overall, 82 percent of European businesses stated that they would benefit from collecting, storing and analysing real-time and historical machine data. So, the appetite is there. However, almost just as many of the companies (83 percent) admit that they would like to improve, or aren't currently getting, real-time business insight from their machine data. A capability gap.

Just to be clear when we are talking about machine data we are talking about information harnessed via IT system logs, which may include what data went via which router, who accessed which application and when, IP addresses, URLs and devices which web sites are accessed.

The report goes on to define the four stages of 'operational intelligence maturity' – where Bob Tarzey claims that most companies start at stage one and then mature over time through stages two, three and finally four. The stages are as follows:

1.  Search and investigate
   1A Capture, store and search machine data
   1B Search and find the root cause of important events
   1C Analyse machine data to find the root cause of important events
2.  Proactive monitoring:
   2A Analyse machine data to notice and manage exceptions before they impact users or service delivery
   2B Use machine data in real time to make decisions about tuning IT systems
   2C Use machine data to provide the business insight it would not otherwise have
3. Operational visibility:
   3A Use machine data to measure service levels and key performance indicators to better serve the business
   3B Use machine data to gain end-to-end visibility of consumer behaviour and business performance
   3C Examine machine data for general intelligence about the business, customers etc. that would not otherwise be available
4. Real-time business insights:
   4A Correlate machine data with business data to provide real-time insight for turning business processes
   4B Use real-time analytics from machine data to detect patterns, identify trends and predict outcomes (e.g. for capacity planning and fraud detection)
   4C Provide the business views from machine data analysis that drive real-time decision making and innovation (customer insights, marketing insights, usage insights, product-centric insights)

Using these principles, the report put together an operational index which marked companies on their capabilities, where the lowest score is 0 (no capability) and the highest score is 3 (strong capability) – bit confusing given that there are four stages of maturity, but anyway,

moving on. Overall, European companies scored an average of 1.92, which sits between poor and some capability. Telcos came out on top again with a score of 2.23, finance second with 1.98, followed by retail, transport and distribution with 1.92, gaming with 1.91 and manufacturing with 1.82. None of them particularly impressive, which suggests that there may be a need to invest in new solutions.

Bob Tarzey stated:

“True operational intelligence uses the data generated by machines to access, tune and improve IT and business processes, identify security threats, highlight performance issues and spot emerging customer trends.

 “Those companies who will gain the most value are those that start using machine data – everything from customer clickstreams and transaction records to logs and feeds from applications, servers, network activity and sensors – to enable practical business decision making. 

“Making sure the right technologies are in place for better data capture and improved analysis of large volumes of machine data is essential to achieve this.”

Verdict

Splunk's report argues – unsurprisingly – that gathering machine data into traditional repositories such as relational databases and data warehouses and using business intelligence tools for analysis will go some way to delivering operational intelligence. But what is really needed for a successful implementation is a unified machine data management tool designed to analyse and report in both real time and from a historic perspective (cue Splunk). However, despite the obvious sales pitch, I tend to agree. There are a number of challenges with gaining a holistic view of machine data, which are only multiplied when you need to pull in data from numerous sources – such as partners

and cloud providers – and a tool designed specifically to gather all of this together and analyse it for you in real time makes sense. Now I know with Splunk there have been issues in the past around usability (surprise, surprise) where the tool itself typically requires a fairly in-depth knowledge of the Splunk system to make requests, which might be fine for IT managers that are just fine-tuning their systems, but not so great for line of business people that want some real customer insights. However, the latest version of Splunk has made some moves to address this and hopefully it will improve in the future.

From conversations I've had in the past with Splunk customers they are often surprised at the use cases for real time operational intelligence, and I get the impression that a lot of the time they start with small pilots because they don't really know what they are looking for. However, there is a need to push the use case out of IT and into the business because this is where the real value for machine data products will be found – a line of business person needs to be able to easily use a dashboard to react to machine data analytics in real time. It sounds dry, but there are plenty of use cases out there and it's a good way of gaining a competitive edge.