Those that had acquired the appropriate skills were referred to, with some relish, as 'Ninjas’. The creators of the technology were often referred to as 'Splunkers’, which can sound like some behavioural idiosyncrasy that one’s mother might disapprove of. They, however, loved that as well.
Most still do, and as the company continues to add to and extend its range of tools that exploit the core technology of system log indexing and analysis to provide extensive security capabilities and a wide range of operational management tools. Then last year, the company hinted strongly that some first examples of business operations management services were cooking on the stove, and this year they were officially served up.
For each announcement was seen as solution to a previously intractable business management problem. And it usually was as well for Splunk had latched onto the trick of listening to the computers themselves. Collecting, indexing and analysing machine logs allowed users to learn what their computers were really doing, not what a developer hopes it should be doing, or what a hacker would want people to believe it is doing.
In one way this year’s conference was no different, there was a goodly selection of new product announcements, and the 8,000-plus Ninjas and Splunkers whooped and hollered their delight. But there was a difference this year, as Splunk CEO Doug Merritt pointed out. There have been some significant changes in important areas of the company’s core marketplace, and he sees a need for change to match it:
Our aim is to help you manage and control the chaos out there so you can use your data to make things happen. Most current tools for such tasks are highly structured. They are systems that won’t adapt and they won’t let you move forward. These days data is very messy, so Splunk is aiming to let you work the way your data works, in a messy data way where you don’t have to try and structure it first. Even Splunk Ninjas are starting to want it a bit easier.
Welcome to the world of chaos
The chaos he now sees surrounding most users is the shear volume of unstructured data and its rate of growth. This brings with it a problem that is growing as fast as the growth in data: there simply are not sufficient people with the right skills to go round so that companies can make some sense of their data. This could be seen in the take-up of places on the company’s pre-conference weekend 'Splunk University’'skills training sessions, which were sold out and over-subscribed. Merritt noted:
The rate of growth in data, and especially unstructured data, means that we are not going to get ahead of it. So we now want to allow ordinary users to act like data scientists.
The corollary of that is the company also wants to make it easier for experts to experiment and innovate.
So the underlying theme of most of the new or updated products announced at the conference is the goal of making access to, and the management of, data more available to a much wider range of tech-savvy `citizen’ developers and data scientists.
It was left to the company’s CTO, Tim Tully, to lead the conference audience through the new and upgraded products emerging this year, with most of them fitting into a portmanteau brand called Splunk Next. This is where the shift in emphasis into the Biz/Ops model is now front and center.
The center piece of this is likely to be Business Flow. This aims to provide users with instant access to on-boarded data from a wide range of sources. This will normally be all the data generated that is relevant to a specific business process and its operation, which is then analysed against time to provide visibility of all contributing channels of the process against a time line. Its main advantage for the users is that it can then provide visualisations of the process flow extracted from the data sets of each contributing channel in the process. This collection of data is normally referred to, rather descriptively, as 'walls of text'.
Because it is based on data derived from the different channels by other Splunk tools it can then be used to ID problems in the process flow, build changes to those areas of the process and test them, and then roll the changes out to the production environment.
Data Stream Processor is one of the tools that will acquire that source data, providing tools for the pre-load stages of data wrangling. And important feature here is the ability to perform such tasks as Source Type and Field extraction, and Redaction of some source data such as credit card information. One important feature here is the ability to identify each version of every data pipeline, so that if a mistake happens it is easy to roll back to an earlier version. It also has the ability to branch the pipelines so that a source of raw data can be kept separately.
The Next suite will also include Data Fabric Search, which provides federated searches across multiple Splunk instances. The company claims this can not only scale to trillions of events but can also carry out searches across all those instances much faster. A demonstration clocked a dataset using DFS searching nearly 1,499% faster than the same dataset running in a non-DFS environment.
There are other elements that map on to the overall Splunk direction of moving into the business operations management arena, including a combination of new steps in the use of augmented reality. The company’s Augmented Reality tools then enhance information delivery, especially using the greater flexibility available with smart phones and other mobile devices.
This is also extended by the steps taken by Splunk earlier this year, when it effectively junked its existing mobile offerings. So the conference gave the company the chance to perform its predicted 'mobile is dead, long live mobile' trick, with a whole new range of tools and services for mobile devices, all based on a new, dedicated, cloud service.
IoT – even bigger chaos
A complementary area of development for the company is IoT, or the Industrial Internet as Americans call it, or Industry 4.0 as Europeans have named it.
The key thing here, of course, is the same old problem in another – and arguably larger – form: the huge gobs of data it generates. This is taking in data from all aspects are areas of an industry’s assets.
There is already a fair level of competition in this marketplace, so Splunk is pitching at using one of its additional capabilities, prediction, as its differentiator. Its new IoT services will provide predictive analytics on all aspects of the business…. from real time monitoring of industrial equipment, through to detection and mitigation of IoT security risks. It also comes with a formula builder that will allow tech savvie business managers to create their own, specific KPIs that are relevant to their business.
The IoT sector is also expected to come up with the most applications for the company’s latest version of its Machine Learning Toolkit, V4.0, which now comes with connections for Apache Spark and for Tensorflow, plus a range of new algorithms available on GitHub.
This is another example of a once straight-up techie company spreading out into the world of business solutions for business problems. It will be interesting to see how far Splunk will want to take this line of development on its own. It already has a well-developed route to market with its existing Channel partners, though there has to be a question of how experienced they may be in working with business solutions rather than technical ones. Expect to see extentions to the channel community, and more focused business solutions appearing from them where the name Splunk may be hard to find, but will none the less be the driving force behind them.