And it's true that when you think about big data and the Internet-of-Things (IoT), a number of open source technologies spring to mind. But is Splunk worried?
I got the chance to ask Brian Gilmore, one of the people heading up Splunk's Internet-of-Things division. Gilmore is pretty well placed to give us a sense of Splunk's position on this. Although he's not about to admit that the company is shaking it its boots, he is an ex-customer, and so has that end-user perspective. Also, dealing with IoT, which are some of the most complicated data projects out there, he is likely to understand the role open source plays.
His response was also balanced. Whilst admitting that Splunk can't ignore the rise of open source in the enterprise community, he also believes that those large customers will mostly turn to companies that can offer a unified, stable and secure experience that can't necessarily be easily built in-house on cheaper alternatives.
When I asked Gilmore about the shift of 'nerdy' data projects into the business areas, he said:
I think we are somewhat isolated from those 'nerdy projects', as you put them. Clearly those projects tend to adopt the open source platforms to start. What we do see though is that as those projects start to drag on and people want to know where the value, output and outcome are, that's when we start to get called in.
We can bring value to an application that's been built all the way up to Hadoop, Hunk sits on top of it, so that we can give people access to the data in search and explore. Or we have actually seen some interest in rewind and erase in terms of open source projects that were built, when partners of ours realised that the end-to-end configurability of Splunk all through one platform [is key].
Even when you use an open source like Lamda architecture, where you use Kafka and you pump Kafka into Spark, then you dump the same data into Hadoop, then you have to build out some top-end SQL like Hive, then you put Tableau on top of it – you build that and you think cool, I've got a very powerful platform. But what about the manageability? Security? Securing that type of platform and making sure that all your Ts are crossed and all your Is are dotted in terms of security of data in transport, security of the web interface for the end users, security to make sure that people aren't sending data onto the bus nefariously. It gets very, very complicated.
Gilmore says that the companies that are beginning to use open source tools to build out these big data projects are beginning to run into the issues that Splunk ran into ten years ago. His implication being that Splunk has now figured out a lot of those issues and has addressed them in the form of the Splunk Enterprise and Cloud platforms.
He added that even though open source tools cost very little to run on paper, Gilmore says that when you take into account all the added management and scale requirements, Splunk can deliver value on a much quicker timescale. Gilmore said:
We can't ignore the fact that people are aware that [these open source products] exist. I think that when customers are looking at total cost of ownership and the difficulty of management, ultimately the goal of any of these IoT projects is to have an environment where people constantly interact with the data, interact with the devices, it's not designed to be a set it and forget it environment.
And even the sort of costs of customisation over the long-term on those open source projects – I mean who updates and manages all of the libraries when your big data warehouse or your HDFS or Hadoop gets updated? How do you make sure it's backwards compatible with your messaging bus, your user interface etc. There's just a huge cost and expense there.
I think when people realise that Splunk is an application that you can install on one server, start small, build out applications, there's a massive ecosystem of app developers, it's a very efficient way to deliver value and it scales to massive scale. When you're building your own architecture from scratch, can you be sure it's going to work for a terabyte, let alone two or three hundred terabytes? We just have an advantage there.
A convincing pitch from Gilmore. And I'm inclined to believe that enterprises are generally going to be more comfortable procuring a product that comes with the support and assurances that come with enterprise software SLAs.
Having said that. It's not one size fits all. We often hear about open source projects that are working well – although these are frequently being integrated by a service provider that guarantees a certain level of service. But either way, open source is becoming mainstream in the enterprise. Splunk can't ignore that.