Hadoop Maturity Survey - closing in on tipping point

AtScale partnered with Cloudera, Tableau and MapR to compile a Hadoop Maturity Survey. There were 2,168 respondents of which 77% of have between 10 and 500+ nodes. Over 80% of them have been using Hadoop for more than 6 months. In short, this is a survey of users who are actively pursuing Hadoop.

The survey results  suggest that Hadoop adoption and use cases are reaching a point where companies are confident about taking Hadoop from the experimental stage to value driven production.

While the results will be encouraging for Hadoop fans, it is not yet seen as strategic by the majority of those surveyed, although you could conclude that if you add in the 15% who see Hadoop as 'game changing.'  (See image at top right.)

As might be expected, a good number believe they have achieved tangible results, but they remain a slight minority at 49%. That may well change in the next 6-9 months because 77% said they plan increasing Hadoop investments.

Although scale out needs dominate as the primary reason to use Hadoop, the variety of use cases is interesting with only 17% citing storage cost savings. Revenue generation at 14% and new applications at 19% are encouraging.

Who is driving Hadoop projects? 57% of respondents who said executive mandate is driving Hadoop have achieved tangible value, compared to 49% of organizations where business is the main driver, and 47% where adoption is driven by IT.

Who is leading the charge? Unsurprisingly, 0ver half of the respondents (52%) view marketing and sales as the biggest beneficiaries from Big Data projects, followed by operations (35%.)

In the five years since Hadoop first started getting attention, the most common workload was for ETL process and data science applications. The AtScale survey clearly indicates a shift in that thinking with business intelligence type applications becoming the dominant broad application category.

I was surprised to see the survey found that Tableau is the number one BI tool for Hadoop while Excel is the number one for non-Hadoop environments. This should be a wakeup call for the traditional BI vendors who lag a long way behind the leading solutions. Qliktech also has plenty to worry about since it is in the no-mans land of low penetration and impaired brand recognition for these workloads.

In a conversation with Bruno Aziza, CMO AtScale he said that:

Hadoop is complex but it should be invisible to the user. Users know that self service is the easiest way to get value both now and into the future but most don’t have that capability. The survey is clear, those that have self-service are much more likely to see value.

Which industries are most advanced? This graph is telling:

In one sense I was surprised by these results. Financial services are usually very early adopters but seeing manufacturing as an early leader should not surprise given the plethora of sensors that are in those environments.

Image credits: all from AtScale survey