Hadoop ODP squabble rumbles on as first compliant products come to market
- Summary:
- This week at the Hadoop Summit in Brussels, Hortonworks, IBM and Pivotal announced distributions based on a new “common core” of open-source technologies, but there's open tension in the air.
Less than two months after the announcement of the Open Data Platform (ODP), an initiative among IT vendors to establish a core set of common technologies for the Hadoop big data platform, Hortonworks, IBM and Pivotal have announced that they are ready to ship products that conform to the new standard.
The ODP’s launch has sparked a lively and at times fractious debate in the big data market, a row that was still rumbling on at this week’s Hadoop Summit in Brussels.
In one corner, the ODP’s proponents claim that establishing a standard version of the core Hadoop stack will make life easier for the developers who build analytic applications to run on it - in areas such as real-time event processing, graph processing, search and text indexing - and who must currently build apps multiple times to run on different vendors’ platforms. They also assert that this will result in “less confusion and friction for enterprise customers”, as well as greater choice.
With these goals in mind, Hortonworks, IBM and Pivotal announced in Brussels that, from this point onward, their respective Hadoop platforms would all be built from a “common core” of components.
Specifically, these components are Hadoop 2.6 (which includes the Hadoop Distributed File System (HFDS), the MapReduce component for parallel processing of very large datasets and cluster management component YARN). The core also includes Ambari 1.7, a management console for Hadoop.
The three distributions - Hortonworks Data Platform 2.2, IBM Open Platform 4.0 with Apache Hadoop, and Pivotal HD 3.0 - are all based on the common ODP core and are now available.
But...
In the opposing corner, however, are those that argue that ODP can only deliver on its bold promises if every company in the Hadoop ecosystem signs up to it - and they haven’t. Nor do they seem likely to do so.
Hortonworks rival Cloudera has dug its heels in, with pugnacious chief strategy officer Mike Olson delivering a blistering attack on the consortium in a blog published hours after its launch was announced. Neither independent software vendors (ISVs) nor enterprise customers are remotely confused about building applications on Hadoop, he argued. He sees the ODP as a closed circle of vendors set on slowing innovation in the sector and open only to those prepared and sufficiently well-heeled to pay its steep membership fees. ODP, he scoffed, should stand for ‘Only Dollars Pay’.
Almost two months on, and with Hadoop Summit in full swing, Olson tells diginomica:
We’ve not had any engagement or invitation on ODP since it was announced. We simply see no value in a programme that competes with the Apache Software Foundation, which is where the standard has been driven for the last decade. No plans to join.
For his own part, Hortonworks CEO Rob Bearden laughs off the idea that the ODP seeks to compete with the ASF, the organisation that oversees the development of Hadoop through the worldwide, voluntary efforts of software developers. He says:
Everything the ODP members do is absolutely based on ASF technology. I hope Cloudera will join. They were absolutely invited to join, but they’ve been violently against it and that’s their decision.
At the heart of this ugly squabble lie fundamental differences in Cloudera’s and Hortonworks’ respective approach to the Hadoop market, laced with personal and political rivalries that data back to the technology’s invention in the software labs at search giant Yahoo! almost a decade ago.
Both are key members and prolific contributors to the ASF, but while Hortonworks only distributes ‘pure’ open source software and makes it money solely on support services, Cloudera earns its keep by building and selling proprietary products to run on Hadoop.
Not least of these is its management console, Cloudera Manager - which competes directly with Ambari, a product largely led by Hortonworks. In other words, Ambari’s inclusion in the “common core” that the ODP is pushing makes membership a non-starter for Cloudera.
In any case, of the 17 companies listed on the ODP website as members, ten are also Cloudera partners. That means that, while vociferously promoting the virtues of a single standard, they’ll still have to build apps to run on Cloudera’s Hadoop distribution, too.
Hortonworks’ Bearden suggests this might not be the case in the long run:
Maybe they don’t, going forward. Maybe they don’t need to. If they’re in ODP, they’ve got one certification on three distributions. When you get that much leverage, where do you think they’re going to put their engineering effort and focus?”
This one will run and run
The dispute around ODP, then, looks set to grind on indefinitely. At research company the 451 Group, analyst Matthew Aslett sees no end in sight.
There’s a lot of talk here [in Brussels] about ‘no vendor lock-in’ and ‘a single distribution’. That’s clearly over-playing it.
Independent software vendors have signed up to ODP, because they saw it as an opportunity to reduce their testing and porting efforts across multiple distributions.
Now, they’re realising that, while ODP will help in developing for Hortonworks, IBM and Pivotal, it does nothing for them in terms of developing for Cloudera and MapR - and those companies are simply not going to join.