But while this move will hurt DataSift in the short term, its CEO Nick Halstead believes the long-term impact will hurt Twitter more, undermining its efforts to sell both advertising and data to big business. I met with Halstead over lunch on Thursday and while he didn't know Twitter's announcement was imminent, in the course of the conversation, he set out three reasons why offering a direct data feed hurts a social network's business.
It was also evident from our conversation that Halstead sees DataSift's future very firmly as serving the broad enterprise market. The company intends to become a platform for analyzing not merely social data but all forms of what the company prefers to call "human data" within an organization.
1. Neutrality matters
Although losing its direct access to the Twitter firehose will hurt DataSift because it will lose that revenue component, it's a small proportion of the company's overall turnover. It has focused on building out a platform that integrates and analyses data from many different sources. Eighty percent of revenues come from what DataSift does with the data rather than simply reselling it, Halstead told me.
We've focused on being a programmatic platform. Our customers can teach our platform how to look at the data.
We are a platform to allow our customers to develop faster and proper social in a unified way without having to do the data integration of 500 different data sources.
Twitter has made a big error of judgement in seeking to make money out of selling its own data, Halstead implied. Businesses will have less trust in the data if it hasn't gone through a neutral analysis platform, which will in turn lead to them buying less advertising.
Yes there's some money to be made in data, but the knock on effect of having your analytics out there really helps ad buying decisions in the end. The networks that don't participate — a good example is Google+ that's never done it — brands don't know what's really going on in the network.
Why we've been successful is by being an independent player. The brands can trust the data because it's us processing them and it's not data that's being supplied.
That is why DataSift has remained at the platform level and never sought to develop its own analytics solutions.
People often ask us, why aren't you trying to be your own social monitoring platform, and we could build the ultimate one because we'd then have more data than everybody else in the world, but we'd then be competing with our thousand plus customers and it's better to be this independent powering them all than try and make a name in that market, which is very, very competitive.
2. Businesses want answers not data
Twitter's approach will require its customers to each build their own data infrastructure to process the data it supplies. Businesses no longer have the appetite to build all of that from scratch, suggested Halstead.
We work at a scale that for most businesses would be beyond their wildest dreams.
You can no longer reinvent the wheel in this market. You could spend three years with fifty developers building your own kit to store and be able to index that kind of scale of data and the text analysis and requirements on the market of how you get value from the data has become so much harder.
Our technology's incredibly defensible because of the scale we work at, the complexities of what we do is not something very easily recreated. It takes a lot of experience, not just lots of programmers.
Businesses aren't interested in buying big data technology for its own sake, he said. They're doing it because of the business outcomes they want to achieve.
The key thing I think people forget about big data is, it should be there to serve a purpose, to answer business questions.
The big problem with most big data solutions is they're technologies, not things that are giving you answers.
You need to start to join that data in some way to the real world to have real-world business impact.
What we care about is how do I make it so that when I see what one person is saying on Twitter, I can say that's the intent of it, that's the positive or negative, and solve that thing for the end business who doesn't want to have to integrate ten different technologies to try and get the same result.
Last month, DataSift announced a new alliance with Facebook that allows it to analyze social data behind the Facebook firewall. The aim is to collect and analyze richer context around the data.
If I'm a marketer or an advertiser, I can actually look what people are saying across the whole of Facebook. I can see what is the age, where are the locations that it's working, and I can start to see how my ad spend on Facebook, on TV, radio, whatever, is being perceived by location.
That level of fine control, you'll be able to make much better business decisions around it. For Facebook, that in the end comes back to advertisers because Facebook wants high quality for both the end user and for the business. If it's better for the user it means more engagement and therefore more clickthroughs for Facebook.
The next step for DataSift will be to take the same technology behind the enterprise firewall and apply it to collaborative and transactional data such as enterprise messaging, he told me.
3. You must respect privacy
An important reason for striking the Facebook deal was to have an offering that complies with European privacy laws, Halstead told me.
What's happening with Facebook is, it's aggregate data that is actually outputted. There is no one individual that is ever analyzed. It is a topic view of a minimum size audience that you're analyzing.
The difference is that the data you get back does then include demographics, which has never been done before.
DataSift has an advantage over US based competitors, said Halstead, because it is closer to those privacy concerns. Even holding publicly available information such as someone's tweets falls foul of European law.
In many European companies, there is no way legally you could supply that data into the end customer because they would then be holding personally identifiable information. You could be tracking an individual's tweets, and even though that individual knows they're public, it doesn't matter under European law.
So we started working on this aggregate model knowing that for the enterprise to get involved, to be able to start to bring this valuable data into a broader market needed this model.
This is a battle between Twitter wanting to own access to its platform and DataSift's multi platform strategy [UPDATE: see useful further reading on the platform strategy published today by board member Mark Suster]. DataSift's I suspect will have more traction in the enterprise market.
Image credits: firehose trickle © fotodrachenei - Fotolia.com; headshot by @datasift.