Gay social networking app Hornet upgrades Cassandra to improve community experience

Hornet is one of the world's most popular social networking apps for the gay community. It has 30 million members worldwide and has become the most popular app of its kind in countries such as France, Russia, Brazil, Turkey and Taiwan. 

Combining popular elements of other social networks, Hornet allows users in the gay community to connect with others through messaging, update stories to their profiles and also post to a newsfeed. And all of these interactive features require petabytes and petabytes of data. 

However, up until November last year, the leadership team at Hornet felt that its existing Cassandra cluster was limiting what functionality and services it could provide to its community, as it was running an old version of the NoSQL database. 

Wanting to expand into areas such as shared interests, locations, and video, Hornet realized that it would need to expand its storage capabilities by upgrading to a newer version of Apache Cassandra. 

Matthew Hirst, Head of Server Side Engineering at Hornet, led the team involved in this project and explained:

We estimated that the volume of messages that we would be supporting would go up by around fifty percent, and alongside the continued growth in the number of people using the app, that would become a problem in the future. We knew we had to make some changes to how we managed our data over time.

As such, Hornet engaged with the DataStax Luna consulting service on Apache Cassandra to help it scale up the existing messaging and newsfeed clusters it was using. 

Without a hitch

We got the chance to speak to Nate Mitchell, Lead DevOps Engineer at Hornet, about the upgrade to the messaging and newsfeed clusters. On the need for the upgrade, he explained: 

Predominantly the issue was mostly just logistical in terms of storage. We didn't have too many performance issues otherwise - we would occasionally have strange blips where a node would drop for a few minutes and then come back. And that was a bit of an issue, which is what actually led us looking into this. 

But predominantly our limitations were with storage space. We're looking at four plus terabytes of data for just messages and tables.

Prior to the upgrade, with the number of users rising, and the number of messages and the use of Hornet's features increasing, so too were costs, given the data storage requirements. 

Working with DataStax Luna consulting service, Mitchell and the rest of the Hornet team were able to free up capacity and reduce this cost burden. He said: 

Being able to modify things and fix the clusters in such a way that we're able to free up a huge amount of disk space from things that we set up back in the legacy days, allowed us to basically give another few years of life to the cluster without any drastic changes being required. 

It also allowed us to provide additional benefits to our users without having to make any changes on the outside. So, our user feed, for instance, used to show the last two weeks - it has now doubled in the last month, because we've got the space to be able to provide that service without having to invest in additional monthly operational expenditure.

Removing stress

As noted already, Hornety worked with the DataStax consulting team to design its upgrade approach, as well as optimize how it managed the clusters. The consulting team also worked with Hornet to implement the upgrade and manage the changes successfully in a phased approach - with the newsfeed cluster being updated first, followed by the messaging cluster. Mitchell said: 

It really takes a lot of the stress out of doing an active upgrade on production systems when you've got DataStax there, who has done this a million times and has the understanding of what to do if something goes wrong. I'm by no means an expert, I have to work on 50 different things every day, I can't specialize in any one of them. So yeah it was extremely beneficial. 

And the benefits have been significant since the upgrades took place. Hornet has been able to reduce the messaging cluster volumes by 50%, it has improved the content service archive from two weeks to one month of messages, and carried out the upgrade without any downtime for users. Mitchell said: 

This is probably going to be the most engineering non-answer ever, which is simply that I haven't really had to care about Cassandra since we made the changes and upgrades. Usually if I was getting paged in the middle of the night, it probably had something to do with a brief Cassandra blip that was causing an increased response time. That has just gone away completely.  

We are also able to accept far more users at this point, before we had to look into doing any sort of infrastructure upgrades - infrastructure meaning disk size, CPU, memory etc.

Mitchell said that Hornet has seen a very decent improvement in the base compute resource usage of the clusters, since the update was made. And this is now helping fuel its community ambitions. He added: 

We are always trying to find new ways to engage with the community. Just from an operating cost point of view, this frees us up to focus resources on new features that we otherwise wouldn't have been able to implement if we'd had to pay to increase the size of our Cassandra clusters. 

We can handle a lot more messages now, we can show you more of your historical messages with less delay, and we can show you double the amount of playback of your user feed than we used to be able to do. And at the same time we are financially able to spend the money we would have allocated for those increases on other features of the app, such as video.