Rumble Entertainment on monetization, heavy lifting, and the cloud
Ho's team is focused on maintaining the Rumble platform and managing cloud services, so our talk gave me a chance to find out how Ho and his cohorts support more than 200,000 game players a day, powered by more than 50 servers globally. Ho has a long time interest in gaming platforms, dating back to work with Microsoft on the Xbox team. More recently, he was at Apple Computers, but two years ago, he made the choice to join the now 50+ employees at Rumble Entertainment, most of whom are based in San Mateo. Fresh off a Series B funding round in the fall of 2013, Rumble Entertainment has ambitious growth plans in play. Ho told me how his team fits into that.
Jon: Why were you drawn to work at Rumble over other gaming companies?
Albert: Aside from the fact that Rumble has passionate employees, Rumble is trying to make a difference in what we call the free-to-play game space. Everyone here comes from a very good pedigree in the gaming industry, so it's not like we haven't been here before. Greg Richardson, our founder and CEO, comes from Electronic Arts. He was able to secure a lot of funding for a vision: he wanted to build free-to-play games that were different from the rest of the market.
Jon: Different in what sense?
Albert: Most free-to-play games you see today are largely simple and transactional-based, like FarmVille or CityVille. The production qualities are, shall we say, limited? Our founder, Greg Richardson, wants to marry design and product innovation with the free-to-play space - we want to build products that people love. I don't think anyone loves FarmVille. They play because it's addictive; you don't get emotionally attached to anything in the game. We're trying to weave narrative and emotional resonance into a free-to-play fabric. It's very challenging, let me tell you, but I think Rumble has the best chance of anyone in the industry to do this.
Jon: And KingsRoad is your first major offering right?
Albert: Yeah. KingsRoad is an action/role playing game. There's a narrative based on the character you choose, but the bigger point is the ability to have a loop-driven game, where I can play through a game with a progressive system, and be able to pick up new items during the game, such as when I slay a monster, and mprove my character as I go along. We have two other games currently in the pipeline: Nightmare Guardians and Ballistic.
Rumble Entertainment's monetization model
Jon: Talk to me about monetization. How does that work in a free-to-play game - are we talking in-game purchases, premium subscriptions?
Albert: That's two of the big ones - we have in-app purchases and we are working on a premium subscription as well. In-app purchase has had many flavors over the years. With free-to-play, the idea was that you were free to play the entire game, but you could purchase quality of life improvements - things that would make game play more enjoyable. Zynga flipped that around by selling bottled time - basically limiting the number of actions each day. Then you had to pay money or spam your Facebook friends to get them to send you energy.
Jon: I seem to recall a spammy message or two. :)
Albert: At Rumble, we are avoiding the 'bottled time' approach because we think it turns off gamers. We've settled on several things. One is: anything you can buy in the game is something you that you can actually grind for - grinding meaning that if you spend enough time, you can get to that same level. Or you can pay to accelerate your advancements, buy skill points, and so on. We also built events into the the game - always adding new features.
There are ways you can monetize around events, by basically either charging an entry fee, or to have optimal gameplay during the three day event, you spend a little money. We make money, but the gamers also feel like they're progressing with their characters. It's also a social lubricant when you have these events, because everyone interacts in the chat area.
Albert's team, cloud, and 'undifferentiated heavy lifting'
Jon: Let's get into the area you're actually focused on at Rumble, which is managing the cloud and platform side. How does your team use cloud services?
Albert: I'm paraphrasing one of the architects I work closely with - Sam - but he has a term, 'undifferentiated heavy lifting.' What's interesting about the cloud is that it has allowed a lot of startups like us to build up server infrastructure quickly - without having to maintain server technology in house. Now, whether that technology runs in Amazon or actually on bare‑metal boxes sitting in some server farm in Texas - it doesn't really matter for these types of services.
Jon: Right.
Albert: There are certain things in our business we need to do that's core to ourselves. For example, we need to provide integration to all these payment gateways for all the games. We need to make payments easy for people. We need to build a UI that makes it easy for all this to happen. We also need to provide chat as a service across games, and that's actually not straightforward. There's actually a lot of complications in getting a scaled chat.
But on the other hand, we don't store credit card information. We would never do that at this juncture in our business. We're too small to be dealing with that kind of data compliance - it's too much red tape to wade through. So we outsource payment providers. When we look at our own platform, there's certain things we do that don't make sense for us to handle ourselves - and that's where the undifferentiated heavy lifting principle comes in. Take logging, for example. We tried building our own logging cluster, but our games are generating on the order of a couple terabytes a day of log data, and a lot of it was noise. We didn't have time to filter all through the log data.
The challenge of Rumble's log files - in terabytes
Jon: What kinds of numbers are we talking about here?
Albert: It was costing us a lot of money - like on the order of five to six figures a month, just to run our logging cluster. We looked around at a lot of logging vendors, and then we found Loggly. I think what the cloud enabled Loggly to do was provide their service cheaply and at a scale, for a variety of customers, and that's what's really changed in the industry.
Jon: So logging is tied to troubleshooting your system performance?
Albert: When a server goes down or when we go down, our revenue impact is huge. Even if there's a slight outage or things are just slow or lagging, we might see as much as a 70 to 80 percent revenue dropoff that day. Sometimes the problem is insidious. You and me, we'll check out the game and find its working fine when we're playing from the United States. But if we had checked in from Singapore or Europe at this time, we would notice these other problems.
Jon: And you use third party tools to track this performance?
Albert: We do. We use New Relic, which provides us a good dashboard view of what's happening in the system. But Loggly has had the most impact on game quality. Typically, it's hard to measure game quality for a ratio of dollars spent versus bugs fixed/found. There are many difficulties building the log cluster and indexing the data. Before we got Loggly, it was really hard to look at the data and say, 'Okay, today we pushed a new build. Is performance better or worse compared to last week?' When you have 50 or 60 servers in each region around the world, you've got to be able to coalesce all the log files in one place and make them searchable. Once we got Loggly set up, we were able to funnel all the log data into Loggly in a way that was structured so we could search on it. We started being able to have high level metrics to measure performance from release to release.
Jon: Assessing solution ROI is notoriously difficult - how do you do it in this case?
Albert: When we used to have a problem, it would probably take us a week to find the root cause. Now a lot of bugs get fixed literally that hour. Any of our engineers can go into Loggly, look at the stack trace, and identify the issue. They don't need a project manager to do it. We can also pull the player ID of the user experiencing the problem, and work with customer support very quickly to address any issues that may have arisen.
It's only as good as the work you put in up front. Our developers put some structure behind it, where they store log data in JSON format, so it's searchable and indexable. You can like filter off of it. . Another thing Loggly has enabled us to do is to reduce the number of servers we actually have to own, because our servers are so efficient.
Jon: Reducing server costs is always a good thing.
Albert: One of the things we realized in our architecture is you have to take a hybrid approach. You can't just be all Amazon. You have to spread your eggs over many baskets, and you have to go where the price is right.
Jon: What are your 2014 goals for your team and your company?
Albert: Growth. The most important thing is that we continue to grow in terms of customer base, while making sure the users are happy. For my team, we need to make sure our system stays operational, to limit errors and bugs, and to use cloud services to cut the cost of goods sold.
Disclosure: Loggly PR helped to set up this interview. Loggly is not a diginomica partner as of this writing.
