How DreamWorks Animation uses Couchbase as a service for microservices at scale
- Summary:
- At Couchbase Connect 2017, I found a memorable use case on Couchbase as a service, via a Dreamworks Animation presentation. The takeaway? The impact of microservices at scale.
I haven't been to a Couchbase show for a couple years. A lot has happened in the NoSQL space since I last kicked tires on them. High on my list of questions: has Couchbase doubled down on its biggest NoSQL strength - enterprise scale?
At Couchbase Connect 2017, a "Couchbase as a service" presentation by DreamWorks' Naveen Kumar Eppa brought the issue of a scale to a head. Eppa, a NoSQL Engineer with DreamWorks Animation, presented an eye-popping slide on their data services:
For context, 3.6 billion hits per day is more than the number of search requests Google processes daily.
Eppa's session on Couchbase as a service also put microservices into play, showing how savvy IT teams are learning to push performance - and reduce downtime exposure - with microservices.
A massive data services environment
For the DreamWorks Animation IT team, the stakes are high. Eppa's data services team is responsible for maintaining and deploying all of DreamWorks databases. Eppa specializes in NoSQL; he has his hands in most of the major NoSQL databases one way or the other.
That over-buzzed phrase "data-driven business" could have been invented for animated movies. As Eppa told the Couchbase Connect Silicon Valley audience:
Data is at the core of making digital movies, which means the data services team is at the core of our business.
Established in 1994 in Los Angeles, DreamWorks needs no introduction, but you can't blame Eppa for singing its praises:
We are the creators of some of these greatest animated characters of all time. Our company was established in 1994. DreamWorks Animation is recognized as one of the most admired family entertainment brands in the world. We have produced 35 movies over the past 23 years. We have made over $14.5 billion worldwide at box office. Some of our all-time hits are Shrek, How to Train Your Dragon, Kung Fu Panda, Madagascar, Boss Baby.
I could have done without some of the Shrek sequels, but yeah - they made more money than I'll see in ten lifetimes, so DreamWorks gets the last laugh there. Add in three Oscars, 41 Emmy Awards and 59 patents, and you have a formidable media institution, with digital tech at its core.
All told, DreamWorks has more than 200 database clusters, supporting more than 15 different types of databases including Couchbase. "Pretty much all the databases that are out in the market," joked Eppa.
"Old ways of working" weren't ideal for modern data needs
Impressive stuff - but what led Eppa's team to microservices? Eppa cited issues galore:
- App deployment - "In the good old days, we used to deploy all our applications manually. This style of deployment is obviously prone to errors."
- Hardware inefficiencies - "We used to have dedicated hardware for each database cluster. And we would end up requiring new hardware when we had to deploy a new type of database."
- Monolithic clusters - "We used to have large monolithic clusters, which used to serve a lot of different applications. Whenever there was an outage, or if you needed to apply a patch, it really affected all our services."
- Development and production discrepancies - "Sometimes our development and deployments did not match in production, because we might have installed the patch in one deployment, and forgot to do the same in the other."
- Crashes were costly across systems - "Whenever there are crashes, we en3ed up losing some functionality for backups and reporting."
- Scaling was difficult - "Scaling of these databases was very painful because we could scale only vertical."
- Monitoring was problematic - Monitoring these databases was complex; making changes to all the database configurations was inefficient.
That's a big laundry list of tech headaches. As Eppa told us, "adapting to Couchbase as a service has allowed us to overcome all these problems." He brought the slide to prove it:
Lifting the Couchbase hood - how microservices got results
Eppa on the before-and-after:
Now we have our automated deployments, efficient resource usage and tracking, multi-tenancy, ease of management, monitoring and alerting, automated backups and restores, and elasticity.
And one more: service discovery, "which got cut" from the slide, joked Eppa. Good result, but what did it take to pull it off? Achieving Couchbase as a service required two things:
- Treat your data center as a cloud - "Even though we are too big to move our data to the cloud, we still have the cloud mentality when we think about our data center. We treat all our servicea as cloud resources and provision them on demand."
- Treat your databases as cloud-native applications - "Because we treat our data center as the cloud, we then can treat our databases as cloud-native applications."
So how does Eppa define cloud-native? "Cloud native is an approach to building and running applications that exploits the advantage of a cloud computing model." What, then, is a cloud-native application? Eppa says it must have three characteristics: it must be containerized; it has to be dynamically orchestrated, and it has to be microservice-oriented.
You could spend an article defining each. Short version: DreamWorks has obsessively fine-tuned its container usage with Docker and Docker plug ins. "Dynamic orchestration" is another area you could geek out with forever; it's about how DreamWorks manages their databases as "highly available clusters." That includes tools like Chef for managing database deployments.
That brings us to microservices. How should we define them?
The microservice architecture is a style of building applications as suite of small independently deployable services, rather than building a giant monolithic application.
Eppa's team uses microservices a bit differently:
Generally a microservice is referred to as a front-end application, or a stateless application, rather than a distributed application like Couchbase.
DreamWorks uses this framework for databases:
We have taken this concept of microservices and applied it to our databases as well. So, our microcluster philosophy says to deploy a suite of small, independently deployable clusters - rather than building a large monolithic database cluster.
That leads us to Couchbase as a service. For each movie, Eppa's team deploys over 35 different services, most of which include databases. Each service now has its own dedicated database clusters:
With this [structure], we achieve true multi-tenancy. With this style of architecture, we are able to isolate not only the movies, but services as well.
Now the list of problems shrinks dramatically:
With this [monocluster style], the impact of an outage is minimal. Only the data being served by that cluster is effected... It allows other movies to remain online. Finally, because these clusters handle a small set of data, the request like reads and writes are very fast.
The biggest concern I hear from developers about microservices is handling complexity, but Eppa doesn't see it that way:
It may sound counterintuitive when I say managing these hundreds of cluster is easier than maintaining a large monolithic cluster, but it makes us much more efficient and agile.
The wrap - adopting a microservices philosophy
One of the more potent before-and-after tech cases I have seen. Referring to the above slide, Apps said:
As you can see, adapting to Couchbase as a service has allowed us to achieve all of our goals. Our clusters are totally elastic; we can scale our clusters horizontally by adding new nodes with another singe click.
The benefits extend to database management:
We are able to maintain over 200 database clusters with only four DBAs.
During the audience Q/A, the DreamWorks Animation team explained their microservices philosophy extends well beyond Couchbase. Therefore, whatever database their developers need, the data services team wants to provide those data services on demand.
It's rare for me to write a tech-focused use case that doesn't tie directly to ROI or customer/user experience gains. But in the case of DreamWorks Animation, performance-at-scale is at the heart of everything. It was good to get a closer look.