Alibaba on open source and cloud business in China - live from the MariaDB user conference
- Summary:
- One of the best things about MariaDB's first annual user conference (M|17) was hearing from open source advocates who flew in from APAC countries to tell their stories. Alibaba presented on how they use open source at monster cloud scale. I also got some interesting views on why some open source database projects are a lot better than others.
When you get a chance to hear from engineers at Alibaba, you seize it. From Alibaba's roots as a Chinese business-to-business portal in 2009, Alibaba has now bypassed Walmart as the world's largest retailer.
What Alibaba said at this week's MariaDB user conference was a little surprising to me. It's about pushing the envelope of open source and cloud, not getting stuck in the database traps of the past. Not long ago, high performance and security compelled companies of this scale to spend big on proprietary databases. Not anymore (here's some of Alibaba's open source projects, and their AliSQL GitHub page).
Alibaba - open source at scale
To say Alibaba's performance requirements are intense would be an understatement:
- Stability at scale - Alibaba must support 100,000 concurrent updates on same row per second on its most popular goods.
- Security and uptime - High availability and "100 percent safety" of customer data is expected - this is addressed through a complex, transparent data encryption plan.
- Black Friday style e-commerce surge - Alibaba hosts the annual November 11 Global Shopping Festival (Singles' Day sale), the world's largest 24 hour online sale. The huge sales volume - $17.8 billion in sales in 2016 - puts huge performance pressure on Alibaba's systems, to the tune of 140,000 orders per second, and 85,000 payments per second.
Open source code sharing - a two way collaboration
During their talk on how Alibaba collaborates with Maria DB, Alibaba's Jianwei Zhao, Senior Database Engineer, and Lixun Peng, Staff Database Engineer, presented details on Alibaba's architecture. That included AliSQL, their cloud-friendly MySQL branch, which enables their work with MariaDB. Zhao and Peng are part of a thirty person Alibaba Cloud engineering team, dedicated to MySQL and MariaDB (Alibaba has 35,000 employees total). Alibaba Cloud now supports numerous "MySQL-like" instances, which includes those Alibaba Cloud customers who choose MariaDB.
During his part of the talk, Peng shared the connection between MariaDB's open source community and AliSQL. It's a two-way code share, with Alibaba contributing to MariaDB and also porting functions from MariaDB. Joint development is also underway, including a sequence engine, which helps companies migrate from large proprietary databases to the public cloud. They are also working on adding columns dynamically, and other column function enhancements.
Alibaba's contributions to MariaDB include multi-source replication, capacity shrink and expansion, and auto-increment persistence and parameters. Ports from MariaDB include max statement time, usage, and expansion.
You don't need to grasp all the database terminology to get this: it's about openly sharing functions and code. That's a pretty radical departure from submitting feature requests and bugs to your enterprise database vendor - and crossing your fingers that a patch will come soon. Peng:
Sometimes what MariaDB wants to do, we have already done. So we can contribute our code to MariaDB. It saves MariaDB time to do other important things. And sometimes, what we want to do, MariaDB has done, so we can port these features into our branch. So we can save lots of time too - and that's great.
Crossing the globe for open source know-how
Given that virtually all of Alibaba's technologies are built on open source, you can see why Peng considers himself an open source enthusiast. Docker is one of the open source technologies that is very popular at Alibaba. Peng says his team meets many community developers via their open source projects - a big reason why Alibaba sent three engineers to the first-ever MariaDB user conference. Peng:
MariaDB is a great open source database, all the activities and development is open. We can see all the details, which is very good.
But wait - aren't all "open source" tools open in this manner? Not in Peng's experience. Names are withheld to protect the not-so-innocent, but Peng says that some big open source projects - similar to MariaDB - are not as open.
Sometimes [with other projects] we don't see the work until after a new release. We waste time doing the same work.
But it gets worse: some projects won't accept Alibaba's patches:
They can't accept other patches, even when we test it. They will not use our patches. MariaDB is different. With MariaDB, we can push our patch into MariaDB directly - that's good.
The wrap - open source should be linked to customer choice
It was refreshing to hear from Chinese companies that are heavily invested in open source - and reaping dividends. Challenges span borders: like the U.S., China is no stranger to hackers. Peng advises any company involved in open source to make sure all security functions are enabled, and don't disable any of them.
Alibaba Cloud's job is to give their customers database choices. But it's clear Peng and his team will be investing in the MariaDB partnership. For them, not all open source isn't created equal. There is a caliber of relationship and a level of transparency that factors in. That makes sense to me - it's hard to imagine getting business results on Alibaba's scale without all those ingredients in place.
I got the impression the Alibaba team will be looking hard at MariaDB's new features and asking customers about offerings they haven't explored, such as the MariaDB ColumnStore. That points to breaking down barriers between OLTP and OLAP - a tantalizing topic that was, alas, outside the scope of these conversations. That's a story I picked up in chats still forthcoming.
End note: I was impressed by Zhao and Peng's guts in front of an American audience; I could not have stood in front of a Chinese audience and presented in Chinese. I've done my utmost to get their story perfect, but if something got lost in the language barrier, I will fix it immediately.