Taming the multi-cloud monster - organic adoption vs. bureaucratic control
- Summary:
- Balancing the need for cloud controls and management with the convenience, speed and dynamism that drew users to cloud services in the first place.
Unfortunately, the term cloud remains sufficiently ambiguous, covering a broad spectrum of services from raw compute and storage infrastructure (IaaS) to packaged applications (SaaS), that depending on how the question is asked, the average enterprise might use dozens of cloud vendors providing more than a thousand distinct cloud services. Indeed, I would wager that at least half of the Top 10 cloud services, as ranked by a McAfee usage analysis of the 30 million users of its cloud service, have been used at one time or another by everyone reading this.
Source: McAfee Cloud Adoption and Risk Report, 2019
It’s noteworthy that each of these are SaaS applications, which are significantly easier to assimilate than new cloud infrastructure, however we’re now seeing cloud polygamy spread to infrastructure services as organizations seek to improve reliability, avoid lock-in, lower costs and take advantage of vendor innovations through a multi-cloud infrastructure.
The annual RightScale State of the Cloud report, which focuses on infrastructure deployments, related tools and management practices finds that the typical organization has deployed or is testing between four and five private and public clouds. However, the trend is clearly to the latter as usage of public clouds is growing three times faster at 24% annually, than private infrastructure. Indeed, half of the respondents from large enterprises now spend more than $1 million per year on public cloud services.
Source: McAfee Cloud Adoption and Risk Report, 2019
As with all such surveys, the data comes with the usual caveats about selection bias, built-in assumptions and loaded questions, particularly since respondents to the RightScale survey are self-selecting and the survey itself had a minuscule 0.64% response rate to more than 100,000 email solicitations.
However, RightScale has been doing substantially the same survey, the same way for many years, and only 20% of respondents are RightScale customers, factors that provide confidence in the demographics, year-to-year consistency and trends. Furthermore, the data confirms many rankings and trends reported elsewhere. Thus, the survey provides a useful snapshot of enterprise cloud usage. Some highlights include:
- Public cloud deployments are the top cloud priority for enterprises (31%) closely followed by hybrid cloud.
- AWS (#1 at 67% adoption) and Azure (#2 at 60%) are by far the most commonly deployed IaaS by enterprises, with Azure closing the gap by 3 points over the past year.
- VMware on AWS and Azure plus Azure Stack are the most widely deployed unified hybrid cloud stacks among enterprises. While some form of VMware stack is used on-premises by virtually every respondent, its AWS embodiment has grown to 18% penetration, showing that almost one-fifth of VMware users have some degree of hybrid deployment. At 28%, Azure Stack has even greater penetration, all of whom certainly also use Azure although RightScale doesn’t provide the cross tab data.
- Aside from raw compute and storage infrastructure, the most commonly deployed cloud services are:
- Relational databases like AWS RDS or Azure SQL
- Push notifications like AWS SNS or Google Cloud Firebase Messaging
- Data warehouses like AWS Redshift or Google Cloud Big Query
- Message queues like AWS SQS
- NoSQL databases like Azure DocumentDB and Table Storage or Google Cloud Datastore
- Managed containers like AWS Fargate and EKS or Azure Container Instances and AKS
- Serverless functions and stream processing are the services experiencing the most rapid growth in adoption with each up 50% in the past year to 36 and 30% penetration respectively.
- The majority of respondents have deployed containers, including 52% of enterprises using one of the AWS container services and 35 percent using their Azure counterparts.
Overall, the data reaffirms the public cloud market to be a quasi-duopoly, where outside of Amazon and Microsoft, all others appear permanently relegated to second-tier status. I would, however, make an exception for Alibaba since the Chinese market is unique and, I suspect, under-represented in this survey. The data also shows IaaS users graduating to higher-level platform and application services while simultaneously building hybrid environments that ideally provide a unified user experience and management platform for on-premises and hosted infrastructure.
The shift from deployment to governance and optimization
Collectively, the RightScale and McAfee data echo earlier reports such as a Deloitte CIO survey showing that enterprises are well past the early stages of cloud adoption characterized by testing the platform’s capabilities, experimenting with low-risk use cases and using the simplest of cloud infrastructure services. Instead, enterprises increasingly treat cloud infrastructure as a strategic, mission-critical extension of traditional on-premises systems and are consequently concentrating on operational excellence, efficiency, governance, security and compliance. Indeed, RightScale categorizes 34 percent of enterprises as heavy, advanced cloud users with 845 of their respondents have some form of multi-cloud strategy.
Vendor surveys understandably highlight issues where they have a vested interest in selling products, thus RightScale outlines the problems with cloud cost management while McAfee focuses on security shortcomings. However, both illustrate different aspects of cloud maturity and demonstrate an increasing awareness of the problems created by unchecked, organic growth of cloud services. While most organizations have yet to implement stricter management practices, they at least acknowledge the need for increased governance and control over their cloud deployments. For example, the RightScale survey finds that:
- Two-thirds of large enterprises have centralized cloud service management under a single IT team while 21 percent are planning for such a cloud control center.
- Among larger organizations, two-thirds (probably the same group with dedicated cloud teams) centrally manage cloud spending. Indeed, respondents understand that there are significant savings to be had, estimating that 27% of their cloud spending is wasted, a number RightScale believes is closer to 35%.
- Nearly 60% of enterprises centrally set and automate the enforcement of policies for cloud use.
- Just under half of enterprise IT organizations select public cloud providers, while only 41% have IT manage all cloud deployments, meaning that there's still flexibility in most organizations for departments and developers to choose the cloud services and deployment configurations that best fit their needs.
It’s instructive to note that these numbers can be significantly different depending upon who is answering the question. The survey found a 10-20% perception gap between respondents working in IT versus business units on the question of IT's role in managing cloud vendors, policies, costs and deployments. Such differences on the question of IT's role and responsibilities are nothing new, but illustrate the need for upper management to detail a cloud governance strategy and clarify lines of authority for its implementation. Highlighting the need for executive oversight, about a quarter of the respondents to RightScale's survey say that cloud governance, security and spending control are significant challenges.
The need for better cloud management and control is also evident in users’ growing risk of data theft or other breaches of security. While the infrastructure and services managed by AWS, Microsoft and Google are almost surely more secure than that of on-premises IT systems, many cloud users don't fully appreciate the shared security model of such services and leave their data and applications exposed to threats. Again, we must consider the source of the following data since McAfee has a vested interest in hyping, if not exaggerating the risk. Thus, I don't take the following as objective, quantified truth, but rather an indication of the types of exposures and growing risks cloud users face. Using aggregated monitoring data from 30 million McAfee MVISION Cloud users worldwide, it estimates that:
- A fifth of all files stored in the cloud contains sensitive data and that the uncontrolled sharing of sensitive data, i.e. accessible via a link that doesn’t require login credentials, has increased 23 percent in the past two years. Indeed, S3 buckets left wide open to public access are a frequent source of massive data breaches.
- Cloud threats from compromised accounts and either accidental or malicious abuse of privileged user rights by insiders have increased by more than 50 percent over the past two years. Indeed, insider threats outnumber those from external attackers hacking an account. Even so, the vast majority (80 percent by McAfee’s estimate) of organizations see at least one privileged cloud account externally compromised each month, most by way of previously stolen credentials available for sale on various Dark Web sites.
- Most enterprises have multiple misconfigured IaaS or PaaS services that expose information to undue risk. Typical mistakes include not automatically encrypting data on storage volumes like EBS or equivalent, leaving outbound network access open and unrestricted, incorrectly using role-based access controls (RBAC) and security groups, and not using multi-factor user authentication.
Such security issues illustrate the relative immaturity of cloud service management and the generally poor state of enterprise security writ large.
My take
Cloud services demonstrate a typical pattern in enterprise IT in which usage of new technology sprouts and spreads organically much faster than the means of governance and control. However, as the use of cloud services — up and down the stack from raw infrastructure to fully managed applications — snowballs to permeate every corner of an organization, consequently becoming a critical element of enterprise operations, the wasted money, reliability risks and security exposure of uncontrolled usage becomes acute.
The trick for enterprises is balancing the need for cloud controls and management with the convenience, speed and dynamism that drew users to cloud services in the first place. IT organizations shouldn’t load up the cloud with layers of bureaucracy and unnecessary limitations in the name of reducing costs or improving efficiency. Cloud governance and management requires a mindset, processes and supporting software that are as efficient, nimble and adaptable as the cloud itself. Otherwise, organizations risk recreating the same bloated, stultifying and cumbersome IT environments in the cloud that led developers and business managers to flee central IT in the first place.