New data shows that DevOps + security = a winning combination

DevOps has proven to be an effective means of reshaping IT and developer organizational culture and processes in ways that improve software quality, release cycles and deployment robustness.

As I recently wrote, integrating security into the DevOps process has emerged as perhaps the best way of stemming the seemingly endless tide of security intrusions, data thefts and application hacks. A new survey supports the assertion that organizations with the most complete, holistic commingling of the three disciplines, application development, cybersecurity and IT operations, have more secure infrastructure. As a subjective measure, the survey can’t link security results with practices, however it does show that those most thoroughly incorporating security into their DevOps processes have the highest confidence in their application security posture.

The amalgam of the three, what some have tagged with the unwieldy moniker DevSecOps, seeks to keep security front and center throughout the application lifecycle. As I previously discussed, a recent report from the Cloud Security Alliance (CSA) summarized the methodology and its benefits this way (emphasis added):

Putting security directly into DevOps substantially improves outcomes by integrating existing security into development and operational processes through modern integrated security paradigms, such as DevSecOps. … The goal is to reduce the complexity in the development and publication of software, ensure that only known and trusted components and services are used, empower software development teams with security resources (both automated and human) integrated directly into their development methods, use development environments that are well secured and monitored, and ultimately deliver the end-product per design and with only the features it was designed to have.

What the 2019 edition of an annual DevOps survey sponsored by software vendors in the DevOps ecosystem, Puppet, CircleCI and Splunk, sought to assess was the degree to which security is integrated into DevOps practitioner's software development and delivery cycles and how much difference it makes in their overall security awareness, rigor and results.

Survey highlights - security posture and perceptions

First, my usual caveat about surveys. The expense and effort required to conduct a user survey means that most, like the one cited here, are sponsored by vendors, a relationship that leaves them ripe for all manner of techniques like skewed samples, loaded questions and Texas sharpshooting to achieve the desired result. In reading through the survey methodology and talking with one of the authors, I found this DevOps survey to be of higher than usual quality, albeit not without its limitations (more on those below). It used a substantial sample, almost 3,000, in which a self-selected online population (and hence, subject to observation bias) was supplemented with a third-party recruited panel composed primarily of executives and managers not necessarily conversant with DevOps. Thus, within its limits, I think the survey presents a reasonably objective measure of enterprise thinking and practice around DevOps.

This year’s Puppet, et.al. survey builds on last year’s results in which the authors developed a 5-level DevOps maturity model that categorizes practitioners based on the degree to which DevOps had been incorporated into their software delivery cycle. For 2019, the authors sought to identify any correlations between cybersecurity preparedness and an organization’s stage of DevOps maturity. The stages of DevOps evolution, which are summarized in the following chart, start with the basics such as development teams using a single software version control system and work up to fully automated DevOps workflows that include self-service infrastructure provisioning and automated incident response.

The report further categorized the extent that security experts and teams are involved in the DevOps process using a similar five-stage scale that is defined by the number of software delivery phases that include security considerations. The following chart summarizes the criteria.

The 2019 survey found that almost half of respondents are at levels 3 or 4, showing that security is generally incorporated at multiple stages of the software delivery pipeline, but not all.

Unsurprisingly, it also found a link between an organization’s security integration and its perception of its security preparedness, with 82% of those at Level 5 agreeing with the proposition that “our security policies and policies significantly improve our security posture.” The report identified five such practices with the greatest effect on security as:

• Security and development teams collaborate on threat models.
• Security tools are integrated into the development integration pipeline to eliminate inadvertent security mistakes.
• Security requirements, both functional and non-functional, are prioritized as part of product updates.
• Infrastructure-related security policies are reviewed before deployment.
• Security experts evaluate automated tests and are called upon to review changes in high-risk areas of the code (such as authentication systems, cryptography, etc.).

Translating from security practices to results is trickier and here the report’s weak point since it didn’t include data that might correlate the number or rate of security incidents with an organization’s integration level. However, it does include some secondary metrics such as:

• A slight relationship between the integration level and the time to fix critical vulnerabilities. For example, half of those at Level 5 can remediate problems within a day, with 11 percent of those doing so within an hour. In contrast, between 33 and 38 percent of those at Levels 1-3 fix serious security incidents within a day.
• 61 percent of those at Level 5 can deploy software patches on-demand versus the 36 to 48 percent of those at Levels 1-4 that can do so. Note however, within that group, there isn't a direct correlation between security level and deployment speed (see chart) indicating some other factors at work.
• Nearly 90 percent of Level 5 organizations prioritize fixing critical or high-severity security issues over new features when making software enhancements, versus only 60 to 65 percent of those at Level 1. Furthermore, the results show a monotonic increase in the willingness to prioritize security over functionality as organizations increase the level of security involvement in software development.

The deployment time is symptomatic of several other surveyed parameters in that results worsen as security is first integrated into DevOps processes only to significantly improve upon achieving complete security integration. The authors offer a reasonable explanation for this counterintuitive result:

The J curve we see for other performance outcomes also describes this ability to remediate critical security vulnerabilities. The cost of integrating security practices is felt most in the middle stages of security integration, and the payoff happens in the later stages. We suspect this is because firms at Level 1 have fewer handoffs and stakeholders to work with, and for those at Level 5, thorough and complex business process is properly modelled and accounted for.

The J curve also appears when assessing the friction between DevOps and security teams at various stages of integration. Since the two have little overlap at first, there’s little conflict, however as the level of integration increases, so do the clashes, only to reduce as the two parties learn to work together.

My take

The Puppet, et.al. DevOps survey adds nuance to the arguments for incorporating security into an organization's core development and IT processes, however, as I mentioned above, it stops short of providing a causal connection between meticulous adherence to procedures and improved security metrics. Thus, it doesn’t provide the link nor degree to which increasing security integration into the software pipeline reduces the number or severity of vulnerabilities, incidents or security-related outages. Indeed, this lack of hard data in part answers a question the report poses in its introduction (emphasis added):

For most companies we’ve spent time with though, integrating security into the software delivery lifecycle is an unrealized ideal, and an obstacle to furthering their DevOps evolution. Why is that the case? When security practices are so well understood, why is it so hard to integrate security into DevOps?

Call us cynical, but we believe it’s because good security practices don’t pay the bills. Good security is not a competitive differentiator. Getting new features out faster, on the other hand, gives you the clear competitive advantage of being early to market. So feature delivery naturally becomes the top priority.

That’s not being cynical, just realistic. Until organizations can quantify the ROI on increased spending on security, adding time, expense and process overhead to software development cycles will be a tough sell to executives who see a more direct link between new products and features and their corporate and personal bottom line. As long as removing executives over security breaches, as Target did, remains the exception and the cost of security incidents amount to a tiny, one-time hit on company profits, don’t expect massive behavioral changes. Sadly, it will take a few more existential, company- and career-ending crises to create rapid, significant improvements in the way organizations integrate and prioritize software and infrastructure security.