It’s that time of year again, the State of DevOps Report is out!

The team at DORA do a great job with this report and it clearly impacts the industry as it’s been all over my Twitter feed since its release. To me, it is my early DevOps Christmas present.

I want to take time and review a few items that stood out to me from my perspective as a tester.

In this I will cover:

• Elite performers vs misguided performers
• Software delivery and operational performance as an indicator of organizational performance
• Cloud
• Open Source
• Outsourcing
• Continuous Testing
• Culture

Elite Performers vs Misguided Performers

This year the State of DevOps Report introduced two new categories:

• Elite Performers: The organizations that are continuing to raise the bar with how they deliver software. Think Google, Amazon, Facebook, CapitalOne etc.
• Misguided Performers: The organizations that are failing to adopt DevOps approaches and instead are sticking to a cautious, large batch approach to delivering software to reduce risk.

When we look at the performance of the Elite Performers compared to Low Performers we see that they have:

• 46 times more frequent code deployments
• 2,555 times faster lead time from commit to deploy
• 7 times lower change failure
• 2,604 times faster at recovering from incidents

Tester takeaway

Often as testers we take a skeptical mindset. We want to slow things down, test more and ensure everything is perfect before release. Situations still exist where it’s unwise to deploy daily, but in the vast majority of situations, working in larger batches reduces the quality of delivered software. As testers, we need to enable the team to deliver software faster and provide faster solutions because ultimately this results in improved product quality.

Software delivery and operational performance as a predictor of organizational performance

The report combines three metrics: software development, software deployment and software operation to form a construct called Software Delivery and Operational Performance (SDO). I paraphrase this construct as “an organization’s ability to deliver software efficiently and keep systems running”. The report finds that SDO is a predictor of organizational performance and that Elite Performers are 1.53 times more likely to meet or exceed organizational performance goals such as:

• Profitability
• Productivity
• Market share
• Customer satisfaction

Tester takeaway

We need to keep in mind that organizations are undergoing significant and difficult changes to the way they deliver software in order to improve their overall performance. Let’s keep our minds open to these changes and keep asking the hard questions but doing so with a mindset of finding solutions. Software operation, or availability is also a key component neglected by testing. How do we work more closely with Ops? Can we better prioritize our testing? What if we understand the production monitoring in place?

Cloud

Cloud has been around for a while now, but this report drives home how critical Cloud is for organizations.

The report found that organizations adopting essential cloud characteristics (i.e. on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service) are 23 times more likely to be Elite Performers.

This is a truly staggering number! And as a tester if you’re not “Cloud Aware” you should be!

Interestingly, there was also a discussion on the differences between “Cloud Native” design vs software designed for traditional on-prem servers.

Tester takeaway

As testers we must continue to up-skill on what is required to test in the cloud. Cloud applications look like traditional applications from the frontend, but we need to understand the differences on the backend and how it impacts testing requirements, such as Performance Testing, Integration Testing, Security etc. We should also be looking for more and more ways to leverage cloud tools for our own testing e.g. mobile device farms and cloud-based performance testing.

Open Source

58% of respondents stated that their teams made extensive use of Open Source components, libraries and platforms, and 50% also agreed that their team planned to expand the use of Open Source software. This coupled with the fact that Elite Performers are 1.75 times more likely to use Open Source show that organizations are increasingly relying on Open Source. There was also an interesting mini case study about Capital One and their “Open Source First” philosophy.

Tester takeaway

While Open Source is still not suitable in every situation it is becoming increasingly common. As testers we should bring ourselves up to speed in Open Source tools, both for testing (e.g. Selenium, Cucumber etc.) and general development practices. We should also ensure that when we evaluate our commercials they will play well with the Open Source development stacks.

Outsourcing

Outsourcing is common in the IT industry. Many organizations rely on it heavily and it is a key way of expanding an organization’s capabilities. The report shows that Low Performers are 3.9 times more likely to use functional outsourcing than Elite Performers, and Low Performers are also 3.2 times more likely to outsource one of these functions - application development, IT operations or testing and QA.

This suggests that Elite Performers rarely outsource by function. These findings mirror the criticisms of outsourcing from many Agile & DevOps advocates and can be related to the impact that outsourcing of a function has on the batch size. Functional outsourcing is most commonly adopted by Misguided Performers.

The report also discusses how functional outsourcing has a detrimental impact on performance, but there are working models with multiple parties which can be effective.

Tester takeaway

Functional outsourcing of testing is a common and often harmful practice in our industry. It doesn’t surprise me that a link exists between wholesale outsourcing and Low/Misguided Performers. For third parties to positively impact an organization’s performance they must align with the client organization and operate in a collaborative manner. Organizations reliant on outsourced suppliers will struggle to shift to DevOps.

Continuous Testing

Previously the State of DevOps report has focused just on Test Automation, which was shown to have a significant impact on Continuous Delivery. This year, the report’s focus expanded to Continuous Testing, a more encompassing idea.

Continuous Testing was defined as including the following practices:

• Automated Testing, which includes fast, reliable test suites.
• Continuously reviewing and improving test suites to find better defects and keep complexity and cost under control.
• Collaboration between testers and developers throughout the software development and delivery process.
• Preforming manual test activities, such as Exploratory Testing, throughout the delivery process.
• Developers using test-driven development techniques such as Unit Testing.
• Receiving feedback from automated tests in less than ten minutes.

Continuous Testing, while a significant investment, provides strong benefits to those who adopt the approach.

Tester takeaway

It’s great to see Continuous Testing becoming a key component of adopting DevOps and to me this indicates a shift away from DevOps being *just* Dev and Ops to a more inclusive interpretation. The practices which form Continuous Testing are spot on. The need to continuously review and improve are vital and are often missed by organizations. Test Automation cannot be a “set and forget” approach. The incorporation of TDD and automated feedback within ten minutes shows that Continuous Testing needs to be a collaborative approach between developers and testers, and the inclusion of Exploratory Testing is particularly important. Continuous Testing is more than just Test Automation. It is a holistic approach to enabling the delivery of quality software.

Culture

The report once again takes a focus on the kind of culture that is conducive to successful software delivery. They use a model called the Westrum organizational model with the generative (performance oriented) culture being the highest performing for DevOps.

This culture type includes:

• Messengers trained
• Risks are shared
• Bridging (collaboration) encouraged
• Failure leads to inquiry and learning
• Novelty implemented

The organizational culture shares much in common with the types of cultural changes that are spoken about by DevOps advocates.

The report also looks at the Employee Net Promoter Score (eNPS). It shows that employees at organizations with a generative culture are 1.8 times more likely to recommend their team as a great place to work.

Tester takeaway

While we often focus on the technical aspects needed for DevOps, culture is the key to making it all work. As testers we must ensure we play our part in creating a positive and collaborative culture within our organizations. We’re an equal part of delivering quality software and we need to be seen as enablers, not a bottleneck.

Summary

Every time I read the State of DevOps report I find another nugget of information that helps me in our DevOps journey.

Having the personal stories and anecdotes backed up by scientific research also provides another angle to the conversation.

It’s pleasing to see that Continuous Testing is now included in the analysis and this serves as a reminder that as testers we need to continue to up our game. The Elite Performers give a high benchmark to strive towards, but it is an incredibly exciting journey to be on!