Manawatu Software Quality Assurance Professionals Meetup: What Gets Measured Gets Managed

Exploring Software Quality Metrics and Their Impact

I bet that quote triggered a few reactions – and the occasional Google search to work out if Peter Drucker really did come up with the quote 😊. Search for the quote online, and you will find all sorts of commentaries, articles, and reactions.

Thankfully, we didn’t get distracted by that at the recent Manawatu Software Quality Assurance Professionals Meetup on 25th February. Our focus was on the ‘what and how’ of software quality metrics – and the ‘I wish’. There was still the inevitable discussion on management, however…

It was a very good session.

Insights from the Meetup

For the Meetup, we were stoked to have four presentations from the community – three discussing what they currently implemented and planned to establish within the next 12 months, and one outlining the desired metrics from the perspective of someone accountable for application delivery but not directly involved in testing. Each perspective contributed greatly to the end-of-event discussion.

Massey University’s Approach to Quality Metrics

Our first speaker, Jarod (Massey University), spoke off the cuff about their organisation’s experiences. Historically, quality assurance reports were regularly generated by environment tooling as software changes progressed through release management environments into production. These reports were aimed at the Heads of Business Analysis, Development, and Application Support

Reports were distributed as PDFs via email, but tracking revealed that few recipients actually reviewed them. This led Massey University to rethink their approach, shifting focus to:

• Is the software delivery on track?
• If issues arise, can we explain why?

To address these needs, they leveraged test execution metrics using TestRail and Tricentis Tosca for test automation. Their current reporting focuses on:

• Test execution results
• Percentage of pass/fail rates
• Test coverage across requirements

With reporting often being time-consuming, the focus remains on delivery efficiency.

FMG’s Journey Towards Effective QA Metrics

Helen from FMG shared insights into FMG’s transformation journey in testing. As part of this journey, they reviewed their quality assurance team’s focus, delivery methods, and tooling. They opted for Xray as their Test Management tool to provide consistent metrics.

At FMG, quality assurance metrics and reporting form part of a broader suite of reports that measure and track performance. Metrics are categorised into three key areas:

1. Delivery
2. Reliability
3. Sustainability

Both absolute scores and trends (whether improving or degrading) are tracked. Helen detailed the current metrics and those planned for future collection, explaining how they contribute to FMG’s overarching strategy for performance reporting and management.

View Helen’s presentation here.

Toyota New Zealand’s Automation-Driven Metrics

Grant from TTC Global presented on Toyota New Zealand’s extensive test automation suite. Unlike Massey University and FMG, Toyota New Zealand, integrates its metrics collection with its test toolchain, incorporating Tricentis Tosca along with PowerShell and Microsoft Power BI to create easily consumable reports.

Key metrics for Toyota New Zealand include:

• Pass/fail rates for automated tests (both scheduled and manually triggered)
• Test automation coverage of key applications

Their focus is on ensuring a resilient test automation suite, providing assurance that regression suites are comprehensive enough to confirm that changes do not negatively impact functionality.

View Grant’s presentation here.

The Concept of Digital Immunity

Shifting gears from current reporting practices, Sujeeth from FMG introduced the concept of Digital Immunity – an application’s ability to withstand user interaction. Digital Immunity is assessed across four main categories:

1. Business Value
2. Organisational Effectiveness
3. Quality and Technical Debt
4. Velocity and Flow

Sujeeth emphasised the importance of incorporating metrics from not just the Quality Assurance function but also business units and those responsible for supporting business processes.

View Sujeeth’s presentation here.

Connecting the Dots: Lessons from Tauranga

This event followed a recent similar meetup in Tauranga, which also covered the role of metrics in demonstrating the value of testing. The discussion highlighted:

• Manual effort saved via test automation
• Business risk reduced by identifying and resolving defects
• The cost to the business if those defects had not been resolved

Read more about the Tauranga event here.

The Future of QA Metrics

While software testing is now a well-established practice, this meetup demonstrated that there are still many perspectives on metrics:

• What is valuable?
• What is useful?
• What should be reported, and when?

What are your key metrics, and how do you ensure they are useful and informative for your audience?

Let us know your thoughts or if you have any questions!