AI, AI, AI – with a little bit of Test Reporting for Good Measure

Interesting times in Testing as there is lots of hype around Artificial Intelligence (AI) in the Software Testing space – both in the ‘How does AI help Software Testing’ and also ‘How do you test AI?’ space. With recent media releases pointing out the legality of AI advice ownership, organisations are on the hook for their AI output. AI maturity within most New Zealand organisations is anecdotally still within the ‘Early Adopter’ (experimental) stage however, so good hard stories on AI use are hard to come by. Many organisations are working on a ‘I’ll try this idea’ basis and still nervous about going public with their efforts.

In parallel, there has been a lot of market commentary on Test Reporting and Metrics and Key Performance Indicators (KPIs). A mature space within the testing world – many standard metrics exist. Equally – the metrics don’t typically talk to the effectiveness of the test team or the value that the test function is providing to the organisation. Or to the risk of the delivery. As one non-test manager recently commented “how is our reporting informing non testing managers/governance. I find often the reporting I see is a list of metrics rather than where are at with reducing the risk of an issue getting into production. How do we tie coverage and impact/risk together?”

We tackled each of these questions in our August Tauranga Test Professionals Meetup.

First up, General Manager Technology and Delivery for TTC Global, Mei Reyes-Tsai, presented on AI in Testing and Testing AI <Link to Presentation PDF>. Mei’s presentation covered Testing Trends, the hype vs. Reality Gap with Test Automation and then overlaid the history and impact (current and potential) of AI over this. AI does have a use in testing – organisations are using AI now to read Use Cases and derive test scripts from the Use Cases as an example. However, a human is still needed to review the results and confirm that they are as expected. The statement was made that even with the potential advantages and adoption of AI – good test management, test governance and test processes were still very much key to effective test outcomes. AI may be an enabler – it does not replace these.

Following Mei, the local community delivered two very good presentations on the use of Reporting Dashboards and Metrics.

For Tauranga City Council, Rodney Colvin provided an awesome overview of their current test reporting and metrics <Link to Presentation>. Azure DevOps based, Rodney’s presentation covered real time metrics used within the Council, as well as the custom metrics that the Council had created to support visibility of testing progress. Rodney also presented on their own internal efforts to benchmark and grow their internal test maturity – and the role that metrics had played in this process. One key takeaway from Rodney’s presentation was the use of a Metrics Wiki Page where all metrics were listed together with what the metric was presenting and how that metric should be used. In an environment with many active projects and many people looking at the metrics that were being reported, it was key that everybody ‘read’ the same information from the metric that was being reported. A great example and takeaway.

For Zespri International, Berny led an interactive discussion during his presentation <Link to Presentation> on the value and the ‘why’ of reporting. Berny walked through the layers of reporting at Zespri – Strategic through to Operational at a Project / Programme Level. Of significant interest were the metrics on team satisfaction, team efficiency and the value from the testing team contributed to the business from the work that they did. Effectively starting that shift in mindset from the Testing Team being a value centre rather than a cost centre. Zespri International’s reporting also covered the value saved through automation over manual testing – a key matric in understanding where the ROI of Test Automation is and if the organisation is living up to that expectation.

Overall – three great presentations that generated a lot of conversation in the drinks and nibbles afterwards.

How does your Reporting and Metrics look – and what is your Technology Adoption Stage with AI……?