Testing and AI. Another warm welcome in the deep south

There are some places in New Zealand where you know there will always be a warm welcome and a keen community. And Invercargill is right up there with them—one of my top five places to visit and collaborate. While a small software testing community, they are active and eager to share. And that was evidenced in our meetup on the 8th of October 2024. Hosted again by the good people at SBS Bank, there was a good turnout for the meetup – and good consumption of the food and drink available.

As with our recent Dunedin Meetup, the focus of the session was on AI in Testing and Testing AI. While some would point out that the hype curve with AI is in the ascendency, the reality is that there is a lot of investment (time and money) in AI. In response to that investment, organisations are looking into what AI means for them and their operations. And that means testing gets involved – both in terms of using AI as well as ensuring that the use of AI (inside and outside of testing) is delivering the intended results for the organisation. Notice I didn’t use the word ‘right’?

Mei Reyes-Tsai again started the meetup with a presentation on the recent research done by TTC on the use and adoption of AI. Covering the history of Testing and Test Automation, Mei covered the hype curves of both Test Automation and AI, noting the similarities and that the more that things change, the reliance on the fundamentals stays the same. Both were hyped to replace manual testing. Both proved the reality that despite the hype and the technology, real testing skills were still needed and in fact are vital to ensuring testing success. Mei did note that there was evidence to show that AI would make some testing easier and reduce some cost – Mei highlighted ten use cases for AI in Testing and presented the current state evaluation and rating of two of these. Evaluations on the remaining eight can be obtained from Mei by contacting her via the TTC Contact Page.

For testing AI, Mei’s presentation discussed the challenges with testing AI (when you cannot predict what the outcome will be, how can you define success and success with repeated executions – each with a different result?). Mei presented on areas to focus on and techniques to reduce the variability, complexity and cost of testing AI. Stubbing and virtualisation are your friends from a cost management perspective and automation enables rapid execution of AI-based test cycles – as long as you allow enough wait time for the response!! The impact of ‘AI gone bad’ was discussed with some real-world examples (many leading to litigation), further evidencing the need for a well-thought-out testing approach.

A testing approach planned and managed by testers.

As much as AI is a current focus for testing, testing isn’t all AI. So, to mix things up a bit we had two very good presentations from SBS Bank on their testing tooling and practices.

Kim Bale delivered a well-prepared and succinct overview of the testing process and toolchain at used within SBS Bank. From Jira and Xray to Confluence and AccelQ, Kim presented how the combination of tooling and process – and good configuration – was delivering results in terms of quicker testing times, good levels of traceability, and a high level of reporting on the state of the applications under test. A great presentation that generated a lot of discussion and comparison with other organisations present.

To reinforce the tooling and processes, Steve Wensel next presented real-life examples of the environment in use. We cannot share the presentation here as real-world SBS Bank data was presented. SBS Bank confirmed that the team is open to site visits however and are very willing to swap information and experiences with those also willing to share their environment.

All up – another great meetup within a great community. Great conversations and great local examples are being presented. Thanks to all those who attended.