Thinking Differently About AI and Testing

A few years ago, I learned an important lesson about how to frame success with automation. A team I was coaching had cut the time a full regression took on their core mainframe system from 3 months to a week. We presented this success to the VP of Development, and he asked me a question I wasn't prepared for: "Great, what can we do with this extra time we have?" 

I went into the meeting expecting to celebrate. I had argued that this might be possible, now we had proof. Instead, I was reminded of my study of Saul Alisky's Rules for Radicals "The price of a successful protest is a constructive alternative."

So over time, we built a very success plan of what to do next. Because we could test faster, more features were shipped in the weekly releases which used to only be for high priority patch fixes. Customers who had paid us to develop functionality received it much faster. While at the same time, because we had some free time, we were able to build a custom mainframe environment with code coverage tracing which allowed us to measure the code coverage of our 15 year+ regression suite. Because we had time to review those results as a team and update our regression suite, in one quarter we were able to go from 40% coverage to more than 70% percent code coverage of the first batch of key programs.

I reflected on this memory, after reading the Claude Skill for exploratory testing that Callum Akehurst-Ryan discussed in his blog, "Yes You Can Run Exploratory Testing with AI". Callum took the methods discussed in Elizabeth Hendrickson's book "Explore It!" and built Claude skills to build test charters and exercise them. In that Callum found he could automate more than he thought was possible. Taking a human process and working to automate parts of it[1] is a natural move. I spent part of the past two weeks helping a colleague do exactly this for the testing workflows they follow at their client. Part of that was connecting them with colleagues in other regions who were solving similar challenges in their context.

But I also wondered, if AI really is a transformative technology, is the right response just to use it to do what we have always done, but faster? So much of what I am seeing from thought leaders in the Dev and Testing communities feels like confirmation bias. One developer (who teaches TDD courses) wrote about how TDD is more important when LLMs are producing most of the code. Another who is a fan of specifications and argues against agile iterative development processes was pointing to Spec Driven Development as proof they were right all along. In the testing space (a thought leader who has been a fan of developers and testers pairing) wrote about using an LLM as a pair partner who allowed them to submit pull requests with their bug reports. Another who advocates for a team approach to quality, asked in an open space: "Anyone have experiences using AI tools to spark more collaboration?" While leaders who spend a lot of time arguing against phrases like "automated testing", now write at length about why AI can't test.

Of course, it’s natural to look for ways that AI can let us do things the way we have found to be most effective and I should note I've learned from each of those leaders and am interested in what they discover - no shade is intended for any of the people mentioned above. Instead, I want to ask: what was not cost effective to do before LLMs that we might want to add now? Now that we have these new tools what should we invest our time and attention in? Let me offer a few ideas:

• Far too many of the teams I work with split their testing into functional testing and non-functional testing. Many of these teams see non-functional testing as a specialist skill and something they are too busy to deal with. For most of the teams I work with, if you ask: "if you had unlimited time, money, and skills what would you test?" they already have a clear answer. Instead of just focusing on keeping up with the increasing pace of software development - I'd encourage those teams to explore how AI can help you do those things as well.
• One thing I have heard from some teams that cannot read code, LLMs excel at translations as a tool that enables them to fully understand the systems they are working in. LLMs can be very helpful at explaining what a piece of syntax is doing or explaining what changes have been made in a specific pull request.
• Another approach is to start with what LLMs excel at[2]. For example LLMs are good at extracting, summarizing, and categorizing text. They do it far better than most programmers could write programs to deterministically do the same job. Some teams spend a lot of time testing changes as they come in, with no time to reflect or notice patterns in what bugs are being found. We rarely have the time to analyze all the bugs found in a large program. There is simply too much to read. Now part of the act of reading is building a map, making sense of this semi-structured information. Leveraging an LLM to get broad brushes - which you then selectively verify is a good way to come to more informed guesses at the underlying pattern. Often that is enough to discover new insights. Jenny Qu shows an example of this kind of analysis in the start of her brief talk looking at all bug fixes in the Linux kernel across 20 years. Understanding what patterns of issues are missed for a long time is a signal which can inform us how we test things and potentially how we design systems to be more resilient.

If you are already doing everything you think you should be I'll share two ideas, I think are worth considering:

• One of the most effective ways to find security bugs in key libraries has been fuzzing. Property based tests bring the same ability to discover unanticipated issues to a broader set of applications - but the shift from example-based testing to thinking about logical statements that are always true is a significant one. Antithesis's research skill  is an interesting approach showing how to think about a specific codebase and suggesting possible properties which might apply. Some programmers at Anthropic adopted a similar approach and used it to test some of the top python packages. They found bugs (in packages like NumPy, Google and Amazon's SDK, and other packages. Combining this with Bombadil an open-source tool for PBT of web (and terminal) applications can help teams adopt a powerful new approach to automatic systematic exploration of their application which can run directly in a CI/CD pipeline.
• Formal verification is a popular topic among computer scientists interested in building trustworthy systems. More widely used in electronic hardware (where the cost of fixing a bug in production is usually much higher) these mathematical approaches allow you to prove that certain things are logically impossible to happen. In the software world this has been used to verify the design of cryptographic protocols, consensus protocols like PAXOS that work underneath most modern distributed systems, and other mission critical activities. "So, why haven’t you used it?" The founders of Midspiral ask, answering: "Because, historically, the cost of writing proofs dwarfed the cost of writing the code." One company I'm watching is Midspiral. They are focused on bringing formal verification to web development. Using an LLM to turn your spec into programmatic claims - which can deterministically prove are met by a specific implementation. This helps give you confidence that what your LLM produced is what you asked for. 
   

So how has your practice of testing changed as you leverage AI? How has it changed your strategic approach?

Citations

[1] Reading of Callum's blog and his comments on LinkedIn the references to “botsitting” points out where the human in the loop is key to what he describes. When I hear "I've automated this" I'd normally expect things which can run in a CI/CD pipeline. While Callum describes a living practice with coaching and feedback - he also describes it running in a pipeline. I'd love to understand how he decides to trust the reporting and turn it into something that can pass and fail in a pipeline.

[2] Wayne Roseberry suggested this approach to thinking of new approaches as he reviewed a draft of this blog.