- 6 November 2023
As a software testing consultant who specializes in helping companies leverage test automation, I frequently have people ask for my opinion on various test automation tools. More specifically, I often encounter people asking which tool they should switch to in order to solve their problems. This is a tempting approach – as a technologist I love diving deeper into tools but, I've often found that it can distract from addressing the root causes of the challenges they are facing.
Jerry Weinburg broke this open for me in his book, “The Secrets of Consulting". He opens the book with three laws of consulting, the second of which is: "no matter how it looks at first; it’s always a people problem." Jerry points out that even when a team is using the wrong tool for the job, people picked that tool, and if you don't understand who made that choice and why they made those choices, you won't be able to frame a better choice in a way they can understand.
I've seen three common people problems which lead teams to look for a new tool:
- The framework author is no longer on the team and the team doesn't understand how to continue to adapt the application.
- The team is not comfortable creating automation in the framework.
- Issues with flapping tests the team doesn't know how to fix.
These issues can be addressed through training, upskilling, and adding individuals with the right skills to the team. Additional issues may arise from the interactions among individuals and the diverse goals that different parties are prioritizing. To address this, we need to look at the processes of the team as a whole and the systems within a company that impact the team. As W. Edwards Deming said, "A bad system will beat a good person every time."
I've seen two common process problems which lead teams to look for a new tool:
- Teams automating low value / low priority scripts and struggling to keep up.
- Teams that did not plan for test maintenance and don't have time to investigate script failures to open bugs.
In each of these cases, changes to the team's process would result in better outcomes. But even if there are problems that might be addressed in other ways, why shouldn't we change tools as well? Due to the cost implications of rewriting test scripts.
Let's say that you calculate that a new tool allows you to be 10% more efficient as you create automation. If you have invested one week creating automation so far, to recreate this in a new framework would take 36 hours and then take another 9 weeks before the time you gained with the extra efficiency makes up for the rework effort. If you've invested a month into your old framework, rewriting in a new framework will require almost 10 months to break even.
This math is why Joel Spolsky declared that rewriting from scratch is: "the single worst strategic mistake that any software company can make." You can read more about this by visiting Joe's blog post.
The economics of rewriting is why I am a big fan of TTC's Quality Maturity Assessment process. We come in and take a comprehensive look at what is going on across 7 categories, giving you feedback in each area and constructing a roadmap of things you can do to improve how you test the crucial software systems in your company.
So, what automation tool should you use? Probably the one you are already using. But perhaps there are some changes we can make to how it’s being used to improve your testing processes overall.
This blog is the first of a series of posts exploring how software testing teams can pick the right tool for their projects. I'll be discussing the following topics:
- If you already have tests in a framework, should you stick with one you are already using?
- What is best: coded or low-code / no-code?
- Open-source test automation and the paradox of choice
- Drinking the pickle juice: should you go BDD?
- Do you need to consider a keyword approach?
- How to pick the right open-source frameworks / tools
Stay tuned to learn more about actionable tool selection recommendations!
Up Next in our Tool Selection Blog Series: Automation in Testing a No Code or Code-Based Approach