15 Years to 5 Weeks with Test Automation
A case study highlighting the results that can be achieved with test automation.
- Neil Fitzgerald
- 3 October 2019
This is the result we achieved for Partners Life when we took a process that was previously manually tested and would have taken 15 years to complete (15.8 years if we’re being exact) and reduced it down to 5 weeks.
No, that’s not a typo. That’s test automation.
Partners Life is a leading provider of health and life insurance in New Zealand. It has a focus on reducing the stress caused by ill health and death by resolving claims quickly and without hassle. To achieve this, Partners Life would regularly release updates to its quoting system, PASS, and enlist its senior management team to manually test each time an update was released. While well intentioned, this process was informal, unstructured and time-consuming - hence the 15.8 years.
It simply was not possible to test every aspect of PASS each time an update was released, which meant that a large volume of regression defects that were introduced as a result of any changes, went undetected. Furthermore, while it was important that the senior leadership team signed off on any changes, manually testing software was not a good use of their time or skillsets.
“The complexity of the system and the granularity of these bugs was impossible to ‘flesh-out’ with manual testing processes, let alone achieving the sheer volume of testing required in the project timeframe. A manual equivalent effort of 5.5 minutes would be required each time one of these tests was run, which equated to 16 years of manual effort at a total cost of approximately NZ$1,700,000.” — Peter Lassen, Partners Life CIO
What did TTC do?
We introduced test automation using Tricentis Tosca. Following a two-week proof of concept, we initiated a three-month pilot, which included build of an expansive test suite within Tosca. Included in this work was static testing of a spreadsheet that the client had developed to calculate limits and quotes during manual testing. The static testing we undertook led to a number of improvements in its design and eventually meant the spreadsheet could be replaced by Tosca.
The results truly speak for themselves
Test automation has been game changing for Partners Life. Test coverage has significantly improved with a large variation of outputs determining expected outcomes. The client can now run regression tests over a matter of days rather than years for any changes in the PASS software, regardless of size.
“The PASS Quote system required 66,000 individual test cases. This set of 66,000 test cases was run 4 times leading up to the PASS Quote release (a total of 264,000 individual tests). Post PASS Quote release we have had to run 19,500 individual tests for regression purposes. Hundreds of ‘severe’ status bugs were resolved as a direct result of the automated testing implemented by TTC.” — Peter Lassen, Partners Life CIO