Automation in Testing a Low/No Code or Code Based Approach

David Malan presents a CS50 lecture to an auditorium of listeners

This weekend, I discovered the video lectures of CS50, one of the most popular courses at Harvard. It is a wildly successful introduction to computer programming with high production values and a lot of thought paid to the pedagogy - how they teach. I watched the first two hour lecture, which began by introducing binary and how you could represent different kinds of information as patterns of 1s and 0s as long as you had some kind of agreement about how to interpret things; expanding to algorithms with the example of a binary search by thinking through how to find a number in a phone book; and then finally building programs in Scratch (https://scratch.mit.edu/).

Scratch is a platform built to teach children programming principles. Users drag different puzzle pieces to build programs which are then executed inside the sandbox. By watching the program evolve from a cat which said ‘hello world’ to games where users fed Oscar the Grouch trash that fell from the ceiling in this visual tool, it became clear that complex systems could be created by breaking them down into smaller problems and thinking logically how to solve them. In 45 minutes, I was reminded of the joy that first got me programming.

As I read more about CS50, I was struck by the lack of debate about if Scratch was really programming or not. The software testing world has seen a wave of low-code and no code tools and push back from experienced SDETs declaring the only way to be successful using automation is with code-based tools. As someone who built automation to aid testing for 7 years with coded tools and who has also spent the last five years helping teams build automation with a no-code tool (Tricentis Tosca), I find the framing of the debate tiresome. Most no code tools have enough primitives to make them Turing Complete. Being Turing Complete means anything you write in one Turing Complete could be written in any other Turing Complete tool / language.

Abstraction Layers

Bas Dijkstra shared a post on LinkedIn that I think is a much more useful frame for evaluating low-code and no-code tools as additional layers of abstraction.

"Abstraction is a good thing. Modern programming languages provide an abstraction layer. Several layers. actually. Without these, we would still be programming in assembly (or maybe even with a magnet and a very steady hand).

But an abstraction layer is also, by nature, a trade-off between ease of use and flexibility. An additional abstraction layer, whether you call it ‘low code’, ‘codeless’ or something else, is meant to increase ease of use, but that comes at the expense of flexibility. Or, in other words: if you’re choosing to adopt an additional abstraction layer, it means you’re now operating a bigger black box.

As with many trade-offs, there is an optimum somewhere, and it’s not often found at an extreme end of the spectrum. At some point, too much abstraction is added and not enough flexibility remaining."

When deciding what kind of tool we are going to use, we need to consider the gains we can get from the ease of use and the opportunity costs we settle for by using a high abstraction tool.

The Power of Being Highly Abstract with Low/No-Code Tools

In my mind, the gains from using a low/no-code tool fall into three categories:

• Efficiency gains due to easier script creation.
• Efficiency gains due to abstraction layers designed for specific SUTs.
• Gains in total work done by expanding the circle of who can participate in automation creation.

Efficiency gains due to easier script creation are easy to evaluate and quantify. Though if you compare someone unfamiliar with one tool and someone familiar with another tool, it’s easy to get the evaluation wrong. During a session where I compared my own efficiency with a coded tool and a no code tool, I found test creation with the no code tool to be about 24% faster. When I measured maintenance over time (across three years of changes), I found that the no code tool again had about the same impact in reducing maintenance. Jerry Weinburg suggests never claiming more than a 10% improvement, so if you don't have good data, I'd use a 10% gain in my model.

Some automation tools include specific abstraction layers to work with key platforms or applications. In the open-source world, a great example of this was the protractor project, which was designed to make testing of websites built with a specific GUI library (AngularJS) easier. In the commercial tools space, this is the value proposition of tools like Provar which specialize in testing a single platform (Salesforce). Platforms like SAP, Oracle Netsuite, Salesforce, Microsoft Dynamics, and ServiceNow, which are used by many large companies, are a popular target for these kind of abstraction layers.

When deciding between Open Source or Commercial, the risk of picking a tool that specializes in one platform is that often the most critical business risks are not in how processes within a tool work, but the edges where these platforms are integrating with other systems in a complex value stream. Trading extra efficiency within a single platform for efficiency across the value stream can lead to teams focusing automation where it is easiest instead of where its most important. I recommend prioritizing the gains from this kind of abstraction when you are dealing with a single system where everything happens and focusing on more generalist tools when testing integrated systems.

Expanding the circle of who can contribute to automation is where I think the biggest potential for gains in productivity with low code tools exists. If your team is struggling with a backlog that only seems to be growing while hiring at your company is frozen - investing in a tool that can double or triple the size of your automation team is a compelling opportunity. To do a great job using automation in testing, I think there are at least three key categories of skills that must be present:

• Domain knowledge about how the software you are testing is used.
• Testing skills to think about where risks might be, and how they might expose problems in the future.
• Tool knowledge/programming knowledge - how to build maintainable and robust automation.

You rarely find folks that are strong in all three areas. A no-code tool doesn't replace the need to think rigorously about what things to make flexible and generic and what things to leave simple and easy to understand; it won't help you break complex flows into simple reproducible steps; and it won't help you spot the minor differences in execution machines which cause scripts to work fine on your machine but fail when executed in unattended mode. Most folks can learn to create programs, and CS50 shows that starting people with high abstraction, visual tools instead of code is a great way to get started on that journey - but the best users of no-code tools I've seen all can write code as well. If you decide to go the low-code / no-code route, I'd recommend ensuring you have someone with experience building software on your team.

What You Miss Without Code

So, what are the opportunities you miss out on if you select a low-code or no-code automation tool? Here are six major ones:

• Code based tools allow you to locate the automation in the same repository as the code being tested. This can aid in breaking down silos between your dev and testing teams.
• Code based tools blend in naturally with the shift to everything as code in more modern cloud native platforms.
• Code based tools allow you to write automation in a language familiar to your developers, potentially expanding your automation team.
• Code based tools have more flexibility and allow for more refined expression. If a low-code or no-code tool has a simplified 'baby talk' language which makes it easy to pick up, code-based languages allow you to invent your own grammar and rhyme schemes to best communicate the way you think is needed for the nuance of this particular process.
• Open-source code based tooling allows you to adjust any part of the system you don't like.
• Open-source code based tooling allows you to start working with it now, instead of requiring you to work within the limits imposed by budgets, procurement processes, and how you scale to fit into the licenses you have acquired.

More then sixty years ago, Melvin Conway observed, "Organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations." As time has passed, practitioners have seen this pattern so often, it has been codified as Conway's Laws. The most basic way software developers communicate is by sharing software code and using a code driven tool that allows all the team that speaks code to be part of the conversation.

On one project, I built test environments using a configuration management tool that was written in a language that none of the dev team had worked with before (puppet/ruby). I used it to make developer environments as well, and initially my team thought what I was doing was magic. Then I committed that code into the same repository as the application and sent my changes through the same code review process as application changes.

Within a week, I got my first pull request with improvements to the dev and test environments - once developers see code and see changes to it, natural curiosity will help break down silos and help you deepen the sense of team. As silos break down, I've worked in teams where developers take up the challenge of adding new automated tests to regression suites (and not just unit tests). This contribution has proved crucial to achieving that oft dreamed about in-sprint automation.

In-sprint automation can be a struggle because testing is fundamentally a response to some action. Even in the most extreme shift left contexts with you having a ‘three amigo’ conversations about how you might meet a need or solve a problem, the testing activities take place once someone has suggested a little bit of an implementation idea. If the team all feels comfortable contributing automated checks as part of each change - you don't have developers handing off work to be tested and pushing to get on with a next task.

Every programming language has idioms, common approaches, and constructs which shape how we think to solve problems. Building automation in the same language your dev team is already using allows them to understand and participate in solving testing challenges. As the famed thinker Pitbull said: "Ask for money, you'll get advice / Ask for advice, you'll get money twice." Asking for advice (code review in their language) has proven an effective method to getting the level of collaboration and shared maintenance (the money you really want).

"The Cathedral & The Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary" by Eric S. Raymond

In his foundational essay "The Cathedral and The Bazar", Eric Raymond shares 19 lessons learned leading an open-source project. His first lesson is: "Every good work of software starts by scratching a developer's personal itch." In my 20+ years in IT, the best software I've used started by scratching a very focused itch. When you work with open-source tools, you have more freedom to scratch your own itch and make a great solution perfectly adapted to your context.

Often, instead of there being one choice to solve a particular problem, there are multiple with different approaches. We will discuss this more in our next blog - but the freedom to adapt things just the way your team likes it is an opportunity you get when you choose to move in an open-source, code-driven direction. The choices require some careful study and judgement, as Eric shares in his second lesson learned: "Good programmers know what to write. Great ones know what to rewrite (and reuse)."

Conclusion

So, after about two thousand words, what choice would I recommend? Low-Code / No-Code or a Code Based approach to using automated software testing? Like most consultants, I tend to hedge when asked for clear recommendations without much context. I firmly believe that the right tool is all about context; this could include the skills and desires of the team, the technologies being tested, the timelines of major projects, etc. I sometimes find myself quoting the great Dr. Suess:

And IF you go in, should you turn left or right... or right-and-three-quarters? Or, maybe, not quite? Or go around back and sneak in from behind? Simple it's not, I'm afraid you will find, for a mind-maker-upper to make up his mind.

If you are a mind-maker-upper, I'm happy to discuss options with you and help you make up your mind.

But after starting my career focused on open-source technologies, and now spending the last five years with commercial low/no-code tools, I've been asked to work in TTC Global's Open-Source Practice in North America. Where it’s best for my clients to work with low/no-code tools, I'll still be around helping them, but I'm excited to work with teams and companies adopting or looking for help improving their use of open-source automation tools. Working together, I'm confident we can do great things. As Dr. Suess writes:

And will you succeed? Yes! You will, indeed! (98 and 3/4 percent guaranteed.)

--

Read the previous installment in my Tool Selection Blog Series: Should You Change the Tool You Already Have