The Do’s and Don’ts When Selecting a Testing Toolset

In order to achieve this, they are leveraging principals taught by DevOps and around 80% of organizations say they plan to move to DevOps.

When companies are asked what their biggest challenge is in their move to DevOps, testing is consistently given as the answer. Increasingly, companies are adopting test automation, continuous integration and other tools to help alleviate some of the bottlenecks that can occur in the testing phase of a project. As software quality professionals, we probably hear these buzzwords several times a day: Agile, DevOps, Continuous Testing, Continuous Integration, Artificial intelligence – it’s most of what people in the industry are talking about.

Today, companies are looking to DevOps-based practices to drive their own internal digital transformation. They are looking to work better, work faster and smarter. This means new ways of thinking and operating, which means a lot of companies are bringing in new sets of tools to facilitate this. New toolsets mean training, configuration and migrations, and these aren’t always things that are done quickly or overnight and typically require a lot of planning and effort to implement. A lot of time and careful thought and consideration has to go into these decisions and a significant amount of planning goes in to how these efforts are carried out

There is an overwhelming number of tools available in market, ranging from enterprise to open sourced. Depending on what tools are eventually selected, there may be different skills required to best leverage the toolsets and frameworks. This includes administration, configuration or even programming.

With this in mind, I think we are ready to jump into the Do’s and Don’ts when selecting a toolset. Let’s start with the Don’ts and get the negatives out of the way first.

Don’t expect a tool to fix all your problems

You may have heard great things about a particular tool – you have friends using it, it’s at the right price point, you were WOWED by the demo… AWESOME! After the purchase, installation and other admin happens, you now have this tool on your hands to configure and help turn your team in to SUPER AWESOME DEVOPSING SUPER HEREOES. …. Problem solved right?

Not necessarily. If all you’re going to do is buy a new tool or set of tools and set them loose within your team, really all that’s doing is putting a band aid on a much larger situation. Especially if you have trouble working under a defined process, or if you are still using older software methodologies.

Don’t expect a tool to dictate how your team will work

It takes far more than just a really good toolset to make your team more Agile. In most cases, these tools are going to need configuration to help you resolve some of these issues. And more times than not, you’re going to have to take a good close look at how your team is actually working and see what it takes to correct that course.

This solution often includes training, knowledge sharing or even consulting to see where those pain points are, and an open line of communication between team members and other teams. On the flip side, over-engineering a solution will lead to low buy-in of a toolset. If it’s perceived as too cumbersome or difficult to use, your team members will ultimately revert to their old ways of working and the value of the toolset will be lost.

Don’t expect test automation to happen overnight

Many teams I work with are trying to move from having little to no test automation to trying to be more Agile and have their regression testing process match this. A lot of times, teams know they need automation but have a lot of difficulty determining where to start or what tests really need to be automated at the same time as celebrating the purchase or adoption of a tool or framework with huge expectations for how it’s going to very quickly make their regression testing much easier.

Automation doesn’t happen overnight – it takes time.

Even with the most straightforward of tools, there is some thought that has to go in to the development, creation,= and organization of your test suite. Because of this lead time, an argument I probably hear the most often for putting off a planned automation implementation is that the team doesn’t have the time to give to automation, even if a budget is in place and there is a strong desire to be more Agile

If you are a team that’s looking to implement automation, please don’t let these comments put you off. Automation is an incredibly powerful tool and has the potential to take your regression testing from days, weeks or months down to a count of mere hours.

And, if you recall, most companies state the number one barrier to DevOps is testing. When done correctly, automation can be an incredible time saver and is an amazing tool to ensure software quality and catch defects as soon as possible after they are introduced to the code. Also, as previously mentioned, there’s a wide variety of tools on the market ranging from code-based frameworks to business readable testing software that can help your team to achieve these outcomes.

Don’t think in silos

DevOps isn’t just about automation and continuous integration – it’s also about communication.

If we look at a definition of DevOps - DevOps is a set of software development practices that combine software development and information technology operations to shorten the systems development lifecycle while delivering features, fixes and updates frequently in close alignment with business objectives.

Having open lines of communication with other teams in the software development lifecycle as well as stakeholders and the business is going to help you alleviate pains and bottlenecks in your development process. I’ve been an advocate for the importance of communication as part of the development and quality process for a very long time, and as DevOps implies, there’s a need for communication in order to effectively deliver frequent software updates.

It could be argued that very often these toolsets can help improve communication between team members. Rather than looking at a tool’s capabilities based on how it can help just your team, think about things like:

• Is this going to be business readable? Will people outside of the testing team understand what’s being tested?
• Can I get information out of this tool, or can this help me communicate information to other teams or stakeholders?

As you learn from these Don’ts, we will now focus on a few Dos!

Do consider what information and metrics are important to your team and stakeholders

A long time ago, I worked with a team that wanted help to fine tune their workflow processes to match a new approach they wanted to standardize to keep their many teams on the same page. We were brought in to help keep them aligned with the tool’s best practices and to make sure all the information made it to the screen of the individuals who needed to see it. Previously, this company had each team acting somewhat independently (and somewhat siloed) with a vast array of tools and ways of working with much information being handed off over email.

For a lot of reasons, you can see why this isn’t ideal. There is no single source of truth and no way to keep track of changes over time. Once we got there and started meeting with the teams, nobody could agree on what was the most important. Discussions often became tense and once progress was made someone new would enter the room and we’d have to start from scratch again. At the end of the project and against our documented objections to configuring the tool this way, the client wound up ignoring all our best practice recommendations and basically decided to track every single thing with a wildly complex workflow. If you looked at the workflow diagram, it looked more like a spider web than a readable and logical diagram. As a result, nobody wanted to use the tool because of how over-engineered it was and was perceived as being difficult to use

This isn’t only an example of why best practices are important, but an example of why you should very carefully consider a few things:

• How are your teams working together (or where should they be working together)
• What processes and tools are they used to using
• What information is important for your team, stakeholders, and management to be keeping track of?

Do create a cut-off date from your legacy system

I wasn’t sure whether to include this or not because there’s a lot of variables and there isn’t generally a clear answer or recommendation, but it really is a big consideration and something that I feel like comes up during almost any kind of implementation. Some of these considerations may include:

• What do you do with all that previous work and old records you have sitting around? Are you going to need to refer to them?
• What about historical test cases and test results?
• What about all those hours of test scripts you spent all of this time maintaining, developing, running, and testing… do those just go away, and you start from scratch?

The thing with this one is that there isn’t really a clean-cut answer. There are really only a few options and there may be a combination of these required:

• Migrate all your data out of your old tool and in to your new one
• Export that historical data to be stored somewhere (or just keep a small instance of your old info around)
• Create a cut-off date to end with the original toolset and start with the new one

The reason why is this cut-off is important is because you don’t want to either be duplicating your work, producing work in two different places or having to re-work things later during the transition period. At some point, you have to say, ‘no more work is being done here’ and create that new source of truth for information related to the project.

Is this necessarily the biggest and most important consideration you’re going to have to make in your implementation? No. Absolutely not. However, it is one that I’ve seen raised during most implementations that I’ve worked on. When do we stop working in the old system and start working in the new one, and/or how are we going to get all our data in to the new system?

Or if you don’t have a system you were using previously, a consideration needs to be made for when all new project work should be stored in a tool rather than say, in Excel sheets or out on a drive somewhere and enforcing that.

Even after a decision has been made, work is usually ongoing in between the initial installation, set up, configuration and training for team members. Business doesn’t stop for your digital transformation! There isn’t usually a lot of down time for teams today to take the time to make the switch, which tends to be an argument for why teams hesitate to move onto new tools or implement something like automation in the first place. There’s just too much to do and not enough time! It’s hard to take the time to implement something that will make your job easier to do when you’re already struggling to find time to manage your current workload.

Here is another place where some careful thought, consideration, and planning can work to your advantage.

Do hire expert help

As someone who is presently working as a consultant and has formed a blog on my experience as a consultant, I probably sound really, really biased … but it’s true, and I mean this. I’ve seen all too often when new software isn’t configured correctly or the correct training isn’t provided, adoption is usually slow or completely stops. Conversely, over-configuration of a tool can also drastically reduce usage due to perceived difficulty.

I’ve seen implementations go badly, I’ve seen teams who did it wrong the first time and brought us in later, which was arguably more difficult, but I’ve also seen teams do things really well. Often, the differentiator came from the teams that asked for help early on rather than letting it fall flat and having to come back from this.

Consultants like me and the team at TTC can help you avoid these mistakes, adhere to best practice and help get you and your team up and running as quickly as possible so you can start to reap the amazing benefits and results that you hear about DevOps so often.

Consulting and professional services can give you powerful tools to help maximize your potential usage from the tool and get your team trained in best practices to help your digital transformation go even more smoothly.

I hope these tips can help you in your journey. There is no one size fits all for testing tools but if you remember these 5 summary points, you can plan to be successful!

• Don’t use a toolset as a band aid
• Consider what metrics matter
• Create a cut-off date where it makes sense
• Communication is vital to success
• Ask for help!