Advice and tips about Test Automation

Almost every software development team would like to have test automation as part of the process. Each team may have different approaches depending on what they understand by test automation and what they think it is for. In this post we are going to describe and analyze some definitions related to test automation to help having a better understanding on what it takes to have test automation in a software development project.

There are two definitions that should be drivers of a test automation mindset and the whole team should be aligned and in the same page regarding these rules of thumb:

Test automation is a software development project
Your automated tests will need to be triaged and maintained

Those teams that do not consider this rules, have a high risk of developing helpless and tricky test automation.

What can a team do to narrow that risk and aim to an aligned and agreed test automation approach? Define goals that would help answer why the team needs test automation and specify a test automation strategy. Once those goals are defined, the team will be able to note the tasks required to work towards the goals. And finding the tasks, the team will wonder how to implement the test automation strategy.

##Goals Most of the times, those goals would be related to regression testing. Also the team would like to deliver a set of tested features at the end of each sprint. And that would make the team move to a continuous testing approach inadvertently (or intentionally).

###Reduce the time that it takes to execute regression testing

Probably all the test automation engineer started working as a tester and moved into test automation because of regression testing. It takes a lot of time to re run the regression test suite over and over again every time a new feature is testable. But don’t confuse, the goal here is to automate a repetitive tasks to allow testers to add value performing other tasks. It is wrong to believe that test automation is to have a faster regression test run. We’ll see later in this post why. Here the advantage is that while running regression testing in an automated manner, your tester can take advantage of that time to add value with other tasks: find gaps, perform exploratory testing, unveil risks and some other tasks that would feed the team backlog with new requirements that aim to improve quality.

###Reduce technical debt by closing your sprints with test automation

Once the team understand the need of an automated regression test suite, it is very common to fall into the “test first, automate later” trap. I have worked on teams that had the test engineers automating on sprint B what was developed on sprint A. That made the team to run in circles and have the test automation left behind, a never ending technical debt. When working together, test engineers and feature developers may work as a team to close sprints with test automation. It is needed to have the whole team understanding that a completed feature is the one developed, built, deployed and tested by the end of the sprint. So the team can make sure for the next iteration the automated regression is ready. In this sense the whole team should agree that test automation is part of the definition of done of each feature developed during the sprint.

###Enable continuous testing as part of your build process

The most experienced teams may have the two previous goals in the mindset already making them thinking on a continuous testing approach where each feature is tested as soon as its development starts. There may be different approaches including unit testing and/or performance testing and some other types of testing to have a shorter feedback loop that allow the team to make firm development steps. In this sense, test automation is a must.

Now that we have a some goals options let’s see what it is needed to do towards them. ##Tasks In order to accomplish the goals, the team should be aware of some key tasks they will perform in a daily basis to keep the goals achievable. We will see later how to perform those tags but here we will analyze what is needed to be done.

###Write the test automation scripts The team needs to find a way to record the automation scripts, not the code yet but a definition of the tests steps to be covered by test automation. These scripts must be shared by the whole team because it will help developers to have a better understanding of what is the expected behavior. It will also help other testers to know what is already covered by test automation and find some other border cases to perform on exploratory sessions. Also, a product team member may point out whether what the team is developing is aligned with the feature requirement.

There are some test management tools to help on this, but basically the it’s up to each team to find the way or tool to accomplish this task.

###Monitor the results every time the tests are ran

Once the team have a bunch of automated test cases ready to run whenever it is needed, there will be a feedback from the test automation to analyze. The information provided by the automated tests results will help the team to find out enhancements opportunities. Test automation results will not unveil feature bugs only. As part of the feedback, the team may find new tasks such as:

test infrastructure errors:
- are the tests running smoothly?
- is the test run completed?
test automation bugs:
- is there any flaky test?
- do the tests have the test data in place and time? (test preconditions)
test error logs:
- is it enough to read a log to find out why a test failed?
- are the errors handled properly?

In order to perform these tasks, it is key to have a reliable reporting tool that helps the team to monitor the feedback provided by test automation. Reporting is one of the most important features a test framework must have to help the team perform the monitoring tasks.

###Update the tests when the behavior of the application changes

Usually a mismatch between an expected value and an actual value is consider a test failure and most likely a bug. However, sometimes that mismatch is just a change detection. Does it mean there is a bug in the application under test? Sometimes yes, sometimes no. The assertion where an expected value is checked will unveil a change. It is part of the monitoring and analysis we reviewed above what will tell whether that assertion failure is actually a bug or not. But if the analysis discovers that the assertion failed due to an expected change there might be a problem in the process. Good to catch the change, but would have been better to point it out before and update the test beforehand to avoid the bottleneck. In order to avoid this, the team must work as a whole. Developers and test engineers must work as a team to make sure the application changes are reflected and covered making the required test automation changes on time.

We described some tasks that the team needs to perform in order to achieve the test automation goals. Let’s see now some tips to help a better approach for these tasks.

##How In the journey of a test automation strategy, it is key to figure out the goals and to acknowledge the tasks need to accomplish those goals. Now it’s time to talk about how to make test automation happen.

###What’s the scenario? Before talking about test automation, the team must make sure every member is aligned and have a good understanding of the solution the application under test will help the end users to solve. A good analysis of the components and capabilities of the application will help the team pointing out the scenarios that would be verified on each feature release or deployment. The actions and expected results should be clearly defined. There are some techniques and methodologies to identify automated scenarios. That is another team activity.

###Which tests should we automate? The criteria to define test automation candidates is not only based on business or customer impact. Technically speaking there are other considerations to take into account. If a test worth be executed every single time after a deploy, it would be automated. But also take into account the cost it has: test automation needs time to be developed and time be maintained. We’ve already mentioned that but it worth be emphasized.

###What about implementation? We agreed that maintenance is expensive so good decisions about implementation will have a huge impact in the test automation journey. What is the best programming language to approach test automation? You can find tons of surveys on internet about the most popular programming languages and which are the most required programmer backgrounds. But do not be fooled and base your decision on those surveys or trends. What will define the programming language for your test automation approach is your team context. Who will be working on test automation? The developers, the testers? All of them together? Those answers will guide your decision. Is the app under test developed in Java and developers will help on automation maintenance? Then it would make sense to chose Java. But if the developers and tester are isolated and test automation is testers only business, you may (but not necessarily) chose another language.

The tools that the team already use to develop the app may also help to define the automation strategy. Is there any continuous integration tool in place? Any code repository? Make sure your test automation project may be onboarded to work on those existing tools. You will need to build your test automation framework as any other application.

Last but not least, following good practices, development standards and conventions will make your life easier. Apply linting policies to the code, and implement some code documentation so any team member would be ready to jump into test automation with the easiest learning curve possible.

###How will the tests be executed? Your team will get to the point where the test automation will be ready to start running. There are some decisions to take now too. Based on the context, you may need your test automation to be triggered manually to support manual testers. Also, you may need to run the tests as soon as test code is checked in to make sure it works properly. In addition, if the team is in the continuous testing way, the team would like to have test automation running as soon as new application code is checked in and deployed. That is what leads to a CICD environment. A third option is to take advantage of the test automation to run tests as kind of a monitoring tool, the nightly test runs is a common practice to make sure both a test environment and the test automation are in good shape.

All these options requires fast and reliable tests that only fail when the application under test has provoked it to. Remember:

Monitor tests consistently
Unmonitored tests are untrustworthy tests