Automation Testing Pros & Cons (+ Why Manual Still Matters)
Automation testing has become so integral to certain routines that some testers wonder if it’ll replace manual testing completely.
Not anytime soon.
When Elon Musk designed the Tesla Model 3, one way he wanted to increase the production rate was to have a fully automated assembly line. AI would assemble the car with almost no supervision from humans.
The plan failed horrifically.
Cars crashed into each other, doors were drilled into windows, and tires wouldn’t properly go on their rims.
What happened? Well, it turns out that robots can’t see very well. The AI in charge of assembling the Model 3 were unable to adapt to unexpected complications or slight misalignments. If everything wasn’t aligned just right, it would make catastrophic mistakes.
The same is true for automation testing in QA. Some forms of testing have too many variables and require the manual tester to be able to course correct and troubleshoot on the fly.
What is Automation Testing?
Automation testing means a QA tester uses a tool to execute a test case. In the development cycle, the same test case will be tested multiple times.
Some of the test cases that would take a team of QA testers hours to manually work through can be done by an automation tool in a few minutes. Several automation testing tools have become the industry standard.
The Dark Side of Automation Testing
Other industries, crafts, and professions have had to deal with the introduction of automation into their field. Whenever that has happened, whether it be in aviation with autopilot, carpet weaving, or testing, the workers in that field lose their understanding of the ‘why’ behind their work.
This is a phenomenon that QA professional, Jan Jaap Cannegieter, noticed began happening with testers. His ultimate fear is that the ease of automation testing and growing push to automate much of the testing process will lead to a generation of testers who know which actions to perform but lack an understanding of why.
Many testers know everything about certain tools or programming languages, but they can’t tell me what they test and why they test it. And that is a bad thing.
Examples of Bad Automation Testing
Many QA testers see the same potential in automation testing. Typically, that means there are observable trends and common mistakes that a QA tester can make. Here are some common examples of bad automation testing.
Nesting Automation Tests
Nesting automation tests are when several automation tests run on top of each other. When this happens it becomes difficult to figure out what has gone wrong when an error appears.
In the short-term, most cases of automation testing are positive. The QA tester uses the right tool and runs it properly.
Many examples of bad automation testing only become bad six months later, when automated tests have been nested inside other automated tests inside other automated tests.
UI Automation Testing Without Supervision
User Interface testing makes sure that there’s nothing a user can do on the UI that will break the program or cause malfunctions. When tested manually, this can call for a lot of testers and a lot of manpower. It is quite frankly inefficient to test that way, especially as the program grows larger and more functions require testing.
With automation testing, the process becomes quicker but can still require the full power a QA tester’s computer for an entire day. To deal with that, QA testers will run the test at the end of the day, let it work overnight, and come back to work the next morning with the results sitting there waiting for them. Makes sense, right?
On the surface, it seems like the smart thing to do. QA tester uses their computer to test other programs during the day, then at night run the automation testing on the UI.
It is because it makes intuitive sense that this is a common mistake. Lots can go wrong when automation testing is performed without supervision.
If something goes wrong early, the rest of the results will be wrong. An entire days worth of testing produced no useful information because of an error that could’ve been easily caught and corrected if someone was there to check that the automation test was performing correctly.
While automated testing can sometimes lead to false positives, choosing the right QA automation tools can significantly reduce such instances.
Automating the wrong things
Automation testing takes time to set-up. QA needs to make sure the automation tools are the right fit for the project and that the QA testers know how to properly use the tools.
All this takes loads of time and organization and if the end result is that a test typically run once a month gets automated, it isn’t worth the effort. Before you begin automation testing, make sure that what you’re going to automate will save QA testers a measurable amount of time.
Not only can you automate tasks that are too rare to be worth the effort, some tasks simply can’t be automated easily.
Replacing Manual Testing
Automated testing can only catch what it’s told to look for. If the fonts on a webpage come back looking weird, but the test was only checking that all the links on the site worked, it’ll solve one problem (the links) but not know anything was the matter with the other (the fonts).
Manual testing can catch things outside of the initial scope of the test. The method of exploratory testing was designed around giving the QA tester the space to find unexpected bugs as they appeared, even if that’s not what they were initially looking for. Set on a singular task, automation testing can be thorough. What it cannot be, is comprehensive.
Examples of Good Automation Testing
Test automation is performed to minimize risk. When a QA tester can minimize the risk and maximize efficiency, automation testing absolutely should be performed. There’s no reason for a QA tester to spend hours manually checking the links on a website when a web crawler can perform the same activity in less time and be unlikely to make a mistake.
A good rule of thumb for deciding when automation testing should be used instead of manual testing is to ask if the test will be quick or continuous. If the test is going to run continuously, automation testing is the way to go.
People are worse than machines when it comes to consistently performing repetitive tasks at a high level. We crave novelty and disengage mentally when we do the same thing for too long. This allows mistakes to slip through.
Just like that, a QA tester will realize they’ve: a) spent more time than if they’d run automation testing and, b) done a worse job and created more headaches later on.
Jason Huggins, found of the popular automation tester, Selenium, developed the program because he found that so many of his days as a tester were being spent running the same tasks - tasks that he thought were simple and straightforward enough that even a robot could do it. So he developed a script that automatically tested browser functionality for him. It was an instant success and quickly became the industry standard.
Even the QA professionals who are skeptical about automation testing know that it in many instances it serves a purpose.
Why Manual Testing Will Never Die
We’ve been over the good, the bad, and the very bad aspects of AT. We know in which situations it works well and where it causes assembly lines to throw car tires around the factory. For a bit, let’s look at manual testing and why it’s still so important for QA testers.
Automation Testing Needs Supervision
As was mentioned above, with Musk’s Model 3 problem, and the risks of running automation testing overnight, things can go very wrong when you leave automation tools alone.
The big benefits of automation testing come when it is performed along with manual testing, or under the supervision of a QA tester. For that reason alone, QA testers should rest easy knowing that AT won’t be taking over their roles in the foreseeable future.
Manual Testing Uses Exploratory Testing
When a tester is performing an exploratory test they are exploring the software without any predefined plan. It’s one of the most popular forms of testing in QA.
Exploratory testing can only be performed through manual testing.
The benefit of exploratory testing is that it allows the tester to adapt to their findings on the fly, without the need to write another test case.
Exploratory testing also allows for collaboration, theory crafting and collaboration all on the fly.
As the agile theory of development has become more prominent, so has exploratory testing.
Automation testing lacks the flexibility and creativity to be agile enough for exploratory testing. It works best in a rigid environment where it knows exactly what to look for. Exploratory testing is the exact opposite - instructing the QA tester to go wherever they’d like.
What Do You Think?
Some people swear by automation testing, while others still think manual is the way to go. How do you think QA teams can get the most out of automation testing? Do you think we're ready for AI in Testing?
Let me know in the comments.