10 Years in Testing, 10 lessons – Lesson 4

This one really grinds my gears, so saddle up.

Lesson 4: Automated GUI tests don’t mimic what a user does

I cannot believe how often I’m having this discussion with fellow testers. Also, this infatuation with GUI automation has got to stop!

Some testers like GUI end-to-end automation a little too much and their reasoning often goes like this: “If I automate this test on the GUI it will mimic what a user does!”

No Padawan, it doesn’t.

An actual human going through a flow in your application will build their context in mere seconds. They observe the entire page (if you’ve ever been at a UX test that measures eye movement you know how quickly this goes), and if the UX is on point it will be clear to them what to do. They will spot silly lay-out bugs at once. The automated script you wrote won’t care.

The script will advance in the application using locators, it doesn’t matter to the computer whether or not those locators are also looking pretty on the screen. A real user doesn’t advance through the flow using locators: they will click on a certain location in the screen because they SEE a button there. If the clickable area of the button is wrong (say, too small), they will be annoyed, the computer would advance as long as the locator is reachable for the script.

The script will do what you tell it to: it’s not hindered by fields that are actually disabled and it doesn’t take a pause in order to build context, it just advances. It has superpowers that a real user doesn’t have: it knows exactly which element in the DOM-tree it should click and doesn’t get distracted by pop-ups that might be in the way.

Then there’s the matter of sensible assertions in these type of tests. Do you assert things on every screen, risking flakiness? Do you just go through the flow and see if you can successfully making it to the end of it? What about timing-issues? Do you use a test environment, and is that environment under your control?

All this hassle can be avoided most of the times by asking yourself the question: do I really need to test this in the UI or am I just testing through the UI (again quoting Mark & Richard here).

Often, the UI is not the best layer for test automation. I’m extremely biased towards API automation, I love that so much. Tests there are much easier to maintain, much quicker to run and those tests can still satisfy some end-to-end needs.

Look, I’ll be the last person to stop you from writing GUI end-to-end tests if that’s your hearts desire. Strip away the end-to-end part and isolate them in the GUI if that’s where your risk is, but I’d really advise against going the end-to-end automation route. That route is filled with maintenance nightmares, false negatives, timing issues and my favorite: sleeps in the code (yikes). Most importantly and the point of this lesson: those tests will not be doing what a user does.

Automation in testing has many purposes, but mimicking humans isn’t one of them.

Thank you for reading, you can find the rest of the blogs here.