Good and Bad UI Test Automation explained - Inspired by Richard Bradshaw's Tweets
Generally speaking there’s a scary trend with the influx of people interested in test automation where (seemingly) everyone wants to automate at the UI level. For example the phrase “Test automation” seems to be synonymous with UI automation which seems to mean using Selenium. To be fair there are numerous reasons for this. First, if you primarily focus on functional testing through the UI, this application of automation is the most relevant and straightforward to understand. Second, there are a lot of resources out there (books, blogs, classes) for Selenium. Third it is much easier to write some blackbox automation at the UI level then to develop competence at writing automation at the API or unit level. (You should still learn to do this lower.)
So Richard took to Twitter to explain his thoughts. Let’s take a look:
Those tests usually weave their way down the stack and end up hitting a database and weave there way back. Multiple times in a single test. Some would call them 'end to end'. A not so helpful term. But what they aren't is targeted.
— Richard Bradshaw (@FriendlyTester) May 8, 2018
Before we go very far, let’s look at what Richard means by targeted tests.
Targeted would be aimed at a risk, or risks, but ideally no more than two.
Seams, is where there is crossover, could be between an interface, protocols, libraries, a model I drew. Some where I can get between, but the systems needs them to be seamless.— Richard Bradshaw (@FriendlyTester) May 8, 2018
Tests are “targeted” if they are designed to reveal a specific risk or failure (we might call this risk-based). If you are doing risk-based testing, your tests should ALWAYS be designed this way, regardless of how you intend to execute the tests (automated or not).
Seams are how you use the system’s implementation to help you build more streamlined and realiable automated tests. Seams can be can be APIs, mocks, stubs, or faked dependecies of your real system.
Want some examples of Seams? Checkout this blog post by Angie Jones.
The industry needs more targeted tests. Understand the risk and behaviours and find the 'lowest' interface/seam you can test it on. Or should test it on.
— Richard Bradshaw (@FriendlyTester) May 8, 2018
Yes, most times we can use better designed tests! Want to test the various ways to get to a particular feature? Use PATH testing. Want to test the feature itself? Use functional or domain testing, etc. Some test techniques will be better at surfacing one type of problem and bad at surfacing another. It’s not always necessary to design your UI tests as long feature rich Scenarios.
When we talk of interfaces and automating at the lowest level it is usually a good time to mention this is what the Test Automation Pyramid tries to describe. There are lots of versions of this but I recommend Alan Page’s Test Automation Snowman.
If you can find the failure by writing a test lower in the stack (at the API level intead of UI level or at the unit level instead at the API level), the faster it runs and the faster you get feedback. If you do have to write the test at a higher level like the UI can you find some way to decrease the setup time / data of the test? (See above for Angie Jone’s blog post.) If you aren’t sure maybe thinking about whether you are testing the UI or testing through the UI will help?
Such questions/research should be asked in planning/story discussions, or similar named sessions. The main one being 'how are we going to test this'. This should trigger a conversation about options, and potential testability changes. Always ask this, even if your 100% YOU know
— Richard Bradshaw (@FriendlyTester) May 8, 2018
Yes! Adding testability like IDs, classes, data-test attributes, setting up test data, mocking, etc. should be done as quickly and dependbly as possible. If you can’t do this on your own (or you have a different process for making changes) bring it up during sprint planning in order to get the development teams help behind building a solution.
If you do some of the things above and only know Selenium, you brain will always see a nail. Fix that. Scan github, follow developers, follow blogs, awesome tools are shared all the time. Find them, do a 'hello world' scenario and bank it.
— Richard Bradshaw (@FriendlyTester) May 8, 2018
This IS a thing. Selenium is great, it’s open source and the WebDriver protocol is the soon-to-be de facto standard for interacting with browsers. However there are lots of tools, some will use Selenium under the hood and others that are completely different (think Cypress.io). Depending on the problem you need to solve, you may find a specific tool much better than Selenium. GitHub is a really good resource for finding open source tools and when you find something that might work, give it a try.
Targeted tests are, well, targeted. So if you don't need to hit a thing below the UI, don't. All the thinking and modelling mentioned earlier should identify this. Work out the risk, identify the lowest interface. Get the right tool.
— Richard Bradshaw (@FriendlyTester) May 8, 2018
If you want to write Good UI Test Automation your tests should be ideally suited for the UI (and not some other layer), targeted, reliable, use seams where possible andcover a specific risk. If you aren’t here yet work to refactor your test code until you can get there.
Did I miss anything about Richard’s tweet storm that was important?