Table of Contents
- What to know about Visual Regression Testing
- Today you’re going to write your first regression test
The content for this book is from learn.visualregressiontesting.com. It is a 6-day email course available for anyone interested in Visual Regression Testing.
Throughout this book, you’ll notice mentions of the current day in the course (e.g. “It’s Day 1!”), or references to previous/later days. This is due to the nature of the original content.
Each chapter in this book matches a day in the course. So six chapters is equal to six days.
With that said, let’s get started!
What to know about Visual Regression Testing
Hey Folks, Micah and Kevin here from visualregressiontesting.com.
Over the next week, you’re going to learn the essentials Visual Regression Testing.
You may be surprised by how much fun writing these tests will be. After all, you’re teaching a computer how to use your website! That’s pretty neat in our book.
Let’s kick it off by expanding our vocabulary. We’ll be using these terms throughout the course:
Change hurts. Especially when it’s unwanted.
“Regressions” are changes, bad ones, to the functionality of your site. To say you’ve found a “regression” means you found code that used to work but no longer does.
“Regression Testing” is a type of testing that checks for these bugs in existing functionality after an update.
While you definitely want to put new code through the wringer, it’s also important to look at existing features to ensure they weren’t adversely affected by the updates. That’s what regression testing does.
Automation comes in many forms, whether through robots or driverless cars. For us, it’s writing code that “automates” actions on a website, similar to how AI is written to “automate” driving a car.
While the goal of website automation is to test without a human clicking links and entering text, in no way can it take the place of hard-working people.
First of all, someone has to write the automation and know how to keep it up to date. That can easily be a full-time job on a large enough application.
But there is also a lot of nuance to websites. Checking to see if an animation worked as designed is very difficult with automated tests. Automation simply aims to handle the boring, repetitive tasks, leaving you time to test the hard stuff.
There are many types of software testing out there; Wikipedia has an extensive list of the various definitions.
Functional testing also goes by “system testing”, “end-to-end tests” and many other monikers, but for our course we’ll use the term “functional testing”.
There’s one last term to cover, and that’s the concept of “visual testing.” In this course, we’ll talk about both standard testing (e.g. make sure certain text is on the page) and visual testing (e.g. make sure the page looks the same as before).
Visual Regression Testing has a unique set of benefits. Just because an element is on a page does not mean it’s in the right location (or the right size or the right color).
With visual testing, we take screenshots of various parts of the page to serve as the “baseline” of how the site should work. Then in subsequent test runs, we take new screenshots and compare to those baselines. Any differences are flagged for review.
The Benefits of Functional Testing
Why take the time to learn all of this? Is it really worth it to write code that only tests the code you really should be writing?
Here’s the thing: you’re already doing this. Every time you refresh your page to see a change you’ve made, you’re running a manual regression test. Throughout the lifetime of a website, hundreds of hours are spent regression testing it.
But the truth is, we’re bad at this. We either get lazy and skip a few tests, or we unknowingly miss a defect or two (we’re especially bad at detecting small visual changes).
With automation, regression testing the site becomes less of a repetitious chore, so we’re more likely to do it on a consistent basis. You can even set up hooks to automatically test your code after every change.
And computers are much better at spotting pixel changes. It’s all numbers to the computer, and numbers are easy to compare.
We really believe in the benefits of automated testing, so we should probably throw out some disclaimers before we get you too excited:
- Functional testing can be finicky, because websites are complicated.
- You can’t test everything because, again, websites are complicated.
- Automation doesn’t work in all browsers, because some browsers just don’t support it. You’re best bets are going to be Firefox, Chrome and Internet Explorer.
Basically, it’s not magic (although it feels like it at times). There are certain things you just won’t be able to automate testing for, so it’s best not to fight that fact.
We’ll go in to installation of the tools tomorrow, but here’s what we’ll be using:
- Selenium - A browser automation tool
- WebdriverIO - A Node.js library to talk to Selenium
- WebdriverCSS - An add-on to WebdriverIO to help with Visual Regression Testing
Course Requirements (aka today’s homework)
There are a few requirements you’ll need to have going before getting started:
- A working NodeJS environment (Learn about Node and NPM if you are new to it)
- A light understanding of the command line
Phew, you’ve made it through the entire first day without any code being shown. Don’t worry, we’ve saved all that for tomorrow, where we’ll take a look at writing your very first automated test (we’re so excited for you!)
Today you’re going to write your first regression test
Day 2 is here; we hope you’re ready to dig in. Let’s start with some installs!
First, create a folder to store all of your code. You can name it whatever you want.
Then, open a command line and navigate to that folder. From there, you’ll create a new
package.json file by running the command:
If you’re new to NodeJS or unfamiliar with NPM, have a look at their about page for a quick intro video.
This file just stores information about your project and the software you need. It’s useful to have on hand if you need to set the project up again later.
Now to run the actual installation:
WebdriverCSS is a Visual Regression Testing plugin for WebdriverIO. It’s a pretty basic tool that boils down to two basic functions:
- Capture images of the specified portion of the website.
- Compare those images to previous ones, creating a “diff” version if differences are found.
You can do a fair amount of configuration with the tool, but for now, those two concepts are all you need to know.
We’re testing in real browsers folks, which is really quite awesome. But it does require some set up. While getting Selenium running is easier than ever, it’s too complex for a single email.
As a worthy alternative, we’re going to use ChromeDriver. ChromeDriver is a Selenium-like tool that allows us to run tests on a real Chrome browser.
To get it going, just open a new command line window (making sure you’re in your project folder), and run:
This will be a constantly-running service. You can stop the it by pressing
- If you’re interested in why we reference
./node_modules/bin/and a way around that, read up on npm scripts.
Let’s make a test!
Now that we have ChromeDriver running, we can use it to run some tests. In your project folder, create a new file called
tests.js. Open it up in your favorite code editor.
The first thing we’ll do is load WebdriverIO in to the file. To do that, we’ll use a require statement:
Disclaimer For the rest of this course we’re going to leave out repetitive bits of the code in our examples. To see the full code, check out the code samples for each day.
With that loaded, our next step is to set up a browser instance.
What just happened here? Well, we told WebdriverIO to start a new Chrome browser. If you want to read more about it, check out the documentation.
Get to our page
Now that we have a browser to play around with, let’s do some damage. We’re going to request a website by using the
url browser command.
Check the title
Now that we have a page loaded, we can do stuff with it. Let’s validate it’s actually the right place.
How about checking the title of the page and logging it through
console.log? Here’s what the syntax looks like:
Here we asked our browser to get the title of the page (the aptly named
getTitle command), and then logged that value out using the generic
End the test
Finally, we need to tell WebdriverIO to shut things down. That’s as simple as adding
.end() at the bottom of our commands:
Just to recap, here’s what your
tests.js file should look like:
Now it’s time to run the test! You can do so by asking Node (via the command line) to ever so kindly execute your code:
If all went well, you should have seen the following message in your command line:
So, how does it feel to have programmatically taken control of a browser? A little awesome, right?
Okay, if you’re underwhelmed, that’s because we didn’t really test much. We only peeked at the page title then closed everything down before we got ourselves in to real trouble.
We promise that over the next couple of days we’ll get our hands extra dirty with mouse clicks, keyboard taps, HTML checks and very real visual tests.
If you can’t wait until then, take a look at the WebdriverIO API page to see a full list of commands available for use.