How to run visual tests in 2023

One of the key inspirations for Storybook was a Chromatic post from 2017. Below is an updated version of the article, current to the frontend landscape today.

Testing is crucial for developing high-quality software. But when testing how UI looks and renders, it’s not really possible to write tests to verify 'correctness'.

The issue stems from the fact that the most important details of your components are hard to express programmatically. Your choice of HTML tags/classes, or the text in the output, for example, don’t provide the full picture that's required for a test to assert whether UI will look good (or not) to human users.

Since the first version of this article emerged in 2017, the discipline of visual testing has taken over the frontend world. It's been embraced by thousands upon thousands of teams to ensure that how their app looks forms a core consideration within its overarching functionality.

In this updated article, we reflect on the journey of UI testing over the past several years. We cover the pros and cons of different historic techniques that have been tried for UI tests, and explore the major impact of the emergence of Storybook, enabling devs to focus solely upon testing the components/states that require their attention the most.

Unit testing UI

A unit test isolates a module and verifies its behaviour by applying a set of inputs. It does this by comparing the module’s output to an expected result.

Unit testing is desirable because testing a module in isolation – rather than the whole system – streamlines the process of covering tricky edge cases. It lets you pinpoint the locations of issues when they occur.

However (and crucially): unit tests don’t have eyeballs.

While we can specify the inputs of our unit tests  (like React props and context ), it’s harder to specify the output in a way that’s robust to the minor changes within component implementation. For example: changing from flexbox to grid will cause many tests to fail, even when the result looks exactly same.

There are great tools like Testing Library that render a single component in a single state and then programmatically inspect the resulting DOM. However, the salient details of the rendered HTML – like color, layout, and scale – are hard to express. So, the tests end up brittle (over-specified) or insufficient (under-specified).

The core issue, though, is that most components’ inherent complexity is visual :  the specifics of how generated HTML and CSS appears on screen. And it has never been easy to write a test for appearance.

📢 Get first access Storybook's upcoming Visual Tests addon!

Snapshot testing

One solution to this problem came in the form of snapshot tests.

Snapshot tests highlight changes in UI by comparing HTML outputs with a recorded baseline. You can then verify whether these changes are intentional .

This approach acknowledges that UI outputs are difficult to specify in unit tests, so it informs developers when any code pertaining to the UI is changed.

Unfortunately, snapshot testing suffers from the same brittleness as unit tests. Minor details like changing a class name on a div can trigger the test to fail, meaning your tests require constant maintenance and checking in order to root out false positives.

...and then came Storybook

Storybook provided a visual approach to traditional testing techniques. It lets you view any component in any specific state, and then capture it as a test case by writing a story. With > 20m downloads each month, it continues to serve that function today!

The key idea behind Storybook is a that a developer can manually verify a story by rendering it within a browser. In practice, this approach is much easier than sifting through the false positives of unit/snapshot tests, updating test cases to match minor UI changes, or working overtime to make tests pass again.

Stories = test cases

Visual tests share the same setup, execution and teardown steps as any other type of testing. However, the final verification is left to the developer.

In Storybook, writing a test case is the same as writing a story:

// src/components/TaskList.stories.jsx

export default {
  component: TaskList,
};
export const Default = {
  args: { 
    tasks: inboxTasks,
  },
};

To manually verify the TaskList in the "Default" state, you would browse to the relevant story to render it in isolation. You can then see that it looks OK and then you’re done!

Automated checks

Storybook's visual testing requires you to capture the relevant component states as stories and verify each one.

But: what happens when you have thousands of stories? Checking each one would take forever.

That's why we created Chromatic, answering precisely that problem by handing developers an automated visual testing tool that runs in CI. Over the past 5 years, it’s been battle-tested by organizations like Adobe, the BBC, Monday.com, Peloton, and Mercado Libre.

The core principle of Chromatic is that you only need to focus on visible changes to the UI. You’re still involved in confirming whether changes are intentional, but a machine has already done the first pass at finding these changes for you.

It works by automatically taking screenshots of every story to compare them – pixel-by-pixel – to past versions of the UI. Next, it builds a changeset of stories that have visual differences and shares them with you for review.

🎉 Storybook’s next big thing

The Storybook team are currently working on a new addon that will let you bring automated visual testing directly into your local Storybook instance: the Visual Tests addon. It pinpoints visual changes in stories across browsers and viewports and is powered by the same technology as Chromatic.

The Visual Test addon is currently in development with six active maintainers. Your feedback can help bring it to life faster. Sign up below to get early access (with free usage) and project updates:

👉 Sign up now