What Is Visual Regression Testing? (with Screenshots)
A practical guide to visual regression testing: deterministic captures, a pixelmatch diff you own, baseline approval in pull requests, and when Playwright or a hosted VRT service is the better fit.
Visual regression testing catches UI bugs that unit and integration tests miss. A button that moved, a font that fell back to a system face, a card that lost its shadow: the code still passes its assertions, but the page looks wrong.
This guide explains what visual regression testing (VRT) is, then builds a small, working loop around a screenshot API: deterministic captures, a pixel diff you control, and a baseline approval step that lives in your pull requests. It is honest about where a capture API stops and where a browser test runner or a hosted VRT service is the better tool.
Commercial tools such as Chromatic, Percy by BrowserStack and Applitools pioneered many of these workflow patterns. The diff in this guide uses the open-source pixelmatch library.
What is visual regression testing?
Visual regression testing compares screenshots of your UI across versions to detect unintended changes. The loop has five steps:
- Baseline: capture each page in a known-good state and store the images.
- Change: develop a feature or fix a bug.
- Capture current: screenshot the same pages, with the same settings, after the change.
- Compare: diff each current image against its baseline.
- Review: a person decides whether each difference is a bug or an intended change. Intended changes become the new baseline.
The capture step is the only one a screenshot API does. Steps 1, 4 and 5 (storing baselines, computing the diff, deciding pass or fail, approving) are yours.
Why compare screenshots instead of the DOM?
Some tools compare DOM structure. Comparing rendered pixels has different strengths:
- CSS changes show up. A changed margin or colour leaves the DOM identical.
- Rendering problems show up. Missing images, font fallbacks and overflowing text are visible only in the pixels.
- It matches what users see, regardless of framework.
The trade-off is noise. Anything that renders differently between runs (dates, ads, animations, data) shows up as a difference too, so most of the work in VRT is making captures repeatable.
Where a screenshot API fits, and where it doesn't
| Part of the loop | Screenshotly | You or another tool |
|---|---|---|
| Load a public URL in a real browser and return PNG/JPEG/WebP | Yes | — |
| Fixed viewport, scale factor, timezone, dark mode | Yes (all plans) | — |
| Hide selectors, AI removal of banners and widgets | Yes (Growth plan and above) | — |
| Store baselines | No | Git, object storage |
| Diff images and decide pass/fail | No | pixelmatch, odiff, ImageMagick, or a VRT service |
| Approve changes | No | Pull request review |
| Clicks, typing, multi-step flows | No (only a javascript string, Growth+) |
Playwright, Cypress |
localhost or private-network URLs |
No (rejected) | Local browser runner, or a public preview URL |
The useful property of a hosted capture is that every run uses the same capture environment rather than whichever laptop or CI image happened to run the test. Playwright's own documentation warns that browser rendering can vary with the host OS, version, settings and hardware. The capture browser can still be updated on the service side, so if every page shifts slightly at once, suspect the environment before your CSS.
Make captures deterministic
A visual test is only as good as its repeatability. These request fields exist for that purpose (field names are exact; the schema rejects unknown keys):
width/height: set them explicitly on every request (320–3840 wide, 240–3840 tall). Without them the default viewport is 1920×1080. Thedevicepresets (desktop,laptop,tablet,mobile) only set a viewport size; they do not switch to a mobile user agent.deviceScaleFactor: pin it (1–3). A 2× capture is a different image from a 1× capture.timezone: pin it (for example"UTC") so dates rendered from the client clock don't drift between runners.darkMode: set it explicitly. It emulatesprefers-color-scheme: dark; test light and dark as separate baselines.waitForNetworkIdle: true: wait for network activity to settle instead of capturing at DOMContentLoaded.waitForSelector: wait for an element that only appears once the page is ready. It waits up to 10 seconds; if the element never appears, the capture still happens. Your diff will catch a half-loaded page, but the API response will not tell you why.fullPage: true: capture the whole document. This matters for typography too: the capture service currently skips web-font requests unlessfullPageistrue, so viewport-only and element (selector) captures render with fallback fonts. If fonts are part of what you test, use full-page captures.hideSelectors(Growth+): hide regions you know are dynamic, such as timestamps, avatars and rotating testimonials.aiRemoval(Growth+): hide cookie banners, newsletter pop-ups, chat widgets, social overlays and ads (cookie-banner,newsletter,chat-widget,social-overlay,ad).blockAds: true(all plans): block common ad requests and hide common ad containers.
On the Free and Basic plans, where hideSelectors and javascript aren't available, the most reliable approach is application-side: serve fixed seed data on your preview environment and add a query flag (say ?vrt=1) that your app uses to stop carousels, disable animations and render a fixed date.
Project structure
visual-tests/
├── vrt.config.json # pages, viewports, diff tolerances
├── capture.mjs # calls the screenshot API
├── compare.mjs # pixelmatch diff, exits non-zero on failure
├── baselines/ # committed, approved images
├── current/ # this run's captures (git-ignored)
└── diffs/ # diff images (git-ignored)
Install the diff dependencies (pixelmatch has been ESM-only since version 6, so the scripts use .mjs and rely on the built-in fetch and top-level await of a current Node.js LTS release):
npm install --save-dev pixelmatch pngjs
Configuration
{
"pages": [
{ "name": "home", "path": "/?vrt=1" },
{ "name": "pricing", "path": "/pricing?vrt=1", "waitForSelector": "[data-testid='pricing-table']" }
],
"viewports": [
{ "name": "desktop", "width": 1440, "height": 900 },
{ "name": "mobile", "width": 390, "height": 844 }
],
"hideSelectors": ["[data-vrt-hide]"],
"pixelThreshold": 0.1,
"maxDiffRatio": 0.001
}
hideSelectors requires the Growth plan. On Free or Basic, remove it from the config and freeze those regions in your app instead.
Capture script
The base URL must be publicly reachable: a preview deployment, staging, or a tunnel. Private and loopback addresses are rejected.
// visual-tests/capture.mjs
import { mkdir, readFile, writeFile } from 'node:fs/promises';
const API_URL = 'https://screenshotly.app/api/screenshot';
const API_KEY = process.env.SCREENSHOTLY_API_KEY;
const BASE_URL = process.env.VRT_BASE_URL; // public preview or staging URL
const outDir = process.argv[2] ?? 'current';
if (!API_KEY || !BASE_URL) {
console.error('Set SCREENSHOTLY_API_KEY and VRT_BASE_URL');
process.exit(1);
}
const config = JSON.parse(await readFile(new URL('./vrt.config.json', import.meta.url), 'utf8'));
function requestBody(page, viewport) {
return {
url: new URL(page.path, BASE_URL).toString(),
width: viewport.width,
height: viewport.height,
fullPage: true,
format: 'png',
deviceScaleFactor: 1,
timezone: 'UTC',
darkMode: false,
waitForNetworkIdle: true,
...(page.waitForSelector && { waitForSelector: page.waitForSelector }),
...(config.hideSelectors?.length && { hideSelectors: config.hideSelectors }), // Growth+
};
}
async function capture(body, attempt = 1) {
const res = await fetch(API_URL, {
method: 'POST',
headers: { 'x-api-key': API_KEY, 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
if (res.status === 429 && attempt < 4) {
const waitSeconds = Number(res.headers.get('retry-after') ?? 60);
if (waitSeconds > 120) throw new Error('Rate limit or quota window too long to wait in CI');
await new Promise((resolve) => setTimeout(resolve, waitSeconds * 1000));
return capture(body, attempt + 1);
}
if (!res.ok) {
const error = await res.json().catch(() => ({}));
throw new Error(`${res.status} ${error.error ?? ''} ${error.message ?? ''}`.trim());
}
return Buffer.from(await res.arrayBuffer()); // binary PNG body
}
await mkdir(new URL(`./${outDir}/`, import.meta.url), { recursive: true });
let failures = 0;
for (const page of config.pages) {
for (const viewport of config.viewports) {
const file = `${page.name}-${viewport.name}.png`;
try {
const png = await capture(requestBody(page, viewport));
await writeFile(new URL(`./${outDir}/${file}`, import.meta.url), png);
console.log(`captured ${file}`);
} catch (error) {
failures++;
console.error(`failed ${file}: ${error.message}`);
}
}
}
process.exit(failures ? 1 : 0);
Captures run one at a time, which keeps you inside the per-minute limit on every plan (5 requests/minute on Free, 40 on Basic, 80 on Growth, 200 on Scale). Failed captures don't count against your quota. For larger suites, see the concurrency and async-job patterns in the CI/CD screenshot guide.
The request bodies this script sends look like this. On the Free plan, drop hideSelectors:
{
"url": "https://preview.example.com/pricing?vrt=1",
"width": 1440,
"height": 900,
"fullPage": true,
"format": "png",
"deviceScaleFactor": 1,
"timezone": "UTC",
"darkMode": false,
"waitForNetworkIdle": true,
"waitForSelector": "[data-testid='pricing-table']",
"hideSelectors": ["[data-vrt-hide]"]
}
Comparison script
// visual-tests/compare.mjs
import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
const dir = (name) => new URL(`./${name}/`, import.meta.url);
const config = JSON.parse(readFileSync(new URL('./vrt.config.json', import.meta.url), 'utf8'));
mkdirSync(dir('diffs'), { recursive: true });
const results = [];
const baselines = readdirSync(dir('baselines')).filter((f) => f.endsWith('.png'));
for (const file of baselines) {
const currentPath = new URL(file, dir('current'));
if (!existsSync(currentPath)) {
results.push({ file, status: 'missing' });
continue;
}
const baseline = PNG.sync.read(readFileSync(new URL(file, dir('baselines'))));
const current = PNG.sync.read(readFileSync(currentPath));
// Full-page captures change height when content is added or removed.
// pixelmatch needs equal dimensions, so report this as a failure for review.
if (baseline.width !== current.width || baseline.height !== current.height) {
results.push({
file,
status: 'size-changed',
detail: `${baseline.width}x${baseline.height} -> ${current.width}x${current.height}`,
});
continue;
}
const { width, height } = baseline;
const diff = new PNG({ width, height });
const diffPixels = pixelmatch(baseline.data, current.data, diff.data, width, height, {
threshold: config.pixelThreshold, // per-pixel colour distance, 0 to 1
});
const ratio = diffPixels / (width * height);
if (diffPixels > 0) writeFileSync(new URL(file, dir('diffs')), PNG.sync.write(diff));
results.push({ file, status: ratio <= config.maxDiffRatio ? 'passed' : 'failed', diffPixels, ratio });
}
for (const file of readdirSync(dir('current')).filter((f) => f.endsWith('.png'))) {
if (!baselines.includes(file)) results.push({ file, status: 'new' });
}
writeFileSync(new URL('./comparison-results.json', import.meta.url), JSON.stringify(results, null, 2));
for (const r of results) console.log(`${r.status.padEnd(12)} ${r.file}${r.ratio !== undefined ? ` (${(r.ratio * 100).toFixed(3)}%)` : ''}${r.detail ? ` ${r.detail}` : ''}`);
const blocking = results.filter((r) => ['failed', 'missing', 'size-changed'].includes(r.status));
process.exit(blocking.length ? 1 : 0);
Two thresholds, not one
pixelmatch's threshold option is easy to misread. It is not a percentage of the image. It is how different two colours must be (0 to 1, where 1 is black versus white) for a single pixel to count as changed; the default is 0.1. Whether the page fails is a separate decision, made here by maxDiffRatio: the share of pixels allowed to differ.
Treat both values in the config above as starting points. Run the suite a few times against an unchanged deployment, look at the ratios you get from pure noise, and set each page's tolerance just above that. pixelmatch already ignores anti-aliased pixels by default (includeAA: false), which removes much of the edge noise. Recent versions also offer a windowSize option that counts the worst N×N area instead of the whole image, which helps separate scattered noise from a compact real change. Check the pixelmatch README for the version you install.
Reading a diff image
The diff PNG is the original image faded towards white, with changed pixels painted red and detected anti-aliasing pixels painted yellow (pixelmatch's defaults). A red block where a button used to be means it moved or disappeared. A thin red outline around text usually means a font or weight change. Red speckle spread across the whole image usually means a rendering-environment change rather than a code change.
Example: one CSS rule, one layout regression

Local demo, not a hosted Screenshotly capture. Two static HTML fixtures were rendered at 480×360 by headless Chrome through Puppeteer on a development machine, then compared with pixelmatch 8 at threshold: 0.1.
The two fixtures share the same card markup. The current version adds one CSS rule, the kind of change a refactor produces when a grid gets scoped to wide screens only:
.features { grid-template-columns: 1fr; } /* was: 1fr 1fr */
The diff shows how far one layout change travels. Every feature item except the first is red, both where it was and where it is now. The button is red in both positions, and the card's bottom border is red where the card grew taller. On the run that produced this image, pixelmatch counted 9,865 changed pixels out of 172,800 (5.7%). Font rendering differs between machines, so the exact count will differ if you reproduce it. All of those pixels come from a single rule, which is why a reviewer reads a diff alongside the code change instead of only checking the count.
Approving changes: the baseline loop
Baselines are approved images, so store them where approval already happens: in git, next to the code.
# .gitignore
visual-tests/current/
visual-tests/diffs/
When a pull request intentionally changes the UI:
# Re-capture baselines from the PR's preview deployment
VRT_BASE_URL=https://pr-123.preview.example.com node visual-tests/capture.mjs baselines
git add visual-tests/baselines/
git commit -m "Update visual baselines for new pricing table"
Reviewers then see the image changes in the pull request's file diff. That is the approval step. Two rules keep this honest:
- Capture baselines and current images from equivalent environments (same seed data, same flags). A baseline from production compared with a preview on test data will always fail.
- Never auto-update baselines in CI on failure. That turns the test into a recorder.
If baselines are large or numerous, keep them in object storage keyed by branch and commit instead of in git; the review step still has to exist somewhere.
Running it in CI
A minimal GitHub Actions job runs both scripts against a preview deployment and uploads the diffs when something fails:
# .github/workflows/visual-tests.yml
name: Visual regression
on:
pull_request:
branches: [main]
jobs:
visual:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 22
- run: npm ci
- name: Capture
env:
SCREENSHOTLY_API_KEY: ${{ secrets.SCREENSHOTLY_API_KEY }}
VRT_BASE_URL: https://pr-${{ github.event.pull_request.number }}.preview.example.com
run: node visual-tests/capture.mjs current
- name: Compare
run: node visual-tests/compare.mjs
- name: Upload diffs
if: failure()
uses: actions/upload-artifact@v7
with:
name: visual-diffs
path: |
visual-tests/current/
visual-tests/diffs/
visual-tests/comparison-results.json
Replace the preview URL pattern with whatever your host uses. Waiting for the preview to be ready, PR comments, GitLab CI and rate-limit-aware concurrency are covered in the CI/CD screenshot automation guide.
Authenticated pages
To test logged-in screens, sign in a dedicated test user, then pass its session cookie. Cookies and custom headers require the Basic plan or higher, and the cookie domain must match the target URL:
{
"url": "https://staging.example.com/dashboard?vrt=1",
"width": 1440,
"height": 900,
"fullPage": true,
"format": "png",
"timezone": "UTC",
"waitForNetworkIdle": true,
"cookies": [
{ "name": "session", "value": "SESSION_FROM_TEST_LOGIN", "domain": "staging.example.com", "secure": true, "httpOnly": true }
]
}
Use a short-lived session for a seeded test account, never a real user's session, and keep the value in CI secrets. The authenticated capture guide covers the options in more detail.
When to use something else
A capture API plus your own diff is a good fit for page-level checks against deployed environments. It is the wrong tool when:
- You need interaction before the capture. Opening a menu, filling a form or stepping through checkout needs a browser test runner. Playwright's
await expect(page).toHaveScreenshot()compares against stored snapshots, supports tolerances such asmaxDiffPixelsandthreshold, and updates baselines with--update-snapshots. It runs wherever your tests run, including againstlocalhost. - You test components, not pages. Storybook-based services such as Chromatic capture each component state.
- You want a hosted review UI, cross-browser rendering or team approval workflows. Percy, Chromatic and Applitools provide these; a capture API does not.
- The page is only reachable on a private network. Screenshotly rejects private and loopback addresses.
Many teams combine them: Playwright snapshots for interactive flows, plus API captures of key public pages on every deploy.
Summary
- VRT is a loop of baseline, capture, compare and review. A screenshot API handles the capture; you own the rest.
- Most effort goes into repeatable captures: pin size, scale, timezone and colour scheme, wait for readiness, and hide or freeze dynamic regions.
- Use
fullPage: truewhen typography matters; other captures currently skip web fonts. - pixelmatch's
thresholdis per pixel; your pass/fail rule is a separate pixel-count or ratio. - Approve baseline changes in pull requests, never automatically.
Ready to add visual regression testing?
Create a free account to get 100 captures for testing the request shape, or compare plans on the pricing page. See how capture fits a wider test setup on the automated testing use case, and check every request field in the API reference.
About the Author

Asad Ali
CTO at ZTABS and the developer behind Screenshotly. Writes about screenshot automation, API design, and building reliable SaaS backends with Next.js, Node.js, and TypeScript.
Credentials: CTO at ZTABS, full-stack developer, builder and maintainer of Screenshotly and its official JavaScript and Python clients
Frequently Asked Questions
Does Screenshotly compare screenshots for me?
No. Screenshotly captures the page and returns the image. Storing baselines, diffing, deciding pass/fail and approving changes are done in your own code (for example with pixelmatch) or in a dedicated visual-testing service.
What diff threshold should I use?
There is no universal number. pixelmatch has two separate knobs: its threshold option (0 to 1, default 0.1) decides how different two colours must be for one pixel to count, and your own rule decides how many differing pixels fail the test. Start with the default per-pixel threshold and a small allowed pixel ratio, then tune each page against real runs.
Why do my visual tests flake in CI?
Usually the page is not in the same state on every capture: data, dates, animations, late-loading content or third-party widgets. Pin the viewport, scale factor and timezone, wait for network idle or a known selector, and hide or freeze dynamic regions. Note that waitForSelector gives up after 10 seconds and the capture still proceeds, so a missing element does not fail the request.
Can I test localhost or click through a flow before capturing?
Not with Screenshotly. Target URLs must be publicly reachable, so point it at a preview or staging deployment. It loads a URL and captures; the only page scripting is the Growth-plan javascript option, so multi-step flows are better tested with Playwright.
Ready to capture your first screenshot?
Get started with 100 free screenshots. No credit card required.