Archived Post: How I Added a pa11y-ci GitHub Action to My Next.js Site

Archived Sunday, September 17, 2023 — 14 minute read

🚨 ATTENTION 🚨

You are currently viewing an archived post. The information here may no longer be accurate or maybe Ashlee just decided not to publish the post as publicly anymore.

If you've been working in web accessibility for a while, you know there are many foundational quick-wins that can go a long way. If you're new to web accessibility, hopefully this doesn't come as a surprise! There are definitely challenging aspects to web accessibility, but it's not all that way. And lucky for us, a lot of the quick-wins are also quick and easy to identify with some automated tools! pa11y-ci is just one of those tools.

In this post, I'll show you how I set up a GitHub Action that runs pa11y-ci as a pull request check. The check will run against a Vercel deploy preview, and then the results will be shared as a comment on the pull request for easy visibility.

If at any point you find yourself stuck or needing to troubleshoot, check out a GitHub repository I created that represents the end-result of this guide: https://github.com/ashleemboyer/pa11y-ci-test-app.

Step 1 (optional): Create a Next.js App

If you don't already have a Next.js App to follow along with, I recommend creating one. There are multiple ways of doing this detailed on the Next.js Getting Started page. To keep things simple for this example, I'm using the Automatic Setup approach.

To create the Next.js app:

Open a terminal
Run npx create-next-app@latest
When prompted, name the app pa11y-ci-test-app
Change directory into the new folder: cd pa11y-ci-test-app
Briefly explore the code and learn your way around

Note: I will be using npm instead of yarn for the rest of this post. If your generated project uses yarn, and you would also like to to use npm instead:

Run rm yarn.lock to delete the file
Run npm install to create a package-lock.json file
Run npm run build so the package-lock.json file can be patched if needed
Run npm install one more time to download any missing dependencies
Run git add . to stage the changed files for a commit
Run git commit -m "Using npm instead of yarn" to commit the changes

Step 2 (optional): Create and Connect a GitHub Repository

If you followed step 1 and created a new Next.js App, you should follow this step as well. You will need a GitHub Repository for your project in order to run the GitHub Action we're creating.

To create the GitHub Repository:

Go to github.com/new
If needed, log in to your account
Under "Owner", select your account
Under "Repository name", enter pa11y-ci-test-app
Select either "Public" or "Private"
Click the "Create repository" button
Leave this page open in your browser

To connect the GitHub Repository to your project:

In a terminal, go to your project folder
Run git remote add origin git@github.com:<YOUR-USERNAME>/pa11y-ci-test-app.git after replacing <YOUR-USERNAME> with the GitHub username you used to create the repository. (Read about Working with Remotes and Remote Branches if you're curious.)
Run git push -u origin main to push the code to the repository.
Verify these steps were successful by refreshing your new repository's page in your browser

Step 3: Install and Configure `pa11y-ci`

It's always good to test new tools locally (on your machine) before testing them remotely in a place like a pull request. Testing remotely can be tedious and time consuming, and the results of the testing can also be more difficult to locate and read. And if you happen to be like me and have ADHD, all of that is a recipe for multiple distractions!

Before doing anything else, let's create a new git branch. That will be useful later when we need to create a pull request for our GitHub Action.

To create a new git branch:

In a terminal, go to your project folder
Run git checkout -b add-pa11y-ci

To install pa11y-ci:

In a terminal, go to your project folder
Run npm install --save-dev pa11y-ci
Open the package.json file in your project
Verify the pa11y-ci package is listed under devDependencies

To configure pa11y-ci for local testing:

In your project folder, create a file called .pa11yci
In the new file, paste the following JSON:

{
  "urls": ["http://localhost:3000/"]
}

This configuration tells pa11y-ci exactly what urls in our app it needs to test.

Step 4: Test `pa11y-ci` locally (on your machine)

Now that we have pa11y-ci installed and configured for local testing, we can get to testing!

In your project folder, open the package.json file
Under the "scripts" property, add a new entry: "pa11y-ci": "pa11y-ci"
Open two separate terminals to your project folder
In the first terminal, run npm run dev to start the app
In the second terminal, run npm run pa11y-ci to run the new script we added in step 2

After pa11y-ci has finished running, this is how the output should read:

> pa11y-ci-test-app@0.1.0 pa11y-ci
> pa11y-ci

Running Pa11y on 1 URLs:
 > http://localhost:3000/ - 1 errors

Errors in http://localhost:3000/:

 • The html element should have a lang or xml:lang attribute which describes the language of the document.

   (html)

   <html><head><meta name="viewport" con...</html>

✘ 0/1 URLs passed

Hooray! We have an error, but the good news is that means pa11y-ci is working! Now we can progress from local testing towards creating our GitHub Action.

Step 5: Deploy to Vercel

Our current configuration for pa11y-ci only works for when we run our app locally with npm run dev. For remote testing, we want pa11y-ci to test against real builds and public-facing urls. In order to do that, we need to deploy our Next.js app. We'll use Vercel in this example.

How to deploy to Vercel:

Go to vercel.com/new
If needed, log in to your account
Under "Import Git Repository", select "Continue with GitHub"
Search for your new repo (mine is called pa11y-ci-test-app)
Click the Import button in-line with the repo name
Click the Deploy button (you should not need to change any configuration)

Once the deploy is complete, you can close this window.

Step 6: Install and Configure `next-sitemap`

There's another way pa11y-ci can be used to test against urls, and that is with the --sitemap option. This option is nice when you have a sitemap.xml file on your site that updates every time a build occurs. It means you don't have to manually update the .pa11yci file's list of urls every time a new page is added. The tool we'll use for generating a sitemap.xml file is called next-sitemap, which is the tool I use on my own website for pa11y-ci testing.

To install next-sitemap:

In a terminal, go to your project folder
Run npm install next-sitemap

To configure next-sitemap:

In your project folder, create a new file called next-sitemap.config.js
In the new file, paste the following JavaScript:

module.exports = {
  siteUrl: process.env.VERCEL_URL
    ? `https://${process.env.VERCEL_URL}`
    : 'http://localhost:3000/',
};

This tells next-sitemap what the main url of your site is. If you use a different port number than 3000, make sure to update the code above to match the port number you use.

Step 7: Test `next-sitemap` locally (on your machine)

To make sure we have everything configured correctly, we're going to test next-sitemap locally just like we did with pa11y-ci.

In your project folder, open the package.json file
Under the "scripts" property, add a new entry: "postbuild": "next-sitemap"
In a terminal, go to your project folder
Run npm run build

The new postbuild script we added will run automatically after the build script has completed. You should see some new sitemap files in the public directory. We will add those to the .gitignore file in the next section.

You should also see something like this output in the terminal after the build script output:

> pa11y-ci-test-app@0.1.0 postbuild
> next-sitemap

✨ [next-sitemap] Loading next-sitemap config: file:///pa11y-ci-test-app/next-sitemap.config.js
✅ [next-sitemap] Generation completed
┌───────────────┬────────┐
│    (index)    │ Values │
├───────────────┼────────┤
│ indexSitemaps │   1    │
│   sitemaps    │   1    │
└───────────────┴────────┘
-----------------------------------------------------
 SITEMAP INDICES
-----------------------------------------------------

   ○ http://localhost:3000/sitemap.xml


-----------------------------------------------------
 SITEMAPS
-----------------------------------------------------

   ○ http://localhost:3000/sitemap-0.xml

Step 8: Prepare for and Create a New Pull Request

Hopefully you created a new git branch called add-pa11y-ci in Step 3, but it's also okay if you didn't! Go ahead and create it now. We'll need a different branch off of the main one in order to create a pull request in our GitHub repository.

To prepare for the pull request:

Make sure there is a /public/sitemap* entry in the .gitignore file. We do not want to commit these files since they need to be generated by the Vercel build process.
Make sure you have all your new pa11y-ci and next-sitemap work committed. I'm committing everything at once with the message: "Adding pa11y-ci and next-sitemap".
Run git push --set-upstream origin add-pa11y-ci to push your new branch up to the repository.

To create the new pull request:

Go to github.com/<YOUR-USERNAME>/pa11y-ci-test-app/compare after replacing <YOUR-USERNAME> with the GitHub username you used to create the repository
Change the compare branch to add-pa11y-ci
The page should update have at least these 5 changed files:
- .gitignore
- .pa11yci
- next-sitemap.config.js
- package-lock.json
- package.json
Click the "Create pull request" button
You should see 2 text boxes: one for the name of the pull request and one for the description
Fill in the title and description however you need
Click "Create pull request"

To verify everything is set up correctly:

Wait for the vercel bot to comment on the new pull request
In the new comment, open the "Visit Preview" link in a new tab
You should see your app built and running
Go to the /sitemap.xml page in the same app
You should see an XML file with contents that look something like this:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>YOUR-APP-URL/sitemap-0.xml</loc>
  </sitemap>
</sitemapindex>

Step 9: Create the pa11y-ci GitHub Action

Now that we have a live website to have pa11y-ci test against, we can create the GitHub Action that runs pa11y-ci.

To create the GitHub Action:

In your project folder, create a new folder called .github
In the new .github folder, create another folder called workflows
In the new workflows folder, create a new file called pa11y-ci.yml
Paste the following content in your new file:

# Wait to run until after the Vercel deployment has finished
# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#deployment_status
on: deployment_status

jobs:
  pa11y-ci:
    # Only run if the Vercel deployment was successful
    # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#deployment_status
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest

    steps:
      # Checkout latest commit of repo
      # https://github.com/actions/checkout#checkout-v3
      - uses: actions/checkout@v3

      # Install project dependencies
      - run: npm install

      # Run pa11y-ci against the generated sitemap
      # https://github.com/pa11y/pa11y-ci#sitemaps
      - run: npm run pa11y-ci -- --sitemap ${{ github.event.deployment_status.target_url }}/sitemap.xml

Before committing and pushing this new file, make sure to delete the .pa11yci file. It's no longer needed.

To verify the GitHub Action works:

Commit and push the new .github/workflows/pa11y-ci.yml and the removed .pa11yci file
Go to your pull request in a browser
Towards the bottom of the page, wait until you see the pa11y-ci check finish
The check should fail
Click the "Details" link to inspect the failed check
You should see the same error that you saw when testing locally in Step 4

Step 10 (optional): Update the pa11y-ci GitHub Action to add a comment on pull requests

When I added this pa11y-ci GitHub Action to my own site's pull requests, I decided to make it also comment results on the pull request. There are a couple of reasons for this:

Normally you have to click the "Details" link next to a failed pull request check. This takes you to another page and away from the pull request. For me, that means an opportunity to get easily distracted.
The formatting of results in pull request checks can sometimes be difficult to read. Putting the results in a pull request comment gives an opportunity to re-format them in a more readable way.

So to update the GitHub Action to comment on pull requests, we'll need to tell the pa11y-ci command to output its results in a different way, and then we'll add a step to the GitHub Action that reads the output, formats it, and creates a comment.

Update the pa11y-ci run step in .github/workflows/pa11y-ci.yml to be:

# Run pa11y-ci against the generated sitemap
# https://github.com/pa11y/pa11y-ci#sitemaps
- run: npm run pa11y-ci -- --json --sitemap ${{ github.event.deployment_status.target_url }}/sitemap.xml |& tee pa11y-output.txt

Add a new step at the end of the file like this:

# Comment the formatted pa11y-ci results on the PR
- uses: actions/github-script@v5
  env:
    BODY_PREFIX: '<!-- pa11y-ci results -->'
  with:
    script: |
      const script = require('./.github/workflows/scripts/pa11y-ci.js')
      await script({ context, core, github })

Add a new folder inside .github/workflows called scripts
Add a new file to scripts called pa11y-ci.js

In the new file, paste the following JavaScript:

const { promises: fs } = require('fs');

module.exports = async ({ context, core, github }) => {
  // Find this PR & do nothing if this isn't a PR
  const { data } =
    await github.rest.repos.listPullRequestsAssociatedWithCommit({
      owner: context.repo.owner,
      repo: context.repo.repo,
      commit_sha: context.sha,
    });
  const prNumber = data[0] && data[0].number;
  if (!prNumber) {
    return;
  }

  // Read the pa11y output and build the comment body
  const pa11yOutput = await fs.readFile('./pa11y-output.txt', 'utf8');
  const lines = pa11yOutput.split('\n');
  const asJSON = JSON.parse(lines.find((line) => line.startsWith('{')));
  const { total, errors, results } = asJSON;

  let commentBody = `${process.env.BODY_PREFIX}\n<h2>:microscope: pa11y-ci results</h2>\n\n`;
  commentBody += `- Number of URLs tested: ${total}\n`;

  if (!errors) {
    commentBody += `- No errors were found! :tada:\n`;
  } else {
    commentBody += `- Number of errors found: ${errors}. :sob:\n`;

    let formattedOutput = '';
    Object.keys(results).forEach((urlKey) => {
      const errors = results[urlKey];
      if (errors.length < 1) {
        return;
      }

      formattedOutput += `- [${urlKey}](${urlKey}):\n\n`;
      errors.forEach((error) => {
        formattedOutput += '  ```\n';
        formattedOutput += `  ${error.message}\n\n`;
        formattedOutput += `  ${error.selector}\n\n`;
        formattedOutput += `  ${error.context}\n\n`;
        formattedOutput += `  ${error.code}\n`;
        formattedOutput += '  ```\n\n';
      });
    });

    commentBody += `<details><summary>See results :eyes:</summary>\n\n${formattedOutput}</details>`;
  }

  // Get the comments on this PR
  const { data: comments } = await github.rest.issues.listComments({
    owner: context.repo.owner,
    repo: context.repo.repo,
    issue_number: prNumber,
  });

  // Try to find an existing pa11y results comment
  const previousComment = comments.find((comment) =>
    comment.body.startsWith(process.env.BODY_PREFIX),
  );
  if (previousComment) {
    // Update the previous comment
    await github.rest.issues.updateComment({
      owner: context.repo.owner,
      repo: context.repo.repo,
      comment_id: previousComment.id,
      body: commentBody,
    });
  } else {
    // Create a new comment
    await github.rest.issues.createComment({
      owner: context.repo.owner,
      repo: context.repo.repo,
      issue_number: prNumber,
      body: commentBody,
    });
  }

  if (errors) {
    core.setFailed('Errors were found by pa11y-ci');
  }
};

Commit and push the changes to .github/workflows/pa11y-ci.yml and the new .github/workflows/scripts/pa11y-ci.js file.

To verify that everything is working as expected:

Go to your pull request in your browser
Wait for the pa11y-ci check to finish
The check should fail just like it did before
Look for a comment by the github-actions bot that says "🔬 pa11y-ci results"
The comment should say that 1 error was found
Expand the "See results 👀" text
There should be one error listed that matches the same error you've been seeing

Step 11. Celebrate! 🎉

Nicely done! You've added a pa11y-ci GitHub Action that runs on pull requests. You should be proud. Give yourself a pat on the back. 😊