Skip Navigation

Archived How I Added a pa11y-ci GitHub Action to My Next.js Site

Archived

🚨 ATTENTION 🚨

You are currently viewing an archived post. The information here may no longer be accurate or maybe Ashlee just decided not to publish the post as publicly anymore.

If you’ve been working in web accessibility for a while, you know there are many foundational quick-wins that can go a long way. If you’re new to web accessibility, hopefully this doesn’t come as a surprise! There are definitely challenging aspects to web accessibility, but it’s not all that way. And lucky for us, a lot of the quick-wins are also quick and easy to identify with some automated tools! pa11y-ci is just one of those tools.

In this post, I’ll show you how I set up a GitHub Action that runs pa11y-ci as a pull request check. The check will run against a Vercel deploy preview, and then the results will be shared as a comment on the pull request for easy visibility.

If at any point you find yourself stuck or needing to troubleshoot, check out a GitHub repository I created that represents the end-result of this guide: https://github.com/ashleemboyer/pa11y-ci-test-app.

Step 1 (optional): Create a Next.js App

If you don’t already have a Next.js App to follow along with, I recommend creating one. There are multiple ways of doing this detailed on the Next.js Getting Started page. To keep things simple for this example, I’m using the Automatic Setup approach.

To create the Next.js app:

  1. Open a terminal
  2. Run npx create-next-app@latest
  3. When prompted, name the app pa11y-ci-test-app
  4. Change directory into the new folder: cd pa11y-ci-test-app
  5. Briefly explore the code and learn your way around

Note: I will be using npm instead of yarn for the rest of this post. If your generated project uses yarn, and you would also like to to use npm instead:

  1. Run rm yarn.lock to delete the file
  2. Run npm install to create a package-lock.json file
  3. Run npm run build so the package-lock.json file can be patched if needed
  4. Run npm install one more time to download any missing dependencies
  5. Run git add . to stage the changed files for a commit
  6. Run git commit -m "Using npm instead of yarn" to commit the changes

Step 2 (optional): Create and Connect a GitHub Repository

If you followed step 1 and created a new Next.js App, you should follow this step as well. You will need a GitHub Repository for your project in order to run the GitHub Action we’re creating.

To create the GitHub Repository:

  1. Go to github.com/new
  2. If needed, log in to your account
  3. Under “Owner”, select your account
  4. Under “Repository name”, enter pa11y-ci-test-app
  5. Select either “Public” or “Private”
  6. Click the “Create repository” button
  7. Leave this page open in your browser

To connect the GitHub Repository to your project:

  1. In a terminal, go to your project folder
  2. Run git remote add origin git@github.com:<YOUR-USERNAME>/pa11y-ci-test-app.git after replacing <YOUR-USERNAME> with the GitHub username you used to create the repository. (Read about Working with Remotes and Remote Branches if you’re curious.)
  3. Run git push -u origin main to push the code to the repository.
  4. Verify these steps were successful by refreshing your new repository’s page in your browser

Step 3: Install and Configure pa11y-ci

It’s always good to test new tools locally (on your machine) before testing them remotely in a place like a pull request. Testing remotely can be tedious and time consuming, and the results of the testing can also be more difficult to locate and read. And if you happen to be like me and have ADHD, all of that is a recipe for multiple distractions!

Before doing anything else, let’s create a new git branch. That will be useful later when we need to create a pull request for our GitHub Action.

To create a new git branch:

  1. In a terminal, go to your project folder
  2. Run git checkout -b add-pa11y-ci

To install pa11y-ci:

  1. In a terminal, go to your project folder
  2. Run npm install --save-dev pa11y-ci
  3. Open the package.json file in your project
  4. Verify the pa11y-ci package is listed under devDependencies

To configure pa11y-ci for local testing:

  1. In your project folder, create a file called .pa11yci
  2. In the new file, paste the following JSON:
{
  "urls": ["http://localhost:3000/"]
}

This configuration tells pa11y-ci exactly what urls in our app it needs to test.

Step 4: Test pa11y-ci locally (on your machine)

Now that we have pa11y-ci installed and configured for local testing, we can get to testing!

  1. In your project folder, open the package.json file
  2. Under the "scripts" property, add a new entry: "pa11y-ci": "pa11y-ci"
  3. Open two separate terminals to your project folder
  4. In the first terminal, run npm run dev to start the app
  5. In the second terminal, run npm run pa11y-ci to run the new script we added in step 2

After pa11y-ci has finished running, this is how the output should read:

> pa11y-ci-test-app@0.1.0 pa11y-ci
> pa11y-ci

Running Pa11y on 1 URLs:
 > http://localhost:3000/ - 1 errors

Errors in http://localhost:3000/:

 • The html element should have a lang or xml:lang attribute which describes the language of the document.

   (html)

   <html><head><meta name="viewport" con...</html>

✘ 0/1 URLs passed

Hooray! We have an error, but the good news is that means pa11y-ci is working! Now we can progress from local testing towards creating our GitHub Action.

Step 5: Deploy to Vercel

Our current configuration for pa11y-ci only works for when we run our app locally with npm run dev. For remote testing, we want pa11y-ci to test against real builds and public-facing urls. In order to do that, we need to deploy our Next.js app. We’ll use Vercel in this example.

How to deploy to Vercel:

  1. Go to vercel.com/new
  2. If needed, log in to your account
  3. Under “Import Git Repository”, select “Continue with GitHub”
  4. Search for your new repo (mine is called pa11y-ci-test-app)
  5. Click the Import button in-line with the repo name
  6. Click the Deploy button (you should not need to change any configuration)

Once the deploy is complete, you can close this window.

Step 6: Install and Configure next-sitemap

There’s another way pa11y-ci can be used to test against urls, and that is with the --sitemap option. This option is nice when you have a sitemap.xml file on your site that updates every time a build occurs. It means you don’t have to manually update the .pa11yci file’s list of urls every time a new page is added. The tool we’ll use for generating a sitemap.xml file is called next-sitemap, which is the tool I use on my own website for pa11y-ci testing.

To install next-sitemap:

  1. In a terminal, go to your project folder
  2. Run npm install next-sitemap

To configure next-sitemap:

  1. In your project folder, create a new file called next-sitemap.config.js
  2. In the new file, paste the following JavaScript:
module.exports = {
  siteUrl: process.env.VERCEL_URL
    ? `https://${process.env.VERCEL_URL}`
    : 'http://localhost:3000/',
};

This tells next-sitemap what the main url of your site is. If you use a different port number than 3000, make sure to update the code above to match the port number you use.

Step 7: Test next-sitemap locally (on your machine)

To make sure we have everything configured correctly, we’re going to test next-sitemap locally just like we did with pa11y-ci.

  1. In your project folder, open the package.json file
  2. Under the "scripts" property, add a new entry: "postbuild": "next-sitemap"
  3. In a terminal, go to your project folder
  4. Run npm run build

The new postbuild script we added will run automatically after the build script has completed. You should see some new sitemap files in the public directory. We will add those to the .gitignore file in the next section.

You should also see something like this output in the terminal after the build script output:

> pa11y-ci-test-app@0.1.0 postbuild
> next-sitemap

✨ [next-sitemap] Loading next-sitemap config: file:///pa11y-ci-test-app/next-sitemap.config.js
✅ [next-sitemap] Generation completed
┌───────────────┬────────┐
│    (index)    │ Values │
├───────────────┼────────┤
│ indexSitemaps │   1    │
│   sitemaps    │   1    │
└───────────────┴────────┘
-----------------------------------------------------
 SITEMAP INDICES
-----------------------------------------------------

   ○ http://localhost:3000/sitemap.xml


-----------------------------------------------------
 SITEMAPS
-----------------------------------------------------

   ○ http://localhost:3000/sitemap-0.xml

Step 8: Prepare for and Create a New Pull Request

Hopefully you created a new git branch called add-pa11y-ci in Step 3, but it’s also okay if you didn’t! Go ahead and create it now. We’ll need a different branch off of the main one in order to create a pull request in our GitHub repository.

To prepare for the pull request:

  1. Make sure there is a /public/sitemap* entry in the .gitignore file. We do not want to commit these files since they need to be generated by the Vercel build process.
  2. Make sure you have all your new pa11y-ci and next-sitemap work committed. I’m committing everything at once with the message: "Adding pa11y-ci and next-sitemap".
  3. Run git push --set-upstream origin add-pa11y-ci to push your new branch up to the repository.

To create the new pull request:

  1. Go to github.com/<YOUR-USERNAME>/pa11y-ci-test-app/compare after replacing <YOUR-USERNAME> with the GitHub username you used to create the repository
  2. Change the compare branch to add-pa11y-ci
  3. The page should update have at least these 5 changed files:
    • .gitignore
    • .pa11yci
    • next-sitemap.config.js
    • package-lock.json
    • package.json
  4. Click the “Create pull request” button
  5. You should see 2 text boxes: one for the name of the pull request and one for the description
  6. Fill in the title and description however you need
  7. Click “Create pull request”

To verify everything is set up correctly:

  1. Wait for the vercel bot to comment on the new pull request
  2. In the new comment, open the “Visit Preview” link in a new tab
  3. You should see your app built and running
  4. Go to the /sitemap.xml page in the same app
  5. You should see an XML file with contents that look something like this:
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>YOUR-APP-URL/sitemap-0.xml</loc>
  </sitemap>
</sitemapindex>

Step 9: Create the pa11y-ci GitHub Action

Now that we have a live website to have pa11y-ci test against, we can create the GitHub Action that runs pa11y-ci.

To create the GitHub Action:

  1. In your project folder, create a new folder called .github
  2. In the new .github folder, create another folder called workflows
  3. In the new workflows folder, create a new file called pa11y-ci.yml
  4. Paste the following content in your new file:
# Wait to run until after the Vercel deployment has finished
# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#deployment_status
on: deployment_status

jobs:
  pa11y-ci:
    # Only run if the Vercel deployment was successful
    # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#deployment_status
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest

    steps:
      # Checkout latest commit of repo
      # https://github.com/actions/checkout#checkout-v3
      - uses: actions/checkout@v3

      # Install project dependencies
      - run: npm install

      # Run pa11y-ci against the generated sitemap
      # https://github.com/pa11y/pa11y-ci#sitemaps
      - run: npm run pa11y-ci -- --sitemap ${{ github.event.deployment_status.target_url }}/sitemap.xml

Before committing and pushing this new file, make sure to delete the .pa11yci file. It’s no longer needed.

To verify the GitHub Action works:

  1. Commit and push the new .github/workflows/pa11y-ci.yml and the removed .pa11yci file
  2. Go to your pull request in a browser
  3. Towards the bottom of the page, wait until you see the pa11y-ci check finish
  4. The check should fail
  5. Click the “Details” link to inspect the failed check
  6. You should see the same error that you saw when testing locally in Step 4

Step 10 (optional): Update the pa11y-ci GitHub Action to add a comment on pull requests

When I added this pa11y-ci GitHub Action to my own site’s pull requests, I decided to make it also comment results on the pull request. There are a couple of reasons for this:

So to update the GitHub Action to comment on pull requests, we’ll need to tell the pa11y-ci command to output its results in a different way, and then we’ll add a step to the GitHub Action that reads the output, formats it, and creates a comment.

  1. Update the pa11y-ci run step in .github/workflows/pa11y-ci.yml to be:

    # Run pa11y-ci against the generated sitemap
# https://github.com/pa11y/pa11y-ci#sitemaps
- run: npm run pa11y-ci -- --json --sitemap ${{ github.event.deployment_status.target_url }}/sitemap.xml |& tee pa11y-output.txt

  2. Add a new step at the end of the file like this:

    # Comment the formatted pa11y-ci results on the PR
- uses: actions/github-script@v5
  env:
    BODY_PREFIX: '<!-- pa11y-ci results -->'
  with:
    script: |
      const script = require('./.github/workflows/scripts/pa11y-ci.js')
      await script({ context, core, github })

  3. Add a new folder inside .github/workflows called scripts

  4. Add a new file to scripts called pa11y-ci.js

  5. In the new file, paste the following JavaScript:

    const { promises: fs } = require('fs');

module.exports = async ({ context, core, github }) => {
  // Find this PR & do nothing if this isn't a PR
  const { data } =
    await github.rest.repos.listPullRequestsAssociatedWithCommit({
      owner: context.repo.owner,
      repo: context.repo.repo,
      commit_sha: context.sha,
    });
  const prNumber = data[0] && data[0].number;
  if (!prNumber) {
    return;
  }

  // Read the pa11y output and build the comment body
  const pa11yOutput = await fs.readFile('./pa11y-output.txt', 'utf8');
  const lines = pa11yOutput.split('\n');
  const asJSON = JSON.parse(lines.find((line) => line.startsWith('{')));
  const { total, errors, results } = asJSON;

  let commentBody = `${process.env.BODY_PREFIX}\n<h2>:microscope: pa11y-ci results</h2>\n\n`;
  commentBody += `- Number of URLs tested: ${total}\n`;

  if (!errors) {
    commentBody += `- No errors were found! :tada:\n`;
  } else {
    commentBody += `- Number of errors found: ${errors}. :sob:\n`;

    let formattedOutput = '';
    Object.keys(results).forEach((urlKey) => {
      const errors = results[urlKey];
      if (errors.length < 1) {
        return;
      }

      formattedOutput += `- [${urlKey}](${urlKey}):\n\n`;
      errors.forEach((error) => {
        formattedOutput += '  ```\n';
        formattedOutput += `  ${error.message}\n\n`;
        formattedOutput += `  ${error.selector}\n\n`;
        formattedOutput += `  ${error.context}\n\n`;
        formattedOutput += `  ${error.code}\n`;
        formattedOutput += '  ```\n\n';
      });
    });

    commentBody += `<details><summary>See results :eyes:</summary>\n\n${formattedOutput}</details>`;
  }

  // Get the comments on this PR
  const { data: comments } = await github.rest.issues.listComments({
    owner: context.repo.owner,
    repo: context.repo.repo,
    issue_number: prNumber,
  });

  // Try to find an existing pa11y results comment
  const previousComment = comments.find((comment) =>
    comment.body.startsWith(process.env.BODY_PREFIX),
  );
  if (previousComment) {
    // Update the previous comment
    await github.rest.issues.updateComment({
      owner: context.repo.owner,
      repo: context.repo.repo,
      comment_id: previousComment.id,
      body: commentBody,
    });
  } else {
    // Create a new comment
    await github.rest.issues.createComment({
      owner: context.repo.owner,
      repo: context.repo.repo,
      issue_number: prNumber,
      body: commentBody,
    });
  }

  if (errors) {
    core.setFailed('Errors were found by pa11y-ci');
  }
};

  6. Commit and push the changes to .github/workflows/pa11y-ci.yml and the new .github/workflows/scripts/pa11y-ci.js file.

To verify that everything is working as expected:

  1. Go to your pull request in your browser
  2. Wait for the pa11y-ci check to finish
  3. The check should fail just like it did before
  4. Look for a comment by the github-actions bot that says ”🔬 pa11y-ci results”
  5. The comment should say that 1 error was found
  6. Expand the “See results 👀” text
  7. There should be one error listed that matches the same error you’ve been seeing

Step 11. Celebrate! 🎉

Nicely done! You’ve added a pa11y-ci GitHub Action that runs on pull requests. You should be proud. Give yourself a pat on the back. 😊

Back to Top