If you've been working in web accessibility for a while, you know there are many foundational quick-wins that can go a long way. If you're new to web accessibility, hopefully this doesn't come as a surprise! There are definitely challenging aspects to web accessibility, but it's not all that way. And lucky for us, a lot of the quick-wins are also quick and easy to identify with some automated tools! [`pa11y-ci`](https://github.com/pa11y/pa11y-ci) is just one of those tools.

In this post, I'll show you how I set up a [GitHub Action](https://github.com/features/actions) that runs `pa11y-ci` as a pull request check. The check will run against a Vercel deploy preview, and then the results will be shared as a comment on the pull request for easy visibility.

If at any point you find yourself stuck or needing to troubleshoot, check out a GitHub repository I created that represents the end-result of this guide: https://github.com/ashleemboyer/pa11y-ci-test-app.

## Step 1 (optional): Create a Next.js App

If you don't already have a Next.js App to follow along with, I recommend creating one. There are multiple ways of doing this detailed on [the Next.js Getting Started page](https://nextjs.org/docs/getting-started). To keep things simple for this example, I'm using the Automatic Setup approach.

**To create the Next.js app:**

1. Open a terminal
2. Run `npx create-next-app@latest`
3. When prompted, name the app `pa11y-ci-test-app`
4. Change directory into the new folder: `cd pa11y-ci-test-app`
5. Briefly explore the code and learn your way around

Note: I will be using `npm` instead of `yarn` for the rest of this post. If your generated project uses `yarn`, and you would also like to to use `npm` instead:

1. Run `rm yarn.lock` to delete the file
2. Run `npm install` to create a `package-lock.json` file
3. Run `npm run build` so the `package-lock.json` file can be patched if needed
4. Run `npm install` one more time to download any missing dependencies
5. Run `git add .` to stage the changed files for a commit
6. Run `git commit -m "Using npm instead of yarn"` to commit the changes

## Step 2 (optional): Create and Connect a GitHub Repository

If you followed step 1 and created a new Next.js App, you should follow this step as well. You will need a GitHub Repository for your project in order to run the GitHub Action we're creating.

**To create the GitHub Repository:**

1. Go to [github.com/new](https://github.com/new)
2. If needed, log in to your account
3. Under "Owner", select your account
4. Under "Repository name", enter `pa11y-ci-test-app`
5. Select either "Public" or "Private"
6. Click the "Create repository" button
7. Leave this page open in your browser

**To connect the GitHub Repository to your project:**

1. In a terminal, go to your project folder
2. Run `git remote add origin git@github.com:<YOUR-USERNAME>/pa11y-ci-test-app.git` after replacing `<YOUR-USERNAME>` with the GitHub username you used to create the repository. (Read about [Working with Remotes](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes) and [Remote Branches](https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches) if you're curious.)
3. Run `git push -u origin main` to push the code to the repository.
4. Verify these steps were successful by refreshing your new repository's page in your browser

## Step 3: Install and Configure `pa11y-ci`

It's always good to test new tools locally (on your machine) before testing them remotely in a place like a pull request. Testing remotely can be tedious and time consuming, and the results of the testing can also be more difficult to locate and read. And if you happen to be like me and have ADHD, all of that is a recipe for multiple distractions!

Before doing anything else, let's create a new git branch. That will be useful later when we need to create a pull request for our GitHub Action.

**To create a new `git` branch:**

1. In a terminal, go to your project folder
2. Run `git checkout -b add-pa11y-ci`

**To install `pa11y-ci`:**

1. In a terminal, go to your project folder
2. Run `npm install --save-dev pa11y-ci`
3. Open the `package.json` file in your project
4. Verify the `pa11y-ci` package is listed under `devDependencies`

**To configure `pa11y-ci` for local testing:**

1. In your project folder, create a file called `.pa11yci`
2. In the new file, paste the following JSON:

```json
{
  "urls": ["http://localhost:3000/"]
}
```

This configuration tells `pa11y-ci` exactly what `urls` in our app it needs to test.

## Step 4: Test `pa11y-ci` locally (on your machine)

Now that we have `pa11y-ci` installed and configured for local testing, we can get to testing!

1. In your project folder, open the `package.json` file
2. Under the `"scripts"` property, add a new entry: `"pa11y-ci": "pa11y-ci"`
3. Open two separate terminals to your project folder
4. In the first terminal, run `npm run dev` to start the app
5. In the second terminal, run `npm run pa11y-ci` to run the new script we added in step 2

After `pa11y-ci` has finished running, this is how the output should read:

```
> pa11y-ci-test-app@0.1.0 pa11y-ci
> pa11y-ci

Running Pa11y on 1 URLs:
 > http://localhost:3000/ - 1 errors

Errors in http://localhost:3000/:

 • The html element should have a lang or xml:lang attribute which describes the language of the document.

   (html)

   <html><head><meta name="viewport" con...</html>

✘ 0/1 URLs passed
```

Hooray! We have an error, but the good news is that means `pa11y-ci` is working! Now we can progress from local testing towards creating our GitHub Action.

## Step 5: Deploy to Vercel

Our current configuration for `pa11y-ci` only works for when we run our app locally with `npm run dev`. For remote testing, we want `pa11y-ci` to test against real builds and public-facing urls. In order to do that, we need to deploy our Next.js app. We'll use Vercel in this example.

**How to deploy to Vercel:**

1. Go to [vercel.com/new](https://vercel.com/new)
2. If needed, log in to your account
3. Under "Import Git Repository", select "Continue with GitHub"
4. Search for your new repo (mine is called `pa11y-ci-test-app`)
5. Click the Import button in-line with the repo name
6. Click the Deploy button (you should not need to change any configuration)

Once the deploy is complete, you can close this window.

## Step 6: Install and Configure `next-sitemap`

There's another way `pa11y-ci` can be used to test against urls, and that is with the `--sitemap` option. This option is nice when you have a `sitemap.xml` file on your site that updates every time a build occurs. It means you don't have to manually update the `.pa11yci` file's list of urls every time a new page is added. The tool we'll use for generating a `sitemap.xml` file is called [`next-sitemap`](https://www.npmjs.com/package/next-sitemap), which is the tool I use on my own website for `pa11y-ci` testing.

**To install `next-sitemap`:**

1. In a terminal, go to your project folder
2. Run `npm install next-sitemap`

**To configure `next-sitemap`:**

1. In your project folder, create a new file called `next-sitemap.config.js`
2. In the new file, paste the following JavaScript:

```javascript
module.exports = {
  siteUrl: process.env.VERCEL_URL
    ? `https://${process.env.VERCEL_URL}`
    : 'http://localhost:3000/',
};
```

This tells `next-sitemap` what the main url of your site is. If you use a different port number than `3000`, make sure to update the code above to match the port number you use.

## Step 7: Test `next-sitemap` locally (on your machine)

To make sure we have everything configured correctly, we're going to test `next-sitemap` locally just like we did with `pa11y-ci`.

1. In your project folder, open the `package.json` file
2. Under the `"scripts"` property, add a new entry: `"postbuild": "next-sitemap"`
3. In a terminal, go to your project folder
4. Run `npm run build`

The new `postbuild` script we added will run automatically after the `build` script has completed. You should see some new `sitemap` files in the `public` directory. We will add those to the `.gitignore` file in the next section.

You should also see something like this output in the terminal after the `build` script output:

```
> pa11y-ci-test-app@0.1.0 postbuild
> next-sitemap

✨ [next-sitemap] Loading next-sitemap config: file:///pa11y-ci-test-app/next-sitemap.config.js
✅ [next-sitemap] Generation completed
┌───────────────┬────────┐
│    (index)    │ Values │
├───────────────┼────────┤
│ indexSitemaps │   1    │
│   sitemaps    │   1    │
└───────────────┴────────┘
-----------------------------------------------------
 SITEMAP INDICES
-----------------------------------------------------

   ○ http://localhost:3000/sitemap.xml


-----------------------------------------------------
 SITEMAPS
-----------------------------------------------------

   ○ http://localhost:3000/sitemap-0.xml
```

## Step 8: Prepare for and Create a New Pull Request

Hopefully you created a new git branch called `add-pa11y-ci` [in Step 3](#step-3-install-and-configure-pa11y-ci), but it's also okay if you didn't! Go ahead and create it now. We'll need a different branch off of the main one in order to create a pull request in our GitHub repository.

**To prepare for the pull request:**

1. Make sure there is a `/public/sitemap*` entry in the `.gitignore` file. We **do not** want to commit these files since they need to be generated by the Vercel build process.
2. Make sure you have all your new `pa11y-ci` and `next-sitemap` work committed. I'm committing everything at once with the message: `"Adding pa11y-ci and next-sitemap"`.
3. Run `git push --set-upstream origin add-pa11y-ci` to push your new branch up to the repository.

**To create the new pull request:**

1. Go to `github.com/<YOUR-USERNAME>/pa11y-ci-test-app/compare` after replacing `<YOUR-USERNAME>` with the GitHub username you used to create the repository
2. Change the `compare` branch to `add-pa11y-ci`
3. The page should update have at least these 5 changed files:
   - `.gitignore`
   - `.pa11yci`
   - `next-sitemap.config.js`
   - `package-lock.json`
   - `package.json`
4. Click the "Create pull request" button
5. You should see 2 text boxes: one for the name of the pull request and one for the description
6. Fill in the title and description however you need
7. Click "Create pull request"

**To verify everything is set up correctly:**

1. Wait for the `vercel` bot to comment on the new pull request
2. In the new comment, open the "Visit Preview" link in a new tab
3. You should see your app built and running
4. Go to the `/sitemap.xml` page in the same app
5. You should see an XML file with contents that look something like this:

```xml
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>YOUR-APP-URL/sitemap-0.xml</loc>
  </sitemap>
</sitemapindex>
```

## Step 9: Create the pa11y-ci GitHub Action

Now that we have a live website to have `pa11y-ci` test against, we can create the GitHub Action that runs `pa11y-ci`.

**To create the GitHub Action:**

1. In your project folder, create a new folder called `.github`
2. In the new `.github` folder, create another folder called `workflows`
3. In the new `workflows` folder, create a new file called `pa11y-ci.yml`
4. Paste the following content in your new file:

```yml
# Wait to run until after the Vercel deployment has finished
# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#deployment_status
on: deployment_status

jobs:
  pa11y-ci:
    # Only run if the Vercel deployment was successful
    # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#deployment_status
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest

    steps:
      # Checkout latest commit of repo
      # https://github.com/actions/checkout#checkout-v3
      - uses: actions/checkout@v3

      # Install project dependencies
      - run: npm install

      # Run pa11y-ci against the generated sitemap
      # https://github.com/pa11y/pa11y-ci#sitemaps
      - run: npm run pa11y-ci -- --sitemap ${{ github.event.deployment_status.target_url }}/sitemap.xml
```

Before committing and pushing this new file, make sure to delete the `.pa11yci` file. It's no longer needed.

**To verify the GitHub Action works:**

1. Commit and push the new `.github/workflows/pa11y-ci.yml` and the removed `.pa11yci` file
2. Go to your pull request in a browser
3. Towards the bottom of the page, wait until you see the `pa11y-ci` check finish
4. The check should fail
5. Click the "Details" link to inspect the failed check
6. You should see the same error that you saw when testing locally in [Step 4](#step-4-test-pa11y-ci-locally-on-your-machine)

## Step 10 (optional): Update the pa11y-ci GitHub Action to add a comment on pull requests

When I added this `pa11y-ci` GitHub Action to my own site's pull requests, I decided to make it also comment results on the pull request. There are a couple of reasons for this:

- Normally you have to click the "Details" link next to a failed pull request check. This takes you to another page and away from the pull request. For me, that means an opportunity to get easily distracted.
- The formatting of results in pull request checks can sometimes be difficult to read. Putting the results in a pull request comment gives an opportunity to re-format them in a more readable way.

So to update the GitHub Action to comment on pull requests, we'll need to tell the `pa11y-ci` command to output its results in a different way, and then we'll add a step to the GitHub Action that reads the output, formats it, and creates a comment.

1. Update the `pa11y-ci` `run` step in `.github/workflows/pa11y-ci.yml` to be:
   ```yml
   # Run pa11y-ci against the generated sitemap
   # https://github.com/pa11y/pa11y-ci#sitemaps
   - run: npm run pa11y-ci -- --json --sitemap ${{ github.event.deployment_status.target_url }}/sitemap.xml |& tee pa11y-output.txt
   ```
2. Add a new step at the end of the file like this:
   ```yml
   # Comment the formatted pa11y-ci results on the PR
   - uses: actions/github-script@v5
     env:
       BODY_PREFIX: ''
     with:
       script: |
         const script = require('./.github/workflows/scripts/pa11y-ci.js')
         await script({ context, core, github })
   ```
3. Add a new folder inside `.github/workflows` called `scripts`
4. Add a new file to `scripts` called `pa11y-ci.js`
5. In the new file, paste the following JavaScript:

   ````js
   const { promises: fs } = require('fs');

   module.exports = async ({ context, core, github }) => {
     // Find this PR & do nothing if this isn't a PR
     const { data } =
       await github.rest.repos.listPullRequestsAssociatedWithCommit({
         owner: context.repo.owner,
         repo: context.repo.repo,
         commit_sha: context.sha,
       });
     const prNumber = data[0] && data[0].number;
     if (!prNumber) {
       return;
     }

     // Read the pa11y output and build the comment body
     const pa11yOutput = await fs.readFile('./pa11y-output.txt', 'utf8');
     const lines = pa11yOutput.split('\n');
     const asJSON = JSON.parse(lines.find((line) => line.startsWith('{')));
     const { total, errors, results } = asJSON;

     let commentBody = `${process.env.BODY_PREFIX}\n<h2>:microscope: pa11y-ci results</h2>\n\n`;
     commentBody += `- Number of URLs tested: ${total}\n`;

     if (!errors) {
       commentBody += `- No errors were found! :tada:\n`;
     } else {
       commentBody += `- Number of errors found: ${errors}. :sob:\n`;

       let formattedOutput = '';
       Object.keys(results).forEach((urlKey) => {
         const errors = results[urlKey];
         if (errors.length < 1) {
           return;
         }

         formattedOutput += `- [${urlKey}](${urlKey}):\n\n`;
         errors.forEach((error) => {
           formattedOutput += '  ```\n';
           formattedOutput += `  ${error.message}\n\n`;
           formattedOutput += `  ${error.selector}\n\n`;
           formattedOutput += `  ${error.context}\n\n`;
           formattedOutput += `  ${error.code}\n`;
           formattedOutput += '  ```\n\n';
         });
       });

       commentBody += `<details><summary>See results :eyes:</summary>\n\n${formattedOutput}</details>`;
     }

     // Get the comments on this PR
     const { data: comments } = await github.rest.issues.listComments({
       owner: context.repo.owner,
       repo: context.repo.repo,
       issue_number: prNumber,
     });

     // Try to find an existing pa11y results comment
     const previousComment = comments.find((comment) =>
       comment.body.startsWith(process.env.BODY_PREFIX),
     );
     if (previousComment) {
       // Update the previous comment
       await github.rest.issues.updateComment({
         owner: context.repo.owner,
         repo: context.repo.repo,
         comment_id: previousComment.id,
         body: commentBody,
       });
     } else {
       // Create a new comment
       await github.rest.issues.createComment({
         owner: context.repo.owner,
         repo: context.repo.repo,
         issue_number: prNumber,
         body: commentBody,
       });
     }

     if (errors) {
       core.setFailed('Errors were found by pa11y-ci');
     }
   };
   ````

6. Commit and push the changes to `.github/workflows/pa11y-ci.yml` and the new `.github/workflows/scripts/pa11y-ci.js` file.

**To verify that everything is working as expected:**

1. Go to your pull request in your browser
2. Wait for the `pa11y-ci` check to finish
3. The check should fail just like it did before
4. Look for a comment by the `github-actions` bot that says "🔬 pa11y-ci results"
5. The comment should say that 1 error was found
6. Expand the "See results 👀" text
7. There should be one error listed that matches the same error you've been seeing

## Step 11. Celebrate! 🎉

Nicely done! You've added a `pa11y-ci` GitHub Action that runs on pull requests. You should be proud. Give yourself a pat on the back. 😊

Use pa11y-ci to test deploy previews before merging pull requests.

Posts Related to the "github-actions" Tag