Up until a few weeks ago, I had to fully rebuild my website to update an article
or note. While this didn't take along after switching to Next.js,
it was a minor peeve. After searching through the Next.js docs, I came across a
concept known as Incremental Static Regeneration.
I opted to use on-demand revalidation, which uses an API route to trigger a revalidation
res.revalidate function which takes the path of the route you want to update.
To get started, I created the API route at
/api/revalidate, then added a few checks
in line with the example. I implemented a HTTP method check to ensure that all requests
POST requests, and to ensure that all requests were authenticated against
a secret key I generated. The source code for the API route is available on GitHub.
After testing the revalidation route with Hoppscotch,
I embarked on the important part: automatically triggering revalidation. Since my
articles and notes are all stored in a GitHub repository,
I decided to use GitHub Actions configured with a
push event to the
as well as a
workflow_dispatch trigger for testing.
I added four stages to the workflow: evaluation, compilation, merging, and revalidation.
The evaluation stage determines if any files that would warrant a revalidation have
been updated. These include all the Markdown files in the
also in the Markdown folder. The compilation stage has two substages: one for Markdown
files and one for the JSON file. These substages retrieve changes from the files,
then compile the paths needed to revalidate. The merge stage merges the output from
both compilation substages. And finally, the revalidation stage actually sends revalidation
requests to Vercel.
The evaluation stage is the simplest stage, because it simply generates a list of
files changed in the last commit. Both evaluation checks
git diff command with two options,
--diff-filter=ACMRT, to check for differences between the current
commit and the previous commit.
The first parameter
git diff to only return the filenames that were changed, as opposed to the
line-level differences. The second parameter
filters the search for files by checking if they were added (
A), copied (
M), renamed (
R), or had their type changed (
T). Then, the commands
filter the file extensions (
.json$ respectively). In the case of Markdown
files, an additional
grep command is in place to verify that all the files found
are in the
grep '^markdown/'. Finally,
compiles the returned files into a single line.
The compilation stage is the next stage, and is split into two substages.
The first substage deals with Markdown files. It first
checks whether any of the Markdown files were changed by checking the evaluation
stage's output. If it does not contain any Markdown files, the substage outputs
If there is a changed Markdown file, the substage starts by replacing the
spaces in the evaluation stage's output with new line characters via the
command. At this point, if the
Changing Google Docs' Default Styles
and Adding Wildcard Subdomain Support to macOS
articles were edited, the output would be the following.
To convert this into a path, the
markdown/, date, and file extension have to go.
The workflow accomplishes this
a stream editor "used to perform basic text transformations on an input stream
(a file or input from a pipeline)". By passing the
sed enables extended
regular expressions, a full description of which can be found in the documentation.
The output is now as follows.
Two more steps remain before finishing the Markdown compilation. One of those steps
is to change
articles (the folder name) to
article (the path name), which the
workflow does through another
The last step is to convert the output to a JSON array. The workflow takes advantage
of GitHub Actions' included software,
jq, a self-described "
JSON data" to do this. It passes two flags:
-R, which tells
jq to not parse the input as JSON, and the
-c, which minifies
the output. In addition to the two flags, the workflow also passes a filter:
'split(" ")', which
interprets as splitting the input on the spaces. The final output is the following.
The second substage, which deals with the
contents.json file, is a lot more complicated.
As with the Markdown substage, this substage first checks the evaluation stage's
output to see if the
contents.json file was updated. If it was not, it outputs
an empty JSON array.
If it was updated, the substage begins
jd, a Go command-line utility
and library for diffing and patching JSON and YAML files, through SupplyPike Engineering's
setup-bin action. The workflow then
outputs the previous commit's version
contents.json file, and compares it to the current version
patch diff format, which is based upon IETF RFC 6902.
The substage uses the
patch format because it outputs a JSON array with objects
to describe each change. To parse this array,
jq is called
..path filter, which gets all the JSON paths that were changed. For
example, if I were to change the
Generate TypeScript Declaration Files for JS Files,
jd would output the following.
jq would then extract the
path key from each object, returning the following.
"/articles/4/title" "/articles/4/title" "/articles/4/title"
The next step that the substage does is to remove any duplicates, leaving only one
entry. The substage uses
to do this, using the
! seen[$0]++ rule.
In the example, using this rule results in the following.
After this, the substage strips the quotes using
then extracts the path
to the object in the
notes array using
These two commands change the
awk output to the following.
To convert this path into a
jq filter, two
sed commands are used,
which converts the above path to the following.
.articles | nth(4) .id
This filter tells
jq to open the
articles object, find the fourth object, and
output the ID. The next part iterates over each filter,
if there are multiple. It uses a
while read function
to do so. The first command in the loop is
jq using the generated filter,
which outputs the following.
tr strips the quotes,
followed by a nested set of
These commands take the generated filter (stored in the
filter variable), extract
the base object (in this case, it's
articles), then convert it to a path prefix,
then prefix the path with a slash, to make it compatible with the Next.js revalidation
function. Finally, just like the Markdown substage, the substage takes all the IDs,
and runs them through
split filter, which outputs the following.
This is not the end of the JSON compilation substage, however. There are two secondary
tasks that run concurrently with the primary task. One of these tasks is to update
all pages where the hero image (also known as a thumbnail) and the publish date
are shown. This does the exact same thing as the primary task, but has two differences.
The first difference is that there is a filter is in place
to make sure only entries where the
published elements have changed
are added, and the second
is to add the homepage and the articles page.
The other secondary task is to update the tag pages. This has a filter for the hero image, publish date, and tags, as well as generating a list of changed tag pages.
The third stage is the merge stage, which merges the four arrays of paths: Markdown,
JSON, JSON (hero image and publish dates), and JSON (tags). The stage uses
passing in the arrays
--argjson parameter. The
--null-input flag, is also passed, and it tells
jq to use
the input. It's useful when constructing JSON data from scratch. Finally, a
filter is passed
which merges the four arrays, then uses the
to remove duplicate paths.
The final stage is the revalidation stage, where the paths are sent to the Next.js
for revalidation. This stage does not run if the output of the merge stage is an
empty JSON array. In terms of the logic, it's quite simple. It outputs
the merge stage's output, runs it through
to get each path on a different line, then iterates over each line
cURL to make a
to the revalidation API route with the appropriate header and secret key.
If you have any questions or need any help, feel free to contact me on Twitter or Mastodon.
If you have any improvements to any of my articles, notes, or this revalidation workflow, please submit a pull request.
Thank you for reading!