Gutenberg release: How to run plugin performance test locally

For Gutenberg 14.0, the release lead noticed some funky test results. As the release of 13.7, I performed was mostly without hick-ups, I was curious how a local performance test would work. This is a record on how this works.

Gutenberg 14.0 release was published on August 31, 2022

The documentation states:

If the GitHub workflow fails, the performance audit can be executed locally using bin/plugin/cli.js perf and passing the branches to run the performance suite against as parameters. In addition, the current major WP version can be passed to avoid running tests against the WP trunk.
Example:

node ./bin/plugin/cli.js perf release/x.y release/x.z wp/a.b --wp-version wp.major

Source: Gutenberg Release Process

Let’s translated to the Gutenberg 14.0 release:

After perf follows a list of branch names for the releases to be compared. Those can be looked up via the branch dropdown on the GitHub repository.

We compare this release with the previous one so the strings are release/13.9 and release/14.0, then we also compare against the latest WordPress major release, here 6.0. We use the version number in the GitHub Branch wp/6.0 and --wp-version 6.0

The full command looked like this:

node ./bin/plugin/cli.js perf release/13.9 release/14.0 wp/6.0 --wp-version 6.0

Now that the command is constructed, in the video you see what happens after you hit enter on that command.

The process can take quite a few minutes. Some contributors run regularly into timeouts.

My equipment: MacOS Pro (13-inch, M1, 2020), Monterey 12.5.1. I didn’t time it but it was more than 30 minutes. The process on GitHub Action runs about an hour.

Test Results

The Test results will be displayed in your terminal window.

For the Post Editor

For the Site Editor:

Release post

Two values of the test results make it into the release post for the plugin release:

Time to the first block (test named firstBlock)

KeyPress Event (typing) (test named type)

Source: Gutenberg Release Process

The table display would show something like this from above results.

Post Editor

Version	Time to the first block	KeyPress Event (typing)
Gutenberg 14.0	9.98 s	43.36 ms
Gutenberg 13.9	3.95 s	54.77 ms
WordPress 6.0	4.31 s	59.53 ms

Site Editor

Version	Time to the first block	KeyPress Event (typing)
Gutenberg 14.0	11.45 s	34.75 ms
Gutenberg 13.9	5.16 s	29.37 ms
WordPress 6.0	4.92 s	20.11 ms

The research on the dramatic changes with Gutenberg 14.0 is on going on the GitHub issue List: use nested blocks

Error messages encountered on local performance testing:

Error: Cannot find module ‘inquirer’

Fix: run npm install to update all dependencies on your local instance

GitError: fatal: couldn’t find remote ref release/x14.0

Fix: fix typo in version branch name

Could not connect to Docker. Is it running?

Fix: start Docker, if it’s not installed go through the installation process.