Pantheon Community

CI Visual Regression Run Logic

I’m wondering if anyone (@ataylorme) could walk me through why we don’t run VR tests except under very specific circumstances? Besides the base logic of needing to be a non-master branch and a branch that has an open PR, why exactly wouldn’t we want VR tests to run if we say, update PHP code in a custom module?

Is this logic more geared towards the assumption that all code changes come from upstream modules? We happen to have several modules that are not in composer.json, but when we make changes to them, they never trigger the VR tests…

LAST_GIT_COMMIT_MESSAGE=$(git log -1 --pretty=%B)

# Always run visual tests if "[vr]" is in the last commit message
if [[ ${LAST_GIT_COMMIT_MESSAGE} != *"[vr]"* ]]
then

# Skip visual tests if there hasn't been a modification to composer.lock
if ! GIT_FILE_MODIFIED 'composer.lock'
then
    echo -e "\nSkipping visual regression tests since composer.lock has NOT changed"
    exit 0
fi

# Skip visual tests if has been a modification to composer.json
if GIT_FILE_MODIFIED 'composer.json'
then
    echo -e "\nSkipping visual regression tests since composer.json HAS changed"
    exit 0
fi

else
echo -e "\nRunning visual regression tests because the latest commit message demands it"
fi

@Adamo the CI set up created from Pantheon Build Tools, such as example-drops-8-composer is meant to be a starting state/example to build upon.

If the logic isn’t what you want then please change it for your project(s). Build tools can create new projects from your own fork/repository if you have multiple customizations from our example start state you want to make.

The reason we have them run on composer.lock changes only is that we are using it to test automated dependency update pull requests made by Composer Lock Updater. This PR is a good example.

You could certainly run the visual regression test more often. My biggest reason for not doing it on every commit is that it can be noisy/slow and the test isn’t always expected to pass (think of a PR with template/CSS changes).

Besides the base logic of needing to be a non-master branch and a branch that has an open PR, why exactly wouldn’t we want VR tests to run if we say, update PHP code in a custom module?

This is what the flexibility of manually adding [vr] to a commit message to trigger the test should provide. If you make a dozen commits working through a PR then you can manually run the visual regression test at the end instead of having to wait for it on each commit and having the report results clutter up the PR comments.

I see – especially the noise bit. Our other app runs backstop on every commit pushed, and with multiple scenarios, it takes a WHILE, even though most pass.

I appreciate you giving context to this default configuration and use case. Since we have a decoupled front-end, we may do away with this and instead use the Behat tests to check if dependency updates broke anything instead, or we may flesh out the VR scenarios for more routine checks against high-traffic admin pages of the site.

I’ll try and update/share when we adjust for others to follow along on :slight_smile: