Pantheon Community

Visual Regression Error with CI workflow

I’m using the CI workflow and have a question about branches that seem to be created by a “CI bot” to update composer dependencies. All the builds keep failing visual regression. Anyone knows why this would be happening? Also, I got an error that I have too many branches now. So I have started to manually delete them on the dashboard.

@ataylorme or @steve.persch might be great folks to help you answer that question!

1 Like

Hi @omega, you can stop those automatic composer updates by removing that part of the CI config if necessary: https://github.com/pantheon-systems/example-drops-8-composer/blob/master/.circleci/config.yml#L205

What is happening in the visual regression failure? Did some part of the website actually break in the composer-updated branches?

1 Like

Hey @steve.persch! Ideally I would love to keep the automatic composer updates. As far as I can tell, the branches that are created are fully functional. This is what I see in CircleCI:

COMMAND | Executing core for "report" See: /root/example_drops_8_composer/backstop_data/bitmaps_test/20190529-171018/failed_diff_052387080_Homepage_0_document_0_phone.png compare | ERROR { requireSameDimensions: false, size: isDifferent, content: 1.58%, threshold: 0.1% }: Homepage 052387080_Homepage_0_document_0_phone.png See: /root/example_drops_8_composer/backstop_data/bitmaps_test/20190529-171018/failed_diff_052387080_Homepage_0_document_1_tablet_v.png compare | ERROR { requireSameDimensions: false, size: isDifferent, content: 3.48%, threshold: 0.1% }: Homepage 052387080_Homepage_0_document_1_tablet_v.png See: /root/example_drops_8_composer/backstop_data/bitmaps_test/20190529-171018/failed_diff_052387080_Homepage_0_document_2_tablet_h.png compare | ERROR { requireSameDimensions: false, size: isDifferent, content: 1.67%, threshold: 0.1% }: Homepage 052387080_Homepage_0_document_2_tablet_h.png See: /root/example_drops_8_composer/backstop_data/bitmaps_test/20190529-171018/failed_diff_052387080_Homepage_0_document_3_desktop.png compare | ERROR { requireSameDimensions: false, size: isDifferent, content: 1.98%, threshold: 0.1% }: Homepage 052387080_Homepage_0_document_3_desktop.png report | Test completed... report | 0 Passed report | 4 Failed report | Writing jUnit Report report | jUnit report written to: backstop_data/ci_report/xunit.xml report | Writing browser report report | Resources copied report | Copied configuration to: /root/example_drops_8_composer/backstop_data/html_report/config.js COMMAND | Executing core for "openReport" openReport | Attempting to ping openReport | Remote not found. Opening backstop_data/html_report/index.html report | *** Mismatch errors found *** COMMAND | Command "report" ended with an error after [1.704s] COMMAND | Error: Mismatch errors found. at /usr/local/lib/node_modules/backstopjs/core/command/report.js:155:17 at <anonymous> at process._tickCallback (internal/process/next_tick.js:189:7) COMMAND | Command "test" ended with an error after [7.714s] COMMAND | Error: Mismatch errors found. at /usr/local/lib/node_modules/backstopjs/core/command/report.js:155:17 at <anonymous> at process._tickCallback (internal/process/next_tick.js:189:7)

Hi @omega,
I would take a look at one of the failing CircleCI build logs. There should be a visual regression report saved as an artifact.

COMMAND | Command "test" ended with an error after [7.714s] COMMAND | Error: Mismatch errors found

That seems like the visual test is failing, causing the build to fail. Hopefully, the report will give you some insight into the failure. If you’ve inspected the multidev manually then you can always merge the pull request for the updates, even with the failed visual regression check.

Note that there was a bug in Composer Lock Updater and it would open multiple pull requests. The auto update should now only open one pull request, hopefully saving you on branches/multidevs.

Feel free to delete all but the most recent Composer update PR and the corresponding multidev environments. Hope this helps.

1 Like

@ataylorme I clicked on the GitHub comment and saw the reference test and diff. The reference is blank, the test has a screenshot of the site and the diff is…well…

.

Ok, I will try to merge a pull request and see what happens. Thanks