Pantheon Community

Using Semaphore Classic instead of CircleCI

We recently needed to migrate the working D8 Composer + CircleCI pipeline to Semaphore due to our company’s policies and resource allocations.

After much trial and error, we were able to (mostly) recreate the workflows as a single job (parallelization works, but with caveats).

Note that Semaphore 2.0 is extremely similar to Circle CI’s YML-based config system, but unfortunately we had to use Classic as we haven’t migrated to 2.0 yet. Therefore jobs have no context as to which other jobs have completed or not (no task dependency control). Secondly, Semaphore Classic can’t boot with a certain Docker image, like CircleCI can, so we needed to bootstrap the image manually with docker-compose.

Here it is in full disclosure:

Setup Task:

# use newer version of Ruby (requirement for octokit dependencies)
rbenv global 2.3.7
# expose the pull request number if there is one (Sempahore doesn't have PR# access natively)
gem install octokit --no-ri --no-rdoc
export PULL_REQUEST_NUMBER=$(ruby pull_request_number.rb)
# make the host build directory writable by the container process
chgrp -R docker .
# start pantheon build tools container
docker-compose up -d ci
# alias the command prefix needed to run subsequent commands in the container
export RUN_IN_CONTAINER="docker-compose exec ci bash -ci"
# make sure Pantheon env vars are available inside the container
$RUN_IN_CONTAINER '/build-tools-ci/scripts/set-environment'
# copy SSH key into the container for use by Terminus commands
$RUN_IN_CONTAINER 'cp $HOME/pantheon_semaphore.pem $HOME/.ssh/id_rsa'
$RUN_IN_CONTAINER 'chmod 600 $HOME/.ssh/id_rsa'
# log in
$RUN_IN_CONTAINER 'terminus -n auth:login --machine-token="$TERMINUS_TOKEN"'
# run composer install to get the vendor directory
$RUN_IN_CONTAINER 'composer install'

Test Job (all tasks):

### Prepare Multidev Environment
# install dev dependencies, build assets, etc.
$RUN_IN_CONTAINER '$SCRIPT_PATH/01-prepare'
# build assets
$RUN_IN_CONTAINER 'composer -n build-assets'
# prepare database for site-under test
$RUN_IN_CONTAINER '$SCRIPT_PATH/02-init-site-under-test-clone-existing'
### Unit Tests
# lint php code for syntax errors
$RUN_IN_CONTAINER 'composer -n lint'
# run unit tests
$RUN_IN_CONTAINER 'composer -n unit-test'
### Functional Tests
# build assets
$RUN_IN_CONTAINER 'composer -n build-assets'
# run composer install again to get dev dependencies
$RUN_IN_CONTAINER 'composer install'
# run functional tests
# $RUN_IN_CONTAINER '$SCRIPT_PATH/03-test'
# post-test actions
$RUN_IN_CONTAINER '$SCRIPT_PATH/04-post-test'
### Visual Regression Tests
# export env vars to a file outside the container
$RUN_IN_CONTAINER 'BASH_ENV=$HOME/project/bash_env /build-tools-ci/scripts/set-environment'
# run visual regression test container
docker-compose up visual-regression

We originally tried parallelization for unit, functional, and visual jobs, but the issue was it was very difficult to tell if the Pantheon MultiDev was initialized AND synced with the latest commits and database updates, since two of the tests run on that Multidev env. We had a working script that tested the output of drush cim -y; drush updb -y for “No changes” to tell if it was ready, but this was very fragile so we abandoned it. There seems to be no way in Terminus to tell if a MultiDev is done building, and then if code/db clone/sync is complete.

We are open to suggestions, but it’s likely we’ll be moving to Semaphore 2.0 in the future which should more natively support the CircleCI style pipeline.

And of course you’ll want to see the docker-compose.yml file :slight_smile:

---
version: '3.7'

x-common-variables: &common-variables
  - ADMIN_EMAIL
  - ADMIN_PASSWORD
  - CI_PROJECT_USERNAME=${DEPLOY_AUTHOR_NAME}
  - CI_PROJECT_NAME=${SEMAPHORE_PROJECT_NAME}
  - CI_PROJECT_REPONAME=${SEMAPHORE_PROJECT_NAME}
  - CI_BRANCH=${BRANCH_NAME}
  - CI_BUILD_NUMBER=${SEMAPHORE_BUILD_NUMBER}
  - GIT_EMAIL
  - GITHUB_TOKEN
  - SCRIPT_PATH=${CI_CONTAINER_HOME}/project/semaphore/scripts
  - PR_NUMBER=${PULL_REQUEST_NUMBER}
  - TERMINUS_SITE
  - TERMINUS_TOKEN
  - TEST_SITE_NAME

services:
  ci:
    image: quay.io/pantheon-public/build-tools-ci:4.x
    container_name: cms_ci
    command: sleep infinity
    working_dir: ${CI_CONTAINER_HOME}/project
    volumes:
      - ${SEMAPHORE_PROJECT_DIR}:${CI_CONTAINER_HOME}/project:rw
      - /home/runner/pantheon_semaphore.pem:${CI_CONTAINER_HOME}/pantheon_semaphore.pem
    environment: *common-variables

  visual-regression:
    image: backstopjs/backstopjs:3.9.5
    container_name: cms_visual_regression
    entrypoint: /src/semaphore/visual-regression-test.sh
    working_dir: /src
    volumes:
      - ${SEMAPHORE_PROJECT_DIR}:/src:rw
    environment: *common-variables