Pantheon Community

When custom upstreams are not the right answer

Custom Upstreams is one of Pantheon’s most powerful features. Custom Upstreams allow an organization (a university, a large company) to provide their own copy of Drupal or WordPress that includes all the modules/plugins/themes/config necessary to achieve a certain use case.

Custom Upstreams are essentially a productized way to distribute a Drupal installation profile. On a literal level, an upstream is a git repo that shares a history with all of the child sites made off the upstream. With clear rules and expectations around which files/directories are updated in the child sites (or in the upstream) you can cleanly git merge in core updates from the upstream into the child sites.

In my job I’m regularly scheduled into meetings with current or potential Pantheon customers where I give guidance on when to use upstreams. I’d like to know if anyone has more caveats/questions/disagreements with my heuristics:

IMO custom upstreams are more helpful when

  • There is a large number of sites running on the shared code base
  • There is a very clear line between which code is expected to change in a site-specific way
  • There are clear expectations around how config changes made in the update will move into pre-existing sites.

I often discourage upstreams for reasons like

  • Not enough sites. If only a handful of sites are going to use the upstream, you might not be saving any time by doing code reuse this way.
  • You only want a start state. Much of the value from custom upstreams comes from the update mechanism they provide. If you just want a boiler plate start state with preferred modules/plugins/themes I think there are better ways.
2 Likes

You only want a start state. Much of the value from custom upstreams comes from the update mechanism they provide. If you just want a boiler plate start state with preferred modules/plugins/themes I think there are better ways.

What are some examples of those better start states?

We used to have a master repo that we would clone that included our preferred custom and contrib code along with a database snapshot that had default settings we preferred. Once the project was started, the team could add/remove/adjust as needed. That worked well for us, but it’s really the only other way of doing this that comes to my mind…

Yeah, I see a few options for start states:

  • Full project boilerplate: I think this is what you’re describing as what you’ve done before. Basically a repo or some artifact that gets the project started with trusted defaults that can be tweaked. Our example-drops-8-composer and example-wordpress-composer repos are both instances of this pattern.
  • External Composer Package: Use a tiny external Composer package to list all the things you want in each new project: https://github.com/stevector/example-acme-d8-recommendations This option has the benefit of update-ability.
  • Just copy the last/most similar project: I would guess nearly every Drupal and WordPress developer has done this one. Just copy the real site of your most recent/similar client and delete stuff until you’re at a decent starting point. This option is probably the most common because it requires the least forethought.

Two more caveats to discourage/encourage Custom Upstreams:

  • Worry more: If you’re going to start a custom upstream, make sure you’re confident the result will put you on the sunny side of these XKCD graphs: https://xkcd.com/1319/ https://xkcd.com/1205/
  • Worry less: If any site starts to break the upstream mold too much you can always take the site off the upstream with terminus site:upstream:set
1 Like

I was just having a verbal conversation about upstreams and another angle came up that I’ve had to repeat often: Please don’t architect a distro/upstream for many Drupal 8 sites until you have experience building one-off Drupal 8 sites.

I’ve talked to a handful of teams (particularly in Higher Ed) who are on Drupal 7 and planning a migration of many sites to D8. They can get stuck on assumptions about code/config reuse that are grounded in D7. Experiencing how Composer/config management work for a single site in D8 is necessary before planning how to use them for many sites.

2 Likes

I’m looking to use custom upstreams for two closely related sites (more potentially in the future), that will share the same custom base custom themes and base functionality.

They’ll use the same modules, but will have unique themes (and DBs).

I’m using a composer-based workflow, and I’m also running each site locally using Lando.

One problem I need to address is how config is managed between sites. I’d like to use CMI for the child sites, but I’d also like to use CMI to set the initial site state and possibly transfer shared feature-specific config.

Would be great to hear about any similar experience others have had trying to achieve the same.

We run 280 sites and honestly wouldn’t be able to do it without custom upstreams. We haven’t tackled the entire CMI-per-site question, and am also curious what experience other people have had.

@lukus, Do you plan to make site-specific Composer changes? If so, I that will make Custom Upstreams harder to use. Changing composer.json or composer.lock in the individual sites will result in git merge conflicts when composer changes are made in the upstream. From what you’re describing of sharing a config start-state, custom modules, base themes, and probably common contrib dependencies you might be better off building an install profile repo and composer requiring it into the individual sites.

@RHL, 280 sites! That’s what custom upstreams are made for! Are they Drupal 8? How do you handle config currently?

D7 at the moment, we are building out our D8 Core over the next several months.

@steve.persch

My motivation behind having a common upstream is to share a common codebase. I’d traditionally look towards a multisite setup, but understand it would be difficult to run (and make decent money from) a SAAS platform when multiple sites can be run from one app instance.

  • While a starter repo would be useful to hit the ground running, we’d have to put processes in place to stop each site from diverging too much from the other.
  • I’m happy for both sites to share composer.json setup. If one of the sites doesn’t require a certain module or library installed via composer, I’ll simply leave it disabled.
  • The base theme will provide shared preprocessor functions, library config and TWIG templates. Each site’s subtheme will then provide site specific modifications (TWIG, CSS, assets etc). I also intend to share custom modules between each site.

Totally get your point about local changes to composer.json … however, if a merge conflict between upstream and the local repo occurred, would it be relatively easy to resolve this?

Also, it sounds like multisite with D8 improves upon D7 by allowing shared CMI config (using https://www.drupal.org/project/config_installer).

I’d be really interested in finding out if Pantheon custom upstreams could be used with a similar setup. Loving Pantheon so far, would be great to share config between sites hosted on Pantheon.

@lukus, I think the install profile concept is your best bet for where to put the shared config. config_installer would let you put it anywhere but from what you’re describing, you are essentially building an install profile/distribution. That will work fine with custom upstreams. And from a directory organization perspective I would recommend putting the base theme and and shared custom modules inside the install profile as well.

1 Like

@RHL, are you building any one-off D8 sites before migrating the 280 sites to D8? I really recommend doing so to get comfortable with Composer and config management.

Given that config installation is supported in Core now, does that change your workflow?

@RHL, I’m glad that feature is now in core. However, I don’t think it changes the workflow compared to how I used it when it in contrib: https://www.stevector.com/2016/06/drupal-7-to-8-migration-diary-part-2-using-configuration-installer-to-delay-a-canonical-database/

For building a new D8 site it is nice to be able to do CI without a canonical DB.

Thanks, that makes a lot of sense. Will try that and report back.

We haven’t touched CI yet, how simple is it to get going on Pantheon?

@RHL, here’s our example repo that shows a CI workflow for a single site: https://github.com/pantheon-systems/example-drops-8-composer

CI with a custom upstream is a little different in that the CI process is probably building the artifact/branch on GitHub that is the upstream brought into child sites.

We’re moving from custom upstreams to a composer workflow. The current plan is to have a common starting point, with updates managed on individual sites using composer locally. Any improvements or additions to the starting point will only apply to new websites.

If an existing website gets a really cool feature we can add it manually to the starter website.

Profiles could be a cleaner option.

1 Like

We’ve now started 3 composer based projects with Pantheon. The worklow is a little trickier if you’re not using an upstream. For example, to obtain the git connection details you need to use your browser’s dev tools to enable the “Connection Info” button so you can click it. Once the code has been committed you can import the database and go from there.

OK the method I described doesn’t work properly as the infrastructure won’t be configured correctly. Instead, create a new site using an empty upstream of your require platform, in our case Drupal 8, to make sure the configurations for NGINX etc. are set up properly.