Pantheon Community

Best SEO Practices for Dev, Test & Live

So, I custom domain names setup for my dev, test, and live environments in Pantheon and need to make sure that dev and test are not being crawled by search engines.

https://dev.example.com
https://test.example.com
https://www.example.com

Do I need to make sure that dev and test are checked to: discourage search engines from crawling them? Will this affect anything when I am pushing from dev to test to live. I’m guessing that is fine since that setting is stored in the database. It just becomes annoying when I clone the db from live back to dev or test and then need to go make that change again.

Any recommendations? I currently have dev and test set to “locked”.

Hi Luke,

Take a look at https://pantheon.io/docs/bots-and-indexing/#indexing-your-pantheon-site. It mentions robots.txt for pantheon environments and also your custom vanity domains.

But, if I update the robots.txt in dev to disallow it will want me to push those code changes to test and then live.

From my understanding you should not have to touch your robots.txt. The one in your codebase should be the one that you want for www.example[dot]com and Pantheon will provide their modified robots.txt for dev/test and dev.example[dot]com and test.example[dot]com.

Ah, you are correct — I just confirmed with Pantheon support that DEV and TEST will not index by default.

2 Likes

Only the Live environment will index. All other environments, including multidev environments, will have a “x-robots-tag: noindex” header.

I would also suggest adding any custom robots.txt rules via the custom function https://pantheon.io/docs/bots-and-indexing/#incorrect-robotstxt-output-in-wordpress instead of adding a robots.txt file. By using the custom function, you can append any robots.txt rules you need after Pantheon’s default robots.txt rules.

Dan,

If I update the functions file in my theme in Dev, wouldn’t that push up to test and live?

The custom function for the robots.txt file will only show up in the Live environment. If you committed the custom function to Dev you will need to deploy it to Test/Live.