Pantheon Community

Tips for making site traffic stats more accurate

Seems like a lot of us are sharing similar concerns with Panethon. Hope they respond very soon or else we’re going to have move several clients off of Pantheon.

Hi @AllysonKapin!

We are investigating ways to expose that data so you can get a fuller picture of your site traffic, but I can’t make any promises about when it will happen or how that will work.

This thread has the most information about the subject, both in terms of struggles other customers are facing and in terms of folks at Pantheon chiming in. I hear your frustration around the pricing changes and around the difficulty getting clarity here. We are working on it and I appreciate you voicing your concerns.

1 Like

Thanks for responding @sparklingrobots . We’ve been hearing this from Pantheon since November and nothing has changed. I realize you are not the one in charge of pricing, but I want to let you know that many of us are losing confidence in Pantheon.

To put in automated price increases without having had a plan to provide clients with detailed traffic breakdowns for each site is not acceptable. Pantheon needs to do the right thing and suspend automated price increases until they figure this out.

1 Like

I 100% agree with @AllysonKapin.

We have also been experiencing similar issues with a few of our websites. All of the IP address’ seem to be coming from some form of *.*.*.*.bc.googleusercontent.com. Does anyone know if these are coming from pantheon or are they possibly malicious?

These 5 ip address’ seem to be causing most of the traffic on of my sites.

104.198.178.229
34.69.27.16
34.69.52.2
35.232.124.130
104.197.66.110
1 Like

@sparklingrobots,

While I realize that the product team is looking into providing tools for us to effectively see our traffic so that we can potentially act to reduce it (or least explain it), I think a lot of us would appreciate a roadmap with target dates. I get that early research means that “you don’t know what you don’t know” and that committing to a date is risky, but I’d say that this is the business we’re all in and you could keep the goal fairly high level to start and the revise the roadmap with more specifics research yields answers. Is this roadmap possible. If so, when can we expect that?

This would at the very least give us something to discuss with our clients in the short term, because as of now telling them that “they’re working on it” isn’t effective.

Thank you @sparklingrobots

5 Likes

I hear you on that and am doing my best to get some more specific info @stephencapellic . Thanks for your candid thoughts and I hope to have more soon.

1 Like

I’m just adding my voice to this discussion, just stumbling on it today (after blasting Josh’s DM with my frustration on this issue. sorry about that!). First, I’m glad to know it’s not just me but I’m also sad to know it’s not just me. I’ve had a few clients now with this issue, who have been bumped. I’m working with a particularly large client that I now have concerns with. They’re already on a large plan and I’m not sure that it will be enough given the gross disparity in figures.

I’m all for fairness in pricing but we’re trying to solve for something that we don’t have the right tools to resolve and taking it in the pocketbook in the meantime.

3 Likes

Hi folks. We talked with the Pantheon team by phone earlier today about the price increases that many of us have been experiencing and expressed concerned about. On the call, Pantheon said that several clients have been able to reduce their traffic and bring their plan back to a normal level. Has anyone experienced this on our slack channel? And if so, what strategies did you implement to achieve this?

3 Likes

With several client sites, we’ve attempted every suggestions provided by Pantheon, with no effect. Many, if not most, efforts are stymied by lack of data. In other words, we still do not know what accounts for the exponential differences between traffic reported by Google Analytics and Pantheon. I, too, would like to hear specific success stories.

3 Likes

Agree @anne. We 're in the same boat as you.

After requesting and receiving a snapshot of our log from Pantheon, we followed their recommendation to block the most common crawlers in our robots.txt file (on Jan. 29 as highlighted on the graph). It hasn’t made any perceptible difference.

1 Like

I hear you @ruby . We have tried several solutions as well. After hearing from Pantheon that several clients have managed to get their traffic back down to normal plan levels, I figured I would post here to see if anyone who did achieve this would share it back out with the community.

1 Like

Hey @AllysonKapin. We have tried to block IP’s and added multiple entries to our robots.txt file and nothing has really worked. In fact this month (which isn’t even over) has more traffic than the subsequent months.

2 Likes

Same issues here - client’s plan was auto-bumped up to performance large from small (x3 in cost per month). There are the nginx server logs, which you can download via SFTP, and there are the global CDN logs, which, as I understand it, determine our cost/plan - please let me know if that’s not correct.

I requested the global CDN logs on Jan 24th and still haven’t received them. So I analyzed the nginx log requests from one of the highest request days, Jan 28th.

Here is the pantheon metrics view for that day via dashboard and the nginx logs (combo of 2 nginx log files - sorry, this image may appear smaller than the 1500px wide one I uploaded)

Sum of total requests for Jan 28th: 7821

Not sure why the metrics are showing 2000 (visits)-10,000 (served) more requests for that day(?)

In analyzing the logs, I found one bot (mj12bot) that I decided to block via robots.txt.

User-agent: MJ12bot
Disallow: /

But this is like whac-a-mole, as other bots will pop up in the future, and the robots.txt file can be ignored, so it will need to be monitored. I thought an .htaccess config allow me to block IPs, but Pantheon uses nginx (not apache), and you have to block IPs via the settings.php file for Drupal sites.

Also, and more importantly, I noticed this bot and others were pinging recursive (yet still accessible) Drupal views URLs, which has been a crawling issue with Drupal views before (/resource/resource/resource/resource…). I found this Drupal.org post that offers a resolution via views contextual filter.

I’m hoping that robots.txt and this views fix will help deal with the extra requests, and possibly help others trying to bring their requests back down. I’ve considered implementing Cloudflare, to easily setup page rules, but not sure I’m at that point yet.

1 Like

I had a conversation with our Pantheon sales rep and an engineer about this topic. Of the several sites my organization has on Pantheon, the least-visited one (according to GA, anyhow) has this huge disparity in metrics. They were kind enough to provide me with a one-day snapshot of the edge logs, in spreadsheet form, that suggests the issue is due to a bot net in Singapore that uses old Android phones. While this traffic isn’t adversely affecting pages served, it is skewing the numbers for visits in Pantheon’s metrics.

The suggested solutions for dealing with the traffic issues were to switch to a Performance XXL plan or deploy Pantheon’s Advanced CDN to filter out the offending traffic. “Right sizing” to the larger plan would be the less expensive option of the two. I suppose another option is to deploy Cloudflare on top of Pantheon’s CDN and try to block the traffic there.

None of these options are things we have budgeted for, however.

1 Like

Here’s what is concerning about this post.

Kind enough? Really? While I appreciate and advocate for kindness, shouldn’t we be concerned that this took an extra effort on Pantheon’s part to provide this data?

“One-day snapshots” may or may not reveal real issues. We need to be able to regularly evaluate all “traffic.”

How many of us would make different decisions if we knew what the real cost of the “right-sized” Pantheon service would be?

Maybe it’s because it’s Monday, or maybe this message has put me over the tipping point. Either way, as much as I love the concept of what Pantheon provides, I’m closer and closer to deciding that it is really more than my small clients can afford and different services are a better match for their needs and budget.

(My opinion only. I do not speak for any company that I do contract work for.)

1 Like

I was just sharing my experience with the matter for the record.

So you don’t misunderstand me: I’m not happy about either of the proffered “solutions” and overall find the responses to the challenges surrounding this issue to be rather lame. Perhaps I was a bit too subtle (I do tend to go back and edit the angry bits before posting to a public forum).

I’ve already moved one project off Pantheon to a competitor due to this. The experience leading up to that was even more frustrating. I’d prefer not to go into it here but I’m happy to share offline.

1 Like

Andrew, indeed thank you for being brave enough to share your experience and commend you for your carefully chosen words. I think I read between the lines and projected my experience on your story.

My primary point was intended to be this: Total cost is a factor in deciding to choose a hosting/development platform. We each made a choice for Pantheon based on assumptions about what the traffic count would be. When those assumptions proved false (which they have invariably have), we’re faced with only unsavory options: expend time and effort to understand where our assumptions were wrong and try to “fix” it, expend time and effort to move somewhere else, expend more money for “right-sizing,” or maybe something else that I haven’t experienced yet.

2 Likes

Hi, all. Just a quick note to say that we are still listening, and still talking internally about how we can help.

General note: If you have concerns about a specific plan right-sizing, reply to the email you received and let us know what your concerns and questions are and we’ll take it from there.

I greatly appreciate the candid discussion happening here. Thank you.

3 Likes

Thanks @sparklingrobots! I feel as though in the meantime the automatic plan “right-sizing” should be suspended if there hasn’t been a consensus reached by pantheon. We are getting closer to the end of this month and a lot of sites will be automatically bumped up without a thorough explanation.

5 Likes