APIs & Integrations

terrance
Participant

What are all of the possible Hubspot User Agents?

We are blocking Hubspot requests for gated application URL's that Hubspot is trying to call, but want to perfect the regex in blocking them.

HubSpot Connect 2.0 (http://dev.hubspot.com/) - UrlBlacklistsKafkaWorkers-urlchecksworker
HubSpot Connect 2.0 (http://dev.hubspot.com/) - WebhooksPlatformService-web
HubSpot Links Crawler 2.0 (http://www.hubspot.com/)

Any more?

0 Upvotes
3 Replies 3
cbarley
HubSpot Alumni
HubSpot Alumni

What are all of the possible Hubspot User Agents?

Hi @terrance! Happy to help in any way I can. My team and I are wondering, though, what exactly are you trying to do? And why exactly would you need the URLs from us?

We'd love to do anything we can to help but just need a bit more info. Would you also be able to respond with your HubID and if you're creating a HubSpot application, your AppID? Thanks so much

0 Upvotes
terrance
Participant

What are all of the possible Hubspot User Agents?

Hey @Connor_Barley, I took this convo offline last year with @Derek_Gervais : HubSpot Links Crawler 2.0 user agent hitting protected pages

Essentially Hubspot isn't respecting the subdomain's robots file (it only looks at root level domain robots) and Hubspot is trying to crawl gated application pages. This is flooding error logs with 403's so we block them at layer 5.

0 Upvotes
cbarley
HubSpot Alumni
HubSpot Alumni

What are all of the possible Hubspot User Agents?

Hey @terrance , welcome to the new forums! The previous forums are Read-only now which is why I picked up our conversation here. Thanks for the additional detail. I chatted with Derek and I'm also seeing that your issue was resolved internally with our team. Are you still actively seeing that the robots.txt file is not being respected? 

 

For the UrlBlacklistsKafkaWorkers-urlchecksworker / WebhooksPlatformService-web links you sent over, those aren't webcrawlers and I'm not totally familiar with what they might do. Can you shed more light on what you're seeing and what kinds of requests you're seeing with respect to those services? 

0 Upvotes