Hi @terrance! Happy to help in any way I can. My team and I are wondering, though, what exactly are you trying to do? And why exactly would you need the URLs from us?
We'd love to do anything we can to help but just need a bit more info. Would you also be able to respond with your HubID and if you're creating a HubSpot application, your AppID? Thanks so much
Essentially Hubspot isn't respecting the subdomain's robots file (it only looks at root level domain robots) and Hubspot is trying to crawl gated application pages. This is flooding error logs with 403's so we block them at layer 5.
Hey @terrance , welcome to the new forums! The previous forums are Read-only now which is why I picked up our conversation here. Thanks for the additional detail. I chatted with Derek and I'm also seeing that your issue was resolved internally with our team. Are you still actively seeing that the robots.txt file is not being respected?
For the UrlBlacklistsKafkaWorkers-urlchecksworker / WebhooksPlatformService-web links you sent over, those aren't webcrawlers and I'm not totally familiar with what they might do. Can you shed more light on what you're seeing and what kinds of requests you're seeing with respect to those services?