HubSpot crawling links in emails


#1

I noticed our servers are getting HEAD requests from the following UA “HubSpot Connect 2.0 (http://dev.hubspot.com/) - UrlBlacklistsKafkaWorkers-urlchecksworker”. Hubspot seems to be crawling links in emails sent through the transactional API.

Why is Hubspot crawling links sent in email? Will a standard HTTP 405 response stop this behavior? Or will the crawler see a HTTP 405 response, from a HEAD request, and start attempting GET or POST requests?


#2

Hi @tomdev,

We crawl all links in emails to make sure they aren’t phishing links; it’s not currently possible to prevent this behavior. The service will only ever send HEAD requests, so you should be able to return a HEAD response for efficiency.