This is driving me a little crazy. I've done extensive tests on our api and know we can handle up to around 300 reqs/s before the server starts lagging.
I tried setting the concurrent requests to 5, 10, 50, and now it's at 100.
The test I do every time is to connect one bare account to the app, and import 600 new contacts with first and last names. Usually around 4700 contacts worth of webhook batches successfully get pushed to my API. The other 1200 contacts timeout and retry like 8-9 times or more before successfully going through.
I'm seeing this message in the Webhook monitoring: "Webhook attempt timed out due to a backlog of concurrent requests."
I don't understand. Does this mean my API timed out? Does this mean there are too many batches being moved in HubSpot and it has to retry later. I assume it's the former.
If so, I have a few questions regarding the nature of the HubSpot Webhook Deamon:
- How many actual requests per second does it send if I set 100 concurrent requests?
- What is the actual timeout of the webhook? (I've seen 1 second here and 2 seconds there, no one really has an answer)
My api simply takes the json, parses it (takes microseconds to do that) then pushes them into our queue system (which also takes 1-10 ms max). The DNS is through cloudflare and I've done ample speed tests on the server. I also have around 6 replications of the API I'm using which load balances the whole thing out.
Need some light shed on these issues. When I have 500 users I don't want to be battling these problems.