Large size migration script issues

companies
contacts
deals

#1

Hi,

I have two questions:

  • How do I know my API request consumption directly from HubSpot ?

  • How do I write a ‘request rate’ efficient migration script for a large amount of data ?

My case is the following:

I’m using the Sales Pro product as of now.

I have a large amount of users to sync, let’s say 5,000, for each one of them I need to create a contact, a company and link them together, also, if a deal exists with either of these objects, create an association between the deal and the two other objects and move it in a specific dealstage. So this is at least 4 to 5 request per business, which would put me (without counting the retries I have to do because of a mysterious “ACCESS DENIED” response I get if I send too many requests to hubapi) at 20-25k requests for the whole migration to run.
I send these to the contacts, companies and deals APIs.

Meanwhile the hubspot integration in our platform continues to sync new data on its own and adds requests to the counter during the whole ordeal.

Also for each of these users I have to transform their transactions from our platform into hubspot managed deals so we can have data to work with.

The thing is that some of these users actually have a lot of transactions linked to their accounts and there is no existing API endpoint to batch their creation so this is at least 1 request per transaction.

All these factors have put me at the DAILY request limit rate of 40k requests which is very blocking because it happened after only 1,500 users synchronisation due to the number of retries I had to do for some users and the inevitably large amount of requests they’ve triggered and I do not have the time to retry this every day until it’s fully synced.

I would appreciate any help or solution provided.

Regards,
Wilfried E.


#2

Hi Wilfried,

I highly recommend taking a look at webhooks.

I believe you can also see your api usage by looking here:
https://app.hubspot.com/developers-alpha/[Hub id]/application/[application id]/monitoring/webhooks

Thanks,
-Keith