Upcoming changes to /recent endpoints for companies and deals

deals
companies

#1

Hi everyone,

We have some upcoming changes that we’re going to be making to the recently modified and created endpoints for companies and deals. Currently, these endpoints will allow you to pull records going back to the creation of your portal.

At the end of this month (October), we’ll be changing the endpoints so that they’ll only return records created or modified in the last 30 days. In addition, the endpoints will be limited to the most recently created or updated 10,000 records, even if there are more than 10k records in the 30 day period.
This affects the following endpoints:

Due to these changes, those endpoints should not be used to get all records from a portal, only the recently modified records. Because of this, we have new endpoints that can be used to get all of the companies or deals from a portal:

If there are any questions about these changes please let us know.


Important update for the Deals and Companies API changes
Get (ALL) Recently Created Companies?
Cannot fetch more than 10,000 records for endpoint
#2

@dadams Thanks for the early warning. This change affects a critical component of our reporting, so I’m glad I wasn’t blindsided by it. :slight_smile:

For http://developers.hubspot.com/docs/methods/deals/get-all-deals can you tell me if any assumptions can be made about the order in which the deals are returned? Thanks.


#3

@dadams It appears that includePropertyVersions=true is not supported by Get All Deals.

This is a critical feature of Get Recent Deals that we rely on for accurate reporting. Without it, Get All Deals cannot serve as a viable replacement. Please advise. Thanks.

(Side note: the ability to specify which properties to retrieve is a nice feature of Get All Deals, and should help us optimize payload sizes. I was surprised, though, that the parameter was called “properties” instead of “property” like other API calls that support this feature, such as /contacts/v1/contact/vids/batch/)


#4

@Bistream630 we’re going to take a look at adding the ability to get property versions to the Get All Deals endpoint, I’ll have an update on that here once that’s sorted out.

The new get all records endpoints are not returned in any reliable order. The records are basically returned in order by ID but that does not imply the order that the records were created in.

In general, you should not fully replace the /recent endpoints with the new endpoints. If you’re running a sync job that looks for new records periodically, you’d should continue using the recent endpoints, since that will be the best way to get recently updated or created records.

When paging through the results of the recent endpoints, you can check the hs_lastmodifieddate property of the records to determine how far back you are, so you know when you have all of updates since the last time you polled for changes.


#5

@dadams Thanks for the update. I’ll wait to hear from you on the addition of property version support. Thanks for explaining the order, as well.

I understand what you’re saying about continuing to use /recent for situations where only updated records need to be accounted for. Our dilemma right now is that we construct our extract using an API that only support append, which forces us to do a full refresh every time to account for any modified data. Our data set is small right now, but the solution will not scale well over time. We will eventually replace our current process with an approach that supports upsert, at which time /recent will be the preferred method of retrieval.


#6

Hi @dadams, any update on adding includePropertyVersions=true to the /get-all-deals endpoint? I’m still using /recent, and that end-of-October deadline to switch over is getting close. Thanks.


#7

@Bistream630 the new get-all endpoints actually use a different query parameter, tied to the individual properties you’re requesting. You can use propertiesWithHistory={property name} instead of properties={property name} to get the history of a specific property, so if you don’t need the history for a property (say for analytics or calculated properties), you can just get the most recent value for those properties to cut down on the response size.

https://api.hubapi.com/companies/v2/companies/paged?hapikey=demo&propertiesWithHistory=name&properties=website&properties=numberofemployees


#8

@dadams Awesome! That works great, and being able to specify whether to pull history per property is so full of win. Thanks!


#9

Previously /recent returns all deals ordered by modified date. If I have 1000 deals in total and need to retrieve 50 most recently modified deals, I am able to get exactly 50 deals from /recent endpoint without scanning through the entire 1000 deals.

Today, /recent endpoint can’t support the above scenario, because it limits to the last 30 days. If I use all deals endpoint, I have to retrieve all 1000 deals, do the order by myself, and get the 50 most recently modified deals. Although functionality wise, we can still use all deals endpoint to replace /recent and make the above scenario work, but in a much less efficient way. It not only increases the burden on 3rd party application (us), but also increases the server load for hubspot. Hubspot server didn’t have to do full deals table scan before, but now it has to. Bytes over the wire increase too, which means higher COGS for both hubspot and us.

By the way, our scenario mentioned above is an end user driven experience, so we can’t afford to build a service to constantly poll /recent endpoint and cache deals that falls off the 30 days window.

I’m wondering what the general direction Hubspot API is heading. Basic filtering and order by is such essential capability for READ API endpoints. Instead of increasing these capabilities, I feel it is actually decreasing. Is it the general guideline to leave filtering/ordering to the 3rd party applications to handle on their own? Is there any plan to add filtering and order by for all deals endpoint in the near future?


#10

@coffeeocean can you tell me more about how you’re using that data? Is part of the requirement for this that you get exactly 50 records, but you’re seeing cases where you have fewer than 50 records updated within 30 days?

Filtering is something that we’re planning to add for multiple objects. Aside from the date, is there any other specific data that you’d want to filter by?


#11

@dadams, thanks for the reply and attention! We read the most recently modified deals for reporting and analytics purpose. We parameterize the number of most recently modified deals, with a default value of top 50 (subject to change based on feedbacks). So we can describe it as “top N most recently modified deals” query instead of “exactly 50 records” :slight_smile: Depending on the purpose of the query, N could be any reasonable integer with 50 being the most frequent.

From our testing, before /recent endpoint will return 50 if there are more than 50 deals in total and it will return all deals if the total number of deals is less than 50. But recently, we noticed that it suddenly started to return nothing on some data set unexpectedly. After debugging and reading about the API change announcement, I realized that it was because the 30 days window kicked in.

It’s great to hear hubspot is planning to add filtering! We are looking forward to it! Besides modified date, filtering based on deal stage (pipeline stage) and close date would be really helpful! Thanks!


#12

Hi David,

We missed this. and it broke some parts of our pipeline. We really appreciate you all posting API changes before they’re shipped here, but there’s also a lot of other discussion in this forum. Is there somewhere we can specifically subscribe to breaking changes (e.g. via email)? Since this change was not released via a versioned endpoint, I believe this is crucial. Thanks again!


#13

@tejasmanohar, I recommend you browse to the Announcements category, click the Notification Mode (little circular icon next to the “New Post” button), and set it to “Watching First Post”. This should allow you to get an email notification anytime a new Announcement is posted. (Average volume is approximately 3 per month.)


#14

I have the same issue as @coffeeocean. I need the most recent 500 deals, some may be further back than one month. Is filtering for get all endpoints on the roadmap?