Companies API not returning all companies


#1

Hello, for an offline process I need to sync the local database with some info from HubSpot companies. My code looks something like this:

const qs = require('querystring');
const rp = require('request-promise');
this.companiesURL = 'https://api.hubapi.com/companies/v2/companies/paged/';

this.companiesParams = {
  hapikey: this.hapikey,
  properties: ['name', 'nit', 'hubspot_owner_id', 'companyId', 'ciclo_de_vida_customer_success', 'hs_num_child_companies']
};
this.companiesOptions = {
  method: 'GET',
  uri: this.companiesURL + '?' + qs.stringify(this.companiesParams),
  json: true
};

async getCompanies() {
let hasMore = true;
while (hasMore) {
  let companiesResponse = await rp(this.companiesOptions);
  for (let company of companiesResponse.companies) {
 // Stuff I need to get
  }
  hasMore = companiesResponse["has-more"];
  this.companiesParams.offset = companiesResponse.offset;
  this.companiesOptions.uri = this.companiesURL + '?' + qs.stringify(this.companiesParams);
}

Even thought I'm correctly handling the 'has-more' and 'offset' features, after around 9-10 loops the 'has-more' variable comes as 'false' but in total I'm only getting 730 out of the 960 companies in my HubSpot. The ones left out seems to always be the same based on some random checks I've done. Is there anything wrong with my code? Am I missing some property of the left out companies?

Thank you!


#3

Welcome, @dramirez!

Nothing in your code jumps out at me at the moment. Can you share the following details so I can investigate further?

  • Your Hub ID.
  • Links to some of the missing companies you've identified.
  • The offset value for which you're consistently seeing "has-more": false returned.

#4

Hello! I can't seem to log into my company's HubSpot at the moment, I'm getting some errors. When I can finally log in I'll PM you the info. Thanks!


#5

Hey, @dramirez

I don't have direct messages enabled at the moment, but you can share the information in this thread — it's not identifying and no one will be able to log into your account!


#6

Hello! Here is the info:

-My Hub ID: 3907023
-2 companies I've notice are missing: https://app.hubspot.com/contacts/3907023/company/752046128/
https://app.hubspot.com/contacts/3907023/company/752045786/
-Offset value: 974550629

Thanks.


#7

Hey, @dramirez.

Apologies for the delayed response.

I'm not quite sure where the breakdown in your code is occurring, but I was able to retrieve all 979 companies from your account both manually and programmatically.

  • I made 10 calls to this endpoint via Postman. Call 9 returned 100 records and "has-more": true, "offset": 974550629, so I made the final call 10 to https://api.hubapi.com/companies/v2/companies/paged?offset=974550629, which returned the last 79 companies and "has-more": false, "offset": 1087573554.
  • I then ran the Node.js code below (based on Connor's repo here) through my console and retrieved all 979 companies, including this record and this record:
require("dotenv").config();
const axios = require("axios");
const fs = require("fs");
const hapikey = "{your_api_key}"; //process.env.API_KEY;

const property = "name";
const object = "companies";
const companiesUrl = `https://api.hubapi.com/companies/v2/companies/paged?hapikey=${hapikey}&limit=100&properties=${property}`;
const arr = [];

const fetchData = (url) => {
  return axios.get(url).then(response => {
    const data = response.data;
    data[object].forEach((item) => {
      if(item.properties[property]){
        if(item.properties[property].value){
          arr.push(item.properties[property].value);
        }
      }
    });
    const newUrl = `${companiesUrl}&offset=${data["offset"]}`;
    if (data["has-more"] === true) return fetchData(newUrl);
    else return data;
  })
};

const main = async () => {
  fetchData(companiesUrl).then((result) => {
    let now = new Date().toString();
    let log = `Time Now: ${now}. ${object.charAt(0).toUpperCase() + object.slice(1)} with value returned: ${arr.length}`
    console.log(log);
    console.log(arr);
    fs.appendFile("server.log", log + "\n\n" + arr, (err) => {
      if(err) {
        console.log("Unable to append to server log");
      }
    });
  }).catch((error) => {
    console.log(error);
  });
}

main()
  .then((response) => {
    console.log("STARTING SERVER");
    console.log("===============");
  });

Screenshot%202