APIs & Integrations

Karthick_Shiva
Member

504 - Gateway Timeout occurs frequently

Earlier one of your developers said that 502/504 timeout errors will be minimised. We have some amount of customers using our HubSpot Integration. Of them, customers having a large volume of data frequently face timeout errors while using get all/recently modified contacts, companies, deals API. Though we’re working within your API limits, we are facing this issue frequently. Some of our customers are feeling dissatisfied with our integration due to the frequent occurrence of such errors. Kindly guide us to make this work on the permanent basis.

0 Upvotes
36 Replies 36
IsaacTakushi
HubSpot Employee
HubSpot Employee

504 - Gateway Timeout occurs frequently

Welcome, @mattb. Apologies for the delayed response.

This topic discusses 504 Gateway Timeout errors, not 405 Method Not Allowed errors. Did you mistype the error code?

Either way, please see Derek's post here:

If you are encountering 504 Gateway Timeout errors, please create a new topic with the following information:

  • The public IP address your traffic is coming from.
  • The full URL of the API you're hitting.
  • What kind of client you're using to hit the API (e.g. Java 8 with HttpClient, curl 7.55, etc).
  • Any error messages you're getting, including the "Ray ID" if it's present.
  • For timeouts and network errors, a traceroute from your machine to our API (e.g. traceroute api.hubapi.com ).

If you are encountering 405 Method Not Allowed errors, please create a new topic with the following information:

  • Your Hub ID.
  • The full URL of the API you're hitting.
  • The method you are using (GET, POST, or PUT).

Isaac Takushi

Associate Certification Manager
0 Upvotes
Mattb3
Member

504 - Gateway Timeout occurs frequently

We received the 405 error while demoing one of our sales pages.

Is there any eta on when this will be fixed?

0 Upvotes
Karthick_Shiva
Member

504 - Gateway Timeout occurs frequently

Hi,
I have found a workaround. Whenever the error occurs, I catch the exception and sleep for 2 seconds and retry again. This works for me every time.

0 Upvotes
mfleming-app
Participant

504 - Gateway Timeout occurs frequently

Ended up doing the same thing... Breaks out of the inner loop once the url is loaded and continues with the rest of the script.

while HasMore:
while True:
try:
xurl = "/engagements/v1/engagements/associated/CONTACT/" + str(c['vid']) + "/paged"
OffsetProperty = '&offset=' + str(OffsetNumber)
url = HS_API_URL + xurl + APIKEY + limit + OffsetProperty
print 'loading url: ' + str(url)
response = urllib2.urlopen(url).read()
engagements = json.loads(response)
if engagements['hasMore'] or engagements['hasMore'] == False:
break
except Exception as E:
print str(e)

0 Upvotes
Derek_Gervais
HubSpot Alumni
HubSpot Alumni

504 - Gateway Timeout occurs frequently

Hey all,

Just wanted to hop in here again since this topic is getting very crowded: It's important to differentiate between a 'normal' and 'abnormal' level of these timeouts. While a 504 or 524 is indicative of a server-side issue, some small number of timeout errors are always expected on the Internet. This is normal and not something we can prevent 100% of the time. It's important to be able to gracefully handle these errors; in the overwhelming majority of cases, we see successful responses after a single retry.

That said, high error rates can be indicative of something else going wrong. If you're seeing success rates lower than ~99.99% (i.e. significantly more than 1-2 504/524 errors for every 10,000 requests) then we should dig into things further to make sure there isn't a more severe issue.

If you're seeing abnormally high error rates, it's likely because the thing you're doing is causing some timeout on the HubSpot side; temporarily slowing down your requests and/or lowering batch sizes is generally the best remedy for these issues.

Going forward, anyone that's experiencing abnormally high rates of 504 and/or 524 errors should create a separate thread with the following information:

  • The public IP address your traffic is coming from
  • The full URL of the API you're hitting
  • What kind of client you're using to hit the API (e.g. Java 8 with HttpClient, curl 7.55, etc)
  • Any error messages you're getting, including the "Ray ID" if it's present
  • For timeouts and network errors, a traceroute from your machine to our API (e.g. traceroute api.hubapi.com)
0 Upvotes
mfleming-app
Participant

504 - Gateway Timeout occurs frequently

Hey Derek,

Getting a large number of 504 errors on the get associated engagements endpoint. It's unusable in its current state.

Error
urllib2.HTTPError: HTTP Error 504: Gateway Time-out

The public IP address your traffic is coming from
50.78.250.202

The full URL of the API you're hitting
doesn't include key
https://api.hubapi.com/engagements/v1/engagements/associated/CONTACT/55126251/paged

What kind of client you're using to hit the API (e.g. Java 8 with HttpClient, curl 7.55, etc)
Python 2.7 - urllib2

For timeouts and network errors, a traceroute from your machine to our API (e.g. traceroute
Traceroute has started…

https://pastebin.com/MjRtPaiZ

-Matt

0 Upvotes
mfleming-app
Participant

504 - Gateway Timeout occurs frequently

Any updates? I'm still encountering 504s. Thanks!

0 Upvotes
pbunce
Member

504 - Gateway Timeout occurs frequently

We are getting 504s frequently for one of our clients, usually when using the API to get contacts, but sometimes when using the bulk API to send back data. Examples from yesterday below.

Request:

ERROR HubspotAdapter: >> GET https://api.hubapi.com/contacts/v1/lists/all/contacts/all?property=country&property=rf_cdm_isthisrec...
operty=rf_techdns&property=rf_dhq_sicname&property=rf_dhq_familymembers&property=firstname&property=rf_emailgrade&property=rf_dhq_locationtype&property=rf_cdm_matchgrade&property=rf_dhq_phone&property=rf_site_annualrev&property=rf_site_co
untrycode&property=rf_dhq_customindustry&property=rf_dhq_employeeshere&property=rf_site_sicname&property=zip&property=rf_dhq_emptotalrange&property=website&property=jobtitle&property=rf_site_tradename&property=rf_site_revenuerange&propert
y=rf_site_country&property=rf_dhq_state&property=rf_site_subcode&property=fax&property=email&property=rf_site_statecode&property=rf_dhq_siccode&property=rf_site_zip&property=rf_dhq_yearstart&property=company&property=salutation&property=r
f_site_url&property=rf_inferredpublic&property=rf_dhq_country&property=rf_site_yearstart&property=phone&property=rf_inferredindustry&property=city&property=rf_cdm_matchscore&property=lastname&property=rf_cdm_duplicateids&property=rf_dhq_a
ddr1&property=rf_dhq_addr2&property=rf_dhq_city&property=rf_site_familymembers&property=rf_dhq_tradename&property=mobilephone&property=rf_site_state&property=rf_site_addr2&property=rf_site_addr1&property=rf_site_customindustry&property=rf
_site_city&property=rf_site_employeestotal&property=rf_dhq_employeestotal&property=rf_site_phone&property=rf_dhq_countrycode&property=rf_cdm_include&property=rf_inferredemployees&property=rf_site_employeeshere&property=rf_cdm_emailquality
&property=rf_dhq_naicsname&property=rf_dhq_naicscode&property=rf_site_emptotalrange&property=state&property=rf_dhq_subcode&property=rf_cdm_modifieddate&property=rf_cdm_junkreason&property=rf_site_naicsname&property=address&property=rf_dhq
_statecode&property=rf_site_company&propertyMode=value_only&count=100&vidOffset=33144899

Error: Caused by: com.reachforce.xds.base.ExternalSystemException: External Server Error: 504 [Gateway Timeout]

          <html>
          <head><title>504 Gateway Time-out</title></head>
          <body bgcolor="white">
          <center><h1>504 Gateway Time-out</h1></center>
          <hr><center>cloudflare</center>
          </body>
          </html>

Example bulk upload:

ERROR HubspotAdapter: >> POST https://api.hubapi.com/contacts/v1/contact/batch/?auditId=10950-1529957960027

Error response:

          <span class="cf-error-type">Error</span>
          <span class="cf-error-code">504</span>
          <small class="heading-ray-id">Ray ID: 430a7b1e286b246e &bull; 2018-06-25 21:20:56 UTC</small>
        <h2 class="cf-subheadline">Gateway time-out</h2>
            <span class="cf-status-desc">You</span>
            <h3 class="cf-status-name">Browser</h3>
            <span class="cf-status-label">Working</span>

            <span class="cf-status-desc">Ashburn</span>
            <h3 class="cf-status-name">Cloudflare</h3>
            <span class="cf-status-label">Working</span>
          
            <span class="cf-status-desc">api.hubapi.com</span>
            <h3 class="cf-status-name">Host</h3>
            <span class="cf-status-label">Error</span>
          
                              <h2>What happened?</h2>
                <p>The web server reported a gateway time-out error.</p>

                <h2>What can I do?</h2>
                <p>Please try again in a few minutes.</p>

              <span class="cf-footer-item">Cloudflare Ray ID: <strong>430a7b1e286b246e</strong></span>
              <span class="cf-footer-separator">&bull;</span>
              <span class="cf-footer-item"><span>Your IP</span>: 52.71.123.54</span>
              <span class="cf-footer-separator">&bull;</span>
              <span class="cf-footer-item"><span>Performance &amp; security by</span> <a               href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
0 Upvotes
Karthick_Shiva
Member

504 - Gateway Timeout occurs frequently

Still these errors are not minimised. We get 502, 504 and 524 errors frequently.

The connection to the origin web server was made_ but the origin web server timed out before responding. The likely cause is an overloaded background task_ database or application_ stressing the resources on your web server. To resolve_ please work with your hosting provider or web development team to free up resources for your database or overloaded application.

This was the error message thrown while we got 524 error.

0 Upvotes
AadDoix
Member

504 - Gateway Timeout occurs frequently

@Derek_Gervais, is there any more information (another topic because this one is about the 502 / 504 error) about that 524 error message? I myself do frequently experience an strange error called "invalid JSON markup" using Cdata's connector which according to Cdata support should be caused by an HTTP-524 error. I can not solve it yet.

Thanks in advance and kind regards,

Aad Dijksman

0 Upvotes
Derek_Gervais
HubSpot Alumni
HubSpot Alumni

504 - Gateway Timeout occurs frequently

Hi @flemingapptopia,

You don't happen to have more details than that, do you? I would expect a bigger error response with more details. Also, are you only seeing this with the Engagements API?

@DavidSickmiller thanks for the info; was there more to the error response? Any 524 errors have different origins than 504 errors, and there's often more info in the full error body. Are you only seeing this with the Deals API?

0 Upvotes
Not applicable

504 - Gateway Timeout occurs frequently

Here's an example response of a recent 524 on https://api.hubapi.com/deals/v1/deal/18175226:

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<meta http-equiv="refresh" content="0">

<title>api.hubapi.com | 524: A timeout occurred</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" 
media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" 
type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>

</head>
<body>
<div id="cf-wrapper">

<div id="cf-error-details" class="cf-error-details-wrapper">
    <div class="cf-wrapper cf-error-overview">
        <h1>
          
          <span class="cf-error-type">Error</span>
          <span class="cf-error-code">524</span>
          <small class="heading-ray-id">Ray ID: 41ec0c39bedb7a05 &bull; 2018-05-22 03:04:46 UTC</small>
        </h1>
        <h2 class="cf-subheadline">A timeout occurred</h2>
    </div><!-- /.error-overview -->
    
    <div class="cf-section cf-highlight cf-status-display">
        <div class="cf-wrapper">
            <div class="cf-columns cols-3">
              
<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">You</span>
  <h3 class="cf-status-name">Browser</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">Seattle</span>
  <h3 class="cf-status-name">Cloudflare</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc">api.hubapi.com</span>
  <h3 class="cf-status-name">Host</h3>
  <span class="cf-status-label">Error</span>
</div>

            </div>
          
        </div>
    </div><!-- /.status-display -->

    <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
            <div class="cf-column">
                <h2>What happened?</h2>
                <p>The origin web server timed out responding to this request.</p>
            </div>
          
            <div class="cf-column">
                <h2>What can I do?</h2>
                      <h5>If you're a visitor of this website:</h5>
  <p>Please try again in a few minutes.</p>

  <h5>If you're the owner of this website:</h5>
  <p><span>The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.</span> <a href="https://support.cloudflare.com/hc/en-us/articles/200171926-Error-524">Additional troubleshooting information here.</a></p>
            </div>
        </div>
          
    </div><!-- /.section -->

    <div class="cf-error-footer cf-wrapper">
  <p>
<span class="cf-footer-item">Cloudflare Ray ID: <strong>41ec0c39bedb7a05</strong></span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Your IP</span>: 52.10.234.30</span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
  </p>
    </div><!-- /.error-footer -->
    </div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>

I do see a 504 recently in our logs as well, for https://api.hubapi.com/deals/v1/deal/115603881:

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<meta http-equiv="refresh" content="0">

<title>api.hubapi.com | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" 
media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" 
type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>

</head>
<body>
<div id="cf-wrapper">

<div id="cf-error-details" class="cf-error-details-wrapper">
    <div class="cf-wrapper cf-error-overview">
        <h1>
          
          <span class="cf-error-type">Error</span>
          <span class="cf-error-code">504</span>
          <small class="heading-ray-id">Ray ID: 41ecc081ef2d7a05 &bull; 2018-05-22 05:06:10 UTC</small>
        </h1>
        <h2 class="cf-subheadline">Gateway time-out</h2>
    </div><!-- /.error-overview -->
    
    <div class="cf-section cf-highlight cf-status-display">
        <div class="cf-wrapper">
            <div class="cf-columns cols-3">
              
<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">You</span>
  <h3 class="cf-status-name">Browser</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">Seattle</span>
  <h3 class="cf-status-name">Cloudflare</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc">api.hubapi.com</span>
  <h3 class="cf-status-name">Host</h3>
  <span class="cf-status-label">Error</span>
</div>

            </div>
          
        </div>
    </div><!-- /.status-display -->

    <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
            <div class="cf-column">
                <h2>What happened?</h2>
                <p>The web server reported a gateway time-out error.</p>
            </div>
          
            <div class="cf-column">
                <h2>What can I do?</h2>
                <p>Please try again in a few minutes.</p>
            </div>
        </div>
          
    </div><!-- /.section -->

    <div class="cf-error-footer cf-wrapper">
  <p>
<span class="cf-footer-item">Cloudflare Ray ID: <strong>41ecc081ef2d7a05</strong></span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Your IP</span>: 52.10.234.30</span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->
</div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>

This seems to happen most often when querying deals, but I've also recently seen:

This was all in the past week.

0 Upvotes
Not applicable

504 - Gateway Timeout occurs frequently

Hi,

I've been seeing 524s since March. I've been monitoring this thread hoping there'd be a resolution. A common endpoint with the issue seems to be https://api.hubapi.com/deals/v1/deal/

Here are a couple recent ray IDs:

  • 419847929aa329f5
  • 419743f37f042a4f

Is there anything I can do to help you debug it? Or a better way for me to report my issue?

Thank you,
David

0 Upvotes
flemingapptopia
Member

504 - Gateway Timeout occurs frequently

Got the error again.

urllib2.HTTPError: HTTP Error 504: Gateway Time-out

0 Upvotes
Derek_Gervais
HubSpot Alumni
HubSpot Alumni

504 - Gateway Timeout occurs frequently

Hi @InProductionDev, @TommyT, and @flemingapptopia,

Can you all post more specifics on the endpoints/APIs you're seeing these errors with? @flemingapptopia it sounds like you're seeing 504s with the Engagements API?
@TommyT and @InProductionDev are you both seeing 504s with the 'Get all contacts' endpoint?

Additionally, are all of you respecting the maximum count for all of your requests? Can you try reducing the number of records you're requesting in the interim? Others have found that reducing the size of the batches you're requesting helps mitigate this problem.

If you can give me more specifics on your errors I can dig into this futher.

0 Upvotes
flemingapptopia
Member

504 - Gateway Timeout occurs frequently

Hey Derek -

Got a timeout again yesterday. Does seem like certain times of day are better to run it (server load?)

I was seeing 504s working through the Engagement API. Batch size of 250 (open to lowering it if that is a potential solution).

-Matt

0 Upvotes
Derek_Gervais
HubSpot Alumni
HubSpot Alumni

504 - Gateway Timeout occurs frequently

Hi @flemingapptopia,

Lowering the limit would likely help alleviate the issue, though batch sizes of 250 should work fine. Do you happen to have the full 504 error you got?

0 Upvotes
flemingapptopia
Member

504 - Gateway Timeout occurs frequently

Got it - sounds good. Next time I'll post the full error here. We run the script everyday, so we should see it soon.

-Matt

TommyT
Member

504 - Gateway Timeout occurs frequently

Since beginning testing with the API, I haven't once been able to iterate through all contacts I wish to retrieve before getting a timeout (504). This is specifically with the test data.

0 Upvotes
flemingapptopia
Member

504 - Gateway Timeout occurs frequently

Getting occasional timeout issues as well. Working with engagements api.

0 Upvotes