504 - Gateway Timeout occurs frequently


#21

Hey Derek -

Got a timeout again yesterday. Does seem like certain times of day are better to run it (server load?)

I was seeing 504s working through the Engagement API. Batch size of 250 (open to lowering it if that is a potential solution).

-Matt


#22

Hi @flemingapptopia,

Lowering the limit would likely help alleviate the issue, though batch sizes of 250 should work fine. Do you happen to have the full 504 error you got?


#23

Got it - sounds good. Next time I'll post the full error here. We run the script everyday, so we should see it soon.

-Matt


#24

Got the error again.

urllib2.HTTPError: HTTP Error 504: Gateway Time-out


#25

Hi,

I've been seeing 524s since March. I've been monitoring this thread hoping there'd be a resolution. A common endpoint with the issue seems to be https://api.hubapi.com/deals/v1/deal/

Here are a couple recent ray IDs:

  • 419847929aa329f5
  • 419743f37f042a4f

Is there anything I can do to help you debug it? Or a better way for me to report my issue?

Thank you,
David


#26

Hi @flemingapptopia,

You don't happen to have more details than that, do you? I would expect a bigger error response with more details. Also, are you only seeing this with the Engagements API?

@DavidSickmiller thanks for the info; was there more to the error response? Any 524 errors have different origins than 504 errors, and there's often more info in the full error body. Are you only seeing this with the Deals API?


#27

@Derek_Gervais, is there any more information (another topic because this one is about the 502 / 504 error) about that 524 error message? I myself do frequently experience an strange error called "invalid JSON markup" using Cdata's connector which according to Cdata support should be caused by an HTTP-524 error. I can not solve it yet.

Thanks in advance and kind regards,

Aad Dijksman


#28

Still these errors are not minimised. We get 502, 504 and 524 errors frequently.

The connection to the origin web server was made_ but the origin web server timed out before responding. The likely cause is an overloaded background task_ database or application_ stressing the resources on your web server. To resolve_ please work with your hosting provider or web development team to free up resources for your database or overloaded application.

This was the error message thrown while we got 524 error.


#29

Here's an example response of a recent 524 on https://api.hubapi.com/deals/v1/deal/18175226:

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<meta http-equiv="refresh" content="0">

<title>api.hubapi.com | 524: A timeout occurred</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" 
media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" 
type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>

</head>
<body>
<div id="cf-wrapper">

<div id="cf-error-details" class="cf-error-details-wrapper">
    <div class="cf-wrapper cf-error-overview">
        <h1>
          
          <span class="cf-error-type">Error</span>
          <span class="cf-error-code">524</span>
          <small class="heading-ray-id">Ray ID: 41ec0c39bedb7a05 &bull; 2018-05-22 03:04:46 UTC</small>
        </h1>
        <h2 class="cf-subheadline">A timeout occurred</h2>
    </div><!-- /.error-overview -->
    
    <div class="cf-section cf-highlight cf-status-display">
        <div class="cf-wrapper">
            <div class="cf-columns cols-3">
              
<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">You</span>
  <h3 class="cf-status-name">Browser</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">Seattle</span>
  <h3 class="cf-status-name">Cloudflare</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc">api.hubapi.com</span>
  <h3 class="cf-status-name">Host</h3>
  <span class="cf-status-label">Error</span>
</div>

            </div>
          
        </div>
    </div><!-- /.status-display -->

    <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
            <div class="cf-column">
                <h2>What happened?</h2>
                <p>The origin web server timed out responding to this request.</p>
            </div>
          
            <div class="cf-column">
                <h2>What can I do?</h2>
                      <h5>If you're a visitor of this website:</h5>
  <p>Please try again in a few minutes.</p>

  <h5>If you're the owner of this website:</h5>
  <p><span>The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.</span> <a href="https://support.cloudflare.com/hc/en-us/articles/200171926-Error-524">Additional troubleshooting information here.</a></p>
            </div>
        </div>
          
    </div><!-- /.section -->

    <div class="cf-error-footer cf-wrapper">
  <p>
<span class="cf-footer-item">Cloudflare Ray ID: <strong>41ec0c39bedb7a05</strong></span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Your IP</span>: 52.10.234.30</span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
  </p>
    </div><!-- /.error-footer -->
    </div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>

I do see a 504 recently in our logs as well, for https://api.hubapi.com/deals/v1/deal/115603881:

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<meta http-equiv="refresh" content="0">

<title>api.hubapi.com | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" 
media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" 
type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>

</head>
<body>
<div id="cf-wrapper">

<div id="cf-error-details" class="cf-error-details-wrapper">
    <div class="cf-wrapper cf-error-overview">
        <h1>
          
          <span class="cf-error-type">Error</span>
          <span class="cf-error-code">504</span>
          <small class="heading-ray-id">Ray ID: 41ecc081ef2d7a05 &bull; 2018-05-22 05:06:10 UTC</small>
        </h1>
        <h2 class="cf-subheadline">Gateway time-out</h2>
    </div><!-- /.error-overview -->
    
    <div class="cf-section cf-highlight cf-status-display">
        <div class="cf-wrapper">
            <div class="cf-columns cols-3">
              
<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">You</span>
  <h3 class="cf-status-name">Browser</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">Seattle</span>
  <h3 class="cf-status-name">Cloudflare</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc">api.hubapi.com</span>
  <h3 class="cf-status-name">Host</h3>
  <span class="cf-status-label">Error</span>
</div>

            </div>
          
        </div>
    </div><!-- /.status-display -->

    <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
            <div class="cf-column">
                <h2>What happened?</h2>
                <p>The web server reported a gateway time-out error.</p>
            </div>
          
            <div class="cf-column">
                <h2>What can I do?</h2>
                <p>Please try again in a few minutes.</p>
            </div>
        </div>
          
    </div><!-- /.section -->

    <div class="cf-error-footer cf-wrapper">
  <p>
<span class="cf-footer-item">Cloudflare Ray ID: <strong>41ecc081ef2d7a05</strong></span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Your IP</span>: 52.10.234.30</span>
<span class="cf-footer-separator">&bull;</span>
<span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->
</div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>

This seems to happen most often when querying deals, but I've also recently seen:

This was all in the past week.


#30

We are getting 504s frequently for one of our clients, usually when using the API to get contacts, but sometimes when using the bulk API to send back data. Examples from yesterday below.

Request:

ERROR HubspotAdapter: >> GET https://api.hubapi.com/contacts/v1/lists/all/contacts/all?property=country&property=rf_cdm_isthisrecordjunk&property=rf_dhq_annualrev&property=rf_site_locationtype&property=rf_inferredrevenue&property=rf_dhq_url&property=rf_dhq_revenuerange&property=rf_site_siccode&property=rf_techcdn&property=rf_techalexarank&property=rf_site_naicscode&property=rf_dhq_company&property=vid&property=rf_dhq_zip&pr
operty=rf_techdns&property=rf_dhq_sicname&property=rf_dhq_familymembers&property=firstname&property=rf_emailgrade&property=rf_dhq_locationtype&property=rf_cdm_matchgrade&property=rf_dhq_phone&property=rf_site_annualrev&property=rf_site_co
untrycode&property=rf_dhq_customindustry&property=rf_dhq_employeeshere&property=rf_site_sicname&property=zip&property=rf_dhq_emptotalrange&property=website&property=jobtitle&property=rf_site_tradename&property=rf_site_revenuerange&propert
y=rf_site_country&property=rf_dhq_state&property=rf_site_subcode&property=fax&property=email&property=rf_site_statecode&property=rf_dhq_siccode&property=rf_site_zip&property=rf_dhq_yearstart&property=company&property=salutation&property=r
f_site_url&property=rf_inferredpublic&property=rf_dhq_country&property=rf_site_yearstart&property=phone&property=rf_inferredindustry&property=city&property=rf_cdm_matchscore&property=lastname&property=rf_cdm_duplicateids&property=rf_dhq_a
ddr1&property=rf_dhq_addr2&property=rf_dhq_city&property=rf_site_familymembers&property=rf_dhq_tradename&property=mobilephone&property=rf_site_state&property=rf_site_addr2&property=rf_site_addr1&property=rf_site_customindustry&property=rf
_site_city&property=rf_site_employeestotal&property=rf_dhq_employeestotal&property=rf_site_phone&property=rf_dhq_countrycode&property=rf_cdm_include&property=rf_inferredemployees&property=rf_site_employeeshere&property=rf_cdm_emailquality
&property=rf_dhq_naicsname&property=rf_dhq_naicscode&property=rf_site_emptotalrange&property=state&property=rf_dhq_subcode&property=rf_cdm_modifieddate&property=rf_cdm_junkreason&property=rf_site_naicsname&property=address&property=rf_dhq
_statecode&property=rf_site_company&propertyMode=value_only&count=100&vidOffset=33144899

Error: Caused by: com.reachforce.xds.base.ExternalSystemException: External Server Error: 504 [Gateway Timeout]

          <html>
          <head><title>504 Gateway Time-out</title></head>
          <body bgcolor="white">
          <center><h1>504 Gateway Time-out</h1></center>
          <hr><center>cloudflare</center>
          </body>
          </html>

Example bulk upload:

ERROR HubspotAdapter: >> POST https://api.hubapi.com/contacts/v1/contact/batch/?auditId=10950-1529957960027

Error response:

          <span class="cf-error-type">Error</span>
          <span class="cf-error-code">504</span>
          <small class="heading-ray-id">Ray ID: 430a7b1e286b246e &bull; 2018-06-25 21:20:56 UTC</small>
        <h2 class="cf-subheadline">Gateway time-out</h2>
            <span class="cf-status-desc">You</span>
            <h3 class="cf-status-name">Browser</h3>
            <span class="cf-status-label">Working</span>

            <span class="cf-status-desc">Ashburn</span>
            <h3 class="cf-status-name">Cloudflare</h3>
            <span class="cf-status-label">Working</span>
          
            <span class="cf-status-desc">api.hubapi.com</span>
            <h3 class="cf-status-name">Host</h3>
            <span class="cf-status-label">Error</span>
          
                              <h2>What happened?</h2>
                <p>The web server reported a gateway time-out error.</p>

                <h2>What can I do?</h2>
                <p>Please try again in a few minutes.</p>

              <span class="cf-footer-item">Cloudflare Ray ID: <strong>430a7b1e286b246e</strong></span>
              <span class="cf-footer-separator">&bull;</span>
              <span class="cf-footer-item"><span>Your IP</span>: 52.71.123.54</span>
              <span class="cf-footer-separator">&bull;</span>
              <span class="cf-footer-item"><span>Performance &amp; security by</span> <a               href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

#31

Hey all,

Just wanted to hop in here again since this topic is getting very crowded: It's important to differentiate between a 'normal' and 'abnormal' level of these timeouts. While a 504 or 524 is indicative of a server-side issue, some small number of timeout errors are always expected on the Internet. This is normal and not something we can prevent 100% of the time. It's important to be able to gracefully handle these errors; in the overwhelming majority of cases, we see successful responses after a single retry.

That said, high error rates can be indicative of something else going wrong. If you're seeing success rates lower than ~99.99% (i.e. significantly more than 1-2 504/524 errors for every 10,000 requests) then we should dig into things further to make sure there isn't a more severe issue.

If you're seeing abnormally high error rates, it's likely because the thing you're doing is causing some timeout on the HubSpot side; temporarily slowing down your requests and/or lowering batch sizes is generally the best remedy for these issues.

Going forward, anyone that's experiencing abnormally high rates of 504 and/or 524 errors should create a separate thread with the following information:

  • The public IP address your traffic is coming from
  • The full URL of the API you're hitting
  • What kind of client you're using to hit the API (e.g. Java 8 with HttpClient, curl 7.55, etc)
  • Any error messages you're getting, including the "Ray ID" if it's present
  • For timeouts and network errors, a traceroute from your machine to our API (e.g. traceroute api.hubapi.com)

#32

Hey Derek,

Getting a large number of 504 errors on the get associated engagements endpoint. It's unusable in its current state.

Error
urllib2.HTTPError: HTTP Error 504: Gateway Time-out

The public IP address your traffic is coming from
50.78.250.202

The full URL of the API you're hitting
doesn't include key
https://api.hubapi.com/engagements/v1/engagements/associated/CONTACT/55126251/paged

What kind of client you're using to hit the API (e.g. Java 8 with HttpClient, curl 7.55, etc)
Python 2.7 - urllib2

For timeouts and network errors, a traceroute from your machine to our API (e.g. traceroute
Traceroute has started…

https://pastebin.com/MjRtPaiZ

-Matt


#33

Any updates? I'm still encountering 504s. Thanks!


#34

Hi,
I have found a workaround. Whenever the error occurs, I catch the exception and sleep for 2 seconds and retry again. This works for me every time.


#35

Ended up doing the same thing... Breaks out of the inner loop once the url is loaded and continues with the rest of the script.

while HasMore:
while True:
try:
xurl = "/engagements/v1/engagements/associated/CONTACT/" + str(c['vid']) + "/paged"
OffsetProperty = '&offset=' + str(OffsetNumber)
url = HS_API_URL + xurl + APIKEY + limit + OffsetProperty
print 'loading url: ' + str(url)
response = urllib2.urlopen(url).read()
engagements = json.loads(response)
if engagements['hasMore'] or engagements['hasMore'] == False:
break
except Exception as E:
print str(e)


#36

We received the 405 error while demoing one of our sales pages.

Is there any eta on when this will be fixed?


#38

Welcome, @mattb. Apologies for the delayed response.

This topic discusses 504 Gateway Timeout errors, not 405 Method Not Allowed errors. Did you mistype the error code?

Either way, please see Derek's post here:

If you are encountering 504 Gateway Timeout errors, please create a new topic with the following information:

  • The public IP address your traffic is coming from.
  • The full URL of the API you're hitting.
  • What kind of client you're using to hit the API (e.g. Java 8 with HttpClient, curl 7.55, etc).
  • Any error messages you're getting, including the "Ray ID" if it's present.
  • For timeouts and network errors, a traceroute from your machine to our API (e.g. traceroute api.hubapi.com ).

If you are encountering 405 Method Not Allowed errors, please create a new topic with the following information:

  • Your Hub ID.
  • The full URL of the API you're hitting.
  • The method you are using (GET, POST, or PUT).