403 Forbidden Error on Files (Codeship CD)


#1

The Setup

We use Codeship to help with continuous deployment of our CSS and JS to Hubspot. We have to load our JS via Files due to separate issue (Uglified Javascript Throwing Errors - Only in Hubspot).

Our CSS gets LFTP’ed over to Coded Files and does not suffer the same issue.

The Errors

When visiting the URL of the file on hubspots CDN - we get a 403 forbidden error. If we upload the file directly via FTP or the GUI - we do not get the same issue.

Attempted Fixes

We’ve tried setting the file permission to 777 before LFTP over to Hubspot from Codeship

Our deploy script is below, this runs after our build and test are completed:

cp ~/clone/public/stylesheets/app.css ~/clone/hubl/work/hubthemes/vast/custom/system/css/
lftp -c "open -p 3200 -u $FTP_USER,$FTP_PASSWORD ftp.hubapi.com; set ssl:verify-certificate no; mirror -R ~/clone/hubl/work/hubthemes/vast/custom/ /portals/3900783-honestfox_com_au/content/templates/custom"
lftp -c "open -p 3200 -u $FTP_USER,$FTP_PASSWORD ftp.hubapi.com; set ssl:verify-certificate no; mirror -R ~/clone/public/javascripts/ /portals/3900783-honestfox_com_au/content/files/"

#2

Hi @dillonbailey,

Is the file you’re showing in that screenshot still returning a 403 for you? I’m able to see the file by navigating directly to that URL.


#3

Hi @Derek_Gervais still shows a 403 if we upload via Codeship - to keep on development we’ve just had to upload via standard FTP with filezilla which removes the error but is manual.

If it’s helpful I can upload a test.js file for you via lftp from codeship for testing on your end?


#4

Hi @Derek_Gervais - we managed to get past the 403 error - two changes were made.

We changed the uglify settings to not compress or beautify the JS.

We also updated the LFTP to just focus on the files we need instead of a mirror it’s now a put

lftp -c "open -p 3200 -u $FTP_USER,$FTP_PASSWORD ftp.hubapi.com; set ssl:verify-certificate no; put -O /portals/3900783-honestfox_com_au/content/templates/custom/system/css/ ~/clone/public/stylesheets/app.css"
lftp -c "open -p 3200 -u $FTP_USER,$FTP_PASSWORD ftp.hubapi.com; set ssl:verify-certificate no; put -O /portals/3900783-honestfox_com_au/content/files/ ~/clone/public/javascripts/app.js"
lftp -c "open -p 3200 -u $FTP_USER,$FTP_PASSWORD ftp.hubapi.com; set ssl:verify-certificate no; put -O /portals/3900783-honestfox_com_au/content/files/ ~/clone/public/images/icons.svg"

#5

May also be important for working around this task to wrap your .js files in {% raw %} tags prior to upload.

Ref: Uglified Javascript Throwing Errors - Only in Hubspot


#6

Hi @Derek_Gervais unfortunately it looks like our problem is back, this time with some more standard file types, JPG and GIF files deployed to File Manager via LFTP are producing the 403 forbidden error.


You can see in this screenshot some other images that were deployed as part of the same task that are working A-OK.


#7

Hi @dillonbailey,

I’m currently looking into this with product. I’ll reach back out here when I have some more information.


#8

Hi @dillonbailey,

So after digging into this, it looks like the file references for those two images were coming up as deleted. Can you give me some more information on how you’re uploading these files? My current theory is that there’s a race condition between a file upload and a file delete, which is causing some weirdness.


#9

Race conditions are always fun.

Essentially this is using an LFTP mirror command from codeship. I’ll send you the exact command when I’m next online. But from what I can gather if a file exists with the same name it will just run a replace on that file, so perhaps the delete, upload of replaced files is the race condition?

@Derek_Gervais here’s the LFTP command, this is placing the contents of images into the directory on hubspot. $FTP_USER and $FTP_PASSWORD are env. variables that you can plain type in your local CLI for testing.

# Images
lftp -c "open -p 3200 -u $FTP_USER,$FTP_PASSWORD ftp.hubapi.com; set ssl:verify-certificate no; mirror -R ~/clone/public/images/. /portals/1745176-jumbla/content/files/jumbla_honestfox_theme/"

#10

Hey @dillonbailey,

Thanks for the info. I’ve just updated our internal issue, and I’ll continue to reach out here with new information.


#11

Thanks @Derek_Gervais!


#12

I am seeing similar issues when trying to FTP files from gitlab using lftp. If the file is new (and contains properly formatted content for it's file type, to get past the validator) it gets uploaded file. I've opened the files from the server through Transmit (FTP) and verified that they contained the code from gitlab as expected.

However, on subsequent commits, which trigger gitlab CI to mirror the files to the server, I am seeing the following error:

"mirror: Access failed: 551 /portals/xxxxxx-domainname/content/templates/custom/page/css/test.css: Error on output file."

The contents of test.css get replaced with HubSpot default html:

<!doctype html>
<html>
    <head>
        <meta charset="utf-8">
        <title>{{ content.html_title }}</title>
        <meta name="description" content="{{ content.meta_description }}">
        {{ standard_header_includes }}
    </head>
    <body>
        {{ standard_footer_includes }}
    </body>
</html>

It could be a race conditions thing, as mentioned. Not sure... but would definitely like to know if there is a way around this issue, as we are trying to verify if this is a process we can rely on in our workflow.


#13

Hey @jvanpelt - o.k. so what I think is happening here, is that your file contains something that the processes at Hubspot are thinking is HuBL...

What I would recommend is ensuring that your file has {% raw %} file contents here {% endraw %} tags. We added ours to JS via gulp as part of the build process.

We did not encounter this issue with our CSS, but it sounds like the same issue ref: Uglified Javascript Throwing Errors - Only in Hubspot


#14

@dillonbailey thanks for your quick response. I saw your comment about that above and it didn't occur to me to try that! I'm testing this now.

One caveat is that we are hoping to share css & js between our HubSpot instance and non-HubSpot pages within the same design system. This makes inserting {% %} tags dicey. I've been testing this morning with the raw tags, with raw tags wrapped in comments, other various scenarios. My results are basically as inconsistent with the raw tags as they were without. Sometimes the upload goes fine, sometimes I end up with the file's content replaced with HubSpot's default HTML.

For what it's worth, my javascript file literally consists of this:

function psyche(){
    /* this is a test js file */
    fakeSomeStuff();
}

Literally just some test code in there, nothing to be misinterpreted by the validator. @dillonbailey has using lftp been working for you over the last three months?

@Derek_Gervais is there anything you guys have discovered on the HubSpot side between November & Now?

Thanks!!


#15

Hey mate, that’s frustrating! But good thinking keeping it simple for testing, we went through the same experience. We have not run any deployments since December 2017. Also that was pre web designer beta..

I would offer two options, if you’re super keen on hubspot hosting the files, try some different file locations in their folder structure.

The better option in my opinion, if you’re using this same code on non hubspot pages, move your CI/CD process to just deploy your css and JS to AWS S3. No weird file processing issues and easy enough to put behind cloud front for a CDN. Will offer a better and more consistent experience. Codeship has a 1 click integration for this sort of deployment that makes it even easier.

Let me know how you go!

DB