Intermittent Behaviour on js of collection of form data

forms
contacts

#1

Hi there,

We have encountered intermittent issues with your online tracking code when forms are submitted.

Currently we use ninja forms to collect data on our clients WordPress website, and this tracking code from Hubspot picks up the form 80% of the time and collects the basic data (first name, last name, email) to create a contact in hubspot. However sometimes we do not get an entry at all.

Is there a reason for this?

As a failsafe we do a curl call to the API on hubspot to check the user exists and update the Source field and assign an owner from the Original Source, however often the data that comeback is an error and that the client does not exist.

We put on a sleeper function to hold off for 4 seconds to give the hubspot time to collet the user data and then update them.
If after 20 pings to get the user data back we try to enter the client manually. This failed as well with one end-user.

Our questions are:

  1. Is there anything that would prevent a user from being picked up by your integrated JS - IE cookies off, javascript switched off. This is our first concern that the data is not being collected at all and is intermittent.

  2. Is there a way to get the hs_original_source value from the front end or back end from your tracking cookie? How is this pulled from the site and populated on a new end user when the form is submitted? If we can get this another way rather than doing multiple requests to Hubspot endpoint then we can set the source manually from that piece of data

  3. how do we decode the tracking cookie _hstc

We are logging the submissions so I can post the logs when I get some information back.

Thanks

Andi Lee Davis


#2

Hi @Andi_Lee_Davis,

There are a few reasons that the collected forms tool might not function as expected, such as:

  • Form not enclosed in <form> tags
  • Form contained inside an <iframe>
  • Form has JavaScript bound to form submit event or submit button click event. This is how the tool captures submissions and any other events, and can prevent HubSpot from knowing when submissions occur.
  • Form was introduced after the page completed loading. The tool scans the page as soon as it loads and listens to submissions on forms that were present at that point.
  • Form contained a sensitive field, such as credit card number.

The following Knowledge Base article has some more details on how the collected forms tool functions. To address your specific questions:

  1. Both disabling JavaScript and blocking cookies could potentially interfere with the collected forms tool, since it uses both JavaScript and cookies to collect form submission information. If you have the ability to use HubSpot forms or the Forms API, I would strongly recommend you do so.
  2. The original source values are calculated based on the activity tied to a hubspotutk tracking cookie. These fields cannot be changed in-app or via the API, and are only populated/updated by HubSpot based on tracking information.
  3. I’m not sure what you mean by ‘decode’. The _hstc cookie contains the domain, utk, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). Check out the cookie article below for more info.

#3

Hi there,

firstly thanks so much for your reply.

There are a few reasons that the collected forms tool might not function as expected, such as:
• Form not enclosed in tags

This is true but it is collecting the details 90-95% of the time ok. Ninja Forms v3 (not yet officially supported by Hubsport) does not contain opening and closing form tags.

•	Form contained inside an <iframe>

Not on this site.

•	Form has JavaScript bound to form submit event or submit button click event. This is how the tool captures submissions and any other events, and can prevent HubSpot from knowing when submissions occur.
•	

We are using a WP hook to process information server-side during an Ajax request.
The hubspot code works front-end separately to this.
During that process we send to an internal curl request to process and sleep for 3 seconds to give enough time for hubspot to collect the data, via its own embed code. There is a loop threshold of 20 attempts with as second interval waiting for the existence of the hs_analytics_source field to become present. * We have seen on each loop log more data on each increment on success, during the time it takes the Hubspot server to process data.

If it continues to fail with this kind of error.
curl called…
?Looped CHECKED DATA IS:
{“status”:“error”,“message”:“contact does not exist”,“correlationId”:“983b713a-3354-44a0-8dca-7773eeb49dd0”,“requestId”:“962439c86f36ea3ef77c50a348cd86ef”}

After 20 attempts (20 seconds) we then use the API to send another curl request to try to add the client. * this sometimes also fails

•	Form was introduced after the page completed loading. The tool scans the page as soon as it loads and listens to submissions on forms that were present at that point.

On average it takes a person more than 3 seconds after the page is loaded to complete the form. So this is not a factor here.

•	Form contained a sensitive field, such as credit card number.

The following Knowledge Base article has some more details on how the collected forms tool functions. To address your specific questions:

There is no sensitive information captured on the form it is a simple contact form with a few other fields.

1	Both disabling JavaScript and blocking cookies could potentially interfere with the collected forms tool, since it uses both JavaScript and cookies to collect form submission information. If you have the ability to use HubSpot forms or the Forms API, I would strongly recommend you do so.

A1	- We do we are forcing adding a client after a number of seconds of not found. Our point being that very rarely the submission neither works on the back end or front end.

1	The original source values are calculated based on the activity tied to a hubspotutk tracking cookie. These fields cannot be changed in-app or via the API, and are only populated/updated by HubSpot based on tracking information.

A2 - No we are looking to get this information and use it - If the data is split by a period . Then how do we decode the parts. 244384154.766efba5f49ce01f056d91071f864e7d.1509700502065.1509974194923.1509977132210.12 - 244384154 datetimestamp came out as 1977, so that did not make any sense.

1	I’m not sure what you mean by ‘decode’. The _hstc cookie contains the domain, utk, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). Check out the cookie article below for more info

Because this is the cookie value = 244384154.766efba5f49ce01f056d91071f864e7d.1509700502065.1509974194923.1509977132210.12 - containing all the information in this string.

thanks again

Andi


#4

Hi @Andi_Lee_Davis,

I’m sorry I didn’t see this in your initial post, but I noticed the fact that you’re using Ninja Forms. Ninja Forms are not wrapped in <form> tags, which means they’re incompatible with the collected forms tool. Collected forms doesn’t collect forms that aren’t contained in a <form> tag, and there aren’t currently plans to support this use case.

I’m confused by the fact that you’re seeing ~80% success with these forms; due to the way the collected forms tool is built, it cannot collect submissions from forms without <form> tags. Can you send me a link to the page in question so that I can do some testing on the form submissions?

Lastly, regarding the cookie value; the __hstc cookie contains the domain (in the form of an id, the first value), the contact’s utk, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). These are the values separated by periods in the cookie value. Given the example you provided, the contact’t utk would be 766efba5f49ce01f056d91071f864e7d, the following 3 numerical values would be the first, last, and current visits, and the session number would be 12.


#5

Hi Derek,

Thanks. I hope you are well today.

We were aware that Hubspot did not cater for Ninja Forms v3, however we needed to upgrade because we also needed the conditional logic plugin. We have found the Original Source to be set to API when this has not worked, however to have a different original source when Hubspot has picked them up.

In the notes we also get information that Hubspot indeed picket up the data via the forms.
xxxx xxxxx submitted Collected form: #nf-form-25-cont .nf-form-cont on REDPILL | Contact

Some are collected data, some through the API, interestingly when the API is sent via the back-end we notice that n the source history but that the cookie also picks up the Original Source and ties it to the client. If this is Set as API then we know that the client was not found and our API integration eventually pushed the details onto the system.
The source is not set because the Original Source is not then populated - Hence why we need to get the information from the cookie if we can and see if we can grab that data - hs_analytics_source.

Seems that the __hstc cookie does not have the information on the analytics source for us, so I am tryng to figure out where Hubspot picks this value up from.

Values we map according to your list:

/// 1) Map as above
	if( $hs_analytics_source ){

		write_log('MAPPING SOURCE : ' . $hs_analytics_source);

		switch(true){
			case ($hs_analytics_source === 'ORGANIC_SEARCH' ):
				$mapped_source = 'Organic';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'PAID_SEARCH' ):
				$mapped_source = 'CPC';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'EMAIL_MARKETING' ):
				$mapped_source = 'Email';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'SOCIAL_MEDIA' ):
				$mapped_source = 'Social';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'REFERRALS' ):
				$mapped_source = 'Referral';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'OTHER_CAMPAIGNS' ):
				$mapped_source = 'Other';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'DIRECT_TRAFFIC' ):
				$mapped_source = 'Direct';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'OFFLINE' ):
				$mapped_source = 'Offline';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			case ($hs_analytics_source === 'PAID_SOCIAL' ):
				$mapped_source = 'Paid Social';
				write_log('MAPPING SOURCE : ' . $mapped_source);
				break;
			default:
				$mapped_source = NULL;
				write_log('MAPPING SOURCE.. : ' . $mapped_source);
		}
	}

So we are trying to map the hs_original_source if the client is found.
If not then post them via the API at which point no hs_original_source is not found on Hubspot so we cannot map it.

If we can somehow glean this information out then we can populate it.

Thanks

Andi


#6

Hi @Andi_Lee_Davis,

So the original source properties are computed based on the tracking parameters (if present) and the referrer. This information is associated with the hubspotutk token, which is just a unique id. It’s not possible to decode or otherwise infer the original source info from the hubspotutk token, since it’s just a unique identifier; the information lives in HubSpot, and you’d need to make a request to the get contact by utk endpoing (below). It’s also not possible to update the original source properties manually (either in-app or via api), since those values are only updated by the HubSpot system.

I’ve included articles on the get contact by utk endpoint and the original source bucketing rules below. While I’m not entirely sure why the collected forms tool is occasionally grabbing submissions from your ninjaforms form, it’s not something we support directly and therefore will be unreliable. HubSpot forms or custom HTML forms using the forms API would be more consistent and reliable, if you have the ability to use them.



#7

Thanks Derek,

This is very useful.

Best regards,

Andi