Enable by default the Heart beat timer to process a more accurate time on page and visit duration

[mattab]

The goal of this issue is to enable by default the Heart beat timer feature, which lets Piwik JavaScript tracking client send ping requests every N second. By enabling this feature by default we will provide more accurate accurate time on site and time on page.

Steps;

• in the JavaScript tracking code, write enableHeartBeatTimer just like we write enableLinkTracking
• Requires a mechanism where by default, ping frequencies will go down so it won't overload servers, eg. Allow JS tracker to change heart beat wait interval exponentially #8213

Follows up #8069 #2041 #8218 #8223

[Tobias-Conrad]

two opinions made earlier #7207:
Hi @Tobias-Conrad, issue #2041 would be the solution to make us notice a difference between single page visits, with visitors probably reading the page and visitors navigating away within a few seconds. The definition that needs to be added is the amount of time a visitor should have stayed on that single page visit before it is calculated as a visit instead of a bounce.

--possible math for user exit rate--
67% User exit rate (left after view one page)
could be turned into
xx% User exit rate (left after view one page and stayed less than 20 seconds)
1 page visit + less than 20 seconds : user exit
1 page visit + more than 20 seconds : user interested
2 page visits + more than 20 seconds : user interested
2+x page visits + less than 20 seconds : user exit (may user interested )

For google e.g. it is important how long the user stays on page.
Think of a landing page (only one page) with video (average view time 8:30 min.).
So Piwik would say user is exit without interest, google would say wow good content, value resource.

So this valuable resource should be honored by Pwiki statistics.

[tsteur]

We should announce this far in advance, maybe a couple of times and in the newsletter as it can cause serious performance problems. Even if you have only 5000 visitors on your website at the same time and suddenly all start pinging every 15 seconds on top (constantly) this adds over 300 tracking requests per second (if they stay for 15 seconds or longer on a page).

If you're unlucky and have eg an ad on TV or so, you might get suddenly 1000-5000 visitors (or more) within 2 or 3 seconds on your website. They would start pinging at the same time so you might have suddenly 1000 or 3000 requests per second more. Same procedure after sending loads of newsletters. Many users will open at the same time and it will be much harder to handle peaks. 15 seconds on page sounds like a lot but can be reached easily, especially when you link to a form. In both cases I had those kinda problems in the past so adding additional load without expecting it can be problematic.

[Tobias-Conrad]

make this option optional: let us set the ping time and to enable the feature, with warning.

If you are happy you can just auto monitor the server load and switch off hearthbeat when an high server load occurs.

@tsteur please keep your workflow fluid, no time for fears that block developing a excellent feature.
Speed of implementation is the goal to success.

[Tobias-Conrad]

HI,
HeartBeat is working nice.
But not the consequenses of this new feature.

When you visit a webpage/one page for 30 seconds are you then a dropout visitor?
When you visit one page for 10 minutes, because you watch a video on site are you a drop out visitor?

I the moment piwik both mark them as droped out.
copied from above:

--possible math for user exit rate/drop out rate--
67% User exit rate (left after view one page)
could be turned into
xx% User exit rate (left after view one page and stayed less than 20 seconds)
1 page visit + less than 20 seconds : user exit
1 page visit + more than 20 seconds : user interested
2 page visits + more than 20 seconds : user interested
2+x page visits + less than 20 seconds : user exit (may user interested)

For google e.g. it is important how long the user stays on page.
Think of a landing page (only one page) with video (average view time 8:30 min.).
Think of ONE PAGE WEBSITES.
So Piwik would say user is exit without interest,
google would say wow good content, value resource.

So this valuable resource should be honored by Pwiki statistics.

[mattab]

I suggest we do this post- 3.0.0 - so moving into Mid term for now

[michaelnordmeyer]

@Tobias-Conrad IIRC, Google considers a visit as bounced if the visitor doesn't access a second page.

[masteranalyze]

Google considers a visit as bounced if the visitor doesn't access a second page.

What google consider,is relevant for the year 2005,not for today anymore,when we have AJAX,and user can practically submit comment,into same page,without reloading the page.An good example of this is FACEBOOK,but they are many nowdays actions that can be performed into an page,like an social share,tweet,print,etc actions that means the user did interact with the webpage.

If the user interact with the webpage,or submit an comment but does not reload the webpage because the webpage is using AJAX,i don`t think this should be considered BOUNCE,as the user did interact with your webpage in an way or another.

More then that imagine an website with lots of pages,an news website,were people just come to read,that news,because only that news they need,in this case it will be only 1 pageview,an google will consider this an bounce,but maybe the user stayed on the page 1 minute or 10 minutes,to read that,neither that visit is not an bounce,thats why we need an accurate system with events triggered to report an better time on page,this is the only way to know if people did interact with that webpage or not. If the user stays more then 15,30 seconds on that page,normally it should not be counted as bounce,because it isnt!

[mattab]

PR #8213 is relevant to this issue but it wasn't merged as is.

[Kendigan]

Hi!

We tried to add "heartbeat" functionality to our website (with Piwik 2.14.1), and we also send custom variables from every page. The issue we faced is that after enabling "heartbeat": when user on the page, the time spent (seen in Visitor Log) is correct, but when user goes to another page, the time spent on previous page is reset to 1s (or sometimes to other very small value). So, the resulting calculations of time spent became even worse that before enabling "heartbeat".

Does anyone else have similar issues?
What could we do wrong if nobody else has similar issues?

Thanks in advance!

[hpvd]

@Kendigan
we are also thinking about enabling heartbeat - do you still have this behaviour with piwik 2.15?

[tsteur]

@Kendigan I suggest to create a new issue for this so someone might have a look at it

[hpvd]

@Kendigan "What could we do wrong if nobody else has similar issues?"
=> I think not many people are using this since it's not enabled by default.

Is there any news on this topic? Is it now working for you?

[hpvd]

There seems to be a issue regarding bounce rate which may also affect the discussion within this one #9199

[mattab]

(fyi Feature is now documented at Accurately measure the time spent on each page)

[hpvd]

many thanks for documenting this!
Is there a setting how often website send this alive information?
not everybody needs it every second, but every 60seconds maybe to long for others...
and of course you may need to find an individual balance between accuracy and server-load

[tsteur]

I just realized it misses a documentation here: http://developer.piwik.org/guides/tracking-javascript-guide (how to change interval)

@hpvd it's partially document here: http://developer.piwik.org/api-reference/tracking-javascript (search for enableHeartBeatTimer). Current default is 15 seconds

[mattab]

@hpvd @tsteur thanks for the review. Doc updated @ http://developer.piwik.org/guides/tracking-javascript-guide#accurately-measure-the-time-spent-on-each-page

[mattab]

also created New FAQ: How do I improve accuracy of “Time spent on page” and “Visit duration” metrics? and improved doc

[hpvd]

maybe,
before making it default we should thing on this topic,
to give a good balance between
the really needed high accuracy to have informative value on this especially for one single page visits
and server load
#9423

[hpvd]

before enabling heartbeat by default, one should have a look on this one:
STRANGE visit times - also in demo: #9504

[hpvd]

please add some information in docu of heartbeat function
https://developer.piwik.org/guides/tracking-javascript-guide#accurately-measure-the-time-spent-on-each-page

regarding

1. being careful enabling heartbeat because of serverload (especially when setting shorter ping periods)
2. how will enabling heartbeat impact on in piwik shown average visit time?
   Will it be lower than before, because with this, also the visits with only one pageview (which should be the shorter ones) are included in statistics? (see also check needed: calculation of average visit time #9506)
3. does heartbeat send ping within each complete visit (always) or only on first visited page of each visit?

edit: just opend a new ticket on this topic: #9516

[mattab]

If we decide to enable Heartbeat by default, we need to update this (new) faq: How do I accurately measure the time spent by visitors on my pages, including the time spent on the last page of their visit?

[dev-101]

I wish to avoid opening new issue, but this is something I was thinking all along.

Plus, with heartbeat, Average Time on Page in Pages table is completely inaccurate (useless, practically), as it still treats single page visits as 0:00 s.

[tsteur]

Note: the title says to process a more accurate time on page and visit but heartbeat does currently not update the time on page.