Scheduled report list of improvements

[mattab]

This ticket is a placeholder to improve the current feature of Scheduled email reports (PDF and HTML).

See the following tickets for new feature requests:

High impact features:

• PDF/HTML Reports: Expand Actions reports (and some others) by default #2137: new Option to expand and show the hierarchy for Pages/downloads /Custom variables/etc. reports
• Possibility to unsubscribe from Email reports (pdf, html, csv) #2197: Add possibility to unsubscribe from the scheduled report list of emails
• PDF Reports - show Conversions instead of Revenue #3493 Scheduled reports should display Conversions as well as Revenue (currently Conversions is not displayed but it is a key metric)
• Email reports should contain percentage values column #6526 Display a new column in each report, with relative percentage Visits values

Other cool features:

• Email reports changes for better UX - List of small ux changes #3183: Various UX improvements to the email reports feature
• Scheduled report : select at which hour reports should be sent #3706: select at which hour reports should be sent
• Scheduled reports: option to display up to 500 rows in email reports instead of just 24 rows #2817: New option to include all rows (no limit)
• PDFReports : Allow customization of mail subject and body #2764: Allow customization of mail subject and body
• List of Scheduled reports: add "View Reports Across All Websites" feature #3336: List of Scheduled reports: add "View Reports Across All Websites" feature
• Scheduled report: "Referrer Type" could have a graph plotting each referrer type evolution #3014: "Referrer Type" could have a graph plotting each type over last periods (would need support for graph with multiple lines which would be great to have for Piwik Mobile of course!)
• Let user include her Custom Dashboards in the Scheduled email reports #6775 Let user include her Custom Dashboards in the Scheduled email reports
• ~~ From Segmentation: implement UI for adding/editing segments, and switching segments #2135 is done, Scheduled Reports with a specific segment (forums) ~~ Done!

Please comment if you have other ideas to improve Scheduled reports.

[anonymous-matomo-user]

Replying to matt:
The Piwik scheduled reports timeout when all the options are selected. Due to which the weekly report are not sent out. The weekly scheduled reports also wont work when there is more data in the reports.

[mattab]

@nouman.qureshi, thanks for the report. Does it also fail if you disable all graphs from the report?

There are 2 issues here:

• Server is misconfigured, or not yet set to a higher php timeout.
• But there is also a Piwik "bug" that a failing report also fails to send all other ones because they are all generated in the same php thread.

I suggest for example:

• PDF/HTML reports should be generated in a separate http request so that when one is too long it does not stop the others from being sent
• Profile the code using Xdebug and find out why it times out.

[anonymous-matomo-user]

Hi Matt

Thanks for the suggestion.

Can you please provide a particular example of how can i do this? i.e:-

     Generating PDF in a separate http request 
     profiling the code using XDebug.

Any help in this matter will be highly appreciated.

Best Regards
Nouman

[mattab]

In [ticket:291#comment:47] it is suggested that we could implement a new very useful feature:

'Compare values to last [day/week/month/year]'

For a CEO receiving the weekly Piwik report by email, it would be extremely useful to be able to have a clear view of the evolution of the metrics compared to the last week.

• The evolution graph above the main metrics already helps with this,
• we could show the +/- percentages also for each table row. For example to show the evolution of bounce rate +14% or the evolution of a particular country: India -55%.
• Above the Country report for example, if graph is enabled, the graph would show the nb of visits for both periods for the top N countries (whatever fits on the screen when comparing periods). This graph improvement is implemented in Static PNG Graph: plot several metrics & multiple row evolutions #3013

There could be a new checkbox in the report Edit "Compare reports to the previous $period".

This would be a great improvement to the report analysis & data driven reporting!

[julienmoumne]

adding reference to #3323

[julienmoumne]

refs #3336

[julienmoumne]

(In [6849]) refs #3323 #3088 #2708 #71 #2318

• generate and compare HTML, PDF & SMS reports in Test_Piwik_Integration_EcommerceOrderWithItems & Test_Piwik_Integration_TwoVisitors_TwoWebsites_DifferentDays
• report content as return value of PDFReports->generateReport() with new output type OUTPUT_RETURN

[mattab]

Adding #2135 related note as a new "high impact" feature!

[mattab]

Added #3493 in the list

[mattab]

Added #2764 in the list

[julienmoumne]

remove #2762 (wontfix)
add #3706

[mattab]

Adding #3336, remove #3023

[anonymous-matomo-user]

Hi guys,

little feature request regarding scheduled reports:

the possibility to limit i.e. referring websites, keyword etc. to display only 10 or 25 items...like it is possible to do this in the dashboard overview.

thanks!

[quba]

Scheduled reports are feature that is widely used by Piwik users (people are lazy so in many cases they don't log into Piwik, but only read e-mail reports). I think that is the reason why we should increase priority of this ticket.

List of things to improve:

• remove hardcoded values (e.g. report truncation https://github.com/piwik/piwik/blob/master/plugins/ScheduledReports/API.php#L55) and move them to config file,
• figure out how we can improve visualisation to make it more similar to the reports generated in the UI (currently generated reports look like those from Piwik 1.*) without using any external library (e.g. wkhtmltopdf),
• make code more reusable, refactor big functions into smaller ones,
• allow to generate custom reports using plugins easily.

[tassoman]

You see, logo is rendered as a linked img but doing this it gets caught by MCA not loading external images (outlook, gmail web).
So Logo image should be sent as attachment and all the icons too.
(search engines and various favicons)

[mattab]

@quba Moved the ticket to Mid Term as you're right it's a key feature of Piwik and it would be high value to improve it.

[timo-bes]

Here's another suggestion:

Users who use mainly custom date ranges in Piwik are used to seeing the sums for the whole period in tables and the days within the period in graphs. The scheduled reports on the other hand behave like days/weeks/months/years in Piwik: The tables show the selected period and graphs show the past periods, e.g. for a week you see the week in the table and the last few weeks in graphs.
Can we offer both behaviors in the scheduled reports?

I suggest adding a selection like this:
A. In graphs, show the evolution across multiple periods (default)
B. In graphs, show the evolution within the selected period

For a weekly report,

• option A calls ScheduledReports.generateReport with period=week and date=2014-12-05
• option B calls ScheduledReports.generateReport with period=range and date=2014-12-01,2014-12-07

@mattab @julienmoumne What do you think?

[timo-bes]

I've looked into the suggestion above and it's not a good idea to change the parameters of ScheduledReports.generateReport. This would impact the regular data tables as well, e.g. there would be no unique visitors.
So we need to adjust only the period/date parameters of the API calls for image graphs. However, this is not possible on a per report basis because the scheduled reports use the imageGraphEvolutionUrl metadata which is generated independently. I think the easiest way would be to add a config option that allows changing the behavior of evolution graphs (e.g., "General.graphs_show_evolution_within_selected_period"). This would impact the imageGraphEvolutionUrl assigned in ImageGraph::getReportMetadata.
I already have an implementation for that. If the feature request is accepted, I can push it.

[mattab]

Can we offer both behaviors in the scheduled reports?

Hi @BeezyT good to see you here :-)
Sure it's a great idea. In fact we need to do the same in the Piwik UI as well: #955
maybe you can create a new issue for this request and link to #955 and #3088 ?

[mattab]

this is not possible on a per report basis because the scheduled reports use the imageGraphEvolutionUrl metadata which is generated independently.

Maybe a good solution would be to add a new parameter to the metadata API to let user change the period being plotted on the graph. There are already several parameters to ImageGraph.get API so it would make sense to add a new one. See: http://developer.piwik.org/api-reference/reporting-api#ImageGraph

[timo-bes]

Matt, what you say also was my first idea. But the problem is that the ImageGraph plugin adds an imageGraphEvolutionUrl to the metadata of all reports (see https://github.com/piwik/piwik/blob/master/plugins/ImageGraph/ImageGraph.php#L163). At this point, it is not known how the image graph is going to be used. The ScheduledReports plugin uses this URL to render the report and does not look at or know about the parameters.
So either the ImageGraph plugin has to offer two URLs for the two different behaviors or the ScheduledReports plugin has to change the ImageGraph parameters. Or we keep everything as it is and add a config option that is used by the ImageGraph plugin when the imageGraphEvolutionUrl is generated.
As I said, I have an implementation for the latter option. If we want to go with this option I can push the code. Otherwise I'll just create a ticket as you suggested.

[mattab]

Hi @BeezyT

add a config option that allows changing the behavior of evolution graphs (e.g., "General.graphs_show_evolution_within_selected_period

OK as there is already setting graphs_default_period_to_plot_when_period_range

In general, and this is what this issue really shows us, we need to refactor the Scheduled reports PHP code and front-end with AngularJS, to make this kind of features available via the UI rather than have to use a new config setting. because the Scheduled Reports feature is the most common way that users will access their data outside of Piwik, it is an essential part of the platform that we can give more love. cc @julienmoumne

[mattab]

Hi guys, closing this issue, feel free to comment in any existing Scheduled Reports issues, or create new ones!