[Telemetry] Use lastReported timestamp on the 'browser' as well

[afharo]

At the moment, the browser telemetry sender stores the lastReported timestamp on the local storage. This means that we report telemetry every 24h from each user's browser (+ the server when it applies).

Should we change the browser implementation to check the lastReported key from the SOs instead?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[Bamieh]

I think it is OK to send multiple reports:

• we dont retry if the browser failed to send the data, this way we can have more chances that we are getting some data.
• We can get some interesting insights from this redundant data.
• We'll have a more updated snapshot of usage in that day if it was reported multiple times throughout the day by multiple users.

The drawback is that we are sending multiple reports which means increased redundancy on our cluster. But I think we've been handling that fairly well so far.

[afharo]

I agree with the benefits of multiple reports. I just want to weight in the cons as well:

• When analysing the time-based trends, clusters with more users have more weight (more reports) over others. Special case: "Publicly-open" Kibanas with hundreds of users accessing.
• The additional insights we might get are spread across events. Maybe they're easier to analyse if they belong in one event? 🤔 A specific collector might help, although we'd need further research/discussion on this.
• Load: As we increase the number of collectors, there's load in the Kibana & ES servers for every report generation will keep increasing in the future as well. Minimizing the number of executions helps with a smoother UX.

Can we do anything to achieve similar benefits to the ones listed above for multiple reports, but sending one report only? i.e.: adding a retry logic, a specific collector for those additional insights, ...

What do you think?

[Bamieh]

When analysing the time-based trends, clusters with more users have more weight (more reports) over others. Special case: "Publicly-open" Kibanas with hundreds of users accessing.

It is advised to nest-aggregate per cluster_uuid for almost all queries ran against our all-* indices.

Load: As we increase the number of collectors, there's load in the Kibana & ES servers for every report generation will keep increasing in the future as well. Minimizing the number of executions helps with a smoother UX.

I agree this might be an issues. I think we can use telemetry to answer this; what is the average number of users sending telemetry per day per cluster. If we find huge numbers then i think it is worth tackling this issue.

i.e.: adding a retry logic

The API will return 200 regardless if we actually store the data to ES or not.

I am also wondering if we have a way to tell how much data we might lose if we only send it once per day from the browser 🤔

[afharo]

I think we can use telemetry to answer this; what is the average number of users sending telemetry per day per cluster. If we find huge numbers then i think it is worth tackling this issue.

It looks like, on average, we do 3 daily reports per cluster. However, I can see clusters that report 1700 per day and some others 600.

[Bamieh]

I think we are OK with that level of redundancy. Especially since we might risk losing usage if we get rid of the reduencancy model we have. What do you think?

[afharo]

I sincerely don't know... on the average level, 3 seems like a fairly small amount of redundancy. On the flip side though, 1700 or 600 seem like a lot of extra load. If the only reason to keep it is the additional insights we get, I'd say we could create collectors to provide those same insights and fix this issue.

If any of the insights is not replaceable with any other collector, then I guess it's OK to close this issue and revise in the future 😇