F-Droid background syncing

[ouchadam]

Centralising the fdroid specific background notification issues - #3263 #3159 #2827 #2669 #2143 #1419 #2519

Context

The fdroid variant of element does not use Google play services which means we cannot receive push notifications (FCM). Instead we must manually poll the server for new events at a scheduled rate whilst the app is in the background to achieve a similar messaging experience.

Element currently provides 2 mechanisms for polling in the background

• Optimised for battery
  • Self rescheduling WorkManager workers
  • The recommended way to do background work, respects battery optimisations like Doze, inexactly triggering at set times or during the Doze maintenance window.
• Optimised for real time
  • Self rescheduling short lived ForegroundService triggered by AlarmManager.setAndAllowWhileIdle
  • AlarmManager usage allows some system battery optimisations to be skipped like executing during Doze however they're limited to once per 9 minutes.

but...

Both mechanisms suffer from OEMs being able to ignore our background scheduled tasks and the system applying restrictions to how often we can schedule.

#3844 approaches the problem by using a long running ForegroundService along with an explicit timer to manually trigger and reapply the background scheduling. We have reports that this resolves the notification issue.

#2921 is similar in approach by using PeriodicWorkManager workers as a way to ensure the polling workers are being scheduled, there's a 15 minute minimum interval. This approach still suffers from the system being able to ignore the scheduling.

Through my local testing simply the existence of a foreground service allows the Element process to avoid being killed early (especially in combination with disabling battery optimisations)

*Pocofone F1 MIUI android 10 & Nexus 6P android 8.1.0
*Unplugged on WIFI
*Battery optimised background sync set to 60 seconds
*Noop foreground service
*Writing a timestamp to disk each time the sync worker is triggered

NEXUS 6P WITHOUT SERVICE	POCO WITHOUT SERVICE	POCO WITH SERVICE
08:31	21:40	10:50
08:32	21:42	10:52
08:34	was running overnight, no other syncs	10:56
08:35		10:57
08:42		11:01
08:52		11:11
09:10		11:28
09:26		11:50
09:43		12:08
10:07		13:05
10:08		14:17
every minute until 12:20		14:54
12:50		15:56
13:51		15:59
15:46		18:14
19:45		18:43
21:33 plugged in to collect results		19:27
		21:31 plugged in to collect results

We can see the WorkManager quickly starts to ignore our 60 second rescheduling requests on the Pocofone, but it takes a few hours for the Nexus 6P to start scaling back.

The foreground service also protects against #2143 (closing the app through recents, only some OEMs ignore WorkManager/AlarmManager jobs when doing this)

Proposed solution

1. A long running ForegroundService with a persistent notification, including when the app is in the foreground. This will stop the system from ignoring our scheduled work when the app is in the background/has been stopped.

An example notification...

PERSISTENT NOTIFICATION

• This has the downside that the system may label us as consuming battery despite our service doing nothing.

2. 1. may not be enough for some OEMs, this is where we provide another background sync option in which we manually reschedule events instead of relying on the system, this will use more battery than the other mechanisms. I would argue for only doing this once we confirm 1. doesn't fix the issue.

CURRENT OPTIONS

[opusforlife2]

This has the downside that the system may label us as consuming battery despite our service doing nothing.

The FairEmail dev had to create an entire FAQ entry just for this: https://github.com/M66B/FairEmail/blob/master/FAQ.md#user-content-faq39

[daniellekirkwood]

For content, what about "We're listening for notifications..."

[ouchadam]

It might be just me but it somehow feels invasive when including "we're", compared to "Listening for notifications"

Like element is personally listening to my notifications 😅

[daniellekirkwood]

Haha, fair point @ouchadam. Cut to "Listening for notifications" then 👍

[ouchadam]

Is part of 1.3.7

[opusforlife2]

Finally (FINALLY) got the 1.3.7 update on F-Droid.

But now I'm confused as to what the default configuration is supposed to be. I was expecting just a single toggle for a permanent sync notification to be created, like Telegram-FOSS has. Maybe Element should also do the same.

Anyway, should I choose battery or real time? What is the recommended sync delay value, and sync timeout value? There is no default button to reset either, and once you change them, you have no idea what the default value was.

If only the real time option will use the sync delay value (that's what I've understood from the description), then the setting should be greyed out for the battery option, no?

---

There is also no check mark or something similar next to the currently selected option in the sync mode menu. You have to dismiss it to see what the current value was. Just a minor QoL issue.

[opk12]

I have not had any notification issue (including late or not arrived notifications, background process auto-closed) on a Samsung phone with Android 7.1.1, Element from F-droid, setting Optimized for battery.

Is it possible to disable the persistent notification for this case?

[ouchadam]

But now I'm confused as to what the default configuration is supposed to be. I was expecting just a single toggle for a permanent sync notification to be created, like Telegram-FOSS has. Maybe Element should also do the same.

Anyway, should I choose battery or real time? What is the recommended sync delay value, and sync timeout value? There is no default button to reset either, and once you change them, you have no idea what the default value was.

this is a good point, I agree these options are confusing (and in my opinion unnecessarily verbose!)

MODE	SETTINGS

• battery is the recommended
• real time is for when battery doesn't work (which might not be needed anymore...)
• Delay between each sync is the polling interval, defaults to 60 seconds (how often we'll wake the device up to check for new messages)
• Sync request timeout is maximum time we'll keep the device awake for each time the sync triggers, if it's too short, or the server is slow at the time of the sync, notifications can be missed
• Start on boot allows the polling to start straight away, otherwise messages can't be received until the first time the app is opened per reboot

Both battery and real time modes use the sync interval butttt... depending on the OS, if the device is dozing, previous of amount of syncs etc, the sync interval can be ignored in different ways for both battery and real time. Some older versions of android are happy to continuously sync in the background, some OEMs can have harsh restrictions to attempt to improve battery life.

---

I have not had any notification issue (including late or not arrived notifications, background process auto-closed) on a Samsung phone with Android 7.1.1, Element from F-droid, setting Optimized for battery.

Is it possible to disable the persistent notification for this case?

We could add the option but it opens the door to users turning it off and encountering errors, on Android 8.0+ it's possible to hide this via app notification channel settings

---

• To simplify the background sync options (do we need to expose control the timings or if we start on boot?)
• Add an option to disable the persistent notification for 7.1.1 and below (android 8.0 has this feature built in to the OS)

cc @daniellekirkwood

[JanZerebecki]

Anyway, should I choose battery or real time? What is the recommended sync delay value, and sync timeout value? There is no default button to reset either, and once you change them, you have no idea what the default value was.

Just delete anything in the input field and save. Then the default will be used and shown.

[opusforlife2]

(how often we'll wake the device up to check for new messages)

This only applies to if the device is left unattended and the screen is turned off, right? I'm assuming if the phone is being actively used there is no "waking" to be done, since there is a permanent foreground notification.

(do we need to expose control the timings or if we start on boot?)

Maybe this could be retained for Developer mode.

[mcg-matrix]

Is part of 1.3.7

I am running Element-FDroid on /e/OS 0.18-o in an "always online" mode (= "optimized for real time", 0 seconds delay, 179 seconds timeout; notifications working perfectly). I can report that version 1.3.7 is using 25 to 30% less energy when idling. Thanks!

Element is now down to using ~8 times as much battery as Conversations.