Raspberry Pi 2: can't launch ot-recorder ("Cannot allocate memory")

[neurolit]

Hi!

I just installed OwnTracks on my Raspberry Pi 2 (with freshly installed Raspberry Pi OS) using quick setup and bootstrap.sh. Everything went well, with no error.

However, part of OwnTracks does'nt work: ot-recorder service isn't started, with this error message:

Jul 26 07:35:23 raspberrypi systemd[1]: Starting ot-recorder.service - OwnTracks Recorder...
Jul 26 07:35:26 raspberrypi systemd[1]: Started ot-recorder.service - OwnTracks Recorder.
Jul 26 07:35:26 raspberrypi ot-recorder[32074]: version 0.9.8 starting with STORAGEDIR=/var/spool/owntracks/recorder/store
Jul 26 07:35:26 raspberrypi ot-recorder[32074]: gcache_open: mdb_env_open: Cannot allocate memory
Jul 26 07:35:26 raspberrypi ot-recorder[32074]: Can't initialize gcache in /var/spool/owntracks/recorder/store/ghash
Jul 26 07:35:26 raspberrypi systemd[1]: ot-recorder.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 07:35:26 raspberrypi systemd[1]: ot-recorder.service: Failed with result 'exit-code'.

I thought using the solution described in #404 and #348 would solve this problem, but it doesn't. I add Environment="OTR_LMDBSIZE=10485760" in [Service] part of /etc/systemd/system/ot-recorder.service, deleted /var/spool/owntracks/recorder/store/ghash/*.mdb files, exported OTR_LMDBSIZE=10485760 in the shell before running ot-recorder --initialize (which created new mdb files), but when I start ot-recorder (with sudo systemctl start ot-recorder), the same error happens.

Do you know what I could try in order to run ot-recorder?

[jpmens]

I'm sorry you're having trouble getting that to start!

I've not seen an Rpi 2 for a bit, so I'm just guessing. Can you please try lowering that cache size a bit, say

rm -f /var/spool/owntracks/recorder/store/ghash/*.mdb
export OTR_LMDBSIZE=1048576  # 1MB
ot-recorder --initialize

ot-recorder   # launch it on the console (for testing, no need to use service)

and see whether it will then start?

[neurolit]

Thanks @jpmens for your help.

It started perfectly:

ot-recorder[10170]: version 0.9.8 starting with STORAGEDIR=/var/spool/owntracks/recorder/store
ot-recorder[10170]: connecting to MQTT on 127.0.0.1:1883 as clientID otrec without TLS
ot-recorder[10170]: HTTP listener started on 127.0.0.1:8083, without browser-apikey
ot-recorder[10170]: HTTP prefix is https://[redacted]/owntracks
ot-recorder[10170]: Using storage at /var/spool/owntracks/recorder/store with precision 7
ot-recorder[10170]: TZDATADB is at /usr/share/owntracks/recorder/timezone16.bin: R_OK
ot-recorder[10170]: Subscribing to owntracks/# (qos=2)

Should I try to change environment variable in /etc/systemd/system/ot-recorder.service and try to start the service?

[jpmens]

Yes, please do, and I don't doubt (famous last words) it will work.

Depending on how many reverse-geo entries you will produce over time (I am assuming you configured OpenCage with a key), the space we've reserved for LMDB might become a bit tight, but we'll have to cross that bridge when we get there.

You might like to try 2MB

export OTR_LMDBSIZE=2097152
...

[neurolit]

I tried 2097152, and it worked when I recreated the mdb files, and when I launched (without the service) ot-recorder. Good.

However, after having updated /etc/systemd/system/ot-recorder.service, launched sudo systemctl daemon-reload and sudo systemctl start ot-recorder, the same problem arises:

Jul 26 14:17:50 raspberrypi systemd[1]: Starting ot-recorder.service - OwnTracks Recorder...
Jul 26 14:17:53 raspberrypi systemd[1]: Started ot-recorder.service - OwnTracks Recorder.
Jul 26 14:17:53 raspberrypi ot-recorder[10653]: version 0.9.8 starting with STORAGEDIR=/var/spool/owntracks/recorder/store
Jul 26 14:17:53 raspberrypi ot-recorder[10653]: gcache_open: mdb_env_open: Cannot allocate memory
Jul 26 14:17:53 raspberrypi ot-recorder[10653]: Can't initialize gcache in /var/spool/owntracks/recorder/store/ghash
Jul 26 14:17:53 raspberrypi systemd[1]: ot-recorder.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 14:17:53 raspberrypi systemd[1]: ot-recorder.service: Failed with result 'exit-code'.

I tried with 1048576, same.

😭

So maybe the environment value isn't used by service? Here is my service file:

# /etc/systemd/system/ot-recorder.service

[Unit]
Description=OwnTracks Recorder
Wants=network-online.target
After=network-online.target

[Service]
Environment="OTR_LMDBSIZE=1048576"
Type=simple
User=owntracks
WorkingDirectory=/
ExecStartPre=/bin/sleep 3
ExecStart=/usr/sbin/ot-recorder

[Install]
WantedBy=multi-user.target

[jpmens]

Did you run

systemctl daemon-reload

after changing the unit file? Yes, you did.

Do you see your change with

systemctl cat ot-recorder

?

[jpmens]

Oopsie!

        if ((p = getenv("OTR_LMDBSIZE")) != NULL) {
                lmdb_size = atol(p);
                if (lmdb_size < 10485760) { // 10 MB
                        lmdb_size = 10485760;
                }
        }

If the configured size is below 10MB we set it to 10MB, hence what we're doing here has no effect.

Are you able to compile code on your Rpi? I could provide a patch quickly.

[neurolit]

Oh oh: systemctl cat ot-recorder gives:

# /etc/systemd/system/ot-recorder.service

[Unit]
Description=OwnTracks Recorder
Wants=network-online.target
After=network-online.target

[Service]
Environment="OTR_LMDBSIZE=1048576"
Type=simple
User=owntracks
WorkingDirectory=/
ExecStartPre=/bin/sleep 3
ExecStart=/usr/sbin/ot-recorder

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/ot-recorder.service.d/override.conf
[Service]
# works as environment variable only (not in ot-recorder defaults)
Environment="OTR_LMDBSIZE=5368709120"

Where does this override.conf comes from?

[jpmens]

It's created during Quicksetup config, but that's not the problem, sadly. See above for my previous "Oopsie" comment.

[neurolit]

If the configured size is below 10MB we set it to 10MB, hence what we're doing here has no effect.

Are you able to compile code on your Rpi? I could provide a patch quickly.

I tried manually with 10MB, and it works! So the problem comes from this override.conf, and I could go with 10MB (without a patch, then).

The value set by override.conf is too large, but 10MB is OK.

[jpmens]

Thanks for finding that! (You know my code better than I do -- tbh I'd forgotten about the override.)
I have submitted a patch now which will in future print the desired size of the lmdb cache; we'll be able to pinpoint the source more easily, so thanks for that!

Very glad you got it sorted.

[jpmens]

Keep in mind, please, that when you re-run ./bootstrap.sh in Quicksetup, your 10MB will be overwritten. I will apply a fix to Quicksetup in order to avoid doing so if the file exists.

[neurolit]

Thank you very much for your help! I updated /etc/systemd/system/ot-recorder.service.d/override.conf (by using 10MB) and it works now 🍾

[jpmens]

Thank you for your help. I think we've added two (very simple but useful) changes.

If you'll please run a git pull in your Quicksetup directory, you'll be set.