Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation to address when to set PRODUCTION_MODE: "true" #111

Open
andrewnhem opened this issue Apr 16, 2021 · 3 comments
Open

Add documentation to address when to set PRODUCTION_MODE: "true" #111

andrewnhem opened this issue Apr 16, 2021 · 3 comments

Comments

@andrewnhem
Copy link
Contributor

We've had a few users address openbalena API crashes by setting PRODUCTION_MODE: "true" to force restarts.

https://forums.balena.io/t/api-container-crashing/273784
https://forums.balena.io/t/openbalena-crash-error-getaddrinfo-enotfound-api-github-com/285697

Not sure if this is just a hack/workaround, or if we want to address this in documentation, since both users and support agents have tried to find a link or context on the variable.

@bartversluijs
Copy link
Contributor

Should production mode be 'true' by default instead of 'false'?
As far as I can tell, looking at the code in the API and VPN, the production mode set to false is kind-of like a debug/development mode.

Most people running openBalena aren't debugging / developing.
I can understand it's still 'false' by default, because of the logs outputted by the system, which are needed when debugging a problem. But the "stability" of the system is lower, because in case of an error in the API, it'll stop and you've to restart it manually.

Maybe some food for thought 🙂

@PaulAnnekov
Copy link

Today found that api crashed by OOM:

Aug 18 16:37:54 e7f399138796 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=37c3cee2673d0b9e70713ddcd7c282e32f6281de1d0d36e4b287807f4b8b75b1,mems_allowed=0,global_oom,task_memcg=/system.slice/containerd.service/system.slice/open-balena-api.service,task=node,pid=48384,uid=0
Aug 18 16:37:54 e7f399138796 kernel: Out of memory: Killed process 48384 (node) total-vm:2028628kB, anon-rss:218764kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:5692kB oom_score_adj:0
Aug 18 16:37:54 e7f399138796 kernel: oom_reaper: reaped process 48384 (node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Aug 18 16:37:54 e7f399138796 api[780]: Program node index.js exited with code null

and not restarted, because PRODUCTION_MODE is false by default and supervisor doesn't restart if exit code !== 0. Toggled mode in config/activate, don't know if that's the best place for toggling, but it works as expected now.

I agree with @bartversluijs that it should be production by default. Also docs should mention somewhere for available env vars and where to modify them.

@dfunckt
Copy link
Member

dfunckt commented Aug 18, 2021

I don't think defaulting to production mode is desirable at this point, considering openBalena isn't supposed to be used in production yet. Apart from restart behaviour, another change this would cause is silencing almost all output from services (definitely the API, maybe others too) which is typically useful for debugging and/or reporting issues. For example, you wouldn't really see that your instance goes OOM without having to dig in much deeper.

Toggled mode in config/activate, don't know if that's the best place for toggling, but it works as expected now.

This is the way to do it 👍 Alternatively, you can selectively override this (or any other environment value) per service in your config/docker-compose.yml file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants