Main UI works, but Device connection for SSH/HTTP doesn't. Logs seem to be intermittent.

[Zebble]

I'm running the latest version of open-balena-admin on a docker setup (not K8S) and able to connect to the admin ui at http://admin.[[domain name]]:8080. I'm running open-Balena 3.4.3.

Everything from an adminitration standpoint seems to work well. I can see Fleets, Devices and I'm really looking forward to the additional Orgs/Users functionality!

When I attempt to access Host or Container logs, more often than not I just get a white screen.

The same goes for any attempt to connect to a Host/Container over SSH or for any containers I've enabled with the corresponding HTTP labels. The option to connect shows up, but any attempt does nothing.

I don't see any corresponding info in the open-balena-admin and related container logs. Where can I start troubleshooting?

Thanks again!

[dcaputo-harmoni]

@Zebble glad you got the admin side working! For SSH troubleshooting, try following the steps from this issue to determine the root cause. In that case it had to do with not being hosted on a secure server, but the debug logging step might help shed light on the issue:

#3

[Zebble]

Thanks @dcaputo-harmoni ! I did some additional digging. I am definitely running devices in Production mode, so that explains the SSH issue. I do get the same errors though when trying to connect via HTTP so is it possible this is also broken because of Production mode?

For Logs, the error seems to be about authorization. I checked the chrome console and the bearer token is nothing I recognized. It's super long and doesn't match any api keys which explains the 401 error. Is this not supposed to be the api-key that is assigned to the logged-in user?

[dcaputo-harmoni]

Actually the production SSH limitation was recently addressed - see this thread:
https://forums.balena.io/t/production-image-ssh-access/355781
As long as you are running the latest builds of open-balena-ui and open-balena-remote it should be generating SSH tokens for production images dynamically. The latest images on dockerhub include those changes.

[Zebble]

Thanks again.

I am running the latest and greatest of openBalena and open-balena-admin and I double-checked by doing a compose down/compose pull/compose up steps for both.

Something still isn't right. If I just test SSH, I still get the white screen. The Chrome console is showing a generateKey error:

Any ideas?

[dcaputo-harmoni]

Looks like the issue is with the crypto.subtle package that generates the RSA keys. According to this:
https://developer.mozilla.org/en-US/docs/Web/API/Crypto/subtle
It requires a https connection - can you confirm if you are using https for your connection to open-balena-ui?

[Zebble]

I just read similar documentation and tried putting the open-balena-ui container behind an nginx reverse proxy. The certs now work but most of the interface is broken ("Failed to fetch" errors). Is there an easier way to enable a secure connect to the open-balena-ui?

[Zebble]

I have an idea to get this working behind nginx but I'm going to need to move the open-balena-admin containers to a different host than open-balena itself. Port 443 conflicts with the open-balena-haproxy container.

I'll give this a shot tomorrow.

[feature request] Would be nice if somehow the open-balena-haproxy container itself could also be leveraged by open-balena-admin!

[dcaputo-harmoni]

regarding haproxy, take a look at the k8s scripts as I have already integrated a haproxy instance for open-balena-admin there. It would be great if you could help build out the same for our docker compose scripts. I believe you are right that you would need to run it on a separate server to do so because of the port 443 conflict. In k8s we are able to just use a separate IP address but I don’t believe we can do this in docker compose, unless you know of a way?

I have thought of integrating our haproxy with open-balena as well but this would require merging the repositories because we would have to modify the config scripts which are part of open-balena. I think we are better off keeping this project separate for now as we won’t have to wait potentially extended timeframes to have each commit approved as would likely happen if we were part of the main open-balena repo. And also because this is an alternative to balena-cloud I don’t think it would make sense to integrate this with open-balena as they also use open-balena as a base for balena-cloud so the functionality would be redundant for them.

[dcaputo-harmoni]

Also be aware with the separate server we might have a problem that open-balena-postgrest needs to be internal to open-balena-db to communicate with it as the db is not publicly exposed. We are able to navigate the issue in k8s with a separate IP, not sure how we can handle this in docked compose.

[Zebble]

I'm a complete noob with k8s, so I'll probably use this as my excuse to finally get up to speed. :)

I don't know of a way that docker could handle a 2nd IP, or any way around this. I did take a look at the open-balena haproxy config and it looks easy enough to modify to add your containers to so I may spin up a custom image and start there. Everything is contained in the single haproxy.cfg file so I wonder if we could submit a PR with the additional configuration required? I obviously wouldn't work for a standard open-balena install but shouldn't break anything either. Then it would just be ready to plug-and-play if anyone wanted to add open-balena-admin.

I don't see this as heavy competition for BalenaCloud as I think that's a different audience so hopefully the PR would be accepted.

[dcaputo-harmoni]

I think it would be fine to try to submit the PR, I just don't expect them to accept it as they don't officially support open-balena-admin. But no harm in trying.

It looks like docker does actually support exposing ports on multiple IP addresses, provided the host has multiple IP addresses to expose them on. Take a look at the first answer in this thread:

https://stackoverflow.com/questions/40296993/docker-publishing-ports-to-multiple-ips

It looks like you can specify an IP when exposing ports, which also works in docker-compose (just add the IP before the port). Might be worth giving this a shot, I suspect you'll find this easier than forking open-balena as with the latter approach you'll need to keep the fork up to date with the master.

Also, just FYI I've developed a K8S version of the helm scripts which use the latest versions of the various open-balena services, the ones that are bundled in the base open-balena package are over a year old. That might be reason enough to get on the K8S bandwagon... For what it's worth, I only picked it up recently myself, it's not that hard to get up and running when using helm scripts.

[Zebble]

Of course docker can use multiple IP's... I've done it before! Not sure how I forgot! :)

Have been doing some digging and I rememer the haproxy + certificate containers for open-balena were a couple of things that made things complicated in my opinion. A single container with Traefik or NGINX Proxy Manager would replace both and be simpler to maintain. Leveraging NGINX Proxy Manager would provide a GUI for configuration/troubleshooting and would make adding something like open-balena-admin a lot simpler.

I think I'll spin up a test instance of open-balena, manually remove the certificate and haproxy containers, add an NGINX Proxy Manager container and see how things go. Then try adding open-balena-admin to the mix.

Thoughts?

[dcaputo-harmoni]

I think that sounds like a good solution - and while I would encourage you to use it if it works for you, I would be reluctant to incorporate it in the repo. My goal with this project from day one has been to make it fully compatible with out of the box open-balena. While you are setting it up from scratch, most people already have running instances of open-balena and will not want to decommission / reconfigure those running instances. Once we get into a situation where we are requiring customizations to the setup of open-balena, the value of this project is diminished. So wherever possible I want to keep compatibility with stock open-balena, and I believe we can here with custom IP’s.

[ArrayIndexOutOfBound]

Hi, as I updated the container and saw that the SSH connection was not working anymore, I came back here.
Quite understandable since I'm not using HTTPS and all.
So I was wondering, does modifing this file by adding by hand and / or via script, would make the already installed haproxy by open-balena redirect to the names admin.somewhere.net and remote.somewhere.net ?
Also, does that mean that we need to change the cert-provider in the openbalena/compose/services.yml with

DOMAINS: "api.${OPENBALENA_HOST_NAME},registry.${OPENBALENA_HOST_NAME},s3.${OPENBALENA_HOST_NAME},vpn.${OPENBALENA_HOST_NAME},tunnel.${OPENBALENA_HOST_NAME},admin.${OPENBALENA_HOST_NAME},remote.${OPENBALENA_HOST_NAME}"

Edit : I do understand that the main objective is to be compatible with most users of open-balena, but I personally think that just 'adding' open-balena-admin specification to the already running instance, without breaking/touching how the main system work, is not a bad idea.

[tchey290]

@Zebble Did you ever figure out a good solution to this?