About SSH

[ArrayIndexOutOfBound]

Hello,

First of all, thanks for this project, it sems to be very usefull for me, but even for futur people who will find this little gem.
Anyways, if I am here to report an issue, it's because it seems that I'm not able to connect via SSH to my device.

The error itself :

After searching a bit in the files of openbalena, and the files if this project, I found this :
The content of openbalena/config/activate

I guess that this specific line shouldn't be empty ?
If I'm trying to do a ssh connection via the balena-cli, it asks for the password of admin@ssh.devices.mydomain.notlocal, which I do not remember setting up anywhere.

Thanks for giving me a bit of your time, have a good day.

[dcaputo-harmoni]

@ArrayIndexOutOfBound thank you for trying it out! Could you provide some additional details on your setup? Also, are your devices running in development or production mode? If you are not able to establish a tunnel to a device using balena tunnel, connect won't work. As currently implemented, SSH only works with devices in development mode - i.e. that it is accessible using root login (no password) on port 22222. We would need to implement some additional logic to pass SSH keys and/or passwords to open-balena-remote for SSH to work on production mode devices. Other services (http, vnc) should work on both development and production mode as long as the appropriate container ports are opened on the host.

[ArrayIndexOutOfBound]

Hello again !

Yep, I do think that I am using the production image instead of the development one, I will change that if needed (I think it's necessary after reading the point about allowing root login).
I guess that explain why I needed to manually add the ssh-keys in the config.json on the boot partition
About the tunnel, I don't know how to mount one, sorry.

Here is a little diagram of how my devices are installed :

As you might have already guessed, english isn't my primary language, sorry if there is some spelling / grammatical errors.

Thanks for your time !

[dcaputo-harmoni]

@ArrayIndexOutOfBound your setup looks fine to me - other than changing to development mode (which hopefully won't be required once we implement the ability to utilize SSH keys) I don't see any issues. What happens if you try running the command from your local computer: balena tunnel <uuid> -p 22222? And if that is successful, try running ssh root@localhost -p 22222. In essence, that is all that remote is doing.

[ArrayIndexOutOfBound]

Hello,
I tried doing the tunnel and then using the ssh : here are the results :
Creating the tunnel (on the server):

Trying to SSH localhost (on the server):

Trying to SSH via the local computer :

Here is what the dashboard is telling me :

It seems that the OS is still on "prod" mode ? But I did this when downloading a second time the os :

BalenaPi4_DEV.img being the image downloaded directly on the balena page, unziped

I will update this post I find something.

[ArrayIndexOutOfBound]

Hello again !

I managed to add "developmentMode": true to the config.json, and now the OS on the web page :

I still am getting the "Error" when trying to SSH via the open-balena-ui, but when creating the tunnel by hand, it seems to be working well.

[dcaputo-harmoni]

@ArrayIndexOutOfBound sounds like we are getting closer. Could you confirm the URL that is being passed to open-balena-remote is properly formed? You can get the URL by right clicking the iframe that connect is running in and inspecting the details. And then can you try running that same URL in a new browser window? I am wondering if this is a CORS issue related to the iframe, and we need to relax your browser settings.

And while you are inspecting the iframe, take a look to see if there are any browser console errors reported. If it is a CORS issue, you will see the errors in the console.

[ArrayIndexOutOfBound]

The link in the iframe is something like this:
http://remote.domain.notlocal:8000/?service=ssh&uuid=string&jwt=superlongstring

The console seems to be empty when loading this page :

I tried some ports :

• 8080 : works fine, I have acess to the UI
• 10000-10009 : gives me the "error" page
• 8000 : seems to be broken, here is the message that is printed
  

If it might help, here is the content of docker ps -a

[dcaputo-harmoni]

OK - a couple of things from your response. I'm not sure why the iframe URL would be referencing port 8000, it should be 10000. Can you try pasting that same URL (with all of the query parameters, including jwt) into a new browser window and change port 8000 to 10000?

If you take a look in the services.yaml file that builds open-balena-admin you will see the following line:

REACT_APP_OPEN_BALENA_REMOTE_URL: http://remote.${OPENBALENA_HOST_NAME}:10000

This is the same environment variable that is used to generate the URL to call connect (see below from src/ui/DeviceConnect.js in the open-balena-ui container:

            let REMOTE_HOST = process.env.REACT_APP_OPEN_BALENA_REMOTE_URL;
            let session = authProvider.getSession();
            let containerChoices = [{id: 0, name: 'host'}];
            let containerServices = [[{id: 0, name: 'SSH'}]];
            let containerLinks = [[`${REMOTE_HOST}?service=ssh&uuid=${props.record.uuid}&jwt=${session.jwt}`]];

I can't think of a reason why your instance would be trying to call port 8000 - I'm assuming you did not modify services.yaml?

[ArrayIndexOutOfBound]

SORRY !
I meant -->
http://remote.domain.notlocal:10000/?service=ssh&uuid=string&jwt=superlongstring
Not -->
http://remote.domain.notlocal:8000/?service=ssh&uuid=string&jwt=superlongstring
As I typed this by hand, and at the same time testing the opened ports, I wrote something wrong.

I did not touch the service.yml file, knowing well that I would break something.

[dcaputo-harmoni]

OK, that makes more sense. So the issue seems to be isolated to the open-balena-remote container. Do you have SSH access to this container? if so, can you ssh into that container, kill the active node process (which you can find using ps -A), modify open-balena-remote.js to change the following line:

const DEBUG = false;

to:

const DEBUG = true;

Then run node open-balena-remote.js, and try to access remote either via the admin user interface or directly from the browser window, and send the debug info you see in the console (sanitized as you see fit) -- thanks.

UPDATE while you are in that container, you should also try using the balena tunnel command to open a connection to your device. You'll need to first use balena login since you are accessing your balena server for the first time from that host.

[ArrayIndexOutOfBound]

Alright !

As I can understand, I have a problem with a self signed certificate

I don't know how certificates works, but does it have to do with the fact that when I'm trying to acess the webpage (admin or api), I sometimes get a warning about the fact that the website is 'dangerous ?'

When doing history on the server, I was able to retrieve the following quickstart of openbalena

bash quickstart -U supersecret@admin.something -P ultrasecret -d domain.notlocal -p

And for openbalena-admin

bash quickstart -j longstring -v 0.139.0 -d same.domain

[dcaputo-harmoni]

OK, looks like you found the issue. It appears your open-balena server is using a self-signed certificate, and balena-cli does not like self signed certificates. I'm surprised your local balena-cli instance is not complaining in the same way - have you made any accommodations on your local machine to address this in the past?

One solution you can try is modifying the following code in open-balena-remote.js:

          sessionData.tunnel.pid = await executeCommand(tunnelCmd, {
            uuid: req.query.uuid, 
            remotePort: remotePort,
            localPort: sessionData.tunnel.port,
            tunnelID: sessionData.tunnel.id
            }, {BALENARC_DATA_DIRECTORY: sessionData.sessionDir}, true);

to this:

          sessionData.tunnel.pid = await executeCommand(tunnelCmd, {
            uuid: req.query.uuid, 
            remotePort: remotePort,
            localPort: sessionData.tunnel.port,
            tunnelID: sessionData.tunnel.id
            }, {BALENARC_DATA_DIRECTORY: sessionData.sessionDir, NODE_TLS_REJECT_UNAUTHORIZED: 0}, true);

And then kill the prior process and re-run it per my earlier instructions. I suspect this would fix your issue, but it's really more of a workaround and I'm not sure we would want that in the main branch of open-balena-remote as it could create security issues for others who utilize valid certificates. I have a broader plan to replace balena tunnel with a custom https proxy tunnel solution, but it's lower on the list as balena really struggles with self signed certificates generally and I suspect you will run into other issues with balena.

[ArrayIndexOutOfBound]

I have tried modifying the open-balena-remote-js, the previous error has dissapeared, but it seems to be once again, blocked by the fact that the certificates are not valid.

I will try to search on how to make the certificates "approuved" by my/any web browser.
It cannot be 'that' complicated, right ...?

As for why my balena-cli on the server seems to be ok, it's because I followed the insctruction on the getting started page on my .profile and .bashrc so it get set on reboot.
export NODE_EXTRA_CA_CERTS='/path/to/ca.crt'

[ArrayIndexOutOfBound]

Update :
I did a fresh install of open-balena, but this time, with the -c parameter, which created good certificates (since now, neither my web browser nor balena-cli is complaining about self signed certificates).

Here are the logs of open-balena-remote.js, with DEBUG set to true :

What htop is showing :

I can confirm that locally, the Raspberry is responding to the SSH connection, on port 22222

Edit :

In the context of the remote docker, when trying to ssh via the port 20000, it works !

Edit 2 :

Here is my /etc/hosts on the server (I didn't edit anything when connecting via my personal computer)

The lines are in the form of *.TheDomainName.fr (so, not local)

[dcaputo-harmoni]

Seems you are making further progress. Can you try inspecting the iframe again, copy the URL and open it in a new browser window to see if it works there. Then try changing http to https in the URL for that new window and see if it makes a difference.

[ArrayIndexOutOfBound]

Copying the URL to another window doesn't work.
With HTTP, I get the classic ERROR page.
With HTTPS, it seems to not be supported since I'm getting an ERR_SSL_PROTOCOL_ERROR page.

I think I need to change some parameters ? Since it works when inside the docker container, it seems to be working well.
Or maybe check the proxy at port 10000, since the error seems to be coming from that "no valid session"

It's very close to working accordingly, at least some people might find this thread quite helping if they encounter the same problem.

Edit : when doing an ssh root@localhost -p 20000 from the docker, as saw before, it works just fine.
Here are the logs :

Does that mean that something went wrong in the process when doing it from the web page ?

[dcaputo-harmoni]

@ArrayIndexOutOfBound can you confirm what browser you are using? open-balena-remote uses localStorage to store the session ID, and it seems that it can’t find your session from the logs. Here is the compatibility chart for localStorage:

https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage

also, can you confirm if you are seeing any errors in the browser console?

[ArrayIndexOutOfBound]

I am using Google Chrome, and is up to date.
I will install Mozilla and post this in a few minutes.
Mozilla is more talkative than Chrome : I am getting a cookie error I think ?

I don't know how they work, but as I undertsand it, it's discarded since it's not considered as "secure" ?

[dcaputo-harmoni]

Ok, further progress. That’s clearly the issue you are having now - without the cookie proxy can’t work. Can you try going to "chrome://flags" and disabling "Cookies without SameSite must be secure"?

[ArrayIndexOutOfBound]

I tried by having these two rules at disabled and then enabled

I am currently searching for equivalent settings on FireFox

[dcaputo-harmoni]

Not sure if it is a translation thing but those don’t look like the right settings. Here is another article that describes which settings to change:

https://www.chromium.org/updates/same-site/test-debug/

[ArrayIndexOutOfBound]

According to the documentation, since my Chrome is version 100 and it's said that :

NOTE: These flags were removed in Chrome 91, as the behavior is enabled by default as of Chrome 85. If you are running Chrome 91 or newer, you can skip to step 3.

I have nothing to do with theses flags that seem to be Enabled by default now.

If I translate the error message, we get something like

Cookie “remote.10000” has been rejected because a non-HTTPS cookie can’t be set as “secure”.

Do I need to remove the secure somewhere, or maybe check on how to setup certificates for your docker images ?

[ArrayIndexOutOfBound]

Alright !
By setting the cookie to false here and exclusively with Mozilla for now, it works !

I won't yet close this issue, because I will try to make it work over HTTPS and with the cookie set to secure AND on Google Chrome.
In the meantime, thanks a lot for your time, I learned things during this couple of days !

[dcaputo-harmoni]

Great! And just to confirm, when you say that you set the cookie to false do you mean you set "httpOnly" to false, or "secure" to false? (or both?) It seems this issue is related to http connections; remote is running fine on my https server otherwise. If this is the case, I can work in a parameter into the quickstart script to specify if the servers will be running via http or https, and if https, it will set all URLs to https and also set secure to true, otherwise all will be http and secure will be false.
Thank you for all of the testing! This will help improve the product for everyone.

[ArrayIndexOutOfBound]

I changed only "secure" to false to make it work on Firefox, I still am searching on why Chrome seems to be reluctant about the cookie.
I think that being able to to choose between http and https is a good idea on your script !
But does that mean that we will need to change some of the openbalena cert-provider configs to generate some certificates for your dockers ?

[dcaputo-harmoni]

Added insecure mode option at build time quickstart -i which sets secure mode to false in remote container.

[anthonyra]

Added insecure mode option at build time quickstart -i which sets secure mode to false in remote container.

./scripts/quickstart -j $OPENBALENA_JWT_SECRET -v $OPENBALENA_API_VERSION_TAG -d $OPENBALENA_HOST_NAME -i
Invalid argument: -i
usage: ./scripts/quickstart [-h] [-d DOMAIN] [-U USERNAME] [-P PASSWORD] [-p PORT] -v VERSION -j JWT_SECRET

[shawaj]

@dcaputo-harmoni is this solved now re SSH with keys in production?

Based on this post? https://forums.balena.io/t/production-image-ssh-access/355781