503 - Service Unavailable

[ersinpw]

I followed the following docs page very precisely: https://www.balena.io/open/docs/getting-started/
This is my third try, however I stuck every time at the step where the openBalena server is tested. The services that runs on port 80 and 443 returns:

   503 Service Unavailable
   No server is available to handle this request.

Therefore the ping with curl fails also and returns the same.
I am running everything on a brand new Ubuntu 18.04 x64 server.

Edit:

I see lots of people with the same issue. I did a deeper investigation and it seems that the "openbalena_haproxy" container is not working properly. When I open a bash to the container:

docker exec -it e7eae59bbe09 /bin/sh

and fetch the page at 0.0.0.0

wget 0.0.0.0

it returns the same 503 Service unavailable page

[luisgs7]

I also have the same problem, did you manage to solve it, @ersinpw how did you solve it?

Thanks.

[matiasAS]

I have the same problem, I am testing everything on localhost, that is, within a virtual machine, the haproxy.conf file I have changed the name of the service to the ip of the vm 192.168.10.14, leaving it like this:

global
tune.ssl.default-dh-param 1024

defaults
timeout connect 5s
timeout client 50s
timeout server 50s

frontend http-in
mode http
option forwardfor
bind *:80
reqadd X-Forwarded-Proto:\ http

acl is_cert_validation path -i -m beg "/.well-known/acme-challenge/"
use_backend cert-provider if is_cert_validation

acl host_api hdr_dom(host) -i "api.${HAPROXY_HOSTNAME}"
use_backend backend_api if host_api

acl host_registry hdr_dom(host) -i "registry.${HAPROXY_HOSTNAME}"
use_backend backend_registry if host_registry

acl host_vpn hdr_dom(host) -i "vpn.${HAPROXY_HOSTNAME}"
use_backend backend_vpn if host_vpn

acl host_s3 hdr_dom(host) -i "s3.${HAPROXY_HOSTNAME}"
use_backend backend_s3 if host_s3

frontend ssl-in
mode tcp
bind *:443
tcp-request inspect-delay 2s
tcp-request content accept if { req.ssl_hello_type 1 }

acl is_ssl req.ssl_ver 2:3.4

acl host_tunnel req_ssl_sni -i "tunnel.${HAPROXY_HOSTNAME}"
use_backend redirect-to-tunnel-in if host_tunnel

use_backend redirect-to-https-in if is_ssl
use_backend vpn-devices if !is_ssl

backend redirect-to-https-in
mode tcp
balance roundrobin
server localhost 127.0.0.1:444 send-proxy-v2

backend redirect-to-tunnel-in
mode tcp
balance roundrobin
server localhost 127.0.0.1:3129

frontend https-in
mode http
option forwardfor
bind 127.0.0.1:444 ssl crt /etc/ssl/private/open-balena.pem accept-proxy
reqadd X-Forwarded-Proto:\ https

acl host_api hdr_dom(host) -i "api.${HAPROXY_HOSTNAME}"
use_backend backend_api if host_api

acl host_registry hdr_dom(host) -i "registry.${HAPROXY_HOSTNAME}"
use_backend backend_registry if host_registry

acl host_vpn hdr_dom(host) -i "vpn.${HAPROXY_HOSTNAME}"
use_backend backend_vpn if host_vpn

acl host_s3 hdr_dom(host) -i "s3.${HAPROXY_HOSTNAME}"
use_backend backend_s3 if host_s3

backend backend_api
mode http
option forwardfor
balance roundrobin
server balena_api_1 192.168.10.14:80 check port 80

backend backend_registry
mode http
option forwardfor
balance roundrobin
server balena_registry_1 192.168.10.14:80 check port 80

backend backend_vpn
mode http
option forwardfor
balance roundrobin
server balena_vpn_1 192.168.10.14:80 check port 80

backend backend_s3
mode http
option forwardfor
balance roundrobin
server balena_s3_1 192.168.10.14:80 check port 80

backend cert-provider
mode http
option forwardfor
balance roundrobin
server balena_cert-provider_1 192.168.10.14:80 no-check

backend vpn-devices
mode tcp
server balena_vpn_1 192.168.10.14:443 send-proxy-v2 check-send-proxy port 443

frontend db
mode tcp
bind *:5432
default_backend backend_db
timeout client 1h

backend backend_db
mode tcp
server balena_db_1 db:5432 check port 5432

frontend redis
mode tcp
bind *:6379
default_backend backend_redis
timeout client 1h

backend backend_redis
mode tcp
server balena_redis_1 redis:6379 check port 6379

listen vpn-tunnel
mode tcp
bind *:3128
server balena_vpn 192.168.10.14:3128 check port 3128

listen vpn-tunnel-tls
mode tcp
bind *:3129 ssl crt /etc/ssl/private/open-balena.pem
server balena_vpn 192.168.10.14:3128 check port 3128

being the solution to these errors:

Building certificate from environment variables...
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:76] : 'server balena_api_1' : could not resolve address 'api'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:82] : 'server balena_registry_1' : could not resolve address 'registry'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:88] : 'server balena_vpn_1' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:94] : 'server balena_s3_1' : could not resolve address 's3'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:104] : 'server balena_vpn_1' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:129] : 'server balena_vpn' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:134] : 'server balena_vpn' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : Failed to initialize server(s) addr.

and now the errors I have left to resolve are:

(input): docker logs -f openbalena_haproxy_1

(output):
Building certificate from environment variables...
Setting up watches. Beware: since -r was given, this may take a while!
[NOTICE] 315/024440 (16) : New worker #1 (18) forked
Watches established.
[WARNING] 315/024442 (18) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 315/024442 (18) : proxy 'vpn-tunnel' has no server available!
[WARNING] 315/024442 (18) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 315/024442 (18) : proxy 'vpn-tunnel-tls' has no server available!

The local domain configured is openbalena.local, if I enter www.openbalena.local configuring that address in /etc/hosts, the error is the following:

503 Service Unavailable
No server is available to handle this request.

I would appreciate your help in solving this problem

Greetings
Matias

@ersinpw @luisgs7 (speak spanish?, i from chile) @wrboyce @relaxdiego @dfunckt

[ersinpw]

@luisgs7 no I didn't managed to get it work..
@matiasAS no I don't speak spanish

Looks like everyone is having the same issue. @fisehara can you prioritise this issue?

[fisehara]

@ersinpw Thanks for the mentioning.

If you have a working setup which results into a haproxy service running in a container can you please share the logs from the failing HAproxy service?
As all services are initialised from a central docker-compose.yml you should be able to run the command manually which should give you the logs or all running services.

Moreover, are we more than happy for contributions to maintain this project, as the current pinned HAproxy version itself is fairly outdated.

Thanks and best regards
Harald

[fisehara]

@matiasAS

This message:

Building certificate from environment variables...
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:76] : 'server balena_api_1' : could not resolve address 'api'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:82] : 'server balena_registry_1' : could not resolve address 'registry'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:88] : 'server balena_vpn_1' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:94] : 'server balena_s3_1' : could not resolve address 's3'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:104] : 'server balena_vpn_1' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:129] : 'server balena_vpn' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:134] : 'server balena_vpn' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : Failed to initialize server(s) addr.

shows that the docker-compose environment isn't able to resolve the service names for the inter-service communication.
As seen here:

open-balena/src/haproxy/haproxy.cfg

Line 76 in ea07d43

server balena_api_1 api:80 check port 80

The haproxy container only knows about name api and the underlying docker / docker-compose environment has to resolve the api to the actual other service running as container.

Please try to narrow down if the docker / docker-compose environment is able to communicate with each other and resolve services names.
eg. with two services and nc the one which listens and the other that tries to connect.