NetBox doesn't generate HTTPS URLs

[Laz80UK]

We are experiencing some weird issue with NetBox running on Docker not generating HTTPS URLs.

Version we use:
netboxcommunity/netbox:v2.9.10-ldap
Python 3.8.6

I have changed the URL scheme in the configuration file to:
ALLOWED_URL_SCHEMES = ('https')

NetBox does read this file as the BANNER_TOP variable is also changed and I can see the banner changed in the UI.

Our setup is as follows:
AWS Load Balancer (port 443) -> Target Group -> NetBox (port 8082)
The SSL certificate is assigned to the Load Balancer.

Request are going through the UI on the secure port with no poblems.
The problem arouse when we try to access assigned objects using Pynetbox.
The first query succeeds through port 443, but Pynetbox tries to access the assigned object through port 80.
https://<netbox_url>/api/docs/ lists only HTTP in Schemes.
The Python output is as follows (with sensitive information removed):

2021-08-02 11:33:44,200 DEBUG - Starting new HTTPS connection (1): <netbox_url>:443
2021-08-02 11:33:44,365 DEBUG - https://<netbox_url>:443 "GET /api/ipam/ip-addresses/?address=<ip_address>&limit=0 HTTP/1.1" 200 894
2021-08-02 11:33:44,370 DEBUG - {'id': 3417, 'url': 'http://<netbox_url>/api/ipam/ip-addresses/3417/', 'family': {'value': 4, 'label': 'IPv4'}, 'address': '<ip_address>/30', 'vrf': None, 'tenant': None, 'status': {'value': 'active', 'label': 'Active'}, 'role': {'value': 'loopback', 'label': 'Loopback'}, 'assigned_object_type': 'dcim.interface', 'assigned_object_id': 11989, 'assigned_object': {'id': 11989, 'url': 'http://<netbox_url>/api/dcim/interfaces/11989/', 'device': {'id': 812, 'url': 'http://<netbox_url>/api/dcim/devices/812/', 'name': '<device_name>', 'display_name': '<device_name>'}, 'name': 'Management', 'cable': None, 'connection_status': None}, 'nat_inside': None, 'nat_outside': None, 'dns_name': '', 'description': '', 'tags': [], 'custom_fields': {'Zendesk Ticket': None}, 'created': '2020-11-05', 'last_updated': '2020-11-12T16:06:51.913181Z'}
2021-08-02 11:33:44,375 DEBUG - Starting new HTTP connection (1): <netbox_url>:80
2021-08-02 11:33:44,494 DEBUG - http://<netbox_url>:80 "GET /api/dcim/devices/812/ HTTP/1.1" 200 1608
2021-08-02 11:33:44,496 DEBUG - {'id': 280, 'url': 'http://<netbox_url>/api/dcim/sites/280/', 'name': '<site_name>', 'slug': 'lab'}

I have tested with the following Docker versions:
Docker version 20.10.7, build f0df350
Docker version 20.10.6, build 370c289
docker-compose version 1.18.0, build 8dd22a9
docker-compose version 1.26.2, build eefe0d31

NetBox env file (sensitive information removed):

CORS_ORIGIN_ALLOW_ALL=True
DB_NAME=netbox
DB_USER=netbox
DB_PASSWORD=<password>
DB_HOST=<db_host>
EMAIL_SERVER=localhost
EMAIL_PORT=25
EMAIL_USERNAME=netbox
EMAIL_PASSWORD=
EMAIL_TIMEOUT=5
EMAIL_FROM=<email>
MEDIA_ROOT=/opt/netbox/netbox/media
PREFER_IPV4=true
NAPALM_USERNAME=root
NAPALM_PASSWORD=<password>
NAPALM_TIMEOUT=10
MAX_PAGE_SIZE=1000
REDIS_HOST=redis
REDIS_PASSWORD=<password>
REDIS_DATABASE=0
REDIS_CACHE_DATABASE=1
REDIS_SSL=false
SECRET_KEY=<secret_key>
SUPERUSER_NAME=<superuser>
SUPERUSER_EMAIL=<email>
SUPERUSER_PASSWORD=<password>
SKIP_STARTUP_SCRIPTS=false
SKIP_SUPERUSER=true
WEBHOOKS_ENABLED=true

None of the production servers work properly. We have a few Dev and Staging servers, but I couldn't reproduce the problem on any of those. I changed the Target Group as well, but didn't help. The problem is with the Production servers.

An identical setup in Dev produces this:

2021-08-02 12:10:09,359 DEBUG - Starting new HTTPS connection (1): <netbox_url>:443
2021-08-02 12:10:09,531 DEBUG - https://<netbox_url>:443 "GET /api/ipam/ip-addresses/?address=<ip_address>&limit=0 HTTP/1.1" 200 885
2021-08-02 12:10:09,531 DEBUG - {'id': 3417, 'url': 'https://<netbox_url>/api/ipam/ip-addresses/3417/', 'family': {'value': 4, 'label': 'IPv4'}, 'address': '<ip_address>/30', 'vrf': None, 'tenant': None, 'status': {'value': 'active', 'label': 'Active'}, 'role': {'value': 'loopback', 'label': 'Loopback'}, 'assigned_object_type': 'dcim.interface', 'assigned_object_id': 11989, 'assigned_object': {'id': 11989, 'url': 'https://<netbox_url>/api/dcim/interfaces/11989/', 'device': {'id': 812, 'url': 'https://<netbox_url>/api/dcim/devices/812/', 'name': '<device_name>', 'display_name': '<device_name>'}, 'name': 'Management', 'cable': None, 'connection_status': None}, 'nat_inside': None, 'nat_outside': None, 'dns_name': '', 'description': '', 'tags': [], 'custom_fields': {'Zendesk Ticket': None}, 'created': '2020-11-05', 'last_updated': '2020-11-12T16:06:51.913181Z'}
2021-08-02 12:10:09,843 DEBUG - https://<netbox_url>:443 "GET /api/dcim/devices/812/ HTTP/1.1" 200 1584
2021-08-02 12:10:09,844 DEBUG - {'id': 280, 'url': 'https://<netbox_url>/api/dcim/sites/280/', 'name': '<device_name>', 'slug': 'lab'}

Has anyone seen such an issue before?
How could I troubleshoot this further?

[jeremystretch]

You'll probably get more of a response over on the netbox-docker project.

[candlerb]

Is this "AWS load balancer" a TCP load balancer, or a HTTP proxy?

If it's a TCP proxy: Netbox has no way of knowing that the incoming request was https, nor will it know that it was on port 443 given that's the default port and so it won't appear in the Host: header. If you're using the nginx frontend in the netbox docker, then you ought to be able to configure it to add a static X-Forwarded-Host and X-Forwarded-Proto header to every request.

If it's a HTTP proxy: it needs to add X-Forwarded-Proto for Netbox to know that it's HTTPS, and preferably X-Forwarded-Host too (although if the proxy preserves the original Host: header rather than putting the backend target host in there, then that may not be necessary).

[Laz80UK]

Hi @candlerb,

Thank you for the quick response.

We use an Application Load Balancer, that's supposed to handle the X-Forwarded-Proto header automatically.
I've done some digging and found that there was recent problem with this, that is now fixed... but not for me.

Anyhow, your answer pointed me to the right direction.
NetBox-Docker uses Nginx, so I changed the following line in nginx.conf from
proxy_set_header X-Forwarded-Proto $scheme;
to
proxy_set_header X-Forwarded-Proto https;
and now it works as expected.