prompt user before changing NetBox database passwords out from underneath existing database

[purplealien51]

Describe the bug
Netbox is not loading. Docker logs for netbox container says "password authentication failed for user netbox".

**Screenshots and/or Logs **

get_for_models(*apps.get_models()).values()', '                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/contrib/contenttypes/models.py", line 91, in get_for_models', '    for ct in cts:', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/query.py", line 400, in __iter__', '    self._fetch_all()', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/query.py", line 1928, in _fetch_all', '    self._result_cache = list(self._iterable_class(self))', '                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/query.py", line 91, in __iter__', '    results = compiler.execute_sql(', '              ^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql', '    cursor = self.connection.cursor()', '             ^^^^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner', '    return func(*args, **kwargs)', '           ^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 316, in cursor', '    return self._cursor()', '           ^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 292, in _cursor', '    self.ensure_connection()', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner', '    return func(*args, **kwargs)', '           ^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 274, in ensure_connection', '    with self.wrap_database_errors:', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__', '    raise dj_exc_value.with_traceback(traceback) from exc_value', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection', '    self.connect()', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner', '    return func(*args, **kwargs)', '           ^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect', '    self.connection = self.get_new_connection(conn_params)', '                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner', '    return func(*args, **kwargs)', '           ^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 277, in get_new_connection', '    connection = self.Database.connect(**conn_params)', '                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', '  File "/opt/netbox/venv/lib/python3.12/site-packages/psycopg/connection.py", line 119, in connect', '    raise last_ex.with_traceback(None)', 'django.db.utils.OperationalError: connection failed: connection to server at "172.18.0.9", port 5432 failed: FATAL:  password authentication failed for user "netbox"', '', " loaded config '/etc/netbox/config/configuration.py'", " loaded config '/etc/netbox/config/extra.py'", " loaded config '/etc/netbox/config/logging.py'", " loaded config '/etc/netbox/config/plugins.py'", '']
django.db.utils.OperationalError: connection failed: connection to server at "172.18.0.9", port 5432 failed: FATAL:  password authentication failed for user "netbox"
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]
⏳ Waiting on DB... (0s / 30s)

Malcolm Version:

• Version [e.g. v25.01.0]

How are you running Malcolm?

• Malcolm running in Ubuntu 22.04 using docker

Additional context
Tried restarting netbox container after reseting password with ./auth_setup script but nothing works

[trwagner1]

Is this a VM deployment? Did you configure the primary ID or did it request an IP with DHCP?

I've seen this when my VM or hardware has a DHCP assigned address and I configured the primary interface, but failed to restart the Malcolm server after completing the configuration and auth_setup script.

@mmguero , I'd recommend inserting a section into the initial configuration that detects if the primary nic is DHCP and prompt to configure the primary IPv4 address manually before the system reboots. That way, when the Malcolm configuration script runs, it's going to do so under the permanent IP address.

[purplealien51]

Hi @trwagner1,
I have it deployed in Ubuntu VM running in Proxmox and using the git repo with docker.
VM is assigned with a static ip through pfsense DHCP server.
All containers are running except Netbox stuck at "starting" state. I can use Opensearch dashboards, Arkime, CyberChef, File Upload is working as well.
I have configured one interface which is mirrored and can see all traffic in Arkime and Dashboards
Only Netbox is giving Bad Gateway error

[mmguero]

Hi, my apologies, I was on travel for DHS all last week and unable to monitor all of this. Getting caught up now. I'm working on seeing if I can reproduce the issue.

Thanks for the ideas @trwagner1, in this case where the networking is all internal to the internal docker network I don't think it's an issue with the IP address/DHCP.

If it's a matter of getting up and running again, I'm pretty sure that shutting down malcolm and running rm -rf ./netbox/postgres/* ./netbox/redis/* and then starting Malcolm back up should get you going again. That would, of course, delete your existing netbox inventory, however. If you've got a backup of it you could then restore it.

I'm working on trying to reproduce the issue locally.

[mmguero]

Okay, I was able to reproduce. Here's the scenario:

1. Use ./scripts/auth_setup to set the netbox-related passwords in their respective environment variable files in ./config/
2. Start Malcolm from a fresh/empty state, let NetBox bootstrap and populate itself
3. Stop Malcolm (do not wipe, just stop)
4. Use ./scripts/auth_setup to set the netbox-related passwords to new values
5. Start Malcolm again, which will use the netbox database populated previously in step 2

Results:

Error messages like:

netbox-postgres-1     | 2025-02-03 16:52:07.184 UTC [57] FATAL:  password authentication failed for user "netbox"

Prognosis: Once the passwords have been changed "out from underneath" the NetBox postgresql database, there's not a whole lot one can do. That's kind of the point of the password. So here's what I propose we do to prevent this from happening:

• During ./scripts/auth_setup, ONLY populate new NetBox-related passwords if either of the following is true:
  • the passwords have not been set yet (empty or all x's)
  • the user is prompted to confirm with a warning that they will need to wipe their netbox database to reset it

[mmguero]

I've added a commit to prompt the user if netbox passwords have already been set:

Kazam_screencast_00000.mp4

I know this doesn't get you up and running if you're already in this state, but it should at least help to prevent users from doing so in the future.

[purplealien51]

By the way my netbox was not started with the fresh install
but what i did now is ./scripts/stop then ./scripts/wipe and then ./scripts/start
and netbox is working now.
Thanks for your support @mmguero. Appreciate it