[URGENT] fixes empty data loading

[rachejazz]

commits that caused the issues:
#207
063e460
and few more cumulative bugs

@Apurv428 @deveshidwivedi till this gets merged into master, you can use this branch to switch and check if data updates for you

[deveshidwivedi]

Hey @rachejazz, I am still unable to get packages on searching for them. I see the landing page and can view FAQs. However, when I try to search for a package after selecting an operating system, I see no updates. I am on the data-update-fixes branch.

[rachejazz]

interesting, lmk if you can come on a debug session. What does the sdt.log say?
It is pretty late for you now, lets try tomorrow @deveshidwivedi

[deveshidwivedi]

interesting, lmk if you can come on a debug session.

Yes, let's have a debug session tomorrow please.

This is from sdt.log:

2024-07-13 20:27:29,393 Exception on /sdt/searchPackages [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 883, in full_dispatch_request
    return self.finalize_request(rv)
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 902, in finalize_request
    response = self.make_response(rv)
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1174, in make_response
    raise TypeError(
TypeError: The view function for 'searchPackages' did not return a valid response. The function either returned None or ended without a return statement.

[rachejazz]

Issue is solved now. Thanks @deveshidwivedi
Solved by renaming .example files to their original.
This can be merged now.

[pleia2]

I'd like to learn a bit more about what was discovered during debugging related to the .json file extensions, since we're not seeing this problem in production, and we transitioned away from referencing them in the configuration file. That will help me review that component of the PR.

But there seems to be a bit of confusion about our database users. I'll put together a visual for our meeting on Wednesday so that's more clear and then we can updated the documentation accordingly, but in the meantime:

• database_build.py needs to be run with a MySQL/MariaDB user with at least the full privileges within the sdtDB databases since it's dropping and creating tables therein. We don't create this user during setup and just assume they are using something like the root user (that's what we do in production). The user that runs this script needs to have these credentials secured off-system in some sort of password management tool, and then inputs them manually when it needs to be run.

• The tool itself when it's reading the database uses the sdtreaduser from the .env file that only has read permissions, so keeping the password in clear text on the file system isn't a data security risk.

[rachejazz]

Got a new bug from the newest 15.6 opensuse data. Solved now. This PR is ready to be merged/reviewed.
cc: @pleia2

[pleia2]

Can you confirm that bin/database_build.py is working for you in the current state? I'm a traceback (below) that are sends me down a whole rabbit hole. It looks like the changes around line 30 seemed to have changed how the user adds the username and password on the command line, and causes problems.

elizabeth@sdt:/opt/software-discovery-tool/bin$ ./database_build.py
Enter password for connecting to MariaDB server : root
Traceback (most recent call last):
File "/opt/software-discovery-tool/bin/./database_build.py", line 126, in
db_init()
File "/opt/software-discovery-tool/bin/./database_build.py", line 39, in db_init
connectdb(username,password,dbName)
File "/opt/software-discovery-tool/bin/./database_build.py", line 22, in connectdb
conn = pymysql.connect(host=HOST,user=username,password=password)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 353, in init
self.connect()
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 633, in connect
self._request_authentication()
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 907, in _request_authentication
auth_packet = self._read_packet()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 725, in _read_packet
packet.raise_for_error()
File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "/usr/lib/python3/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.OperationalError: (1698, "Access denied for user 'elizabeth'@'localhost'")

[rachejazz]

Hey @pleia2 couple of questions. Firstly, yeah this is working for me even after changing localhost to a remote db (I'm using docker and mariadb so different networks)

1. Since you're giving password as root, does it mean elizabeth is your root account?
2. Since it says access denied, could you manually check logging into sql on terminal, then use information_schema then select * from USER_PRIVILEGES

[pleia2]

1. My administrative user is root, so I input that on the command line. But then the script completely ignores that input and falls back to my local user on the system (elizabeth) when it fails, which is why 'elizabeth'@'localhost' shows up
2. Due to 1, this isn't the problem 😄

It really just totally falls over (traceback) after I type "root" when the expected behavior is asking for a password. I'm using Ubuntu 24.04.

[pleia2]

Works now! Thank you.