Investigate If Changing Pypi URL in settings is working

[ryanmelt]

Investigate this:

I tried changing my "Pypi URL" in Admin Console to my company's mirror, but when I upload a .whl package in Packages in Admin Console it still tries the typical one. I did click "save" and refresheed the page to see it was there. Do I need to do anything else to get it to use the mirror?

Running Podman on RHEL 9.2, COSMOS 5.14.2. USER/GROUP are both root.

Error:

Stdout:
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))) - skipping

Stderr:
WARNING: The directory '/openc3/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl is not a supported wheel on this platform.

[tsjordan-eng]

Little update here on 5.15.2:

• Now I set the PyPi url in the .env file
  • Note that this is not reflected in the Admin GUI
• When I upload a .whl now I still get an error:

  Stdout:
  Looking in indexes: <company-url>/repository/pypi-proxy/pypi/simple
  Processing /gems/python_packages/cache/orbit_predictor-1.15.0-py3-none-any.whl
  INFO: pip is looking at multiple versions of orbit-predictor to determine which version is compatible with other requirements. This could take a while.
  
  Stderr:
  ERROR: Could not find a version that satisfies the requirement numpy>=1.8.2 (from orbit-predictor) (from versions: none)
  ERROR: No matching distribution found for numpy>=1.8.2
  

When I try putting the package in the requirements.txt file I get the following similar error:

Failed to retrieve pypi_url: NoMethodError : undefined method `get_setting' for OpenC3::PluginModel:Class
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.15.2/lib/openc3/models/plugin_model.rb:185:in `install_phase2'
/openc3/bin/openc3cli:346:in `update_plugin'
/openc3/bin/openc3cli:465:in `load_plugin'
/openc3/bin/openc3cli:656:in `<main>'
Installing python packages from requirements.txt with pypi_url=<company-url>/repository/pypi-proxy/pypi/simple

When I start an interactive session in the operator container and try to install with the command line like the code does I get the same kind of error:

~/lib/openc3/operators $ pip install --index-url <company-url>/repository/pypi-proxy/pypi/simple orbit_predictor
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: <company-url>/repository/pypi-proxy/pypi/simple
ERROR: Could not find a version that satisfies the requirement orbit_predictor (from versions: none)
ERROR: No matching distribution found for orbit_predictor

But when I set the index and index-url in ~/.config/pip/pip.conf

[global]
index = <company-url>/repository/pypi-proxy/pypi
index-url = <company-url>/repository/pypi-proxy/simple

It downloads the package and attempts to install. But then I hit the next problem:

~/lib/openc3/operators $ pip install orbit_predictor
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: <company-url>/repository/pypi-proxy/simple
Collecting orbit_predictor
  Downloading <company-url>/repository/pypi-proxy/packages/orbit-predictor/1.15.0/orbit_predictor-1.15.0-py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.2/42.2 kB 13.7 MB/s eta 0:00:00
Collecting numpy>=1.8.2 (from orbit_predictor)
  Downloading <company-url>/repository/pypi-proxy/packages/numpy/1.26.4/numpy-1.26.4-cp311-cp311-musllinux_1_1_x86_64.whl (18.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.1/18.1 MB 72.1 MB/s eta 0:00:00
Collecting sgp4>=2.5 (from orbit_predictor)
  Downloading <company-url>/repository/pypi-proxy/packages/sgp4/2.23/sgp4-2.23.tar.gz (176 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 176.9/176.9 kB 41.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests in /usr/lib/python3.11/site-packages (from orbit_predictor) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/lib/python3.11/site-packages (from requests->orbit_predictor) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.11/site-packages (from requests->orbit_predictor) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3.11/site-packages (from requests->orbit_predictor) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.11/site-packages (from requests->orbit_predictor) (2023.7.22)
Building wheels for collected packages: sgp4
  Building wheel for sgp4 (pyproject.toml) ... done
  Created wheel for sgp4: filename=sgp4-2.23-cp311-cp311-linux_x86_64.whl size=229163 sha256=aced5becb8bddc6d7e23fa985a1f22a49f34127721419ea5bb271bdd02e0f6ab
  Stored in directory: /openc3/.cache/pip/wheels/dc/ca/02/111b0e98c90a51d40ad77390649e904ce4449ad078f96e76ad
Successfully built sgp4
Installing collected packages: sgp4, numpy, orbit_predictor
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/gems/python_packages'

I think this is due to the compose.yaml. Maybe I'm trying this in the wrong container?

But I think this comes down to a Python pip problem where the -i flag doesn't work.

[ryanmelt]

I think the PYPI_URL in the .env file is currently only used when building the containers to handle offline environments with their own package mirrors.

The get_setting stack trace looks like a bug.

Maybe we need to use the flags to set the index and the index-url when we shell out to pip?

/gems is readonly in most containers (from compose.yaml). The cmd-tlm-api container has permissions to write to it.

[tsjordan-eng]

I tried again after setting the pipy url in the admin panel but I get the same response:

{"time":1714431928670283033,"@timestamp":"2024-04-29T23:05:28.670283Z","level":"ERROR","container_name":"9606364ad7fe","message":"Failed to retrieve pypi_url: NoMethodError : undefined method `get_setting' for OpenC3::PluginModel:Class\n/usr/lib/ruby/gems/3.2.0/gems/openc3-5.15.2/lib/openc3/models/plugin_model.rb:185:in `install_phase2'\n/openc3/bin/openc3cli:346:in `update_plugin'\n/openc3/bin/openc3cli:465:in `load_plugin'\n/openc3/bin/openc3cli:656:in `<main>'","type":"log"}
{"time":1714431928672708756,"@timestamp":"2024-04-29T23:05:28.672708Z","level":"INFO","container_name":"9606364ad7fe","message":"Installing python packages from requirements.txt with pypi_url=<company-url>/repository/pypi-proxy/pypi/simple","type":"log"}
Looking in indexes: <company-url>/repository/pypi-proxy/pypi/simple
Command failed - retrying with --no-index

...

Stderr:
WARNING: The directory '/openc3/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Could not find a version that satisfies the requirement orbit_predictor (from versions: none)
ERROR: No matching distribution found for orbit_predictor
WARNING: The directory '/openc3/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Could not find a version that satisfies the requirement orbit_predictor (from versions: none)
ERROR: No matching distribution found for orbit_predictor

[tsjordan-eng]

Testing in 5.16.1

I changed the Admin Panel PyPi URL setting to remove the /simple at the end as I saw from logs that it is added.

Here's a plugin installation:

...
{"time":1715373535437746061,"@timestamp":"2024-05-10T20:38:55.437746Z","level":"INFO","container_name":"52abc9eafbaa","message":"Installing python packages from requirements.txt with pypi_url=<company-url>/pypi/simple","type":"log"}
Looking in indexes: <company-url>/pypi/simple
Command failed - retrying with --no-index
...
Stderr:
WARNING: The directory '/openc3/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that
directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Could not find a version that satisfies the requirement orbit_predictor (from versions: none)
ERROR: No matching distribution found for orbit_predictor
WARNING: The directory '/openc3/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that
directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Could not find a version that satisfies the requirement orbit_predictor (from versions: none)
ERROR: No matching distribution found for orbit_predictor

I tried running in the cmd-tlm-api container and got the same directory error. The /openc3/.cache/pip did not exist. I created it and tried again and it worked!

[ryanmelt]

@tsjordan-eng All you did was create the directory /openc3/.cache/pip and that fixed your issue?

[tsjordan-eng]

Yes. I also had to correct my PyPi URL. Note, though, that I still get a warning when I install:

Stderr:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[ryanmelt]

Was that warning during the build phase, or are you running in rootless podman?

[tsjordan-eng]

We are running docker now, but as root. (Perhaps I should switch this back to running as a user, we just had permission troubles)