`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt` #1664

DnPlas · 2024-04-23T09:33:08Z

Bug Description

I have noticed that the list of packages in charm-python-packages may not guarantee they will be installed before charm requirements, as the docs suggest.

The behaviour I am getting is that the packages listed in charm-python-packages get collected and it looks like they are being installed as the very first thing, for example:

:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jinja2,jsonschema,lightkube,lightkube-models,markupsafe,oci-image,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip>23.0', 'setuptools>69.0', 'wheel']    
::    :: Collecting pip>23.0                                                                                                                                                                 
::    ::   Using cached pip-24.0.tar.gz (2.1 MB)                                                                                                                                             
::    ::   Installing build dependencies: started                                                                                                                                            
::    ::   Installing build dependencies: finished with status 'done'                                                                                                                        
::    ::   Getting requirements to build wheel: started                                                                                                                                      
::    ::   Getting requirements to build wheel: finished with status 'done'                                                                                                                  
::    ::     Preparing wheel metadata: started                                                                                                                                               
::    ::     Preparing wheel metadata: finished with status 'done'                                                                                                                           
::    :: Collecting setuptools>69.0                                                                                                                                                          
::    ::   Using cached setuptools-69.5.1.tar.gz (2.3 MB)                                                                                                                                    
::    ::   Getting requirements to build wheel: started                                                                                                                                      
::    ::   Getting requirements to build wheel: finished with status 'done'                                                                                                                  
::    ::   Installing backend dependencies: started                                                                                                                                          
::    ::   Installing backend dependencies: finished with status 'done'                                                                                                                      
::    ::     Preparing wheel metadata: started                                                                                                                                               
::    ::     Preparing wheel metadata: finished with status 'done' 
::    :: Collecting jinja2==3.1.2                                                                                                                                                            
::    ::   Using cached Jinja2-3.1.2.tar.gz (268 kB)

but in reality they get installed after collecting all packages, including the ones listed in requirements.txt.

# There is a long list of `::    :: Collecting <package-name>...` before the following lines
::    :: Successfully built wheel setuptools pip                                                                                                                                             
::    :: Installing collected packages: wheel, setuptools, pip                                                                                                                               
::    ::   Attempting uninstall: setuptools                                                                                                                                                  
::    ::     Found existing installation: setuptools 44.0.0                                                                                                                                  
::    ::     Uninstalling setuptools-44.0.0:                                                                                                                                                 
::    ::       Successfully uninstalled setuptools-44.0.0                                                                                                                                    
::    ::   Attempting uninstall: pip                                                                                                                                                         
::    ::     Found existing installation: pip 20.0.2                                                                                                                                         
::    ::     Uninstalling pip-20.0.2:                                                                                                                                                        
::    ::       Successfully uninstalled pip-20.0.2

This could result in errors if any of the charm requirements depends on a charm-python-package to be installed (or upgraded) a priori.

Please NOTE I concluded this was an issue after trying to solve this other issue with the suggestions from this post. Upgrading setuptools fixes the issue with markupsafe and jinja2, I tested it in a virtual env, but it was not possible in the charm.

To Reproduce

Take this charm.
charmcraft pack -v
Check the build logs and observe when the setuptools and pip packages get actually installed and upgraded.

Environment

Ubuntu 20.04
charmcraft latest/candidate

charmcraft.yaml

type: "charm"
bases:
  - build-on:
    - name: "ubuntu"
      channel: "20.04"
    run-on:
    - name: "ubuntu"
      channel: "20.04"
parts:
  charm:
    charm-python-packages: [setuptools>69.0, pip>23.0]  # Fixes install of some packages

Relevant log output

Logs: https://pastebin.canonical.com/p/DhpBRNn7pz/

The text was updated successfully, but these errors were encountered:

syncronize-issues-to-jira · 2024-04-23T09:33:16Z

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-2848.

This message was autogenerated

DnPlas · 2024-04-23T09:49:06Z

Another way in which this issue could be reproduced is by running pip install commands in a virtual environment directly and observing what happens first. I have this:

ubuntu@charm-dev-focal:~$ source jinja2-test/bin/activate
(jinja2-test) ubuntu@charm-dev-focal:~$ pip list | grep setuptools
setuptools    44.0.0 
(jinja2-test) ubuntu@charm-dev-focal:~$ pip install --no-binary=jinja2,pip,setuptools --requirement=requirements.txt pip>23.0 setuptools>69.0 wheel # <--- this is the command that gets run during a build
    ERROR: Command errored out with exit status 1:
     command: /home/ubuntu/jinja2-test/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pa0uqgo3/jinja2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pa0uqgo3/jinja2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-pa0uqgo3/jinja2/pip-egg-info
         cwd: /tmp/pip-install-pa0uqgo3/jinja2/
    Complete output (35 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-pa0uqgo3/jinja2/setup.py", line 4, in <module>
        setup(
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.8/distutils/core.py", line 121, in setup
        dist.parse_config_files()
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/dist.py", line 701, in parse_config_files
        parse_configuration(self, self.command_options,
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 121, in parse_configuration
        meta.parse()
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 426, in parse
        section_parser_method(section_options)
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 399, in parse_section
        self[name] = value
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 184, in __setitem__
        value = parser(value)
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 515, in _parse_version
        version = self._parse_attr(value, self.package_dir)
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 349, in _parse_attr
        module = import_module(module_name)
      File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 848, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/tmp/pip-install-pa0uqgo3/jinja2/src/jinja2/__init__.py", line 8, in <module>
        from .environment import Environment as Environment
      File "/tmp/pip-install-pa0uqgo3/jinja2/src/jinja2/environment.py", line 14, in <module>
        from markupsafe import Markup
    ModuleNotFoundError: No module named 'markupsafe'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

kimwnasptd · 2024-07-19T09:11:33Z

This is still a problem for us, even after a latest revision of charmcraft 3.0.0.post77+g8f64d66. Cross referencing also the issues from the Kubeflow repo
canonical/bundle-kubeflow#1005

We see the exact same errors that come from setuptools (you can see the integration-tests runs that try to build the charms)

canonical/istio-operators#476
canonical/kfp-operators#541

kimwnasptd · 2024-07-22T09:10:29Z

I've also tried to use the edge/pr-1727 channel from #1727 which should resolve the issue, but the CI is still failing in my tests
canonical/kfp-operators#393

This time though I see different kinds of errors (in all tests that try to build charms)

:: Installing packages from requirements files and charm lib dependencies.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt', 'cosl']
::    :: ERROR: Double requirement given: cosl==0.0.12 (from -r requirements.txt (line 20)) (already in cosl, name='cosl')
:: Traceback (most recent call last):
::   File "/snap/charmcraft/4183/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 473, in <module>
::     main()
::   File "/snap/charmcraft/4183/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 468, in main
::     builder.build_charm()
::   File "/snap/charmcraft/4183/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 92, in build_charm
::     self.handle_dependencies()
::   File "/snap/charmcraft/4183/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 344, in handle_dependencies
::     self._install_dependencies(staging_venv_dir)
::   File "/snap/charmcraft/4183/lib/python3.10/site-packages/charmcraft/instrum.py", line 152, in _f
::     return func(*args, **kwargs)
::   File "/snap/charmcraft/4183/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 276, in _install_dependencies
::     _process_run(
::   File "/snap/charmcraft/4183/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 404, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt', 'cosl'] execution failed with retcode 1
Failed to run the build script for part 'charm'.
Recommended resolution: Check the build output and verify the project can work with the 'charm' plugin.

kimwnasptd · 2024-07-22T16:00:53Z

Some more context on the error above:

cosl is defined both in the prometheus library used by our charm, and also in the charm's requirements.txt
Using 2.7.0 we don't see this error for cosl

Also if I'd run locally the command pip install --no-binary=:all --requirement=requirements.txt cosl then I can't reproduce this error

kimwnasptd · 2024-07-22T16:14:58Z

I tried to remove cosl from the requirements.txt, since it will be downloaded from the library but now am bumping into the same issue since ops is getting added into the mix

:: Installing packages from requirements files and charm lib dependencies.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt', 'cosl', 'ops>=2.0.0']

https://github.com/canonical/kfp-operators/actions/runs/10044196423/job/27758697033?pr=393#step:5:284

lengau · 2024-07-22T18:56:19Z

Thanks! I missed de-duplicating the values in the charmlibs: #1750

kimwnasptd · 2024-07-23T10:22:42Z

Thanks for the update @lengau! We tried also with edge/pr-1750 but we saw the initial error of this issue happen again

canonical/istio-operators#476
https://github.com/canonical/istio-operators/actions/runs/10055636878/job/27792722000?pr=476#step:4:233

:: Installing packages from requirements files and charm lib dependencies.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt']
...
...
::    :: Collecting jinja2==3.1.2
::    ::   Downloading Jinja2-3.1.2.tar.gz (268 kB)
::    ::     ERROR: Command errored out with exit status 1:
::    ::      command: /root/parts/charm/build/staging-venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-afsosfx0/jinja2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-afsosfx0/jinja2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-afsosfx0/jinja2/pip-egg-info
::    ::          cwd: /tmp/pip-install-afsosfx0/jinja2/
::    ::     Complete output (35 lines):
::    ::     Traceback (most recent call last):
::    ::       File "<string>", line 1, in <module>
::    ::       File "/tmp/pip-install-afsosfx0/jinja2/setup.py", line 4, in <module>
::    ::         setup(
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
::    ::         return distutils.core.setup(**attrs)
::    ::       File "/usr/lib/python3.8/distutils/core.py", line 121, in setup
::    ::         dist.parse_config_files()
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/dist.py", line 701, in parse_config_files
::    ::         parse_configuration(self, self.command_options,
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 121, in parse_configuration
::    ::         meta.parse()
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 426, in parse
::    ::         section_parser_method(section_options)
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 399, in parse_section
::    ::         self[name] = value
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 184, in __setitem__
::    ::         value = parser(value)
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 515, in _parse_version
::    ::         version = self._parse_attr(value, self.package_dir)
::    ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 349, in _parse_attr
::    ::         module = import_module(module_name)
::    ::       File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
::    ::         return _bootstrap._gcd_import(name[level:], package, level)
::    ::       File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
::    ::       File "<frozen importlib._bootstrap>", line 991, in _find_and_load
::    ::       File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
::    ::       File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
::    ::       File "<frozen importlib._bootstrap_external>", line 848, in exec_module
::    ::       File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
::    ::       File "/tmp/pip-install-afsosfx0/jinja2/src/jinja2/__init__.py", line 8, in <module>
::    ::         from .environment import Environment as Environment
::    ::       File "/tmp/pip-install-afsosfx0/jinja2/src/jinja2/environment.py", line 14, in <module>
::    ::         from markupsafe import Markup
::    ::     ModuleNotFoundError: No module named 'markupsafe'
::    ::     ----------------------------------------
::    :: ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
:: Traceback (most recent call last):
::   File "/snap/charmcraft/4216/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 480, in <module>
::     main()
::   File "/snap/charmcraft/4216/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 475, in main
::     builder.build_charm()
::   File "/snap/charmcraft/4216/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 93, in build_charm
::     self.handle_dependencies()
::   File "/snap/charmcraft/4216/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 351, in handle_dependencies
::     self._install_dependencies(staging_venv_dir)
::   File "/snap/charmcraft/4216/lib/python3.10/site-packages/charmcraft/instrum.py", line 152, in _f
::     return func(*args, **kwargs)
::   File "/snap/charmcraft/4216/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 283, in _install_dependencies
::     _process_run(
::   File "/snap/charmcraft/4216/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 411, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt'] execution failed with retcode 1

kimwnasptd · 2024-07-23T10:39:46Z

The main difference in the repo that succeeds canonical/kfp-operators#393 and in a repo that fails canonical/istio-operators#476 is that in the succeeding case we have this code:

parts:
  charm:
    charm-python-packages: [setuptools, pip]  # Fixes install of some packages
    # Install jinja2 (a dependency of charmed-kubeflow-chisme) from binary to avoid build-time issues
    # See https://github.com/canonical/bundle-kubeflow/issues/883
    # Remove when https://github.com/canonical/charmcraft/issues/1664 is fixed
    charm-binary-python-packages: [jinja2]

While in the failing case we have this code:

parts:
  charm:
    charm-python-packages: [setuptools, pip, jsonschema, cryptography, cffi]
    build-packages: [git, rustc, cargo, libffi-dev, libssl-dev, pkg-config]

So in the succeeding case, we have defined jinja2 to be a charm-binary-python-package

DnPlas · 2024-07-23T10:56:36Z

@kimwnasptd maybe you want to take a look at #1135 (comment)

DnPlas · 2024-07-23T11:02:58Z

@lengau could this issue also be related to #1456?

lengau · 2024-09-20T20:18:42Z

@DnPlas I believe this was fixed in Charmcraft 3.2 - can you confirm?

DnPlas added the Bug Something isn't working label Apr 23, 2024

DnPlas mentioned this issue Apr 23, 2024

ModuleNotFoundError: No module named 'markupsafe' error at build time canonical/bundle-kubeflow#883

Closed

kimwnasptd mentioned this issue Jul 22, 2024

fix(builder)!: remove single-step non-strict install #1727

Merged

lengau self-assigned this Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt` #1664

`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt` #1664

DnPlas commented Apr 23, 2024 •

edited

Loading

syncronize-issues-to-jira bot commented Apr 23, 2024

DnPlas commented Apr 23, 2024

kimwnasptd commented Jul 19, 2024

kimwnasptd commented Jul 22, 2024

kimwnasptd commented Jul 22, 2024

kimwnasptd commented Jul 22, 2024

lengau commented Jul 22, 2024

kimwnasptd commented Jul 23, 2024

kimwnasptd commented Jul 23, 2024

DnPlas commented Jul 23, 2024

DnPlas commented Jul 23, 2024

lengau commented Sep 20, 2024

charm-python-packages does not guarantee these packages are installed before packages in requirements.txt #1664

charm-python-packages does not guarantee these packages are installed before packages in requirements.txt #1664

Comments

DnPlas commented Apr 23, 2024 • edited Loading

Bug Description

To Reproduce

Environment

charmcraft.yaml

Relevant log output

syncronize-issues-to-jira bot commented Apr 23, 2024

DnPlas commented Apr 23, 2024

kimwnasptd commented Jul 19, 2024

kimwnasptd commented Jul 22, 2024

kimwnasptd commented Jul 22, 2024

kimwnasptd commented Jul 22, 2024

lengau commented Jul 22, 2024

kimwnasptd commented Jul 23, 2024

kimwnasptd commented Jul 23, 2024

DnPlas commented Jul 23, 2024

DnPlas commented Jul 23, 2024

lengau commented Sep 20, 2024

`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt` #1664

`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt` #1664

DnPlas commented Apr 23, 2024 •

edited

Loading