[charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency #551

heitorPB · 2021-09-22T17:35:39Z

Charmcraft 1.3 fails to pack our slurmd charm. Reverting to 1.2 works again.

Here's part of the log:

[...]
2021-09-22 17:22:58,259  craft_parts.executor.step_handler        DEBUG    :: :: Collecting jinja2
2021-09-22 17:22:58,261  craft_parts.executor.step_handler        DEBUG    :: ::   Using cached Jinja2-3.0.1.tar.gz (267 kB)
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::     ERROR: Command errored out with exit status 1:
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::      command: /root/parts/charm/build/staging-venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-nbidbuh_/jinja2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-nbidbuh_/jinja2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-nbidbuh_/jinja2/pip-egg-info
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::          cwd: /tmp/pip-install-nbidbuh_/jinja2/
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::     Complete output (35 lines):
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::     Traceback (most recent call last):
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::       File "<string>", line 1, in <module>
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::       File "/tmp/pip-install-nbidbuh_/jinja2/setup.py", line 4, in <module>
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::         setup(
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::         return distutils.core.setup(**attrs)
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::       File "/usr/lib/python3.8/distutils/core.py", line 121, in setup
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::         dist.parse_config_files()
2021-09-22 17:22:58,388  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/dist.py", line 701, in parse_config_files
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         parse_configuration(self, self.command_options,
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 121, in parse_configuration
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         meta.parse()
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 426, in parse
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         section_parser_method(section_options)
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 399, in parse_section
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         self[name] = value
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 184, in __setitem__
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         value = parser(value)
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 515, in _parse_version
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         version = self._parse_attr(value, self.package_dir)
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/setuptools/config.py", line 349, in _parse_attr
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         module = import_module(module_name)
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         return _bootstrap._gcd_import(name[level:], package, level)
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "<frozen importlib._bootstrap_external>", line 848, in exec_module
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/tmp/pip-install-nbidbuh_/jinja2/src/jinja2/__init__.py", line 8, in <module>
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         from .environment import Environment as Environment
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::       File "/tmp/pip-install-nbidbuh_/jinja2/src/jinja2/environment.py", line 15, in <module>
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::         from markupsafe import Markup
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::     ModuleNotFoundError: No module named 'markupsafe'
2021-09-22 17:22:58,389  craft_parts.executor.step_handler        DEBUG    :: ::     ----------------------------------------
2021-09-22 17:22:58,401  craft_parts.executor.step_handler        DEBUG    :: :: ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
2021-09-22 17:22:58,427  craft_parts.executor.step_handler        DEBUG    :: Traceback (most recent call last):
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: File "/snap/charmcraft/x24/lib/charmcraft/charm_builder.py", line 345, in <module>
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: main()
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: File "/snap/charmcraft/x24/lib/charmcraft/charm_builder.py", line 341, in main
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: builder.build_charm()
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: File "/snap/charmcraft/x24/lib/charmcraft/charm_builder.py", line 93, in build_charm
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: self.handle_dependencies()
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: File "/snap/charmcraft/x24/lib/charmcraft/charm_builder.py", line 241, in handle_dependencies
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: _process_run(cmd)
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: File "/snap/charmcraft/x24/lib/charmcraft/charm_builder.py", line 292, in _process_run
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: raise CommandError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
2021-09-22 17:22:58,428  craft_parts.executor.step_handler        DEBUG    :: charmcraft.cmdbase.CommandError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip3', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt'] execution failed with retcode 1
2021-09-22 17:22:58,446  charmcraft                               ERROR    Parts processing error: Failed to run the build script for part 'charm'. (full execution logs in '/tmp/charmcraft.log')
2021-09-22 14:22:58,511  craft_providers.lxd.lxc                  DEBUG    Executing in container: lxc --project charmcraft exec local:charmcraft-slurmd-11534686-0-0-amd64 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin CHARMCRAFT_MANAGED_MODE=1 test -f /tmp/charmcraft.log
2021-09-22 14:22:58,598  craft_providers.lxd.lxc                  DEBUG    Executing on host: lxc --project charmcraft file pull local:charmcraft-slurmd-11534686-0-0-amd64/tmp/charmcraft.log /home/h/snap/charmcraft/common/charmcraft-ib5f_flp

The text was updated successfully, but these errors were encountered:

sergiusens · 2021-09-23T14:49:05Z

1.3.0 no builds with --no-binary which is the most plausible root cause for this.

sergiusens · 2021-09-23T14:52:53Z

sergiusens@imladris:~$ python3 -m venv bar
sergiusens@imladris:~$ . bar/bin/activate
(bar) sergiusens@imladris:~$ pip install --upgrade --no-binary :all: jinja2
Collecting jinja2
  Using cached Jinja2-3.0.1.tar.gz (267 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/sergiusens/bar/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xxf5o888/jinja2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xxf5o888/jinja2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-xxf5o888/jinja2/pip-egg-info
         cwd: /tmp/pip-install-xxf5o888/jinja2/
    Complete output (35 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-xxf5o888/jinja2/setup.py", line 4, in <module>
        setup(
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.8/distutils/core.py", line 121, in setup
        dist.parse_config_files()
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/dist.py", line 701, in parse_config_files
        parse_configuration(self, self.command_options,
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/config.py", line 121, in parse_configuration
        meta.parse()
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/config.py", line 426, in parse
        section_parser_method(section_options)
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/config.py", line 399, in parse_section
        self[name] = value
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/config.py", line 184, in __setitem__
        value = parser(value)
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/config.py", line 515, in _parse_version
        version = self._parse_attr(value, self.package_dir)
      File "/home/sergiusens/bar/lib/python3.8/site-packages/setuptools/config.py", line 349, in _parse_attr
        module = import_module(module_name)
      File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 848, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/tmp/pip-install-xxf5o888/jinja2/src/jinja2/__init__.py", line 8, in <module>
        from .environment import Environment as Environment
      File "/tmp/pip-install-xxf5o888/jinja2/src/jinja2/environment.py", line 15, in <module>
        from markupsafe import Markup
    ModuleNotFoundError: No module named 'markupsafe'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

facundobatista · 2021-09-23T17:07:53Z

Note that older Jinja installs ok. I opened this issue in their project.

@heitorPB may you try using Jinja 2.11.3? thanks!

heitorPB · 2021-09-23T17:15:35Z

Thanks for looking into that!

Curiously, installing markupsafe before installing jinja2 solves the issue...

cmatsuoka · 2021-09-23T18:09:49Z

Curiously, installing markupsafe before installing jinja2 solves the issue...

How did you install it?

We were considering to add a new property to the charm plugin to allow installation of python packages before requirements, but maybe you found a different way to solve the issue?

heitorPB · 2021-09-23T18:34:09Z

Curiously, installing markupsafe before installing jinja2 solves the issue...

How did you install it?

I mean, manually it works:

$ virtualenv venv
created virtual environment CPython3.8.10.final.0-64 in 145ms
  creator CPython3Posix(dest=/tmp/asdfasdf/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/h/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator


$ source venv/bin/activate
(venv)

$ pip install --no-binary :all: markupsafe
Collecting markupsafe
  Using cached MarkupSafe-2.0.1.tar.gz (18 kB)
Skipping wheel build for markupsafe, due to binaries being disabled for it.
Installing collected packages: markupsafe
    Running setup.py install for markupsafe ... done
Successfully installed markupsafe-2.0.1
(venv)

$ pip install --no-binary :all: jinja2
Collecting jinja2
  Using cached Jinja2-3.0.1.tar.gz (267 kB)
Requirement already satisfied: MarkupSafe>=2.0 in ./venv/lib/python3.8/site-packages (from jinja2) (2.0.1)
Skipping wheel build for jinja2, due to binaries being disabled for it.
Installing collected packages: jinja2
    Running setup.py install for jinja2 ... done
Successfully installed jinja2-3.0.1
(venv)

We were considering to add a new property to the charm plugin to allow installation of python packages before requirements, but maybe you found a different way to solve the issue?

That's already possible for "normal" packages, right? Having something similar for Python sounds awkward to me, but if it works: it works.

cmatsuoka · 2021-09-23T18:42:15Z

Ah, manually, ok. Our plan is to add a charm-python-packages property to the charm plugin that will be similar to python-packages that already exists in the python plugin (and serving the same purpose). This will install Python packages directly from PyPI and avoid situations like this one.

cmatsuoka · 2021-09-23T21:07:17Z

Proposed implementation in PR #553

cmatsuoka · 2021-09-24T20:17:01Z

This should be fixed by updating setuptools using charm-python-packages: [setuptools] in the charm part configuration.

heitorPB · 2021-09-28T12:38:42Z

@cmatsuoka and @facundobatista, what are your thoughts on always including setuptools and pip in the charm-python-packages field? Or including them by default it there is at least one entry in the requirements.txt?

This was not the first time I see python packages breaking because pip/setuptools are not the most recent releases...

cmatsuoka · 2021-09-28T13:05:33Z

@cmatsuoka and @facundobatista, what are your thoughts on always including setuptools and pip in the charm-python-packages field? Or including them by default it there is at least one entry in the requirements.txt?

IIRC there were situations (in Snapcraft?) where the opposite happened and packages didn't install correctly if the tools are newer than expected. The virtual environment is created with the Python interpreter from the base (the same that will be used at runtime) so the default tools would keep everything consistent.

We could do some magic to check if the version needs update, but we're trying to avoid having too much magic happening behind the scenes in order to let the developer understand what's happening during the packing process. If the outdated setuptools problem starts to happen with other packages or if we notice that it's too much of an annoyance we can pursue some other solution.

@facundobatista btw charm-python-packages: [setuptools] to fix jinja2 issues is a good thing to add to the extra properties documentation along with build-packages: [git].

facundobatista · 2021-10-04T11:29:27Z

Related forum post: https://discourse.charmhub.io/t/install-or-update-python-packages-before-packing-a-charm/5158

Jinja2 requires a very up to date setuptools package, that must be available before starting the charm build container. See canonical/charmcraft#551

New minor release of charmcraft requires us to change the code, as it introduced a breaking change. This patch adds git and updates setuptools before packing the charm. See canonical/charmcraft#551 for discussion

heitorPB changed the title ~~Cannot pack a charm with Jinja2 as dependency~~ [charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency Sep 22, 2021

heitorPB mentioned this issue Sep 24, 2021

parts,charm_builder: add charm-python-packages plugin property (CRAFT-552) #553

Merged

cmatsuoka closed this as completed Sep 24, 2021

cmatsuoka added parts Related to craft-parts lib usage triaged labels Sep 24, 2021

heitorPB mentioned this issue Oct 7, 2021

Failed to run the build script for part 'charm'. nextcloud-charmers/nextcloud-charms#28

Closed

facundobatista mentioned this issue Oct 12, 2021

Error when building with charmcraft from edge #566

Closed

knkski mentioned this issue Nov 13, 2021

Add metacontroller for kubeflow 1.4 canonical/metacontroller-operator#1

Merged

Cynerva mentioned this issue Jan 27, 2022

Fix failure to build operator-framework test charm charmed-kubernetes/pytest-operator#45

Merged

heitorPB mentioned this issue Aug 15, 2022

Crash with charmcraft pack #738

Closed

ijansky mentioned this issue Sep 23, 2022

CSS-2177: Implement pgsql relation canonical/temporal-k8s-operator#3

Merged

This was referenced Dec 9, 2022

Enable TLS in filebeat canonical/layer-filebeat#101

Merged

Add option to upgrade core deps in build venv juju/charm-tools#650

Merged

No control over virtualenv used to build wheels juju/charm-tools#646

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency #551

[charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency #551

heitorPB commented Sep 22, 2021

sergiusens commented Sep 23, 2021

sergiusens commented Sep 23, 2021

facundobatista commented Sep 23, 2021

heitorPB commented Sep 23, 2021

cmatsuoka commented Sep 23, 2021 •

edited

Loading

heitorPB commented Sep 23, 2021

cmatsuoka commented Sep 23, 2021

cmatsuoka commented Sep 23, 2021

cmatsuoka commented Sep 24, 2021

heitorPB commented Sep 28, 2021

cmatsuoka commented Sep 28, 2021

facundobatista commented Oct 4, 2021

[charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency #551

[charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency #551

Comments

heitorPB commented Sep 22, 2021

sergiusens commented Sep 23, 2021

sergiusens commented Sep 23, 2021

facundobatista commented Sep 23, 2021

heitorPB commented Sep 23, 2021

cmatsuoka commented Sep 23, 2021 • edited Loading

heitorPB commented Sep 23, 2021

cmatsuoka commented Sep 23, 2021

cmatsuoka commented Sep 23, 2021

cmatsuoka commented Sep 24, 2021

heitorPB commented Sep 28, 2021

cmatsuoka commented Sep 28, 2021

facundobatista commented Oct 4, 2021

cmatsuoka commented Sep 23, 2021 •

edited

Loading