Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python: update pyyaml to 6.0.0 #4367

Merged
merged 1 commit into from
Jul 18, 2023

Conversation

uniquefine
Copy link
Contributor

@uniquefine uniquefine commented Jul 18, 2023

Update the pyyaml dependency to 6.0.0.
6.0.0 includes this commit that removes the use of the deprecated license_file field in the setup.cfg.

This addresses the following warning:

The license_file parameter is deprecated, use license_files instead.
      
By 2023-Oct-30, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
      
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.

This change is Reviewable

Copy link
Contributor

@matzf matzf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Thanks!

Reviewed 2 of 2 files at r1.
Reviewable status: all files reviewed (commit messages unreviewed), all discussions resolved (waiting on @uniquefine)

Copy link
Contributor Author

@uniquefine uniquefine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @uniquefine)

@matzf matzf merged commit 5de1e58 into scionproto:master Jul 18, 2023
@matzf
Copy link
Contributor

matzf commented Jul 18, 2023

Note: while the update pyyaml was the right thing to fix the failing builds, the license_file deprecation warning was a red herring. The actual error was the stacktrace just below:

      Traceback (most recent call last):
        File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/43ab9e1896251f6982e33e72692349be/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/43ab9e1896251f6982e33e72692349be/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/43ab9e1896251f6982e33e72692349be/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 314, in run
          self.find_sources()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
          mm.run()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-_npw2glq/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources

This is related to the recent Cython 3.0.0 release, which is silently picked up during the build of the pyyaml wheel. See the sort of related discussion in yaml/pyyaml#702 (perhaps we should have picked 6.0.1).

This highlights that some internal dependencies are not pinned (which is bad). I currently don't know if this is a mistake on our side, or with rules_python, or somewhere deep in the python wheel-house.

@uniquefine uniquefine deleted the update-pyyaml-dependency branch July 18, 2023 13:17
matzf pushed a commit that referenced this pull request Jul 21, 2023
Update pyyaml dependency from 6.0.0 -> 6.0.1, as amendment to #4367.

The actual issue that had caused our builds to fail suddenly, was not
related to the license_file warning, but an AttributeError  related to a
"cython_sources" option. It appears that pyyaml had an unconstrained
version of for Cython as a build system  dependency. When Cython 3 was
released, this started to fail.
Pyyaml 6.0.1 now has a constraint for Cython version < 3, which should
keep things working.

Somewhat worrying, and not addressed by this change, is the fact that
the build system dependencies do not seem to be pinned by our
requirements.txt file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants