opensuse-tumbleweed: python3 build fails because of openssl #34273

mkoeppe · 2022-08-03T23:56:27Z

Our OpenSUSE system package info leads to the installation of LibreSSL, which is not compatible with the python3 build.

[python3-3.10.6] checking whether compiling and linking against OpenSSL works... yes
[python3-3.10.6] checking for --with-openssl-rpath... 
[python3-3.10.6] checking whether OpenSSL provides required APIs... no
[python3-3.10.6] checking for --with-ssl-default-suites... python
[python3-3.10.6] checking for --with-builtin-hashlib-hashes... md5,sha1,sha256,sha512,sha3,blake2
[python3-3.10.6] checking for --with-experimental-isolated-subinterpreters... no
[python3-3.10.6] checking for --with-static-libpython... yes
[python3-3.10.6] checking for --disable-test-modules... no
[python3-3.10.6] configure: creating ./config.status

Our configure script does not know that.

We update the distro file and add an additional check to openssl spkg-configure.m4

To test: tox -e docker-opensuse-tumbleweed-standard -- V=1 python3

CC: @sagetrac-tmonteil @dimpase

Component: packages: standard

Author: Matthias Koeppe

Branch/Commit: d46f190

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/34273

The text was updated successfully, but these errors were encountered:

mkoeppe · 2022-08-04T00:05:11Z

comment:2

configure:17891: gcc -o conftest  -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden   -Wl,-rpath-link,/sage/local/lib -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-rpath-link,/sage/local/lib -L/sage/local/lib -Wl,-rpath,/sage/local/lib   conftest.c -lcrypt -ldl  -lm -lssl -lcrypto  >&5
conftest.c: In function 'main':
conftest.c:442:12: error: 'NID_sha3_512' undeclared (first use in this function); did you mean 'NID_sha512'?
  442 | OBJ_nid2sn(NID_sha3_512);
      |            ^~~~~~~~~~~~
      |            NID_sha512
conftest.c:442:12: note: each undeclared identifier is reported only once for each function it appears in
conftest.c:443:12: error: 'NID_blake2b512' undeclared (first use in this function)
  443 | OBJ_nid2sn(NID_blake2b512);
      |            ^~~~~~~~~~~~~~
conftest.c:444:1: error: implicit declaration of function 'EVP_PBE_scrypt'; did you mean 'EVP_PKEY_decrypt'? [-Werror=implicit-function-declaration]
  444 | EVP_PBE_scrypt(NULL, 0, NULL, 0, 2, 8, 1, 0, NULL, 0);
      | ^~~~~~~~~~~~~~
      | EVP_PKEY_decrypt
cc1: some warnings being treated as errors
configure:17891: $? = 1

mkoeppe · 2022-08-04T00:12:11Z

comment:3

We should probably explicitly install libopenssl-3-devel

mkoeppe · 2022-08-04T00:15:01Z

Branch: u/mkoeppe/opensuse_tumbleweed__python3_build_fails_because_of_openssl

mkoeppe · 2022-08-04T00:22:48Z

comment:5

We should probably get the more detailed configure check from Python:
https://github.com/python/cpython/blob/main/configure.ac#L6823

New commits:

`de3e0f3`	`build/pkgs/openssl/distros/opensuse.txt: Use libopenssl-3-devel`

mkoeppe · 2022-08-04T00:22:48Z

Commit: de3e0f3

mkoeppe · 2022-08-04T00:25:23Z

comment:6

or rather https://github.com/python/cpython/blob/3.10/configure.ac#L5845

sagetrac-git · 2022-08-04T00:33:00Z

Branch pushed to git repo; I updated commit sha1. New commits:

`d46f190`	`build/pkgs/openssl/spkg-configure.m4: Add API check from python3 configure.ac`

sagetrac-git · 2022-08-04T00:33:00Z

Changed commit from de3e0f3 to d46f190

mkoeppe · 2022-08-04T00:43:34Z

Author: Matthias Koeppe

sagetrac-git · 2022-08-04T01:35:52Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a9eaae5`	`tox.ini, build/bin/write-dockerfile.sh: Add 'tox -e docker-...-incremental'`
`b9bfbf9`	`tox.ini: Add comment`
`4c0d7f5`	`tox.ini: Use FROM_DOCKER_REPOSITORY`
`a07874d`	`build/bin/write-dockerfile.sh: In incremental build, keep logs`
`ae269d1`	`tox.ini (docker-incremental): Do not include '-incremental' in the Docker image name`
`b354a38`	`tox.ini, .github/workflows/docker.yml: Reimplement -incremental via SKIP_SYSTEM_PACKAGES`
`5decadd`	`build/bin/write-dockerfile.sh: Do not use persistent env var PACKAGES (except on nix)`
`49ebf1b`	`Merge #34228`

sagetrac-git · 2022-08-04T01:35:52Z

Changed commit from d46f190 to 49ebf1b

sagetrac-git · 2022-08-04T01:36:43Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

sagetrac-git · 2022-08-04T01:36:43Z

Changed commit from 49ebf1b to d46f190

mkoeppe · 2022-08-05T01:25:34Z

Reviewer: https://github.com/mkoeppe/sage/runs/7682738998

mkoeppe · 2022-08-05T03:11:15Z

comment:16

This fixes it. Let's get it in

kwankyu · 2022-08-08T11:10:19Z

comment:18

I ran the test and got: docker-opensuse-tumbleweed-standard: commands succeeded.

I am new to this business, and let me ask some naive questions. What was the sign of failure (of LibreSSL)? From the tox test, I still get

[python3-3.10.5] checking whether compiling and linking against OpenSSL works... yes
[python3-3.10.5] checking for --with-openssl-rpath... 
[python3-3.10.5] checking whether OpenSSL provides required APIs... yes
[python3-3.10.5] checking for --with-ssl-default-suites... python
[python3-3.10.5] checking for --with-builtin-hashlib-hashes... md5,sha1,sha256,sha512,sha3,blake2
[python3-3.10.5] checking for --with-experimental-isolated-subinterpreters... no
[python3-3.10.5] checking for --with-static-libpython... yes
[python3-3.10.5] checking for --disable-test-modules... no

which seems not much different from the excerpt in the ticket description... How do I check if libopenssl-3-devel works with python3 well? Is the message Sage build/upgrade complete! enough?

mkoeppe · 2022-08-08T14:00:27Z

comment:19

Thanks for testing!

The crucial line in python3's configure output is this:

[python3-3.10.5] checking whether OpenSSL provides required APIs... yes

... which has changed from "no".

I should have added a link to a failing build for reference. In https://github.com/sagemath/sage/actions, selecting "CI Linux", there is a completed run for 9.7.beta7, https://github.com/sagemath/sage/actions/runs/2777704621

Selecting "docker / linux (opensuse-tumbleweed, standard)" from the sidebar, one gets to https://github.com/sagemath/sage/runs/7619552751, which shows the end of the failing build of the python3 SPKG:

  [python3-3.10.5]   
  [python3-3.10.5]   Failed to build these modules:
  [python3-3.10.5]   _hashlib              _ssl                                     
  [python3-3.10.5]   
  [python3-3.10.5]   
  [python3-3.10.5]   Could not build the ssl module!
  [python3-3.10.5]   Python requires a OpenSSL 1.1.1 or newer
  [python3-3.10.5]   
  ...
  [python3-3.10.5]   Testing importing of various modules...
  [python3-3.10.5]   ctypes module imported OK
  [python3-3.10.5]   math module imported OK
  [python3-3.10.5]   hashlib module imported OK
  [python3-3.10.5]   crypt module imported OK
  [python3-3.10.5]   socket module imported OK
  [python3-3.10.5]   zlib module imported OK
  [python3-3.10.5]   sqlite3 module imported OK
  [python3-3.10.5]   Traceback (most recent call last):
  [python3-3.10.5]     File "<string>", line 1, in <module>
  [python3-3.10.5]     File "/sage/local/var/lib/sage/venv-python3.10.5/var/tmp/sage/build/python3-3.10.5/src/Lib/ssl.py", line 99, in <module>
  [python3-3.10.5]       import _ssl             # if we can't import it, let the error propagate
  [python3-3.10.5]   ModuleNotFoundError: No module named '_ssl'
  [python3-3.10.5]   ssl module failed to import
  [python3-3.10.5]   Error: One or more modules failed to import.
  [python3-3.10.5]   
  [python3-3.10.5]   real	5m46.942s
  [python3-3.10.5]   user	2m41.648s
  [python3-3.10.5]   sys	0m17.654s
  [python3-3.10.5]   ************************************************************************
  [python3-3.10.5]   Error building package python3-3.10.5
  [python3-3.10.5]   ************************************************************************
  [python3-3.10.5] Full log file: /sage/logs/pkgs/python3-3.10.5.log
make[2]: *** [Makefile:2930: python3-SAGE_VENV-no-deps] Error 1

What happened there is that Python did not build the _ssl module because it did not find a suitable openssl. Python reports this with the message "Could not build the ssl module!" This is only an info message.

Our script https://github.com/sagemath/sage-prod/blob/develop/build/pkgs/python3/spkg-build.in#L120 then checks again for the required modules and exits with an error.

How do I check if libopenssl-3-devel works with python3 well? Is the message Sage build/upgrade complete! enough?

I would say it is not necessary to do more testing than what our build script does.

kwankyu · 2022-08-08T14:51:29Z

comment:20

Replying to @mkoeppe:

I would say it is not necessary to do more testing than what our build script does.

Okay. Now I understand the situation better. So the success of the tox test has shown that your branch works well (otherwise python build script would have failed).

The relevant github workflow is still running. So let us wait for the result to double-check.

The branch itself seems good as far as I can judge.

kwankyu · 2022-08-08T15:03:30Z

comment:21

Another test would be to install pkgconfig(libssl) system package, and to check if the configure correctly decide to install openssl from source. Right?

It is quite out of my technical reach though. Have you tried it?

mkoeppe · 2022-08-08T15:11:13Z

comment:22

Yes, I tested this. If you want to test it, you can just git revert de3e0f3d5d405ef0f255aaf6d9934f86c72fcca7 and then run tox -e docker-opensuse-tumbleweed-standard -- config.status

kwankyu · 2022-08-09T00:18:11Z

comment:23

Replying to @mkoeppe:

Yes, I tested this. If you want to test it, you can just git revert de3e0f3d5d405ef0f255aaf6d9934f86c72fcca7 and then run tox -e docker-opensuse-tumbleweed-standard -- config.status

Ah, easy. I did it, and the configure worked correctly.

kwankyu · 2022-08-09T00:30:50Z

comment:24

Replying to @kwankyu:

The relevant github workflow is still running. So let us wait for the result to double-check.

It would take too long to get the result. As this is simply double-checking the correct result I got locally, I will skip it.

Then, positive review to this ticket.

kwankyu · 2022-08-09T00:30:50Z

Changed reviewer from https://github.com/mkoeppe/sage/runs/7682738998 to Kwankyu Lee

mkoeppe · 2022-08-09T00:37:02Z

comment:25

Thank you!

vbraun · 2022-08-29T11:24:36Z

Changed branch from u/mkoeppe/opensuse_tumbleweed__python3_build_fails_because_of_openssl to d46f190

mkoeppe added this to the sage-9.7 milestone Aug 3, 2022

mkoeppe added c: packages: standard labels Aug 3, 2022

This comment has been minimized.

Sign in to view

mkoeppe added s: needs review and removed s: needs review labels Aug 4, 2022

mkoeppe added s: needs review and removed s: needs work labels Aug 4, 2022

This comment has been minimized.

Sign in to view

mkoeppe added p: critical / 2 and removed p: major / 3 labels Aug 4, 2022

mkoeppe removed the p: critical / 2 label Aug 7, 2022

mkoeppe added the p: blocker / 1 label Aug 7, 2022

kwankyu added s: positive review and removed s: needs review labels Aug 9, 2022

vbraun removed the s: positive review label Aug 29, 2022

vbraun closed this as completed in 20d7f18 Aug 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

opensuse-tumbleweed: python3 build fails because of openssl #34273

opensuse-tumbleweed: python3 build fails because of openssl #34273

mkoeppe commented Aug 3, 2022

This comment has been minimized.

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

This comment has been minimized.

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

This comment has been minimized.

mkoeppe commented Aug 5, 2022

mkoeppe commented Aug 5, 2022

kwankyu commented Aug 8, 2022

mkoeppe commented Aug 8, 2022

kwankyu commented Aug 8, 2022

kwankyu commented Aug 8, 2022

mkoeppe commented Aug 8, 2022

kwankyu commented Aug 9, 2022

kwankyu commented Aug 9, 2022

kwankyu commented Aug 9, 2022

mkoeppe commented Aug 9, 2022

vbraun commented Aug 29, 2022

opensuse-tumbleweed: python3 build fails because of openssl #34273

opensuse-tumbleweed: python3 build fails because of openssl #34273

Comments

mkoeppe commented Aug 3, 2022

This comment has been minimized.

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

mkoeppe commented Aug 4, 2022

This comment has been minimized.

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

sagetrac-git mannequin commented Aug 4, 2022

This comment has been minimized.

mkoeppe commented Aug 5, 2022

mkoeppe commented Aug 5, 2022

kwankyu commented Aug 8, 2022

mkoeppe commented Aug 8, 2022

kwankyu commented Aug 8, 2022

kwankyu commented Aug 8, 2022

mkoeppe commented Aug 8, 2022

kwankyu commented Aug 9, 2022

kwankyu commented Aug 9, 2022

kwankyu commented Aug 9, 2022

mkoeppe commented Aug 9, 2022

vbraun commented Aug 29, 2022