Port to Python cryptography module #27

chris-allan · 2022-10-12T16:10:08Z

Ports the plugin to use the Python cryptography module rather than calling out to the OpenSSL command line tools which can be error prone, is not cross platform, and for which error conditions are hard to control for.

Note that this commit also changes the format of `omero.certificates.owner` from the OpenSSL command line `/` separated format to RFC 4514 (which supercedes RFC 2253) `,` separated format. As this plugin saves the owner in OMERO server configuration the upgrade user experience will have to be considered in a follow up commit.

sbesson

Two immediate comments from my initial testing in a OSX 12 / Python 3.8 (server) / OS X 12 / Python 3.10 (client) context. I'll expand my testing on Linux environments.

omero_certificates/certificates.py

sbesson · 2022-10-18T15:34:36Z

Successfully tested in an Ubuntu 20.04 environment with a deployed OMERO.server.

After stopping the server, moving the existing certs and installing this new version of the omero-certificates plugin

(OMERO.venv) root@omero-certificates:/opt/omero# pip install git+https://github.com/ome/omero-certificates@refs/pull/27/merge
...
Successfully built omero-certificates
Installing collected packages: omero-certificates
  Attempting uninstall: omero-certificates
    Found existing installation: omero-certificates 0.2.0
    Uninstalling omero-certificates-0.2.0:
      Successfully uninstalled omero-certificates-0.2.0
Successfully installed omero-certificates-0.2.1.dev59+ga697857
(OMERO.venv) root@omero-certificates:/opt/omero# exit
(OMERO.venv) omero@omero-certificates:~$ pip freeze | grep omero
omero-certificates @ git+https://github.com/ome/omero-certificates@a697857f61bd090dcce591e6beaf8b0cacdc528e
omero-figure==4.4.3
omero-marshal==0.7.0
omero-parade==0.2.3
omero-plus==0.2.2
omero-py==5.12.1
omero-web==5.13.0
(OMERO.venv) omero@omero-certificates:~$ omero certificates -v
INFO:omero_certificates.certificates:Using existing key: /OMERO/certs/server.key
INFO:omero_certificates.certificates:Creating self-signed certificate: /OMERO/certs/server.pem
WARNING:omero_certificates.certificates:'omero.certificates.owner' configuration setting '/L=OMERO/O=OMERO.server' not a valid RFC 4514 string!  Attempting to convert.
INFO:omero_certificates.certificates:Creating PKCS12 bundle: /OMERO/certs/server.p12
certificates created: /OMERO/certs/server.pem /OMERO/certs/server.p12
(OMERO.venv) omero@omero-certificates:~$ sudo service omero-server start

Both local connections and imports worked as expected

(OMERO.venv) omero@omero-certificates:~$ omero login root@localhost 
Previously logged in to localhost:4064 as root
Password:
Created session for root@localhost:4064. Idle timeout: 10 min. Current group: system
(OMERO.venv) omero@omero-certificates:~$ omero import test.fake --errs test.err
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Image:1902

jburel · 2022-10-20T11:51:08Z

I have now modified the OMERO-test-integration job on merge-cito install this PR instead of the release version of omero-certificates.

jburel · 2022-10-20T14:06:19Z

ome/omero-install#269 tests the installation and import of image on Ubuntu 20.04, Debian 10 and Centos7 with scl Python 3.8

jburel · 2022-10-21T06:55:33Z

https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/1263/ is green

jburel · 2022-10-21T06:59:04Z

List of actions derived from this change

openssl will need to be removed from:
- https://omero.readthedocs.io/en/stable/sysadmins/unix/server-centos7-ice36.html
- devspace
add section to upgrade server page

joshmoore · 2022-10-25T14:35:36Z

README.md

-For full information see the output of:
+## Upgrading
+
+Since version 0.3.0 this plugin uses portable RFC 4514 (supercedes RFC 2253) formatted strings for the `omero.certificates.owner` configuration option.  If you have ran `omero certificates` before you may have OpenSSL command line formatted strings in your configuration that should be updated before you can run `omero certificates` again.  In most cases this means taking a string such as `/L=OMERO/O=OMERO.server` and reformatting it to `L=OMERO,O=OMERO.server`; remove the leading `/` and replace separator `/`'s with `,`'s.


Quick note before I need to head out: there is a place in omero-py for upgrading properties in case we want to make use of it. (And this is a really long line...)

jburel · 2022-11-22T13:45:18Z

This PR has been included in daily build for several weeks.
Proposing to merge and tag as 0.3.0
The installation doc should be updated accordingly

sbesson · 2022-11-22T15:51:45Z

As briefly mentioned at the weekly meeting today, my main caveat is that the testing so far has taken place on several operating systems/environment but has been restricted to client/server connections within the server environment itself.

I will work on cross-environment testing and report in the upcoming days.

setup.py

sbesson

Set-up two servers with different versions of Python/OpenSSL with omero-certificates including this PR:

	Ubuntu 20.04	Ubuntu 22.04
Python	3.8.10	3.10.6
Java	11.0.17	11.0.17
OpenSSL	1.1.1	3.0.215
omero-py	5.13.1	5.13.1

Tested the import against both servers using the same systems as well as a CentOS 7 as client. All connections worked as expected.

Client / server	Ubuntu 20.04	Ubuntu 22.04
Ubuntu 20.04	pass	pass
Ubuntu 22.04	pass	pass
CentOS 7	pass	pass

The CentOS 7 / Python 3.6 / OpenSSL 1.0.x client/server should have been tested by the latest builds of the OME CI infrastructure. Deferring to @jburel on whether we want to some cross-environment testing using this environment.

The only remaining issue that was highlighted from my testing is the minimum cryptography requirement. Importantly, this came up when running an upgrade against a deployed system with an older version of cryptography so there is probably a case for testing the upgrade scenario if we cut a new release of omero-certificates.

sbesson · 2023-03-21T08:41:35Z

@jburel with OMERO.server 5.6.7 behind us, I propose we come back to this. What are the next steps to get this released as 0.3.0?

jburel · 2023-03-29T09:46:27Z

Several warning in the merge-ci build

/home/omero/workspace/OMERO-test-integration/.venv3/lib64/python3.6/site-packages/omero_certificates/certificates.py:11: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
02:17:28   from cryptography import x509

sbesson · 2023-03-29T11:08:23Z

This warning communicates the upstream decide to drop Python 3.6 support with the current 40.x series being the last one where this version is supported.
From my side, this is inline with OME's intent to set Python 3.8+ as the recommended minimum Python version for OMERO deployments although I understand the documentation and the CI infrastructure still need to be adjusted accordingly. Independently, Python 3.6 deployments should still be able to install and use the last compatible release of cryptography.

imagesc-bot · 2023-04-06T08:45:40Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ezomero-on-windows/73163/12

sbesson · 2023-06-09T09:14:07Z

Coming back to this as the underlying migration of OME CI infrastructure is getting upgraded to Python 3.8+. This means the concerns echoed in #27 (comment) should be lifted soon. Are there any other outstanding blockers to moving forward with this migration?

Update (2023-06-16): from a discussion with @chris-allan, the current proposal would be to:
1- hold off on this PR for now
2- develop and release a version omero-certificates using openssl generating SSL certificates fixing the Cipher issues seen when newer openssl systems are connecting to CentOS 7/RHEL7
3- once support for CentOS 7/ RHEL7 has been removed, switch to pure Python cryptography as per this PR and remove the OpenSSL 1.0.x workarounds

chris-allan added 8 commits October 12, 2022 10:34

Depend on a version of cryptography that ships universal2 wheels

243ccc9

Port openssl genrsa to use cryptography

1125110

Port openssl pkcs12 to use cryptography

6ec9f05

Remove use of OpenSSL command line tools from the plugin

0a40d93

Remove use of OpenSSL command line tools from test cases

7eb1bda

Remove references to OpenSSL command line tools from README

9ef9399

Better UX for OpenSSL command line DN strings

284619a

chris-allan requested review from jburel, joshmoore and sbesson October 12, 2022 16:10

chris-allan mentioned this pull request Oct 13, 2022

Existing configuration properties may need to be overwritten. Document or add option. #3

Open

sbesson reviewed Oct 13, 2022

View reviewed changes

omero_certificates/certificates.py Show resolved Hide resolved

omero_certificates/certificates.py Outdated Show resolved Hide resolved

OpenSSL < 3.0.0, macOS security framework and Windows compat

050c3c9

chris-allan force-pushed the py-cryptography branch from 3c66b69 to 050c3c9 Compare October 14, 2022 09:16

Add compatibility mode

019bbe9

chris-allan force-pushed the py-cryptography branch from a0d7c08 to 019bbe9 Compare October 14, 2022 12:11

jburel mentioned this pull request Oct 20, 2022

Test certificates ome/omero-install#269

Closed

joshmoore reviewed Oct 25, 2022

View reviewed changes

sbesson reviewed Nov 29, 2022

View reviewed changes

setup.py Outdated Show resolved Hide resolved

sbesson requested changes Dec 2, 2022

View reviewed changes

chris-allan added 2 commits December 16, 2022 09:28

Formatting

1825d99

Update to cryptography>=38.0.0 as suggested

afd2a8e

sbesson approved these changes Dec 16, 2022

View reviewed changes

jburel approved these changes Apr 5, 2023

View reviewed changes

sbesson mentioned this pull request May 12, 2023

Set IceSSL Ciphers to exclude DH #33

Closed

sbesson mentioned this pull request Jun 9, 2023

add dependencies remove by error ome/devspace#204

Closed

sbesson mentioned this pull request Sep 15, 2023

Check whether ffdhe2048.pem exists before writing #38

Merged

chris-allan mentioned this pull request Oct 3, 2023

Use OpenSSL 3 pkcs12 defaults for forwards compatibility #39

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port to Python cryptography module #27

Port to Python cryptography module #27

chris-allan commented Oct 12, 2022

sbesson left a comment

sbesson commented Oct 18, 2022 •

edited

Loading

jburel commented Oct 20, 2022

jburel commented Oct 20, 2022

jburel commented Oct 21, 2022

jburel commented Oct 21, 2022 •

edited

Loading

joshmoore Oct 25, 2022

jburel commented Nov 22, 2022

sbesson commented Nov 22, 2022

sbesson left a comment

sbesson commented Mar 21, 2023

jburel commented Mar 29, 2023

sbesson commented Mar 29, 2023

imagesc-bot commented Apr 6, 2023

sbesson commented Jun 9, 2023 •

edited

Loading

Port to Python cryptography module #27

Are you sure you want to change the base?

Port to Python cryptography module #27

Conversation

chris-allan commented Oct 12, 2022

sbesson left a comment

Choose a reason for hiding this comment

sbesson commented Oct 18, 2022 • edited Loading

jburel commented Oct 20, 2022

jburel commented Oct 20, 2022

jburel commented Oct 21, 2022

jburel commented Oct 21, 2022 • edited Loading

joshmoore Oct 25, 2022

Choose a reason for hiding this comment

jburel commented Nov 22, 2022

sbesson commented Nov 22, 2022

sbesson left a comment

Choose a reason for hiding this comment

sbesson commented Mar 21, 2023

jburel commented Mar 29, 2023

sbesson commented Mar 29, 2023

imagesc-bot commented Apr 6, 2023

sbesson commented Jun 9, 2023 • edited Loading

sbesson commented Oct 18, 2022 •

edited

Loading

jburel commented Oct 21, 2022 •

edited

Loading

sbesson commented Jun 9, 2023 •

edited

Loading