Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to py311 #12

Open
wants to merge 4 commits into
base: SP1
Choose a base branch
from
Open

Move to py311 #12

wants to merge 4 commits into from

Conversation

schaefi
Copy link
Collaborator

@schaefi schaefi commented Jul 3, 2024

Move to python 3.11 stack for SLE/Leap

Since SLE and Leap has python 3.11 we can now finally move to
a newer python version for this target

The required functionality is provided by the core os library
@schaefi schaefi requested a review from davidcassany July 3, 2024 13:06
@schaefi schaefi self-assigned this Jul 3, 2024
@schaefi
Copy link
Collaborator Author

schaefi commented Jul 3, 2024

Copy link
Collaborator

@davidcassany davidcassany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I struggle to follow the spec changes, many macros changed...

By looking the package builds I see it results into a different package name:

SLE-15-SP1 has python3-kiwi-9.24.43
The QA:Maintenance project builds python311-kiwi-9.24.43

Different package name with different provision with the same exact version.... isnt this so confusing? I doubt this is what we want, I might miss some details, but at a glance this looks wrong to me and resulting into two different packages that might conflict to to install both together and that the new one might not be seen as an update from the package manager.

@schaefi
Copy link
Collaborator Author

schaefi commented Jul 5, 2024

@davidcassany Actually this is what we want to achieve. I'm not sure if you have followed the conversation around python in SLES15 but beginning with SLES15-SP5 onwards a full python 3.11 stack exists in combination with the old python 3.6 stack. This gives us finally the opportunity to move kiwi in SLE15 to python 3.11.

In order to do this the SLE team has provided new macros to allow an easy way to package python modules in SLES. This is what you see with %{?sle15_python_module_pythons} and it also finally allows to get rid of all the extra macros we have defined in the spec.

As a consequence of using the macros the package name changes intentionally

So kiwi for python 3.11 in SLES has to be named differently because all old 3.6 builds stays as python3-xxx whereas all 3.11 builds are named python311-xxx. That's how they were able to add the 3.11 stack without breaking anything

SLE-15-SP1 has python3-kiwi-9.24.43
The QA:Maintenance project builds python311-kiwi-9.24.43

Correct and the wanted behavior

Different package name with different provision with the same exact version.... isnt this so confusing?

different python interpreters on the same distro makes this inevitable. On TW you will see even more of this because there is 310, 311 and 312

resulting into two different packages that might conflict to to install both together

nope no conflicts, as they differentiate the binary names and the location of files by the version of the python interpreter. And all this is handled by the macro and fortunately out of my hands, it just worked :)

With this background information in mind could you review again ?

Thanks much

Since SLE and Leap has python 3.11 we can now finally move to
a newer python version for this target
@davidcassany
Copy link
Collaborator

@schaefi thanks much for the explanations.

@davidcassany Actually this is what we want to achieve. I'm not sure if you have followed the conversation around python in SLES15 but beginning with SLES15-SP5 onwards a full python 3.11 stack exists in combination with the old python 3.6 stack. This gives us finally the opportunity to move kiwi in SLE15 to python 3.11.

Oh I see, I did not follow the conversation around python packaging sorry. I see, so correct me if I am wrong, but I understand that both packages py3.6 and py3.11 will be installable together and both will be launched at the same version, at least for the first release. My doubt is what happens from now on for the py3.6 version of it, is it now deprecated? To my understanding with the current change KIWI sources will no longer build py3.6 version of it. I don't have any issue with that setup, just trying to understand maintenance implications of the change.

In order to do this the SLE team has provided new macros to allow an easy way to package python modules in SLES. This is what you see with %{?sle15_python_module_pythons} and it also finally allows to get rid of all the extra macros we have defined in the spec.

Yes, this is nice, in fact the spec is significantly simplified in this PR.

As a consequence of using the macros the package name changes intentionally

So kiwi for python 3.11 in SLES has to be named differently because all old 3.6 builds stays as python3-xxx whereas all 3.11 builds are named python311-xxx. That's how they were able to add the 3.11 stack without breaking anything

👍

All looking good from my side then. As I suspected I just missed many details on the packaging 😅

@schaefi
Copy link
Collaborator Author

schaefi commented Jul 8, 2024

Oh I see, I did not follow the conversation around python packaging sorry. I see, so correct me if I am wrong, but I understand that both packages py3.6 and py3.11 will be installable together and both will be launched at the same version, at least for the first release.

I'm not aiming for two packages (3.6 and 3.11). In fact the changes here will cause the python code to be no longer working with 3.6 because of the other commit where I moved the xattr stuff to os.getxattr which does not exist in 3.6. So when this PR gets merged we move kiwi in SLES to py311 and thinking further I believe we need an Obsoletes: setting in the spec for the old package python3-kiwi. What do you think ?

My doubt is what happens from now on for the py3.6 version of it, is it now deprecated?

yes that's what I want to achieve

To my understanding with the current change KIWI sources will no longer build py3.6 version of it. I don't have any issue with that setup, just trying to understand maintenance implications of the change.

It's good that you ask all the questions so we can think it through. Thanks much

Obsolete python3-kiwi which is the version that
builds against the old 3.6 stack
@davidcassany
Copy link
Collaborator

davidcassany commented Jul 8, 2024

I'm not aiming for two packages (3.6 and 3.11). In fact the changes here will cause the python code to be no longer working with 3.6 because of the other commit where I moved the xattr stuff to os.getxattr which does not exist in 3.6. So when this PR gets merged we move kiwi in SLES to py311 and thinking further I believe we need an Obsoletes: setting in the spec for the old package python3-kiwi. What do you think ?

Sounds good. I guess this requires some further testing though, I'll try to check if I can easily run a test on Leap 15.6, should be a reasonable OS for testing all this.

To my understanding what we want is making the upgrade of kiwi transparent, hence if anyone runs zypper up having python3-kiwi this will result on installing python311-kiwi and its new dependencies (probably python3.11 stack if not installed already) and removing python3-kiwi. Also a manual install of zypper in python311-kiwi while having python3-kiwi installed should end up with the same result as an upgrade.

To my eyes this two cases above is what needs to be tested and those require the Obsoletes. I never now by heart how this two RPM spec directives, Obsoletes and Conflicts, work together so I'll try to run some tests.

@schaefi
Copy link
Collaborator Author

schaefi commented Jul 8, 2024

To my understanding what we want is making the upgrade of kiwi transparent, hence if anyone runs zypper up having python3-kiwi this will result on installing python311-kiwi and its new dependencies (probably python3.11 stack if not installed already) and removing python3-kiwi. Also a manual install of zypper in python311-kiwi while having python3-kiwi installed should end up with the same result as an upgrade.

yes exactly

To my eyes this two cases above is what needs to be tested and those require the Obsoletes. I never now by heart how this two RPM spec directives, Obsoletes and Conflicts, work together so I'll try to run some tests.

Same here 😄

Thanks much for testing. I have only tested in the QA projects, but that doesn't cover the update scenario you described

@davidcassany
Copy link
Collaborator

davidcassany commented Jul 9, 2024

Alright these are the results of my tests:

Running zypper up on Leap 15.6 having python3-kiwi already installed in it:
image

Running zypper in python311-kiwi on Leap 15.6 having python3-kiwi already installed in it
image

Running zypper in python311-kiwi on Leap 15.6 without any KIWI installed:
image

So first surprising result, which is probably not that bad (for now), but IMHO not desired, is that some of the subpackages that are not python related are getting mixed (e.g. man pages). Since both are in the same version it should not be a problem and the content should be the same, however, they come from two independent source rpms... I'd say this comes from this lines on the main package:

Conflicts:      kiwi-man-pages < %{version}

This might not be strict enough as higher versions of kiwi-man-pages are not conflicting. I have no good ideas on how to improve it. I think that making python311-kiwi to be seen as an upgrade of python3-kiwi from the package manager PoV should be a good enough solution.

I don't think this is a big issue now, but it might be in later on versions once kiwi py3.11 source code is no longer aligned with kiwi py3.6 package. We don't want that a zypper up updates kiwi-man-pages to a higher version of currently installed kiwi. To my understanding this could happen silently by anyone that simply keeps calling zypper up on regular basis.

@schaefi do my tests make sense to you? Have you done some other sort of trials?

@schaefi
Copy link
Collaborator Author

schaefi commented Jul 10, 2024

@davidcassany Thanks for testing.

  • I think that zypper up does not move to the 311 variant is expected, and the update of the non-python packages is ok too
  • The issue with the -solv package I have fixed by making it a Recommended package. kiwi does not require the solv plugin and Michael seems to not have it ported to py311 for SLES anyway
  • The zypper in on a system without any kiwi looks good

I think what we want to test is on a system with python3-kiwi installed and zypper dup because that should move to the 311 variant and drop the other

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants