Make `libprotobuf` a `build` only dependency #74

charlesbluca · 2023-10-12T14:56:55Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

conda-forge-webservices · 2023-10-12T14:57:01Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

charlesbluca · 2023-10-12T14:57:11Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2023.10.12.13.22.36

charlesbluca · 2023-10-12T15:00:42Z

@conda-forge-admin, please rerender

github-actions · 2023-10-12T15:02:42Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you but ran into some issues. Please check the output logs of the latest rerendering GitHub actions workflow run for errors. You can also ping conda-forge/core for further assistance or try re-rendering locally.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/dask-sql-feedstock/actions/runs/6497409473.

jakirkham · 2023-10-12T15:18:29Z

If the re-render didn't do anything, this change isn't working. Would suggesting taking a closer look

charlesbluca · 2023-10-12T15:18:56Z

@conda-forge-admin, please rerender

github-actions · 2023-10-12T15:21:07Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/dask-sql-feedstock/actions/runs/6497631962.

jakirkham · 2023-10-12T15:33:25Z

Might want to look at extended_keys

…nda-forge-pinning 2023.10.12.13.22.36

bdice · 2023-10-12T16:20:16Z

Might want to look at extended_keys

@jakirkham I looked over extended_keys but I'm not sure if I understand how it would be used here. libprotobuf is already part of a zip_keys with libgrpc, which seems like it might act differently than the example in the docs which is only one key. I pushed some commits that seem to give the right rerendering, but I'm very open to alternatives if this isn't the best approach.

… conda-smithy 3.27.1, and conda-forge-pinning 2023.10.12.13.22.36

charlesbluca · 2023-10-12T16:33:25Z

@conda-forge-admin, please rerender

jakirkham · 2023-10-12T17:25:58Z

recipe/conda_build_config.yaml

+    - 1.56
+libprotobuf:
+    - 4.24.3
+    - 4.23.3


Should we keep the global pinning for libprotobuf if we are overriding libprotobuf here?

Suggested change

- 4.23.3

- 4.23.4

- 4.23.3

The entries in this matrix are:

Global pinning (4.23.4) plus migrations (which make it 4.24.3)

Older libprotobuf pinning that we need (4.23.3)

jakirkham · 2023-10-12T17:27:23Z

recipe/conda_build_config.yaml

+# replace migrators for libprotobuf 1.57 / 1.58.
+libgrpc:
+    - 1.58
+    - 1.56


Also libgrpc is pinned globally at 1.57. Do we want to include this as well?

Suggested change

- 1.56

- 1.57

- 1.56

The entries in this matrix are:

Global pinning (1.57) plus migrations (which make it 1.58)

libgrpc matching the older libprotobuf pinning that we need

libgrpc 1.56 was originally pinned with libprotobuf 4.23.3, from what I see here: conda-forge/conda-forge-pinning-feedstock@621d4a7

You'll need to zip and match libprotobuf with libgrpc and also libabseil; they're not interrelated on source-level, but since we now need to migrate even for protobuf patch versions, we cannot feasibly build all combinations, and so we coupled them 1:1 as follows:

abseil grpc protobuf

20230125 1.54 3.21

20230125 1.56 4.23.3

20230802 1.57 4.23.4

20230802 1.58 4.24.3

Though I guess it would be conceivable to rebuild grpc 1.56 & protobuf 4.23.3 with newer abseil.

More details here

Thanks Axel! 🙏

Currently we are hoping none of this is needed and that it was an oversight libprotobuf was in host. So are now trying to add it to build only

Though it is possible that won't work, in which case we will go back to adding in another set of versions. We had a working solution later in the commits based on a custom migrator, which was more ergonomic. This is probably what we would go back to if we need to add these in version pinnings in

jakirkham · 2023-10-12T17:30:02Z

Might want to look at extended_keys

@jakirkham I looked over extended_keys but I'm not sure if I understand how it would be used here. libprotobuf is already part of a zip_keys with libgrpc, which seems like it might act differently than the example in the docs which is only one key. I pushed some commits that seem to give the right rerendering, but I'm very open to alternatives if this isn't the best approach.

Am worried that what we are doing here is a bit fragile since we are overriding global pinnings and manually adding values from migrators

Think it would be better if we used some method to append our values to the keys (instead of overriding them). Either using something from conda-build or by manually adding a migrator of our own to add the missed values

jakirkham · 2023-10-12T17:47:25Z

In terms of a migrator (which may wind up being the best/simplest solution), would try...

Copying one of the existing migrators to a new file
Adding use_local: true
Tweaking it to match whatever versions we are wanting to add
Re-rendering to apply the changes

This should add new builds with the expected versions (if not repeat 3-4)

bdice · 2023-10-12T17:56:28Z

In terms of a migrator (which may wind up being the best/simplest solution), would try...
1. Copying one of the existing migrators to a new file

2. Adding [`use_local: true`](https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4400/commits/8c5323d2a65a49be297c216a9b4b3cd0135a3776)

3. Tweaking it to match whatever versions we are wanting to add

4. Re-rendering to apply the changes
This should add new builds with the expected versions (if not repeat 3-4)

This is a good idea. I like it!

…nda-forge-pinning 2023.10.12.13.22.36

jakirkham · 2023-10-12T22:08:48Z

May also need to make this change (in requirements/build):

-   - libprotobuf                            # [build_platform != target_platform]
+   - libprotobuf

…nda-forge-pinning 2023.10.12.13.22.36

jakirkham · 2023-10-12T22:15:33Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2023.10.12.13.22.36

Rebuild for libprotobuf 4.23.3

4def034

charlesbluca requested review from ayushdg, galipremsagar, nils-braun and quasiben as code owners October 12, 2023 14:56

conda-forge-webservices[bot] and others added 2 commits October 12, 2023 14:59

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

5abcf49

…nda-forge-pinning 2023.10.12.13.22.36

Define libprotobuf matrx in conda build config

8909e80

Modify conda build config to define libgrpc variants

ebbbb81

bdice added 3 commits October 12, 2023 11:11

Remove zip_keys.

a6635cf

Remove migrations.

c5a9a97

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

f315333

…nda-forge-pinning 2023.10.12.13.22.36

Add MACOSX_DEPLOYMENT_TARGET and re-rendered with conda-build 3.27.0,…

de9f921

… conda-smithy 3.27.1, and conda-forge-pinning 2023.10.12.13.22.36

bdice force-pushed the libprotobuf4233 branch from fdd85d2 to de9f921 Compare October 12, 2023 16:44

jakirkham reviewed Oct 12, 2023

View reviewed changes

charlesbluca and others added 2 commits October 12, 2023 12:12

Attempt to update build matrix with migrator file

6fee414

Revert changes to migrators.

7a662d4

bdice added 2 commits October 12, 2023 15:21

Update migrator to use operation: key_add.

1c985af

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

35d2ca7

…nda-forge-pinning 2023.10.12.13.22.36

charlesbluca mentioned this pull request Oct 12, 2023

See what happens if we ignore libprotobuf's runtime exports dask-contrib/dask-sql#1242

Closed

charlesbluca added 2 commits October 12, 2023 15:03

Remove libprotobuf from host deps

17a73ae

MNT: Re-rendered with conda-build 3.24.0, conda-smithy 3.27.1, and co…

4cce344

…nda-forge-pinning 2023.10.12.13.22.36

charlesbluca and others added 3 commits October 12, 2023 15:08

Unconditionally include libprotobuf in build deps

a9dc936

MNT: Re-rendered with conda-build 3.24.0, conda-smithy 3.27.1, and co…

5f7c60a

…nda-forge-pinning 2023.10.12.13.22.36

Delete unneeded custom migrator

5775146

conda-forge-webservices[bot] and others added 2 commits October 12, 2023 22:17

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

8c6f29d

…nda-forge-pinning 2023.10.12.13.22.36

Readd EOF newline to conda_build_config.yaml

689afdd

jakirkham changed the title ~~Rebuild for libprotobuf 4.23.3~~ Make libprotobuf a build only dependency Oct 12, 2023

jakirkham approved these changes Oct 12, 2023

View reviewed changes

vyasr approved these changes Oct 12, 2023

View reviewed changes

bdice approved these changes Oct 12, 2023

View reviewed changes

charlesbluca merged commit ce6c8b8 into conda-forge:main Oct 12, 2023
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `libprotobuf` a `build` only dependency #74

Make `libprotobuf` a `build` only dependency #74

charlesbluca commented Oct 12, 2023

conda-forge-webservices bot commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

jakirkham commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

jakirkham commented Oct 12, 2023

bdice commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

jakirkham Oct 12, 2023

bdice Oct 12, 2023

jakirkham Oct 12, 2023

bdice Oct 12, 2023

h-vetinari Oct 12, 2023 •

edited

Loading

h-vetinari Oct 12, 2023

h-vetinari Oct 12, 2023

jakirkham Oct 12, 2023

jakirkham commented Oct 12, 2023

jakirkham commented Oct 12, 2023

bdice commented Oct 12, 2023

jakirkham commented Oct 12, 2023 •

edited

Loading

jakirkham commented Oct 12, 2023

abseil	grpc	protobuf
20230125	1.54	3.21
20230125	1.56	4.23.3
20230802	1.57	4.23.4
20230802	1.58	4.24.3

Make libprotobuf a build only dependency #74

Make libprotobuf a build only dependency #74

Conversation

charlesbluca commented Oct 12, 2023

conda-forge-webservices bot commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

jakirkham commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

jakirkham commented Oct 12, 2023

bdice commented Oct 12, 2023

charlesbluca commented Oct 12, 2023

jakirkham Oct 12, 2023

Choose a reason for hiding this comment

bdice Oct 12, 2023

Choose a reason for hiding this comment

jakirkham Oct 12, 2023

Choose a reason for hiding this comment

bdice Oct 12, 2023

Choose a reason for hiding this comment

h-vetinari Oct 12, 2023 • edited Loading

Choose a reason for hiding this comment

h-vetinari Oct 12, 2023

Choose a reason for hiding this comment

h-vetinari Oct 12, 2023

Choose a reason for hiding this comment

jakirkham Oct 12, 2023

Choose a reason for hiding this comment

jakirkham commented Oct 12, 2023

jakirkham commented Oct 12, 2023

bdice commented Oct 12, 2023

jakirkham commented Oct 12, 2023 • edited Loading

jakirkham commented Oct 12, 2023

Make `libprotobuf` a `build` only dependency #74

Make `libprotobuf` a `build` only dependency #74

h-vetinari Oct 12, 2023 •

edited

Loading

jakirkham commented Oct 12, 2023 •

edited

Loading