Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate libabseil, libgrpc & libprotobuf #4814

Merged
merged 3 commits into from
Sep 10, 2023
Merged

Conversation

h-vetinari
Copy link
Member

Follows plan in #4075; currently on hold until we move to MacOS 10.13, because abseil requires it, and affects a lot of packages.

@conda-forge-webservices
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@h-vetinari
Copy link
Member Author

h-vetinari commented Aug 21, 2023

Redoing the analysis from #4768, this time with libabseil, libgrpc, libprotobuf and some relevant first-order dependencies, we get:

mamba repoquery whoneeds libabseil -c conda-forge -p linux-64 > impact.txt
mamba repoquery whoneeds libprotobuf -c conda-forge -p linux-64 >> impact.txt
mamba repoquery whoneeds protobuf -c conda-forge -p linux-64 >> impact.txt
mamba repoquery whoneeds libgrpc -c conda-forge -p linux-64 >> impact.txt
mamba repoquery whoneeds grpcio -c conda-forge -p linux-64 >> impact.txt
mamba repoquery whoneeds libarrow -c conda-forge -p linux-64 >> impact.txt
mamba repoquery whoneeds pyarrow -c conda-forge -p linux-64 >> impact.txt
mamba repoquery whoneeds pytorch -c conda-forge -p linux-64 >> impact.txt
# light editing to remove headers
python
>>> q = open("impact.txt", "r").readlines()
>>> p = {x.strip().split(" ")[0] for x in q} - {"", "abseil-cpp", "arrow-cpp", "grpc-cpp", "ortools-cpp", "cpp-opentelemetry-sdk"}  # remove aliases
>>> len(p)
819
>>> print("\n".join(sorted(list(p))))
Lower bound for list of affected packages
>>> print("\n".join(sorted(list(p))))
_dvc-hdfs
abacus
ablang
accelerate
acsconv
activitysim
adaptive-scheduler
aim
aimodelshare
airflow-provider-xlsx
airflow-with-grpc
airflow-with-pandas
airflow-with-papermill
alchemlyb
alibi-detect
allennlp
allennlp-models
ansys-api-mapdl-v0
apache-airflow-providers-google
apache-airflow-providers-grpc
apache-airflow-providers-papermill
apache-beam
api2db
arcos-gui
arcticdb
argopandas
arize-phoenix
arrow-c-glib
arrow_odbc
asdfghjkl
austin-python
auto3d
autogluon.core
autogluon.multimodal
autogluon.tabular
autogluon.timeseries
av2
avalanche-lib
awkward-pandas
awswrangler
axial-positional-embedding
axisregistry
azure-quantum
baal
backpack-for-pytorch
bazel
bear
bentoml
bert_score
bertopic
bigearthnet-common
bigearthnet-gdf-builder
bindsnet
bio-embeddings
bio-embeddings-bepler
bio-embeddings-cpcprot
bio-embeddings-deepblast
bio-embeddings-esm
bio-embeddings-plus
biobear
bitsandbytes
blitz-bayesian-pytorch
bloptools
boa-framework
botorch
brax
bufrtools
bytewax
caffe
captum
carterbox-torch-radon
cartographer
cartwright
catalystcoop.pudl
causalegm
causalml
causalnex
cdt
cellpose
census-parquet
chainer
chemprop
ci-sdr
cirq-google
classy_vision
cliffordlayers
clouddrift
coffea
cofi
coiled-runtime
compass-interface
compass-interface-pandas
condastats
contrastive-learner
coqui-trainer
coremltools
cox
crypten
ctgan
ctrl-benchmark
cylc-flow
cylc-flow-base
czmodel
d3rlpy
dagster
dagster-duckdb-polars
dagster-duckdb-pyspark
darts
dask
dask-bigquery
dask-deltatable
dask-expr
dask-ms
dask-snowflake
dask-sql
dask-yarn
databricks-sql-connector
datafusion
datapane
dataprofiler
datasets
db-dtypes
dbt-bigquery
dbt-core
dcgp-python
ddsketch
ddtrace
deepecho
deepforest
deeppurpose
deeprobust
deepspeed
delimited-protobuf
delta-sharing-python
deltalake
descarteslabs
detecto
detectron2
dgl
diffusers
dingo-gw
dirty_cat
dlhub-sdk
dm-control
dm-tree
dm_control
docquery
dpt
dremio-arrow
dsc
dtoolai
dvc
dvc-hdfs
dye-score
e3nn
earth-osm
earthformer
easyocr
efficientnet-pytorch
emmental
empanada-dl
eo-dataflow-report
espaloma
espnet
etcdpy
evotorch
ezyrb
facetorch
facexlib
facilyst
fair-esm
fairscale
fairseq
farm
fastai
fcd_torch
fcos
feast
feather-format
featherhelper
featurewiz
felyx-processor
felyx-report
finetuning-scheduler
firelight
fitsnap3
flaightidl
fletcher
flowtorch
flwr
flyteidl
flytekit
flytekitplugins-modin
fml
fonduer
foundry-dev-tools
foyer
fugue
functorch
fvgp
gandlf
gazebo
gazebo-yarp-plugins
gdptools
geobuf
geosnap
geospatial-ml
geotorch
gflanguages
glasflow
gluoncv
gluonts
gmpyinfr_dbutils
gnss-sdr
go-cockroach
google-ads
google-api-core
google-api-core-grpc
google-cloud-aiplatform
google-cloud-appengine-logging
google-cloud-artifact-registry
google-cloud-audit-log
google-cloud-automl
google-cloud-bigquery
google-cloud-bigquery-core
google-cloud-bigquery-datatransfer
google-cloud-bigquery-storage
google-cloud-bigquery-storage-core
google-cloud-bigtable
google-cloud-billing-budgets
google-cloud-build
google-cloud-compute
google-cloud-container
google-cloud-core
google-cloud-cpp
google-cloud-datacatalog
google-cloud-dataflow-client
google-cloud-dataform
google-cloud-dataplex
google-cloud-dataproc
google-cloud-dataproc-metastore
google-cloud-datastore
google-cloud-dlp
google-cloud-error-reporting
google-cloud-firestore
google-cloud-kms
google-cloud-language
google-cloud-life-sciences
google-cloud-logging
google-cloud-memcache
google-cloud-monitoring
google-cloud-orchestration-airflow
google-cloud-os-login
google-cloud-pubsub
google-cloud-pubsublite
google-cloud-recommendations-ai
google-cloud-redis
google-cloud-resource-manager
google-cloud-scheduler
google-cloud-secret-manager
google-cloud-spanner
google-cloud-speech
google-cloud-storage
google-cloud-storage-transfer
google-cloud-tasks
google-cloud-texttospeech
google-cloud-trace
google-cloud-translate
google-cloud-videointelligence
google-cloud-vision
google-cloud-workflows
googleapis-common-protos
googleapis-common-protos-grpc
googleapis-cpp
gpcam
gpytorch
grad-cam
gradsflow
graphistry
graphium
graphormer-pretrained
graphvite
graphysio
groundingdino-py
grpc-google-iam-v1
grpc-stubs
grpc_java_plugin
grpcio
grpcio-gcp
grpcio-health-checking
grpcio-reflection
grpcio-status
grpcio-tools
gunpowder
gymnasium-other
gz-sim7-python
hangar
heavyai
higher
hippynn
hloc
hover
hummingbird-ml
hydrofunctions
hypernets
ibis-bigquery
ibis-dask
ibis-framework
ibis-framework-core
ibis-heavyai
ibis-omniscidb
ibis-parquet
ibis-postgres
ibis-pyspark
ibis-substrait
ibis-trino
ilamb
imagedataextractor
imaginairy
imp
incognita
inferno
instructorembedding
intake-dremio
intake-odbc
intake-parquet
invariant-point-attention
ipumspy
isofit
jax
jaxlib
jina
jina-core
jina-perf
json2parquet
jupyter-hdfscm
kartothek
kats
kedro
kedro-vertexai
kelp-o-matic
keras-radam
keras-tuner
keras2onnx
kfp
kfp-pipeline-spec
koalas
koopa-viz
kornia
label-maker
labours
laplace-torch
larch
lava-dl
lenskit
libabseil-tests
libarrow
libgazebo-yarp-plugins
libgdal-arrow-parquet
libgoogle-cloud
libgrpc
libgz-fuel-tools8
libgz-gui7
libgz-launch6
libgz-msgs9
libgz-sensors7
libgz-sim7
libgz-transport12
libhdfs3
libignition-fuel-tools
libignition-fuel-tools4
libignition-fuel-tools5
libignition-fuel-tools6
libignition-fuel-tools7
libignition-gazebo4
libignition-gazebo5
libignition-gazebo6
libignition-gui4
libignition-gui5
libignition-gui6
libignition-launch3
libignition-launch4
libignition-launch5
libignition-msgs1
libignition-msgs5
libignition-msgs6
libignition-msgs7
libignition-msgs8
libignition-sensors4
libignition-sensors5
libignition-sensors6
libignition-transport10
libignition-transport11
libignition-transport4
libignition-transport8
libignition-transport9
libopencv
libopentelemetry-cpp
libortools
libprotobuf
libprotobuf-static
libpulsar
libsentencepiece
libtensorflow
libtensorflow_cc
lightning
lightning-bolts
lightning-flash
lightwood
linear_operator
local-attention
loralib
lovely-tensors
lpips
lsst_dashboard
lsstdesc-env
lsstdesc-gcr-catalogs
lstchain
mage-ai
mapbox-vector-tile
mapshader
materials-learning-algorithms
matrixprofile
maxsmi
mbuild
medigan
micro_sam
min-dalle
mitmproxy
ml-research
mlflow
mlflow-pipelines
mlflow-skinny
mmcv
mmcv-full
mmdet
mne-icalabel
modin-hdk
modin-omnisci
modin-ray
modzy-sdk
molearn
molecule-generation
molfeat
molgrid
molsets
monai
mongodb
monotonicnetworks
mosartwmpy
mosh
mujoco-cxx
mujoco-python
muon
mup
mvlearn
mypy-protobuf
mysql-connector-python
mysql-router
mysql-server
nagl
nannos
nannyml
napari-affinities
napari-cellseg3d
napari-czann-segment
napari-locpix
napari-n2v
napari-philow
napari-picasso
napari-pssr
napari-sam
napari-sam4is
napari-segment-anything
napari-stable-diffusion
napari-unicell
napari-yolov5
nengo-loihi
nequip
nessai
neuralforecast
neuralprophet
nevergrad
ngboost
nlp
nlpaug
nlptest
nncf
nnpops
norse
nteract-scrapbook
numbers-parser
nutpie
nvidia-apex
nystrom-attention
nyxus
obake
obake-devel
ocsmesh
ogb
omicverse
onnx
onnx2torch
onnxconverter-common
onnxmltools
onnxruntime
onnxruntime-cpp
onnxruntime-novec
onnxruntime-novec-cpp
opacus
open-clip-torch
opencensus-proto
opendp-smartnoise-core
openff-nagl
openml
openmm-torch
openmmtorchplugin
opentelemetry-exporter-jaeger-proto-grpc
opentelemetry-exporter-opencensus
opentelemetry-exporter-otlp-proto-grpc
opentelemetry-exporter-zipkin-proto-http
opentelemetry-proto
opt_einsum_fx
optimum
optimum-graphcore
optimum-habana
orbit-ml
orc
ortools-python
ortools-samples
pado
pandas-gbq
pandera
pandera-base
pandera-core
paraview
parquet-tools
parsmurf
patchnetvlad
peft
performer-pytorch
pgmpy
pixsfm
plateau
platelet-unet-watershed
plio
ploomber
premise
pretrainedmodels
proto-plus
protobuf
protobuf3-to-dict
protoc-gen-swagger
protoc-gen-validate
protoletariat
protozfits
provenance
ptgnn
ptychonn
py-grpc-prometheus
pyarrow
pyarrow-tests
pyaudi
pychromecast
pycroscopy
pydiverse-pipedag
pyfitparquet
pyg-lib
pygadm
pygaul
pygeohydro
pygod
pygwalker
pyhdk
pyhdx
pyiceberg
pyiem
pyinaturalist-convert
pyinterpolate
pymace
pymapd
pymde
pymilvus
pymongoarrow
pymovements
pymrio
pynhd
pyobsplot
pyomnisci
pyomniscidbe
pypots
pyquokka
pyramids
pyro-ppl
pysipfenn
pyspark
pysurvival
pythae
python-dremio-client
python-duckdb
python-flair
python-substrait
pytorch
pytorch-cpu
pytorch-dp
pytorch-forecasting
pytorch-gpu
pytorch-ignite
pytorch-lightning
pytorch-metric-learning
pytorch-model-summary
pytorch-pretrained-bert
pytorch-ranger
pytorch-spiking
pytorch-tabnet
pytorch-wpe
pytorch3d
pytorch_cluster
pytorch_geometric
pytorch_revgrad
pytorch_scatter
pytorch_sparse
pytorch_spline_conv
pytorch_tabular
pytorch_warmup
pytorchcv
qcfractal-core
qcportal
qgis
qp
qp-prob-full
quilt
quilt3
r-arrow
r-protolite
ray-client
ray-core
ray-data
ray-tune
refinenet
refinitiv-data
rerun-sdk
res-mlp-pytorch
rift-gpu
river-cpp
river-ingester
robotframework-browser
robustness
rpy2-arrow
rubicon-ml
rubin-env
rubin-env-nosysroot
rubinenv
rvc3python
s2geography
s2geometry
sagemaker
sagemaker-python-sdk
sahi
salesforce-merlion
saqc
sas7bdat-converter
schnetpack
scmorph
scrapbook
scrapbook-build
sctour
scvi-tools
sdgym
sdmetrics
segment-anything
segment-anything-hq
segmentation-models-pytorch
semantic-code-search
sentence-transformers
sentencepiece-python
sentencepiece-spm
sepal-ui
sharrow
shogun
shogun-cpp
sinabs
singlejar
siphon
skein
skl2onnx
skorch
sliderule
smif
snakebite
snakebite-py3
snntorch
snorkel
snowflake-connector-python
solaris
spacy-model-fr_dep_news_trf
spacy-transformers
sparkflow
spatialpandas
spikingjelly
sqlalchemy-bigquery
sqlalchemy-dremio
squirrel-core
squirrel-datasets-core
stable-baselines3
stac-geoparquet
stanza
stglib
streamlit
string2string
superset
syft-proto
tables-io
tables-io-full
tango-fairscale
tango-pytorch_lightning
tango-torch
tango-transformers
tango-wandb
tape_proteins
tensorboard
tensorboardx
tensordict
tensorflow
tensorflow-base
tensorflow-datasets
tensorflow-estimator
tensorflow-hub
tensorflow-metadata
tensorly-torch
text_extensions_for_pandas
textattack
tf2onnx
tf_object_detection
theseus-ai
tianshou
tiara
tiatoolbox
tiled-base
tiled-dataframe
tiled-sparse
timm
tip
tomviz
torch-complex
torch-ema
torch-fidelity
torch-hd
torch-optimizer
torch-runstats
torch-scatter
torch_em
torchani
torchcam
torchdata
torchdiffeq
torchgeo
torchinfo
torchio
torchlie
torchmd
torchmetrics
torchquad
torchsde
torchsnapshot
torchsparse
torchtext
torchtime
torchtnt
torchts
torchtuples
torchview
torchvision
towhee
tpot-torch
traffic
traja
transformer-smaller-training-vocab
transformers
treelite
triad
triton
tritonclient-grpc
ttach
tts
turbodbc
tweetynet
txtai
u8darts-all
u8darts-torch
ultralytics
unfccc_di_api
vaex-arrow
vaex-core
vak
vector-quantize-pytorch
vegafusion
verta
vit-pytorch
wandb
wannierberri
web3
webdataset
wetterdienst
whyqd
wilds
woodwork
wsinfer
xbbg
xformers
xilinx-runtime
xitorch
xlntpyarrow
xopt
yandexcloud
yggdrasil.pytorch
ylearn
zaber-motion
zcollection

@h-vetinari h-vetinari marked this pull request as ready for review September 7, 2023 23:11
@h-vetinari h-vetinari requested a review from a team as a code owner September 7, 2023 23:11
@h-vetinari
Copy link
Member Author

Following discussion with @isuruf in conda-forge/conda-forge-ci-setup-feedstock#274, the best path (IIUC) to unblock this is as follows:

  • bump the MACOSX_DEPLOYMENT_TARGET as part of this migration
  • add a __osx run-export to packages requiring 10.13 at runtime; done here for abseil.

With that, we ensure that CI doesn't break, and that the artefacts we produce have both correct metadata, and are kind to the solver.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants