Update to conda 22.9 with tests for boa compatibility #361

hmaarrfk · 2022-10-08T02:04:36Z

Explicit test for conda/mamba/boa compatibility for Mambaforge
Reduced test matrix for Mambaforge-pypy3 on emulated architectures (aarch64 and ppc64le) -- ensures CI pass in time?
Updated micromamba on osx

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

hmaarrfk · 2022-10-08T03:03:10Z

I think anaconda is just blocking us from downloading the package index:

23:02 $ wc -l /home/mark/mambaforge/pkgs/cache/a08d06e9.json
2311247 /home/mark/mambaforge/pkgs/cache/a08d06e9.json
(base) ✔ ~/git/feedstocks/miniforge [fixup_2209|✔]
23:02 $ wc -l /home/mark/mambaforge/pkgs/cache/568c2e3d.json
7067307 /home/mark/mambaforge/pkgs/cache/568c2e3d.json

about 7 million lines, not 3000.

hmaarrfk · 2022-10-14T04:02:47Z

@wolfv are you able to help look into why when I try to install boa, it seems to be unable to find packages.

For reference, this is a similar PR that simply removes to boa install tests.

isuruf · 2022-10-14T04:28:25Z

The package cache is the one with only the dependencies of the packages in the installer (in this case python, conda and mamba) so that the packages inside the installer can be installed.
These caches should have an old timestamp so that when new packages are installed, this cache is invalidated. Not sure why that is.

isuruf · 2022-10-14T04:35:59Z

You need conda/constructor@85bca74

isuruf · 2022-10-14T16:59:56Z

scripts/build.sh

@@ -12,16 +12,25 @@ cd "${CONSTRUCT_ROOT}"
 # Constructor should be latest for non-native building
 # See https://github.com/conda/constructor
 echo "***** Install constructor *****"
-conda install -y "constructor>=3.3.1" jinja2 curl libarchive -c conda-forge --override-channels
+conda install --yes jinja2 curl libarchive --channel conda-forge --override-channels


You can install constructor here and override it with pip later on so that you don't have to manually install the dependencies of constructor.

I'm personally always confused about how that plays out. There are also some new nsis dependencies.

In either case the errors seem to be unrelated to the removal of this

How about we carry that patch (the one line with used_repodata.pop("_mod", None)) in constructor-feedstock?

ok, let me try to fork and apply that patch only, and see what happens.

I guess i'm ok with debugging constructor, but this isn't any worse than before.

Can we move forward with: #362

What's the point of releasing something that doesn't work?

It works just as well as the other things used to.

The problem is just as well hidden as it always has been.

wolfv · 2022-10-16T19:18:48Z

@hmaarrfk is there still something that I could do to help here?

hmaarrfk · 2022-10-16T19:29:43Z

If you can help us understand why the following index cache isn't being invalidated that would be great:

cat /root/miniforge/pkgs/cache/09cdf8bf.json
{"_mod": "Mon, 07 Jan 2019 15:22:15 GMT", "_url": "https://conda.anaconda.org/conda-forge/noarch",
  "_etag": "W/\"ded6fb8204936e8acedca121bd4b42e3\"",
  "_cache_control": "public, max-age=1200",

I'm not sure what "max-age" refers to.

hmaarrfk · 2022-10-17T11:55:53Z

@wolfv My latest modification shows that in fact, this seems to be a peculiarity that is observed with mamba and not conda.

It seems at least that mamba is holding on to the cache for too long, or conda ignores it entirely. I'm not too sure.

isuruf · 2022-10-22T23:20:57Z

Looks like mamba doesn't look at _mod timestamp in the json and instead looks at the timestamp in the file attributes.
See https://github.com/mamba-org/mamba/blob/e3ab841b2c1349afb75af156eee88fc3a1bc125e/libmamba/src/core/subdirdata.cpp#L269

hmaarrfk · 2022-10-22T23:36:46Z

wow. excellent work!

isuruf · 2022-10-22T23:36:57Z

See conda/constructor#579

scripts/test.sh

hmaarrfk · 2022-10-22T23:42:48Z

I updated my branch. lets see.

isuruf · 2022-10-23T00:33:39Z

scripts/build.sh

 fi
-# pip install git+git://github.com/conda/constructor@3.3.1#egg=constructor --force --no-deps
+
+constructor_rev=3.3.1_patch_for_cache


If the tests pass, can you change this to a commit instead of a branch?

I was going to patch conda forges version with the two patches.

hmaarrfk · 2022-10-23T16:23:20Z

We might have to trim down some of the test matrix.

hmaarrfk · 2022-10-23T17:40:02Z

Thank you for your help Isuru!

isuruf · 2022-10-23T17:42:03Z

scripts/build.sh

+conda install --yes \
+    --channel conda-forge --override-channels \
+    jinja2 curl libarchive \
+    "constructor>=3.3.1=*_1"


This means version>=3.3.1 and build=*_1. Is that what you mean?

yeah. do i need == in the second one?

Actually, i mean "3.3.2" and "build=0" would be ok.

I'm not sure if syntax exists for that.

Yeah, there's no syntax for that. Let's keep it constructor>=3.3.1 and hope for the best.

ok, i hope the smaller test matrix passes now for the ppc64le. it has been quite slow.

isuruf · 2022-10-23T19:25:51Z

.github/workflows/ci.yml

@@ -132,6 +132,8 @@ jobs:
            DOCKERIMAGE: condaforge/linux-anvil-aarch64
            MINIFORGE_NAME: "Mambaforge-pypy3"
            OS_NAME: "Linux"
+            # Reduce the test matrix because the builds timeouts on emulated architectures
+            TEST_IMAGE_NAMES: "ubuntu:22.04 ubuntu:16.04 centos:7"


Let's keep only one. The one where the timeout happened barely ran two tests. https://pipelines.actions.githubusercontent.com/serviceHosts/e400fe88-afca-422a-abd9-23e00445dee4/_apis/pipelines/1/runs/6160/signedlogcontent/16?urlExpires=2022-10-23T19%3A25%3A43.9598492Z&urlSigningMethod=HMACV1&urlSignature=WWLz9fSOwtV%2F6x%2BVpSGa47kLLBUt7FOlRi7Z%2FOxGBaM%3D

K. one has a modern ubuntu, the other has centos 7. I feel like that is a good spread for these.

hmaarrfk · 2022-10-24T11:26:42Z

Seems like the thorough check is just a little too much on these emulated architectures.

hmaarrfk · 2022-10-26T00:15:19Z

Maybe with your fix i can remove the heavy handed skipping of the tests.

hmaarrfk · 2022-10-26T02:19:43Z

Thanks again!

jakirkham · 2022-10-26T15:56:21Z

Thank you both! 🙏

Are we comfortable publishing this release as final? Or what still needs to be done to get to that point?

hmaarrfk · 2022-10-26T15:58:08Z

Honestly, i'm lost in the number of assets generated. So if somebody can confirm that 74 i the correct count, we can publish.

isuruf · 2022-10-26T16:02:38Z

Yes, 74 is correct.

jakirkham · 2022-10-27T02:37:27Z

Noticed 2 releases showed up:

https://github.com/conda-forge/miniforge/releases/tag/22.9.0-0 (marked pre-release)
https://github.com/conda-forge/miniforge/releases/tag/4.14.0-2 (marked latest)

Which one should we be using?

hmaarrfk · 2022-10-27T11:57:28Z

I guess 22.9.0. They both point to the same sha. I think the version was always meant to follow the conda version, and as such, should be 22.9.0

jakirkham · 2022-10-27T22:06:14Z

Sounds good. Updating the Docker images with PR ( conda-forge/docker-images#224 )

isuruf · 2022-10-27T22:07:11Z

You can use either. Miniforge3-4.14.0-2 gives you conda=4.14.0 and Miniforge3-22.9.0-0 gives you conda=22.9.0.

nyngwang · 2023-04-01T14:38:08Z

@isuruf Out of curious why the version "jumps" from 4.x to 22.x on the conda side. (I thought I was a time traveler when I saw the version jumped like that)

Appreciate your reading and I'm sorry for commenting on an old issue.

hmaarrfk · 2023-04-01T14:42:01Z

Jus following conda
https://github.com/conda/conda/releases/tag/22.9.0

hmaarrfk force-pushed the fixup_2209 branch from 7636c73 to 7ff5279 Compare October 8, 2022 19:21

hmaarrfk changed the title ~~Fixup 2209~~ Update to conda 22.9 with tests for boa compatibility Oct 8, 2022

hmaarrfk mentioned this pull request Oct 8, 2022

Latest anvil image installs incompatible conda and mamba versions conda-forge/docker-images#223

Closed

hmaarrfk closed this Oct 14, 2022

hmaarrfk reopened this Oct 14, 2022

hmaarrfk force-pushed the fixup_2209 branch from d9bef48 to 456ec19 Compare October 14, 2022 15:54

isuruf reviewed Oct 14, 2022

View reviewed changes

hmaarrfk commented Oct 22, 2022

View reviewed changes

scripts/test.sh Outdated Show resolved Hide resolved

isuruf reviewed Oct 23, 2022

View reviewed changes

hmaarrfk mentioned this pull request Oct 23, 2022

Add nearly upstreamed patches to help with cache conda-forge/constructor-feedstock#62

Merged

5 tasks

hmaarrfk force-pushed the fixup_2209 branch from fa48cfc to 34dc255 Compare October 23, 2022 17:38

hmaarrfk marked this pull request as ready for review October 23, 2022 17:38

hmaarrfk requested a review from a team as a code owner October 23, 2022 17:38

isuruf reviewed Oct 23, 2022

View reviewed changes

hmaarrfk force-pushed the fixup_2209 branch from 34dc255 to 29dabc9 Compare October 23, 2022 17:47

isuruf reviewed Oct 23, 2022

View reviewed changes

hmaarrfk force-pushed the fixup_2209 branch from 29dabc9 to bb4d4b8 Compare October 23, 2022 19:32

hmaarrfk added 4 commits October 25, 2022 07:55

Update micromamba for osx

c5787db

Build with newer version of constructor

e64fefa

Update Conda to 22.9

0e61b33

Explicitely test for conda/mamba/boa compatibility

0950521

hmaarrfk force-pushed the fixup_2209 branch from 957ec21 to 642f76f Compare October 25, 2022 11:56

Reduce test matrix for emulated architecture

bc8b082

hmaarrfk force-pushed the fixup_2209 branch from 650c990 to bc8b082 Compare October 25, 2022 21:23

export TEST_IMAGE_NAMES

8de9ab1

Remove extra hack

ab5e888

isuruf merged commit 2e25aec into conda-forge:main Oct 26, 2022

jakirkham mentioned this pull request Oct 26, 2022

Update to 22.9.0 #353

Closed

5 tasks

jakirkham mentioned this pull request Oct 27, 2022

Update to Mambaforge 22.9.0-0 conda-forge/docker-images#224

Merged

Update to conda 22.9 with tests for boa compatibility #361

Update to conda 22.9 with tests for boa compatibility #361

Conversation

hmaarrfk commented Oct 8, 2022 • edited Loading

hmaarrfk commented Oct 8, 2022

hmaarrfk commented Oct 14, 2022

isuruf commented Oct 14, 2022

isuruf commented Oct 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfv commented Oct 16, 2022

hmaarrfk commented Oct 16, 2022

hmaarrfk commented Oct 17, 2022

isuruf commented Oct 22, 2022

hmaarrfk commented Oct 22, 2022

isuruf commented Oct 22, 2022

hmaarrfk commented Oct 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hmaarrfk commented Oct 23, 2022

hmaarrfk commented Oct 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hmaarrfk commented Oct 24, 2022

hmaarrfk commented Oct 26, 2022

hmaarrfk commented Oct 26, 2022

jakirkham commented Oct 26, 2022

hmaarrfk commented Oct 26, 2022

isuruf commented Oct 26, 2022

jakirkham commented Oct 27, 2022

hmaarrfk commented Oct 27, 2022 • edited Loading

jakirkham commented Oct 27, 2022

isuruf commented Oct 27, 2022

nyngwang commented Apr 1, 2023 • edited Loading

hmaarrfk commented Apr 1, 2023

hmaarrfk commented Oct 8, 2022 •

edited

Loading

hmaarrfk commented Oct 27, 2022 •

edited

Loading

nyngwang commented Apr 1, 2023 •

edited

Loading