Use strict #50

ocefpaf · 2019-03-07T14:18:27Z

We should use the strict channel option in conda-forge since we are recommending it as the preferred way to use the channel. Also, this will act like a good test to see if strict is ready to be the default in conda 4.7.

The rationale behind this is that strict is the only way to install the GIS stack safely at the moment.

There may be problem on Windows due to the lack of scipy. One would expect scipy to be installed from defaults but depending on the dependencies we may get an unsolvable env. (IMO an unsolvable env is not that bad, we should focus our efforts on finding and fixing them.)

See conda-forge/osmnx-feedstock#40 (comment)

conda-forge-linter · 2019-03-07T14:18:36Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

isuruf · 2019-03-07T15:20:04Z

-1 for this. Entire R stack on Windows relies on defaults' mingw packages

ocefpaf · 2019-03-07T15:21:48Z

-1 for this. Entire R stack on Windows relies on defaults' mingw packages

Yep. We know. This PR is to start a discussion on how to solve that. Vendor mingw? Never use strict on Windows? Start packaging everything?

ocefpaf · 2019-03-07T16:49:10Z

@isuruf also note that, if strict channel works as expected, we don't have to do anything b/c the packages that don't exist on conda-forge will be pulled from defaults anyway.

jjhelmus · 2019-03-07T17:35:16Z

The issue with using strict on Windows is not the missing scipy package but rather the numpy packages in the conda-forge cannot meet the requirements of the scipy package from defaults since they use a different blas implementation, OpenBLAS vs MKL.

isuruf · 2019-03-07T17:41:34Z

Thanks @jjhelmus for the info. conda-forge/blas-feedstock#15 would fix this.

ocefpaf · 2019-03-07T17:59:44Z

The issue with using strict on Windows is not the missing scipy package but rather the numpy packages in the conda-forge cannot meet the requirements of the scipy package from defaults since they use a different blas implementation, OpenBLAS vs MKL.

That is what I thought but it does boil down to a missing scipy (with conda-forge deps 😄).

BTW, I made a few tests trying to get a single package from the second channel using strict and virtually all of them, except the scipy/openblas windows problem, worked out just fine. One example is getting altair, whch does not exist in defaults, by adding conda-forge as the second channel and activating strict. The installation works as expected by getting everything from defaults but altair.

In light of that I do believe we should start using strict globally. Any issue we face are probably "fixable" by adding missing packages and or adding variants with the same deps as defaults.

CJ-Wright · 2019-03-07T18:18:06Z

My understanding that the issues with windows and strict are also somewhat broken without strict, as they cause large scale movement to defaults which might not be compatible. This just makes this brokenness more obvious. Is this correct?

ocefpaf · 2019-03-17T13:15:31Z

@isuruf would you be OK to not use strict on Windows until we fix our blas/numpy/scipy situation there but use it for Linux and macOS? It would really help those installing packages that have many dependencies, like gdal. (Which is currently broken for macOS thanks to mixing of packages.)

scopatz · 2019-03-18T16:33:26Z

I think strict for mac and linux at this point in time makes a lot of sense. If we have to hold off on windows that's OK.

ocefpaf · 2019-03-18T16:37:57Z

I think strict for mac and linux at this point in time makes a lot of sense. If we have to hold off on windows that's OK.

Actually, thanks to @isuruf I guess we'll be able to do it on Windows too. The problem we are hitting now is conda-forge/gdal-feedstock#270 (comment)

scopatz · 2019-03-18T17:47:50Z

Should this be merged then?

isuruf · 2019-03-18T17:50:14Z

On windows, we should wait for the BLAS migrator to finish. (It's only numpy that I have migrated manually)

ocefpaf · 2019-04-25T14:47:38Z

On windows, we should wait for the BLAS migrator to finish. (It's only numpy that I have migrated manually)

Thanks for that info @isuruf. I'll follow the BLAS migration closer and see if there are any PRs I can unblock.

PS: I foresee a few issues that will rise when we do this, for example, I could not start a qt build with strict enabled om Windows. We should identify these issues and fix them before conda 4.7 lands.

ocefpaf · 2019-06-01T11:48:47Z

An example where this is needed now: conda-forge/blas-feedstock#32

borupdaniel · 2019-07-11T16:43:31Z

Not sure if you're collecting examples where this is needed, but please feel free to delete this comment if not. Strict priority was required to successfully build/test with linux and qt5.6 in conda-forge/gpi-feedstock#5.

scopatz · 2019-07-17T19:19:39Z

recipe/run_conda_forge_build_setup_win.bat

@@ -8,6 +8,7 @@ set PYTHONUNBUFFERED=1
 conda.exe config --set show_channel_urls true
 conda.exe config --set auto_update_conda false
 conda.exe config --set add_pip_as_python_dependency false
+conda.exe config --set channel_priority strict


If we got rid of this for the moment, it would probably be OK

No, osx has issues with libcxx as well.

No, osx has issues with libcxx as well.

That is "fixed" with the choice of vendorizing the old libcxx and may be solved for good with conda/conda#8961.

That's a terrible fix as I mentioned in another issue that I will revert soon. Correct fix is AnacondaRecipes/aggregate#162

If we got rid of this for the moment, it would probably be OK

Can you elaborate why @scopatz? If we can list the foreseeable issues on Windows I can start working on them. (BTW, I've been using strict on Windows and Linux for a while now with more success than macOS, which should also be OK-ish now.)

minrk · 2019-07-22T13:00:35Z

See conda/conda#8960 for one possible proposal to relax strict priority in the event of exact version requirements not met by the top priority channel, and related issues for repo2docker investigating strict priority (for frozen envs, where the broken label is our problem) and openmpi-mpicxx where compilers coming from defaults and requiring libcxx==4.0.1 which is missing from conda-forge.

isuruf · 2019-07-22T13:03:09Z

libcxx==4.0.1 is broken and getting it from defaults or vendoring it into conda-forge is a terrible idea. Correct fix is at AnacondaRecipes/aggregate#162 if anybody cares to review.

ocefpaf · 2019-07-22T13:10:53Z

libcxx==4.0.1 is broken and getting it from defaults or vendoring it into conda-forge is a terrible idea. Correct fix is at AnacondaRecipes/aggregate#162 if anybody cares to review.

We are already doing the bad idea ;-p

I'll ping people on AR to see if we can get that merge. Thanks for the ping!

alimanfoo · 2019-09-03T08:09:10Z

Not sure it's relevant here, but wanted to mention I'm having problems using strict channel priority and trying to install packages from bioconda and conda-forge: bioconda/bioconda-recipes#17186

ocefpaf · 2019-09-03T13:02:12Z

Not sure it's relevant here, but wanted to mention I'm having problems using strict channel priority and trying to install packages from bioconda and conda-forge: bioconda/bioconda-recipes#17186

Any issues when using strict are "real" issues we can address. When not using strict you may get a solution but a broken env.

scopatz · 2019-09-04T16:19:43Z

This has conflicts now 😢

…da-forge-pinning 2019.05.31

…da-forge-pinning 2019.08.27

conda-forge-linter · 2019-09-04T16:50:49Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

license_file entry is missing, but is expected.

ocefpaf · 2019-09-04T17:33:54Z

This has conflicts now

Not anymore!

scopatz · 2019-09-04T21:08:01Z

Are the blocking issues with this resolved yet?

ocefpaf · 2019-09-04T21:46:58Z

They are not really issues but I would no merge this until more people are on board with the idea.

mwcraig

I'm on board as long as this is just affecting our build system. I still occasionally hit cases where strict is not what I want.

ocefpaf · 2019-09-05T13:40:06Z

I'm on board as long as this is just affecting our build system. I still occasionally hit cases where strict is not what I want.

It won't affect user options but it should make it easier to fix those cases when you cannot use strict that you just mentioned. Also, we need to do that before conda 5, when strict will be the default option.

ocefpaf requested a review from mariusvniekerk March 7, 2019 14:20

ocefpaf mentioned this pull request Mar 7, 2019

re-render feedstock, bump build number conda-forge/osmnx-feedstock#40

Merged

5 tasks

This was referenced Mar 8, 2019

Add gerrychain recipe conda-forge/staged-recipes#7871

Merged

strict-channel-priority does not fall back on lower priority channels conda/conda#8398

Closed

CJ-Wright mentioned this pull request Mar 11, 2019

xray-vision v0.0.9 conda-forge/xray-vision-feedstock#10

Merged

ocefpaf mentioned this pull request Mar 17, 2019

fiona v1.8.5 conda-forge/fiona-feedstock#122

Merged

ocefpaf mentioned this pull request Mar 17, 2019

Re render conda-forge/gdal-feedstock#270

Closed

ocefpaf force-pushed the use_strict branch from 8830df2 to 246c045 Compare May 31, 2019 23:34

ocefpaf closed this Jun 1, 2019

ocefpaf reopened this Jun 1, 2019

scopatz reviewed Jul 17, 2019

View reviewed changes

scopatz mentioned this pull request Jul 19, 2019

Interplay of conda-forge/broken channel and the channel ordering strict jupyterhub/repo2docker#731

Open

ocefpaf added 3 commits September 4, 2019 13:46

enforce strict channel in conda-forge

7d793f6

MNT: Re-rendered with conda-build 3.18.2, conda-smithy 3.3.4, and con…

24fa8f5

…da-forge-pinning 2019.05.31

MNT: Re-rendered with conda-build 3.18.9, conda-smithy 3.4.1, and con…

ff78ca8

…da-forge-pinning 2019.08.27

ocefpaf force-pushed the use_strict branch from e1ac2c9 to ff78ca8 Compare September 4, 2019 16:50

ocefpaf requested review from CJ-Wright, jakirkham, jjhelmus, msarahan, mwcraig, patricksnape and pelson as code owners September 4, 2019 16:50

mwcraig approved these changes Sep 4, 2019

View reviewed changes

isuruf mentioned this pull request Mar 10, 2020

Add strict priority for pypy builds #70

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use strict #50

Use strict #50

ocefpaf commented Mar 7, 2019 •

edited

Loading

conda-forge-linter commented Mar 7, 2019

isuruf commented Mar 7, 2019

ocefpaf commented Mar 7, 2019

ocefpaf commented Mar 7, 2019

jjhelmus commented Mar 7, 2019

isuruf commented Mar 7, 2019

ocefpaf commented Mar 7, 2019

CJ-Wright commented Mar 7, 2019

ocefpaf commented Mar 17, 2019

scopatz commented Mar 18, 2019

ocefpaf commented Mar 18, 2019

scopatz commented Mar 18, 2019

isuruf commented Mar 18, 2019

ocefpaf commented Apr 25, 2019

ocefpaf commented Jun 1, 2019

borupdaniel commented Jul 11, 2019 •

edited

Loading

scopatz Jul 17, 2019

isuruf Jul 17, 2019

ocefpaf Jul 22, 2019 •

edited

Loading

isuruf Jul 22, 2019

ocefpaf Jul 22, 2019

minrk commented Jul 22, 2019

isuruf commented Jul 22, 2019

ocefpaf commented Jul 22, 2019

alimanfoo commented Sep 3, 2019

ocefpaf commented Sep 3, 2019 •

edited

Loading

scopatz commented Sep 4, 2019

conda-forge-linter commented Sep 4, 2019

ocefpaf commented Sep 4, 2019

scopatz commented Sep 4, 2019

ocefpaf commented Sep 4, 2019

mwcraig left a comment

ocefpaf commented Sep 5, 2019

Use strict #50

Are you sure you want to change the base?

Use strict #50

Conversation

ocefpaf commented Mar 7, 2019 • edited Loading

conda-forge-linter commented Mar 7, 2019

isuruf commented Mar 7, 2019

ocefpaf commented Mar 7, 2019

ocefpaf commented Mar 7, 2019

jjhelmus commented Mar 7, 2019

isuruf commented Mar 7, 2019

ocefpaf commented Mar 7, 2019

CJ-Wright commented Mar 7, 2019

ocefpaf commented Mar 17, 2019

scopatz commented Mar 18, 2019

ocefpaf commented Mar 18, 2019

scopatz commented Mar 18, 2019

isuruf commented Mar 18, 2019

ocefpaf commented Apr 25, 2019

ocefpaf commented Jun 1, 2019

borupdaniel commented Jul 11, 2019 • edited Loading

scopatz Jul 17, 2019

Choose a reason for hiding this comment

isuruf Jul 17, 2019

Choose a reason for hiding this comment

ocefpaf Jul 22, 2019 • edited Loading

Choose a reason for hiding this comment

isuruf Jul 22, 2019

Choose a reason for hiding this comment

ocefpaf Jul 22, 2019

Choose a reason for hiding this comment

minrk commented Jul 22, 2019

isuruf commented Jul 22, 2019

ocefpaf commented Jul 22, 2019

alimanfoo commented Sep 3, 2019

ocefpaf commented Sep 3, 2019 • edited Loading

scopatz commented Sep 4, 2019

conda-forge-linter commented Sep 4, 2019

ocefpaf commented Sep 4, 2019

scopatz commented Sep 4, 2019

ocefpaf commented Sep 4, 2019

mwcraig left a comment

Choose a reason for hiding this comment

ocefpaf commented Sep 5, 2019

ocefpaf commented Mar 7, 2019 •

edited

Loading

borupdaniel commented Jul 11, 2019 •

edited

Loading

ocefpaf Jul 22, 2019 •

edited

Loading

ocefpaf commented Sep 3, 2019 •

edited

Loading