Figure out better way for dealing with CPU arches

[h-vetinari]

While debugging #22, I found out that the upstream build-refactor that I tried to follow in #17 changed things so that upstream always compiles for both AVX2 & default profiles, and then switches at runtime using:
https://github.com/facebookresearch/faiss/blob/v1.6.5/faiss/python/loader.py#L27-L39

Of course we could mimick that, but given the infrastructure added in conda/conda#9930 by @isuruf, I'm thinking we could do better. Not sure if this is ready for primetime yet -- as in, does CF infra support this already?

I see the following todos (restricted to this feedstock):

• Figure out if we want to build for AVX2, resp. other profiles
• If yes, decide trade-off between packaging size impact for packaging several (probably negligible) and multiplication of CI jobs
• Patch the upstream loader depending on outcome
  • If we keep the upstream approach, add windows detection (how?) and upstream the patch

CC @beckermr @mbargull @jakirkham

[isuruf]

I'd suggest going with upstream and shipping both. archspec based packages are not ready yet.

[h-vetinari]

Thanks @isuruf for that update! Is there any place to track the archspec-based approach so I can subscribe (or possibly help) and test/apply it once done?

[h-vetinari]

Fortunately, there's now an issue to track progress on this: conda-forge/conda-forge.github.io#1261

[ngam]

I'm thinking we could do better

In what way? Size and unnecessary complexity?

I am often tempted to think about this in general like the noarch situation --- one package for all, but maybe that's the naive way to think about this 😅

[h-vetinari]

I'm thinking we could do better

In what way? Size and unnecessary complexity?

Maximize performance & minimize binary size, by building different package variants (e.g. for the x86-v2 - x86-v4 levels) and having each user automatically download the highest-supported binary according to their CPU, via the __archspec virtual package.

one package for all, but maybe that's the naive way to think about this 😅

Doing that doubles (or triples, if building for AVX512 as well) build time & binary size for no other reason other than that we cannot (currently) do better infrastructurally, and don't want to completely forgo the performance benefits that newer CPU arches have via-à-vis SSE4.

[h-vetinari]

Now that the feedstock has been revived, this can be tackled. Should be easy to do (c.f. the docs), but need to figure out how to do this on windows. We'll also need to patch the upstream builds a bit, but we were already doing that anyway (and there's a bunch of patches to drop as well...)