Detect dynamically AVX2 within npm

[lemire]

When can detect AVX2 dynamically within npm without user intervention, see for example...

https://github.com/SalvatorePreviti/roaring-node/blob/master/binding.gyp#L24-L29

[luizperes]

That sounds good! I am still to test these bindings in all three platforms (we got an error with windows #1) and will totally add the AVX2 checking when I get there!

Thanks @lemire

[luizperes]

Hi @lemire, I think I fixed this issue. However, because I do not have an AVX2 Linux machine, I would like to ask if you could download the branch issue-6 (https://github.com/luizperes/simdjson_nodejs/tree/issue-6) and test if it is still compiling for Linux.

It was needed because we were having a bug where the bindings were only working for Linux (please see #1 and #6). Oh! I used the lead that you gave me https://github.com/SalvatorePreviti/roaring-node/blob/master/binding.gyp! It helped a lot!

I would appreciate if you could give me an update on this issue. I will merge the code onto master when this issue has been fixed. One more thing, please make sure to run the benchmarks just to double-check if it is recognizing your system as AVX2 (I created an simdjson as an alias to JSON when the avx2 flag checking fails...)

Thanks!

[lemire]

@luizperes I have just emailed to your sfu address instructions for how to access (by ssh/linux) Linux servers with AVX2.

Will that help?

[lemire]

(If you don't get my email, please reach out to me at lemire@gmail.com.)

[lemire]

Idea: it is 10x harder to support a platform you cannot directly access.

[luizperes]

@lemire That helps a lot, I received your email, will work on that and will update you here! Thanks!

[luizperes]

Hi @lemire, it is working properly on Linux as well, thanks for the help. Also, if you kept my user would be good so I can test in the future. I realized that the Nan Google V8, might actually be worse than the previous Napi, so I will need to cherry pick some of the commits before I merge it. Thank you again!

[lemire]

Sure. I also have other Linux machines if you want further testing.

@TkTech : I can give you the same access if you need it.

[TkTech]

@lemire thanks for the offer, I luckily have a large collection of random boxes from my osdev days. Did you see the new POWER9 blackbird? First POWER9 I've seen around ~$1k for PCU+board.

@luizperes There is also the Intel Software Development Emulator for win/mac/linux hosts. Will be slow as the dickens, but if you're just doing development and testing it's perfect for emulating AVX2/AVX-512 for a single process.

[lemire]

Anyhow, I am happy to provide access to our machines for this type of research.

Evidently, one can also get access to fun machines on cloud providers like Packet.

(Yes. I have a POWER9 machine. Was way more expensive than that... but now that I have it...)

[luizperes]

Thanks for the info about the emulator @TkTech !

[luizperes]

Hi @lemire, I am detecting AVX2 now on the (merged) branch https://github.com/luizperes/simdjson_nodejs/tree/proper-fix-issue-6. I've tested so far on Mac OS and Linux and it looks good. I am waiting for @davaloper to update me on Windows.

Other than that, please let me know if this issue can be closed. Cheers

[lemire]

Upstream, simdjson will provide runtime dispatch. So this should not be necessary in the future.

cc @ioioioio

[luizperes]

I have updated to 0.2.0. Could you confirm that @lemire and @ioioioio?

[lemire]

@luizperes I think that your code still check for AVX2. You don't need that anymore.

• On 64-bit ARM, no check should be necessary. We don't support 32-bit ARM.
• We don't support 32-bit x86...
• We support x64 with a minimum of SSE 4.2 + CLMUL. This is what fairly old processors have (Westmere, Sandy Bridge...).

What you do (check for AVX2 and then bail out) should work fine if you are not on ARM, but you could weaken quite a bit the checks.

[luizperes]

So, would it suffice if we said we don't support 32-bit at all?

I will likely use the same directives simdjon is using then, good to know, thanks!

[lemire]

Right... and to be prudent check for sse 4.2 and clmul on x64.

[luizperes]

I just realized I did not update that (with #23). Hi @lemire, is it correct that we don't need to use the flags -mavx2 anymore?

[lemire]

Please be advised that we no longer require SSE 4.2. The upcoming version of simdjson will run everywhere.

[lemire]

@luizperes You should update to version 0.3 in a few days. You will just compile without any flag. It will run everywhere.

[luizperes]

PR #27 fixes this issue