Illegal Instruction crash in sodium_malloc on Intel CPUs in v1.0.19 (from prebuilt binaries)

[enclave-alistair]

Hi,

Since upgrading our dependency on libsodium to v1.0.19 we've seen illegal instruction errors reported on multiple Intel CPUs when running linux, one on the Jasper Lake (Tremont) micro-architecture, and one on the Apollo Lake (Goldmont) micro-architecture (so far).

The failing code path appears to be sodium_malloc (please forgive the image, had to capture an image on a customer call):

Here's a procinfo output for two of the failing devices, one CPU each:

Intel N5095:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 156
model name      : Intel(R) Celeron(R) N5095 @ 2.00GHz
stepping        : 0
microcode       : 0x24000024
cpu MHz         : 2000.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 27
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault epb cat_l2 cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms rdt_a rdseed smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req umip waitpkg gfni rdpid movdiri movdir64b md_clear flush_l1d arch_capabilities
vmx flags       : vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid ple shadow_vmcs ept_mode_based_exec tsc_scaling usr_wait_pause
bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs srbds mmio_stale_data
bogomips        : 3993.60
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

Intel J3455

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 92
model name      : Intel(R) Celeron(R) CPU J3455 @ 1.50GHz
stepping        : 9
microcode       : 0x3c
cpu MHz         : 1501.000
cache size      : 1024 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 21
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch intel_pt ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms mpx rdseed smap clflushopt sha_ni xsaveopt xsavec xgetbv1 dtherm ida arat pln pts md_clear arch_capabilities
bugs            : spectre_v1 spectre_v2
bogomips        : 2995.02
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

Now, I can see that in b5bd5b8 @jedisct1 configured the zig builds to target sandybridge instead of baseline. I'm going to assume that this may be the cause of the incompatibility with these CPUs.

Two questions then:

• Why would targeting the relatively old sandybridge architecture break newer architectures?
• If I manually build a version targeting baseline instead (happy to do so), what do I lose? Is there an issue fixed by building for that target that we would lose? I can't find any additional context for that commit.

Thanks

[enclave-alistair]

Oh, I'm guessing that this line from the release notes should probably have been a tip-off:

However, on x86_64, targets are expected to support at least the AVX instruction set.

[enclave-alistair]

OK, so these two CPUs don't support AVX, hence the error. If I take baseline, what do I lose? Is it only performance?

[jedisct1]

Hi!

The AVX instruction set has been around for a very long time (it was introduced circa 2011), so I blindly assumed that the .NET framework didn't even support CPUs that didn't have it.

The legacy build system enables support for AVX, AVX2, etc. only for specific files where these instructions are required.

But in Zig builds, everything is currently using the same compilation flags. Sandy bridge looked like a reasonable baseline target for .NET. But apparently not.

The best way to solve this would be to port the old logic to the zig build system.

If you take baseline (or better, tremont or goldmont), you will lose performance, and AES-GCM. Everything else will work the same.

[enclave-alistair]

Thanks; is AES-GCM depending on AVX and AVX2 a part of the 19 version as part of the re-write? Or has AES-GCM required AVX for a while?

I may have to bundle both native binaries, and load the one I want at runtime based on detecting AVX availability.

[jedisct1]

baseline doesn't include the AESNI extensions. With the tremont or goldmont targets, AES-GCM should work. Not AEGIS, though. Without AVX, AEGIS will be very slow, though.

[enclave-alistair]

Oh, also worth noting that these processors are quite new (2021-ish), but they are processors for embedded hardware and micro PCs that may just have less features.

[jedisct1]

Yes, my bad, I didn't consider that new processors couldn't include AVX.

I need to find some time to change the build files to only apply non-baseline flags to relevant files.

[enclave-alistair]

Hi again @jedisct1; I've got past the illegal instruction error by creating a build for the goldmont CPU, however getting past that I now realise that as of 1.0.19 these CPUs now can't use AESGCM because 4388ef3 added a requirement for AVX for AESGCM to be used.

In 1.0.18 crypto_aead_aes256gcm_is_available() returned true on these architectures.

These CPUs are typically lower-powered IoT devices, so hardware-accelerating AES is pretty important for us; what's the impact of removing the check for avx on AESGCM on our build? Am I correct in that the rewrite of AES-GCM adds the hard dependency on AVX?

[jedisct1]

Only performance impact on CPUs that could have used AVX. You can remove the check in AEGIS implementations, too (and if you do, update the #pragma clang attribute push(__attribute__((target("aes,avx"))), apply_to = function pragma as well).

[enclave-alistair]

OK, thanks, I'll try that locally; we only use AESGCM with a fallback to ChaChaPoly, so I'll leave AEGIS alone for now.

[enclave-alistair]

OK, that works. I'm now bundling two builds of libsodium with my .NET app, libsodium.so and libsodium-noavx.so, so we get better performance on CPUs that support AVX, but have the fallback.

For future arrivals at this discussion, here's the .NET code that lets me switch out the libsodium binary, called from Main:

protected static void InitialiseLibsodiumSelection()
{
   static bool PlatformRequiresNoAvxFallback()
   {
       // On x64 linux, on processors that do not support the AVX2 instruction set,
       // we need to use our alternate version of libsodium.
       return OperatingSystem.IsLinux() &&
              RuntimeInformation.ProcessArchitecture == Architecture.X64 &&
              !Avx2.IsSupported;
   }

   static IntPtr ResolveFallbackLibsodium(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
   {
       if (libraryName == "libsodium")
       {
           return NativeLibrary.Load("libsodium-noavx.so", assembly, searchPath);
       }

       // fall back to the default resolver.
       return IntPtr.Zero;
   }

   if (PlatformRequiresNoAvxFallback())
   {
       Console.WriteLine("No AVX2 support detected, falling back to libsodium version with no AVX requirement.");

       NativeLibrary.SetDllImportResolver(Assembly.GetExecutingAssembly(), ResolveFallbackLibsodium);
   }
}

[lambdaupb]

Hit this Problem as well, twice in fact.

Missing AVX can easily happen on VMs where the selected guest CPU feature-set is conservative for live migration (often the default).

Or on long term support platforms such as the mentioned Goldmont, Tremont Generation (Jasper Lake CPUs), these are Products launched in 2021 and everywhere in embedded PCs, NAS systems, cheap computers etc.
Some of those are very long lived devices.

This makes shipping software with the .net library very difficult.

Maybe there could be a -compat version of the libsodium nuget package or just include a fallback .so and switch in the .net code as mentioned above.

The stable lightweight implementation of libsodium makes it attractive for embedded use, requiring AVX turns out to make this quite a hassle.

[jedisct1]

I'm working on this. Essentially enabling CPU-specific flags only on required files, as in the old build system.

[jedisct1]

9b369db should address this.

[jedisct1]

Ok, it took much more work than anticipated, but that should finally be fixed, including in stable.

[lambdaupb]

Thank you for spending your time on this, I appreciate it a lot and I really think it was worthwhile to do, saving trouble for users.

[jedisct1]

Version 1.0.19.1 is now available on NuGet!