Meta Issue for proper Enzyme Integration into Lux

[avik-pal]

• CPU Support: We have tests covering several models.
  • PolyAlg selecting the correct broadcast mechanism for fast_activation!! fails Segfault with Assertion 'isa<LoadInst>(orig) == isa<LoadInst>(val)' failed EnzymeAD/Enzyme.jl#1408 (Fixed upstream needs to be verified here)
  • JVP fails for a simple Dense Layer Error for JVP by Enzyme #628 (Needs Enzyme runtimeActivity enabled)
  • Dense works after Remove Polyester (generates incorrect LLVM) LuxLib.jl#68
  • Simple Chains integration SimpleChains integration doesn't work with Enzyme #644
  • Broadcasting related (runtime activity) Simple MLP requires Enzyme runtimeActivity #647
  • ComponentArrays Enzyme Assertion with ComponentArrays EnzymeAD/Enzyme.jl#1447
    • Partially work -- see tests
  • Recurrent Layers are broken
• CUDA Support
  • cuBLAS: getting a minimal Lux example working (with CUDA) EnzymeAD/Enzyme.jl#1392
  • cuBLASLt: For the matmul kernels
  • Minimal CUDA version for Lux Dense Layer EnzymeAD/Enzyme.jl#1442
  • Enzyme rules for cuBLASLt LuxLib.jl#148
• AMDGPU Support
  • rocBLAS
• Performance
  • Slow Broadcasting (compared to Zygote) EnzymeAD/Enzyme.jl#1434

Current Status: All (native lux) CPU Models should work. If not, that is a bug and please open issues.

[wsmoses]

@avik-pal the fast_activation!! issue is closed, let me know if anything else is blocking there.

[avik-pal]

Yes, I have been playing around with it, so I will open any issue upstream as and when they show up.

[avik-pal]

After LuxDL/LuxLib.jl#69 and #640 are merged, larger models from Boltz.jl seem to work seamlessly 🎉