implicit layers with neuralODEs ?

[yewalenikhil65]

Came across application of implicit function theorem to neural-nets
General idea is "need only jacobian at fixed point "

http://implicit-layers-tutorial.org/introduction/
https://github.com/locuslab/deq

Might be useful thing for DiffEqFlux ?

[ChrisRackauckas]

Already exists. Just use SteadyStateProblem.

[yewalenikhil65]

That would require data at steady-state or require availibility of ODEProblem ,right ?

Is the whole implicit-layers thing different than the way we normally approach in training neuralODEs as per DiffEqFlux docs. I am not much clear whether its better or not compared to our normal approach

[ChrisRackauckas]

It's literally the same. DEX is just a steady state problem, or a nonlinear solve 0 = f(u) which is equivalent to finding u' = 0 or a steady state of the ODE u' = f(u). The adjoint on it is https://math.mit.edu/~stevenj/18.336/adjoint.pdf which is exactly the same as the DEX paper, just written 15 years earlier as equation (3).

[ChrisRackauckas]

We made a whole separate package around this: https://github.com/SciML/DeepEquilibriumNetworks.jl