Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimization Issue in FFJORD with AutoZygote #635

Closed
prbzrg opened this issue Oct 11, 2021 · 11 comments · Fixed by #642
Closed

Optimization Issue in FFJORD with AutoZygote #635

prbzrg opened this issue Oct 11, 2021 · 11 comments · Fixed by #642

Comments

@prbzrg
Copy link
Member

prbzrg commented Oct 11, 2021 •
edited
Loading

When I run this code:

using DiffEqFlux, DifferentialEquations, GalacticOptim, Distributions

nn = Chain(
    Dense(1, 3, tanh),
    Dense(3, 1, tanh),
) |> f32
tspan = (0.0f0, 10.0f0)
ffjord_mdl = FFJORD(nn, tspan, Tsit5())

data_dist = Normal(6.0f0, 0.7f0)
train_data = rand(data_dist, 1, 100)

function loss(θ)
    logpx, λ₁, λ₂ = ffjord_mdl(train_data, θ)
    -mean(logpx)
end

function cb(p, l)
    @show l
    false
end

adtype = GalacticOptim.AutoZygote()
res1 = DiffEqFlux.sciml_train(loss, ffjord_mdl.p, ADAM(0.1), adtype; cb, maxiters=25)

I get this output:

julia> res1 = DiffEqFlux.sciml_train(loss, ffjord_mdl.p, ADAM(0.1), adtype; cb, maxiters=25)
l = 25.784435f0
l = 25.782549f0
l = 25.78082f0
l = 25.778957f0
l = 25.772545f0
l = 25.766665f0
l = 25.781792f0
l = 25.783306f0
l = 25.776882f0
l = 25.777037f0
l = 25.779358f0
l = 25.779919f0
l = 25.77038f0
l = 25.77084f0
l = 25.773977f0
l = 25.783195f0
l = 25.77303f0
l = 25.779255f0
l = 25.77898f0
l = 25.776768f0
l = 25.783546f0
l = 25.75965f0
l = 25.779638f0
l = 25.781958f0
l = 25.770374f0
l = 25.75965f0
u: 10-element Vector{Float32}:
 -0.40228903
 -1.0076219
  0.97539043
  0.0
  0.0
  0.0
 -0.44423598
  0.16586965
 -0.16390069
  0.0

julia> ffjord_mdl.p
10-element Vector{Float32}:
 -0.40228903
 -1.0076219
  0.97539043
  0.0
  0.0
  0.0
 -0.44423598
  0.16586965
 -0.16390069
  0.0

In this environment:

(ffjord-report-issues) pkg> status
      Status `C:\Users\Hossein Pourbozorg\Code Projects\Mine\ffjord-report-issues\Project.toml`
  [aae7a2af] DiffEqFlux v1.43.0
  [0c46a032] DifferentialEquations v6.19.0
  [31c24e10] Distributions v0.25.18
  [a75be94c] GalacticOptim v2.0.3

(ffjord-report-issues) pkg> status --manifest
      Status `C:\Users\Hossein Pourbozorg\Code Projects\Mine\ffjord-report-issues\Manifest.toml`
  [621f4979] AbstractFFTs v1.0.1
  [1520ce14] AbstractTrees v0.3.4
  [79e6a3ab] Adapt v3.3.1
  [ec485272] ArnoldiMethod v0.1.0
  [4fba245c] ArrayInterface v3.1.33
  [4c555306] ArrayLayouts v0.7.7
  [13072b0f] AxisAlgorithms v1.0.1
  [ab4f0b2a] BFloat16s v0.1.0
  [aae01518] BandedMatrices v0.16.11
  [e2ed5e7c] Bijections v0.1.3
  [62783981] BitTwiddlingConvenienceFunctions v0.1.0
  [764a87c0] BoundaryValueDiffEq v2.7.1
  [fa961155] CEnum v0.4.1
  [2a0fbf3d] CPUSummary v0.1.5
  [00ebfdb7] CSTParser v3.2.4
  [052768ef] CUDA v3.3.6
  [7057c7e9] Cassette v0.3.9
  [082447d4] ChainRules v1.11.6
  [d360d2e6] ChainRulesCore v1.7.2
  [fb6a15b2] CloseOpenIntervals v0.1.2
  [944b1d66] CodecZlib v0.7.0
  [3da002f7] ColorTypes v0.11.0
  [5ae59095] Colors v0.12.8
  [861a8166] Combinatorics v1.0.2
  [a80b9123] CommonMark v0.8.3
  [38540f10] CommonSolve v0.2.0
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v3.39.0
  [b152e2b5] CompositeTypes v0.1.2
  [88cd18e8] ConsoleProgressMonitor v0.1.2
  [187b0558] ConstructionBase v1.3.0
  [a8cc5b0e] Crayons v4.0.4
  [754358af] DEDataArrays v0.2.0
  [9a962f9c] DataAPI v1.9.0
  [82cc6244] DataInterpolations v3.6.1
  [864edb3b] DataStructures v0.18.10
  [e2d170a0] DataValueInterfaces v1.0.0
  [bcd4f6db] DelayDiffEq v5.31.1
  [2b5f629d] DiffEqBase v6.73.2
  [459566f4] DiffEqCallbacks v2.17.0
  [5a0ffddc] DiffEqFinancial v2.4.0
  [aae7a2af] DiffEqFlux v1.43.0
  [c894b116] DiffEqJump v7.3.1
  [77a26b50] DiffEqNoiseProcess v5.9.0
  [055956cb] DiffEqPhysics v3.9.0
  [41bf760c] DiffEqSensitivity v6.59.0
  [163ba53b] DiffResults v1.0.3
  [b552c78f] DiffRules v1.3.1
  [0c46a032] DifferentialEquations v6.19.0
  [c619ae07] DimensionalPlotRecipes v1.2.0
  [b4f34e82] Distances v0.10.4
  [31c24e10] Distributions v0.25.18
  [ced4e74d] DistributionsAD v0.6.31
  [ffbed154] DocStringExtensions v0.8.5
  [5b8099bc] DomainSets v0.5.9
  [7c1d4256] DynamicPolynomials v0.3.21
  [da5c29d0] EllipsisNotation v1.1.0
  [7da242da] Enzyme v0.7.0
  [d4d017d3] ExponentialUtilities v1.10.0
  [e2ba6199] ExprTools v0.1.6
  [7a1cc6ca] FFTW v1.4.5
  [7034ab61] FastBroadcast v0.1.8
  [9aa1b823] FastClosures v0.3.2
  [1a297f60] FillArrays v0.12.6
  [6a86dc24] FiniteDiff v2.8.1
  [53c48c17] FixedPointNumbers v0.8.4
  [587475ba] Flux v0.12.7
  [59287772] Formatting v0.4.2
  [f6369f11] ForwardDiff v0.10.20
  [069b7b12] FunctionWrappers v1.1.2
  [d9f16b24] Functors v0.2.6
  [0c68f7d7] GPUArrays v8.1.2
  [61eb1bfa] GPUCompiler v0.12.9
  [a75be94c] GalacticOptim v2.0.3
  [af5da776] GlobalSensitivity v1.2.1
  [3e5b6fbb] HostCPUFeatures v0.1.4
  [0e44f5e4] Hwloc v2.0.0
  [7869d1d1] IRTools v0.4.3
  [615f187c] IfElse v0.1.0
  [d25df0c9] Inflate v0.1.2
  [a98d9a8b] Interpolations v0.13.4
  [8197267c] IntervalSets v0.5.3
  [92d709cd] IrrationalConstants v0.1.0
  [42fd0dbc] IterativeSolvers v0.9.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.2
  [98e50ef6] JuliaFormatter v0.17.1
  [e5e0dc1b] Juno v0.8.4
  [5ab0869b] KernelDensity v0.6.3
  [929cbde3] LLVM v4.6.0
  [b964fa9f] LaTeXStrings v1.2.1
  [2ee39098] LabelledArrays v1.6.5
  [23fbe1c1] Latexify v0.15.6
  [a5e1c1ea] LatinHypercubeSampling v1.8.0
  [73f95e8e] LatticeRules v0.0.1
  [10f19ff3] LayoutPointers v0.1.3
  [1d6d02ad] LeftChildRightSiblingTrees v0.1.2
  [093fc24a] LightGraphs v1.3.5
  [d3d80556] LineSearches v7.1.1
  [2ab3a3ac] LogExpFunctions v0.3.3
  [e6f89c97] LoggingExtras v0.4.7
  [bdcacae8] LoopVectorization v0.12.84
  [1914dd2f] MacroTools v0.5.8
  [d125e4d3] ManualMemory v0.1.6
  [e89f7d12] Media v0.5.0
  [e1d29d7a] Missings v1.0.2
  [961ee093] ModelingToolkit v6.6.0
  [46d2c3a1] MuladdMacro v0.2.2
  [f9640e96] MultiScaleArrays v1.8.1
  [102ac46a] MultivariatePolynomials v0.3.18
  [d8a4904e] MutableArithmetics v0.2.20
  [d41bc354] NLSolversBase v7.8.1
  [2774e3e8] NLsolve v4.5.1
  [872c559c] NNlib v0.7.29
  [a00861dc] NNlibCUDA v0.1.9
  [77ba4419] NaNMath v0.3.5
  [8913a72c] NonlinearSolve v0.3.11
  [d8793406] ObjectFile v0.3.7
  [6fe1bfb0] OffsetArrays v1.10.7
  [429524aa] Optim v1.4.1
  [bac558e1] OrderedCollections v1.4.1
  [1dea7af3] OrdinaryDiffEq v5.64.1
  [90014a1f] PDMats v0.11.1
  [65888b18] ParameterizedFunctions v5.12.1
  [d96e819e] Parameters v0.12.3
  [69de0a69] Parsers v2.0.5
  [e409e4f3] PoissonRandom v0.4.0
  [f517fe37] Polyester v0.5.3
  [1d0040c9] PolyesterWeave v0.1.0
  [85a6dd25] PositiveFactorizations v0.2.4
  [d236fae5] PreallocationTools v0.1.1
  [21216c6a] Preferences v1.2.2
  [33c8b6b6] ProgressLogging v0.1.4
  [92933f4c] ProgressMeter v1.7.1
  [1fd47b50] QuadGK v2.4.2
  [8a4e6c94] QuasiMonteCarlo v0.2.3
  [74087812] Random123 v1.4.2
  [e6cf234a] RandomNumbers v1.4.0
  [c84ed2f1] Ratios v0.4.2
  [3cdcf5f2] RecipesBase v1.1.2
  [731186ca] RecursiveArrayTools v2.19.1
  [f2c3362d] RecursiveFactorization v0.2.4
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.1.3
  [ae5879a3] ResettableStacks v1.1.1
  [37e2e3b7] ReverseDiff v1.9.0
  [79098fc4] Rmath v0.7.0
  [7e49a35a] RuntimeGeneratedFunctions v0.5.3
  [3cdde19b] SIMDDualNumbers v0.1.0
  [94e857df] SIMDTypes v0.1.0
  [476501e8] SLEEFPirates v0.6.27
  [1bc83da4] SafeTestsets v0.0.1
  [0bca4576] SciMLBase v1.19.1
  [efcf1570] Setfield v0.7.1
  [699a6c99] SimpleTraits v0.9.4
  [ed01d8cd] Sobol v1.5.0
  [a2af1166] SortingAlgorithms v1.0.1
  [47a9eef4] SparseDiffTools v1.16.5
  [276daf66] SpecialFunctions v1.7.0
  [860ef19b] StableRNGs v1.0.0
  [aedffcd0] Static v0.3.3
  [90137ffa] StaticArrays v1.2.13
  [82ae8749] StatsAPI v1.0.0
  [2913bbd2] StatsBase v0.33.11
  [4c63d2b9] StatsFuns v0.9.12
  [9672c7b4] SteadyStateDiffEq v1.6.6
  [789caeaf] StochasticDiffEq v6.37.3
  [7792a7ef] StrideArraysCore v0.2.4
  [53d494c1] StructIO v0.3.0
  [c3572dad] Sundials v4.6.0
  [d1185830] SymbolicUtils v0.16.0
  [0c5d862f] Symbolics v3.4.3
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.6.0
  [8ea1fca8] TermInterface v0.1.8
  [5d786b92] TerminalLoggers v0.1.4
  [8290d209] ThreadingUtilities v0.4.6
  [a759f4b9] TimerOutputs v0.5.13
  [0796e94c] Tokenize v0.5.21
  [9f7883ad] Tracker v0.2.16
  [3bb67fe8] TranscodingStreams v0.9.6
  [592b5752] Trapz v2.0.3
  [a2a6695c] TreeViews v0.3.0
  [d5829a12] TriangularSolve v0.1.6
  [5c2747f8] URIs v1.3.0
  [3a884ed6] UnPack v1.0.2
  [1986cc42] Unitful v1.9.0
  [3d5dd08c] VectorizationBase v0.21.13
  [19fa3120] VertexSafeGraphs v0.1.2
  [efce3f68] WoodburyMatrices v0.5.4
  [a5390f91] ZipFile v0.9.4
  [e88e6eb3] Zygote v0.6.26
  [700de1a5] ZygoteRules v0.2.2
  [7cc45869] Enzyme_jll v0.0.19+0
  [f5851436] FFTW_jll v3.3.10+0
  [e33a78d0] Hwloc_jll v2.5.0+0
  [1d5cc7b8] IntelOpenMP_jll v2018.0.3+2
  [dad2f222] LLVMExtra_jll v0.0.11+0
  [856f044c] MKL_jll v2021.1.1+2
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [fb77eaff] Sundials_jll v5.2.0+1
  [0dad84c5] ArgTools
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [f43a241f] Downloads
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions
  [44cfe95a] Pkg
  [de0858da] Printf
  [9abbd945] Profile
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML
  [a4e569a6] Tar
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll
  [deac9b47] LibCURL_jll
  [29816b5a] LibSSH2_jll
  [c8ffd9c3] MbedTLS_jll
  [14a3606d] MozillaCACerts_jll
  [4536629a] OpenBLAS_jll
  [05823500] OpenLibm_jll
  [bea87d4a] SuiteSparse_jll
  [83775a58] Zlib_jll
  [8e850ede] nghttp2_jll
  [3f19e933] p7zip_jll

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

The parameters don't optimize with AutoZygote and If we use AutoFiniteDiff instead, we get:

l = 25.781557f0
l = 5.857797f0
l = 1.8612224f0
l = 2.1685507f0
l = 2.094833f0
l = 2.041414f0
l = 2.1555657f0
l = 2.2074337f0
l = 2.372404f0
l = 2.7590957f0
l = 3.127139f0
l = 3.3681376f0
l = 3.6843796f0
l = 3.994985f0
l = 4.092485f0
l = 4.156217f0
l = 4.2265596f0
l = 4.2244806f0
l = 4.1855636f0
l = 4.198212f0
l = 4.182091f0
l = 4.106333f0
l = 3.8979046f0
l = 3.6430976f0
l = 3.4097815f0
l = 1.8612224f0
u: 10-element Vector{Float32}:
 -0.47955412
 -0.48196423
  0.55147487
  0.28039846
  0.053077944
 -0.12632409
 -0.52188706
  0.24633482
 -0.61024064
 -0.076528005

I think two weeks ago, it was fine but after a couple of manifest updates, the issue appeared.
@ChrisRackauckas
Copy link
Member

Does it work if you force Zygote 0.6.25?

@prbzrg
Copy link
Member Author

prbzrg commented Oct 11, 2021

I tried it with Zygote@0.6.25 and Zygote@0.6.21, it doesn't work.

@ChrisRackauckas
Copy link
Member

pull back DiffEqSensitivity?

@prbzrg
Copy link
Member Author

prbzrg commented Oct 11, 2021

I tried DiffEqSensitivity@6.58.0

l = 64.55258f0
l = 64.5528f0
l = 64.55275f0
l = 64.554695f0
l = 64.5521f0
l = 64.55299f0
l = 64.551994f0
l = 64.55238f0
l = 64.552505f0
l = 64.55315f0
l = 64.55369f0
l = 64.55284f0
l = 64.55311f0
l = 64.55194f0
l = 64.55305f0
l = 64.55184f0
l = 64.5527f0
l = 64.55287f0
l = 64.55246f0
l = 64.55297f0
l = 64.55229f0
l = 64.552635f0
l = 64.55217f0
l = 64.55173f0
l = 64.55325f0
l = 64.55173f0
res1.u = Float32[0.5981966, 0.6247843, 0.33043903, 0.0, 0.0, 0.0, 0.046019506, 0.88301414, -0.3543966, 0.0]
ffjord_mdl.p = Float32[0.5981966, 0.6247843, 0.33043903, 0.0, 0.0, 0.0, 0.046019506, 0.88301414, -0.3543966, 0.0]

(ffjord-report-issues) pkg> st
      Status `C:\Users\Hossein Pourbozorg\Code Projects\Mine\ffjord-report-issues\Project.toml`
  [aae7a2af] DiffEqFlux v1.43.0
  [41bf760c] DiffEqSensitivity v6.58.0
  [0c46a032] DifferentialEquations v6.19.0
  [31c24e10] Distributions v0.25.14
  [a75be94c] GalacticOptim v2.0.3

(ffjord-report-issues) pkg> st -m
      Status `C:\Users\Hossein Pourbozorg\Code Projects\Mine\ffjord-report-issues\Manifest.toml`
  [621f4979] AbstractFFTs v1.0.1
  [1520ce14] AbstractTrees v0.3.4
  [79e6a3ab] Adapt v3.3.1
  [ec485272] ArnoldiMethod v0.1.0
  [4fba245c] ArrayInterface v3.1.33
  [4c555306] ArrayLayouts v0.7.5
  [13072b0f] AxisAlgorithms v1.0.1
  [ab4f0b2a] BFloat16s v0.1.0
  [aae01518] BandedMatrices v0.16.11
  [62783981] BitTwiddlingConvenienceFunctions v0.1.0
  [764a87c0] BoundaryValueDiffEq v2.7.1
  [fa961155] CEnum v0.4.1
  [2a0fbf3d] CPUSummary v0.1.5
  [00ebfdb7] CSTParser v3.2.4
  [052768ef] CUDA v3.3.6
  [7057c7e9] Cassette v0.3.9
  [082447d4] ChainRules v0.8.25
  [d360d2e6] ChainRulesCore v0.10.13
  [fb6a15b2] CloseOpenIntervals v0.1.2
  [944b1d66] CodecZlib v0.7.0
  [3da002f7] ColorTypes v0.11.0
  [5ae59095] Colors v0.12.8
  [861a8166] Combinatorics v1.0.2
  [a80b9123] CommonMark v0.8.3
  [38540f10] CommonSolve v0.2.0
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v3.39.0
  [b152e2b5] CompositeTypes v0.1.2
  [88cd18e8] ConsoleProgressMonitor v0.1.2
  [187b0558] ConstructionBase v1.3.0
  [a8cc5b0e] Crayons v4.0.4
  [754358af] DEDataArrays v0.2.0
  [9a962f9c] DataAPI v1.9.0
  [82cc6244] DataInterpolations v3.6.1
  [864edb3b] DataStructures v0.18.10
  [e2d170a0] DataValueInterfaces v1.0.0
  [bcd4f6db] DelayDiffEq v5.31.1
  [2b5f629d] DiffEqBase v6.73.2
  [459566f4] DiffEqCallbacks v2.17.0
  [5a0ffddc] DiffEqFinancial v2.4.0
  [aae7a2af] DiffEqFlux v1.43.0
  [c894b116] DiffEqJump v7.3.1
  [77a26b50] DiffEqNoiseProcess v5.9.0
  [055956cb] DiffEqPhysics v3.9.0
  [41bf760c] DiffEqSensitivity v6.58.0
  [163ba53b] DiffResults v1.0.3
  [b552c78f] DiffRules v1.3.1
  [0c46a032] DifferentialEquations v6.19.0
  [c619ae07] DimensionalPlotRecipes v1.2.0
  [b4f34e82] Distances v0.10.4
  [31c24e10] Distributions v0.25.14
  [ced4e74d] DistributionsAD v0.6.29
  [ffbed154] DocStringExtensions v0.8.5
  [5b8099bc] DomainSets v0.5.9
  [7c1d4256] DynamicPolynomials v0.3.21
  [da5c29d0] EllipsisNotation v1.1.0
  [7da242da] Enzyme v0.7.0
  [d4d017d3] ExponentialUtilities v1.10.0
  [e2ba6199] ExprTools v0.1.6
  [7a1cc6ca] FFTW v1.4.5
  [7034ab61] FastBroadcast v0.1.8
  [9aa1b823] FastClosures v0.3.2
  [1a297f60] FillArrays v0.11.9
  [6a86dc24] FiniteDiff v2.8.1
  [53c48c17] FixedPointNumbers v0.8.4
  [587475ba] Flux v0.12.7
  [59287772] Formatting v0.4.2
  [f6369f11] ForwardDiff v0.10.20
  [069b7b12] FunctionWrappers v1.1.2
  [d9f16b24] Functors v0.2.6
  [0c68f7d7] GPUArrays v8.1.2
  [61eb1bfa] GPUCompiler v0.12.9
  [a75be94c] GalacticOptim v2.0.3
  [af5da776] GlobalSensitivity v1.2.1
  [3e5b6fbb] HostCPUFeatures v0.1.4
  [0e44f5e4] Hwloc v2.0.0
  [7869d1d1] IRTools v0.4.3
  [615f187c] IfElse v0.1.0
  [d25df0c9] Inflate v0.1.2
  [a98d9a8b] Interpolations v0.13.4
  [8197267c] IntervalSets v0.5.3
  [92d709cd] IrrationalConstants v0.1.0
  [42fd0dbc] IterativeSolvers v0.9.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.2
  [98e50ef6] JuliaFormatter v0.15.11
  [e5e0dc1b] Juno v0.8.4
  [5ab0869b] KernelDensity v0.6.3
  [929cbde3] LLVM v4.6.0
  [b964fa9f] LaTeXStrings v1.2.1
  [2ee39098] LabelledArrays v1.6.5
  [23fbe1c1] Latexify v0.15.6
  [a5e1c1ea] LatinHypercubeSampling v1.8.0
  [73f95e8e] LatticeRules v0.0.1
  [10f19ff3] LayoutPointers v0.1.3
  [1d6d02ad] LeftChildRightSiblingTrees v0.1.2
  [093fc24a] LightGraphs v1.3.5
  [d3d80556] LineSearches v7.1.1
  [2ab3a3ac] LogExpFunctions v0.3.0
  [e6f89c97] LoggingExtras v0.4.7
  [bdcacae8] LoopVectorization v0.12.84
  [1914dd2f] MacroTools v0.5.8
  [d125e4d3] ManualMemory v0.1.6
  [e89f7d12] Media v0.5.0
  [e1d29d7a] Missings v1.0.2
  [961ee093] ModelingToolkit v6.4.7
  [46d2c3a1] MuladdMacro v0.2.2
  [f9640e96] MultiScaleArrays v1.8.1
  [102ac46a] MultivariatePolynomials v0.3.18
  [d8a4904e] MutableArithmetics v0.2.20
  [d41bc354] NLSolversBase v7.8.1
  [2774e3e8] NLsolve v4.5.1
  [872c559c] NNlib v0.7.29
  [a00861dc] NNlibCUDA v0.1.9
  [77ba4419] NaNMath v0.3.5
  [8913a72c] NonlinearSolve v0.3.11
  [d8793406] ObjectFile v0.3.7
  [6fe1bfb0] OffsetArrays v1.10.7
  [429524aa] Optim v1.4.1
  [bac558e1] OrderedCollections v1.4.1
  [1dea7af3] OrdinaryDiffEq v5.64.1
  [90014a1f] PDMats v0.11.1
  [65888b18] ParameterizedFunctions v5.12.1
  [d96e819e] Parameters v0.12.3
  [69de0a69] Parsers v2.0.5
  [e409e4f3] PoissonRandom v0.4.0
  [f517fe37] Polyester v0.5.3
  [1d0040c9] PolyesterWeave v0.1.0
  [85a6dd25] PositiveFactorizations v0.2.4
  [d236fae5] PreallocationTools v0.1.1
  [21216c6a] Preferences v1.2.2
  [33c8b6b6] ProgressLogging v0.1.4
  [92933f4c] ProgressMeter v1.7.1
  [1fd47b50] QuadGK v2.4.2
  [8a4e6c94] QuasiMonteCarlo v0.2.3
  [74087812] Random123 v1.4.2
  [e6cf234a] RandomNumbers v1.4.0
  [c84ed2f1] Ratios v0.4.2
  [3cdcf5f2] RecipesBase v1.1.2
  [731186ca] RecursiveArrayTools v2.17.2
  [f2c3362d] RecursiveFactorization v0.2.4
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.1.3
  [ae5879a3] ResettableStacks v1.1.1
  [37e2e3b7] ReverseDiff v1.9.0
  [79098fc4] Rmath v0.7.0
  [7e49a35a] RuntimeGeneratedFunctions v0.5.3
  [3cdde19b] SIMDDualNumbers v0.1.0
  [94e857df] SIMDTypes v0.1.0
  [476501e8] SLEEFPirates v0.6.27
  [1bc83da4] SafeTestsets v0.0.1
  [0bca4576] SciMLBase v1.19.1
  [efcf1570] Setfield v0.7.1
  [699a6c99] SimpleTraits v0.9.4
  [ed01d8cd] Sobol v1.5.0
  [a2af1166] SortingAlgorithms v1.0.1
  [47a9eef4] SparseDiffTools v1.16.5
  [276daf66] SpecialFunctions v1.7.0
  [860ef19b] StableRNGs v1.0.0
  [aedffcd0] Static v0.3.3
  [90137ffa] StaticArrays v1.2.13
  [82ae8749] StatsAPI v1.0.0
  [2913bbd2] StatsBase v0.33.11
  [4c63d2b9] StatsFuns v0.9.9
  [9672c7b4] SteadyStateDiffEq v1.6.6
  [789caeaf] StochasticDiffEq v6.37.3
  [7792a7ef] StrideArraysCore v0.2.4
  [53d494c1] StructIO v0.3.0
  [c3572dad] Sundials v4.6.0
  [d1185830] SymbolicUtils v0.13.2
  [0c5d862f] Symbolics v3.2.3
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.6.0
  [5d786b92] TerminalLoggers v0.1.4
  [8290d209] ThreadingUtilities v0.4.6
  [a759f4b9] TimerOutputs v0.5.13
  [0796e94c] Tokenize v0.5.21
  [9f7883ad] Tracker v0.2.16
  [3bb67fe8] TranscodingStreams v0.9.6
  [592b5752] Trapz v2.0.3
  [a2a6695c] TreeViews v0.3.0
  [d5829a12] TriangularSolve v0.1.6
  [5c2747f8] URIs v1.3.0
  [3a884ed6] UnPack v1.0.2
  [1986cc42] Unitful v1.9.0
  [3d5dd08c] VectorizationBase v0.21.13
  [19fa3120] VertexSafeGraphs v0.1.2
  [efce3f68] WoodburyMatrices v0.5.4
  [a5390f91] ZipFile v0.9.4
  [e88e6eb3] Zygote v0.6.17
  [700de1a5] ZygoteRules v0.2.2
  [7cc45869] Enzyme_jll v0.0.19+0
  [f5851436] FFTW_jll v3.3.10+0
  [e33a78d0] Hwloc_jll v2.5.0+0
  [1d5cc7b8] IntelOpenMP_jll v2018.0.3+2
  [dad2f222] LLVMExtra_jll v0.0.11+0
  [856f044c] MKL_jll v2021.1.1+2
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [fb77eaff] Sundials_jll v5.2.0+1
  [0dad84c5] ArgTools
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [f43a241f] Downloads
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions
  [44cfe95a] Pkg
  [de0858da] Printf
  [9abbd945] Profile
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML
  [a4e569a6] Tar
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll
  [deac9b47] LibCURL_jll
  [29816b5a] LibSSH2_jll
  [c8ffd9c3] MbedTLS_jll
  [14a3606d] MozillaCACerts_jll
  [4536629a] OpenBLAS_jll
  [05823500] OpenLibm_jll
  [bea87d4a] SuiteSparse_jll
  [83775a58] Zlib_jll
  [8e850ede] nghttp2_jll
  [3f19e933] p7zip_jll

@ChrisRackauckas
Copy link
Member 

@show DiffEqFlux.Flux.Zygote.gradient(loss, ffjord_mdl.p)
# (Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],)

Zygote is dropping the gradients here. @DhairyaLGandhi could you help us find the issue here?

@DhairyaLGandhi
Copy link
Member

Could you point me to a certain version of zygote where this works and one where it doesn't? (Ideally both environments should have the same versions of the sciml packages)

This was referenced Oct 14, 2021
Merged
Closed
@ChrisRackauckas
Copy link
Member

@prbzrg could you help find the versions for that?

@prbzrg
Copy link
Member Author

prbzrg commented Oct 21, 2021

I will be glad to help
why [compat] in Project.toml doesn't apply when adding DiffEqFlux.jl?

# in an empty environment

(tp1) pkg> add DiffEqFlux
    Updating registry at `C:\Users\Hossein Pourbozorg\.julia\registries\General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `C:\Users\Hossein Pourbozorg\Code Projects\Mine\test-opt-p\tp1\Project.toml`
  [aae7a2af] + DiffEqFlux v1.43.0
    Updating `C:\Users\Hossein Pourbozorg\Code Projects\Mine\test-opt-p\tp1\Manifest.toml`
  [621f4979] + AbstractFFTs v1.0.1
  [1520ce14] + AbstractTrees v0.3.4
  [79e6a3ab] + Adapt v3.3.1
  [ec485272] + ArnoldiMethod v0.1.0
  [4fba245c] + ArrayInterface v3.1.34
  [13072b0f] + AxisAlgorithms v1.0.1
  [ab4f0b2a] + BFloat16s v0.2.0
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.0
  [fa961155] + CEnum v0.4.1
  [2a0fbf3d] + CPUSummary v0.1.5
  [052768ef] + CUDA v3.5.0
  [7057c7e9] + Cassette v0.3.9   # must be Cassette = "0.3.7"
  [082447d4] + ChainRules v1.11.6
  [d360d2e6] + ChainRulesCore v1.10.1
  [fb6a15b2] + CloseOpenIntervals v0.1.2
  [944b1d66] + CodecZlib v0.7.0
  [3da002f7] + ColorTypes v0.11.0
  [5ae59095] + Colors v0.12.8
  [38540f10] + CommonSolve v0.2.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.39.0
  [88cd18e8] + ConsoleProgressMonitor v0.1.2
  [187b0558] + ConstructionBase v1.3.0
  [754358af] + DEDataArrays v0.2.0
  [9a962f9c] + DataAPI v1.9.0
  [82cc6244] + DataInterpolations v3.6.1   # must be DataInterpolations = "3.3"
  [864edb3b] + DataStructures v0.18.10
  [e2d170a0] + DataValueInterfaces v1.0.0
  [2b5f629d] + DiffEqBase v6.74.0   # must be DiffEqBase = "6.41"
  [459566f4] + DiffEqCallbacks v2.17.0
  [aae7a2af] + DiffEqFlux v1.43.0
  [c894b116] + DiffEqJump v7.3.1
  [77a26b50] + DiffEqNoiseProcess v5.9.0
  [41bf760c] + DiffEqSensitivity v6.60.2   # must be DiffEqSensitivity = "6.52"
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.3.1
  [b4f34e82] + Distances v0.10.5
  [31c24e10] + Distributions v0.25.20
  [ced4e74d] + DistributionsAD v0.6.31
  [ffbed154] + DocStringExtensions v0.8.5
  [7da242da] + Enzyme v0.7.2
  [d4d017d3] + ExponentialUtilities v1.10.0
  [e2ba6199] + ExprTools v0.1.6
  [7a1cc6ca] + FFTW v1.4.5
  [7034ab61] + FastBroadcast v0.1.8
  [9aa1b823] + FastClosures v0.3.2
  [1a297f60] + FillArrays v0.12.7
  [6a86dc24] + FiniteDiff v2.8.1
  [53c48c17] + FixedPointNumbers v0.8.4
  [587475ba] + Flux v0.12.7
  [f6369f11] + ForwardDiff v0.10.21
  [069b7b12] + FunctionWrappers v1.1.2
  [d9f16b24] + Functors v0.2.7
  [0c68f7d7] + GPUArrays v8.1.2
  [61eb1bfa] + GPUCompiler v0.13.6
  [a75be94c] + GalacticOptim v2.2.0
  [af5da776] + GlobalSensitivity v1.2.1
  [3e5b6fbb] + HostCPUFeatures v0.1.4
  [0e44f5e4] + Hwloc v2.0.0
  [7869d1d1] + IRTools v0.4.3
  [615f187c] + IfElse v0.1.0
  [d25df0c9] + Inflate v0.1.2
  [a98d9a8b] + Interpolations v0.13.4
  [3587e190] + InverseFunctions v0.1.1
  [92d709cd] + IrrationalConstants v0.1.1
  [42fd0dbc] + IterativeSolvers v0.9.1
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.3.0
  [e5e0dc1b] + Juno v0.8.4
  [5ab0869b] + KernelDensity v0.6.3
  [929cbde3] + LLVM v4.6.0
  [2ee39098] + LabelledArrays v1.6.5
  [a5e1c1ea] + LatinHypercubeSampling v1.8.0
  [73f95e8e] + LatticeRules v0.0.1
  [10f19ff3] + LayoutPointers v0.1.3
  [1d6d02ad] + LeftChildRightSiblingTrees v0.1.2
  [093fc24a] + LightGraphs v1.3.5
  [d3d80556] + LineSearches v7.1.1
  [2ab3a3ac] + LogExpFunctions v0.3.4
  [e6f89c97] + LoggingExtras v0.4.7
  [bdcacae8] + LoopVectorization v0.12.87
  [1914dd2f] + MacroTools v0.5.8
  [d125e4d3] + ManualMemory v0.1.6
  [e89f7d12] + Media v0.5.0
  [e1d29d7a] + Missings v1.0.2
  [46d2c3a1] + MuladdMacro v0.2.2
  [d41bc354] + NLSolversBase v7.8.1
  [2774e3e8] + NLsolve v4.5.1
  [872c559c] + NNlib v0.7.29
  [a00861dc] + NNlibCUDA v0.1.9
  [77ba4419] + NaNMath v0.3.5
  [8913a72c] + NonlinearSolve v0.3.11
  [d8793406] + ObjectFile v0.3.7
  [6fe1bfb0] + OffsetArrays v1.10.7
  [429524aa] + Optim v1.4.1
  [bac558e1] + OrderedCollections v1.4.1
  [1dea7af3] + OrdinaryDiffEq v5.64.1
  [90014a1f] + PDMats v0.11.1
  [d96e819e] + Parameters v0.12.3
  [e409e4f3] + PoissonRandom v0.4.0
  [f517fe37] + Polyester v0.5.3
  [1d0040c9] + PolyesterWeave v0.1.1
  [85a6dd25] + PositiveFactorizations v0.2.4
  [d236fae5] + PreallocationTools v0.1.1
  [21216c6a] + Preferences v1.2.2
  [33c8b6b6] + ProgressLogging v0.1.4
  [92933f4c] + ProgressMeter v1.7.1
  [1fd47b50] + QuadGK v2.4.2
  [8a4e6c94] + QuasiMonteCarlo v0.2.3
  [74087812] + Random123 v1.4.2
  [e6cf234a] + RandomNumbers v1.5.3
  [c84ed2f1] + Ratios v0.4.2
  [3cdcf5f2] + RecipesBase v1.1.2
  [731186ca] + RecursiveArrayTools v2.20.0
  [f2c3362d] + RecursiveFactorization v0.2.4
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.1.3   # must be Requires = "0.5, 1.0"
  [ae5879a3] + ResettableStacks v1.1.1
  [37e2e3b7] + ReverseDiff v1.9.0
  [79098fc4] + Rmath v0.7.0
  [3cdde19b] + SIMDDualNumbers v0.1.0
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.27
  [0bca4576] + SciMLBase v1.19.2
  [efcf1570] + Setfield v0.8.0
  [699a6c99] + SimpleTraits v0.9.4
  [ed01d8cd] + Sobol v1.5.0
  [a2af1166] + SortingAlgorithms v1.0.1
  [47a9eef4] + SparseDiffTools v1.17.0
  [276daf66] + SpecialFunctions v1.8.0
  [860ef19b] + StableRNGs v1.0.0
  [aedffcd0] + Static v0.3.3
  [90137ffa] + StaticArrays v1.2.13
  [82ae8749] + StatsAPI v1.0.0
  [2913bbd2] + StatsBase v0.33.12
  [4c63d2b9] + StatsFuns v0.9.12
  [789caeaf] + StochasticDiffEq v6.40.0
  [7792a7ef] + StrideArraysCore v0.2.6
  [53d494c1] + StructIO v0.3.0
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.6.0
  [5d786b92] + TerminalLoggers v0.1.4
  [8290d209] + ThreadingUtilities v0.4.6
  [a759f4b9] + TimerOutputs v0.5.13
  [9f7883ad] + Tracker v0.2.16
  [3bb67fe8] + TranscodingStreams v0.9.6
  [592b5752] + Trapz v2.0.3
  [a2a6695c] + TreeViews v0.3.0
  [d5829a12] + TriangularSolve v0.1.6
  [3a884ed6] + UnPack v1.0.2
  [3d5dd08c] + VectorizationBase v0.21.15
  [19fa3120] + VertexSafeGraphs v0.1.2
  [efce3f68] + WoodburyMatrices v0.5.4
  [a5390f91] + ZipFile v0.9.4
  [e88e6eb3] + Zygote v0.6.29
  [700de1a5] + ZygoteRules v0.2.2
  [7cc45869] + Enzyme_jll v0.0.19+0
  [f5851436] + FFTW_jll v3.3.10+0
  [e33a78d0] + Hwloc_jll v2.5.0+0
  [1d5cc7b8] + IntelOpenMP_jll v2018.0.3+2
  [dad2f222] + LLVMExtra_jll v0.0.11+0
  [856f044c] + MKL_jll v2021.1.1+2
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [f50d1b31] + Rmath_jll v0.3.0+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [9abbd945] + Profile
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [05823500] + OpenLibm_jll
  [83775a58] + Zlib_jll
  [8e850ede] + nghttp2_jll
  [3f19e933] + p7zip_jll
Precompiling project...
  38 dependencies successfully precompiled in 201 seconds (131 already precompiled)

@ChrisRackauckas
Copy link
Member

Those are lower bounds

@prbzrg
Copy link
Member Author

prbzrg commented Oct 22, 2021

I looked into the CI and found out

working                                  ,not working
[82cc6244] + DataInterpolations v3.6.0   ,[82cc6244] + DataInterpolations v3.6.1
[41bf760c] + DiffEqSensitivity v6.58.0   ,[41bf760c] + DiffEqSensitivity v6.60.1
[31c24e10] + Distributions v0.25.14      ,[31c24e10] + Distributions v0.25.19
[ced4e74d] + DistributionsAD v0.6.29     ,[ced4e74d] + DistributionsAD v0.6.31
[587475ba] + Flux v0.12.6                ,[587475ba] + Flux v0.12.7
[f6369f11] + ForwardDiff v0.10.19        ,[f6369f11] + ForwardDiff v0.10.21
[a75be94c] + GalacticOptim v2.0.3        ,[a75be94c] + GalacticOptim v2.1.0
[731186ca] + RecursiveArrayTools v2.17.2 ,[731186ca] + RecursiveArrayTools v2.19.1
[0bca4576] + SciMLBase v1.19.0           ,[0bca4576] + SciMLBase v1.19.2
[90137ffa] + StaticArrays v1.2.12        ,[90137ffa] + StaticArrays v1.2.13
[e88e6eb3] + Zygote v0.6.17              ,[e88e6eb3] + Zygote v0.6.27
[700de1a5] + ZygoteRules v0.2.1          ,[700de1a5] + ZygoteRules v0.2.2

And tested the working versions and it is working.
I will try to make the list shorter.

@prbzrg
Copy link
Member Author

prbzrg commented Oct 22, 2021

It looks like the problem is SciMLBase@1.19.1 (SciMLBase@1.19.0 is working)
I will experiment more to be sure and make a PR.

@prbzrg prbzrg mentioned this issue Oct 22, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

update Project.toml file
3 participants
@ChrisRackauckas @DhairyaLGandhi @prbzrg