Some tweaks to the Getting Started docs

[MasonProtter]

I'd like to get around to some more comprehensive documentation contributions, but thought I'd start with some very low hanging fruit / minor tweaks to the getting started page:

• I removed the pattern autodiff(mode, f, activity, args...) and replaced it with autodiff(mode, f, args...) since telling it the activity mode is no longer needed
• Before delving into various things, I just made it clear that we expect rosenbrock(1.0, 2.0) == 100.0 and the derivative w.r.t. x should be -400.0 and the derivative w.r.t. y should be 200.0. This should hopefully help users orient themselves a bit quicker when they're trying to interpret the (perhaps confusing) outputs of the various autodiff calls.
• I mentioned what we mean by 'primal' since this isn't necessarily a word everyone who took a calculus course knows

[wsmoses]

so telling it the activity is still helpful. Enzyme does its best job to deduce the return activity, but if something is unstable (e.g. like in #2194) it may fail. In the case of the issue I presume marking the return activity as Active explicitly would allow the type unstable case to succeed.

That said, that doesn't mean we need to introduce the more complex version at the start

[mcabbott]

Consider showing this as code instead of prose?

julia> rosenbrock(x, y) = (1.0 - x)^2 + 100.0 * (y - x^2)^2;

julia> rosenbrock(xy) = (1.0 - xy[1])^2 + 100.0 * (xy[2] - xy[1]^2)^2;

julia> rosenbrock(1.0, 2.0) == rosenbrock([1.0, 2.0]) == 100.0
true

I also think you should not call the input of rosenbrock_inp the same thing, x == [x, y] is weird. The name rosenbrock_inp also seems a bit weird, maybe it can just be another method, or if that's too confusing, add a suffix more informative than "inp"? (I'm not sure what INP means, maybe input, but why?)

[wsmoses]

I think it was originally in place (@michel2323 were you the one to originally author this doc, just by virtue of it being rosenbrock?)

But either way sure!

[mcabbott]

Ah ok. But this function isn't in-place, it's just going to be used somewhere below in a demonstration that Enzyme likes to handle functions which accept Vector by mutating something else. The reader doesn't know that yet.

[wsmoses]

very true, maybe rosenbrok_array or something? or even just rosenbrock2

[mcabbott]

Consider not using "value" to mean so many things here?

is a tuple that contains as a first element the derivatives of ..., and optionally the primal value (i.e. what the function returns).

[mcabbott]

Maybe "optionally" also seems a bit odd to describe the output not the input. It's not that you may omit this. It's that ReverseWithPrimal tells it to.

[wsmoses]

yeah we definitely don't need to say "derivative value" and can just say "the derivative of"

[wsmoses]

and "the value of the undifferentiated function" -> "the result of the original function without differentiation"

[mcabbott]

Also consider putting the ReverseWithPrimal case first, as without it, ((-400.0, 200.0),) seems like a puzzle to count the brackets & guess why.

Perhaps also write it with destructuring syntax, like:

derivs, y = autodiff(ReverseWithPrimal, rosenbrock, Active(1.0), Active(2.0))

[wsmoses]

oh yeah totally fair, if you want to put that in this PR that would be fine with me!

[mcabbott]

I can't make suggestions across deleted lines :/ so this is going to be messy...

[mcabbott]

This wording seems weird. The inplace version returns ((nothing,),), it's written right there. That's what return means. And inplace / "normal" are new terms here, which aren't the terms you need to learn to understand Enzyme.

The version with Active arguments (for immutable inputs like x::Float64) returns the gradient. The version with Duplicated (for mutable inputs like x::Vector{Float64}) instead writes the gradient into the Duplicated object, and returns nothing in the corresponding slot of the returned derivs. In fact it accumulates the gradient, i.e. if you run it again it will double dx. (See make_zero! perhaps.) In general a function may accept any mix of Active, Duplicated, and Const arguments.

IDK how much of the end goes here, but the reader should not get the impression that all arguments must have the same type.

[wsmoses]

perhaps we should say both compute the gradient.

And we can use whatever function names are here for clarity

[mcabbott]

Suggested change

	Both the inplace and "normal" variant return the gradient. The difference is that with [`Active`](@ref)
	Both versions calculate the same derivatives. The difference is that with [`Active`](@ref) arguments

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.70%. Comparing base (037dfed) to head (d856c12).
Report is 286 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2195      +/-   ##
==========================================
+ Coverage   67.50%   70.70%   +3.20%     
==========================================
  Files          31       55      +24     
  Lines       12668    16302    +3634     
==========================================
+ Hits         8552    11527    +2975     
- Misses       4116     4775     +659     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mcabbott]

This is messy but I made many suggestions.

• Should it still mention autodiff(Reverse, f, Active, Duplicated()) i.e. annotating the return type, somewhere? Maybe it's not the first thing to know, but it is surprising to encounter later.

• Should this page also introduce gradient(Reverse, f, ...)? Should it do this before or after autodiff?

[mcabbott]

Suggested change

	julia> rosenbrock_inp(x) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2;
	```
	where we note for future reference that the value of this function at `x=1.0`, `y=2.0` is `100.0`, and its derivative
	with respect to `x` at that point is `-400.0`, and its derivative with respect to `y` at that point is `200.0`.
	julia> rosenbrock(xy::Vector) = (1 - xy[1])^2 + 100 * (xy[2] - xy[1]^2)^2;
	julia> z = rosenbrock(1.0, 2.0)
	100.0
	julia> z == rosenbrock([1.0, 2.0]) # Vector method
	true
	```
	We note for future reference that the value of this function at `x=1.0`, `y=2.0` is `z=100.0`. Its derivative with respect to `x` at that point is `-400.0`, and its derivative with respect to `y` is `200.0`.

I've also removed 100.0 from the definition, as IMO this is idiomatic Julia -- rosenbrock can take & return Float32 without promoting.

[mcabbott]

Suggested change

	The return value of reverse mode [`autodiff`](@ref) is a tuple that contains as a first value
	The return value of reverse mode [`autodiff`](@ref) is a tuple that contains as a first element

[mcabbott]

I can't make suggestions across deleted lines :/ so this is going to be messy...

[mcabbott]

Suggested change

	the derivative value of the active inputs and optionally the _primal_ return value (i.e. the
	value of the undifferentiated function).
	the derivatives with respect to the inputs. The tuple's second element is the _primal_ value (i.e. the result of the original function without differentiation),
	but this is omitted if you use `Reverse` instead of `ReverseWithPrimal`:

[mcabbott]

Suggested change

	julia> autodiff(Reverse, rosenbrock, Active(1.0), Active(2.0))
	julia> derivs, z = autodiff(ReverseWithPrimal, rosenbrock, Active(1.0), Active(2.0))
	((-400.0, 200.0), 100.0)
	julia> autodiff(Reverse, rosenbrock, Active(1.0), Active(2.0))

[mcabbott]

Suggested change

	julia> autodiff(ReverseWithPrimal, rosenbrock, Active(1.0), Active(2.0))
	((-400.0, 200.0), 100.0)

[mcabbott]

Suggested change

	julia> autodiff(Reverse, rosenbrock_inp, Duplicated(x, dx))
	julia> autodiff(Reverse, rosenbrock, Duplicated(x, dx))

[mcabbott]

Suggested change

	Both the inplace and "normal" variant return the gradient. The difference is that with [`Active`](@ref)
	Both versions calculate the same derivatives. The difference is that with [`Active`](@ref) arguments

[mcabbott]

Suggested change

	the gradient is returned and with [`Duplicated`](@ref) the gradient is accumulated in-place into `dx`,
	and a value of `nothing` is placed in the corresponding slot of the returned `Tuple`.
	(for immutable inputs like `x::Float64`) it returns them as `derivs`, while the version with `Duplicated` (for mutable inputs like `x::Vector{Float64}`) instead writes the gradient into the `Duplicated` object, and returns `nothing` in the corresponding slot of the returned `derivs`.
	In fact it accumulates the gradient, i.e. if you run `autodiff` again it will double `dx`.
	In general, `autodiff` accepts any mix of `Active` and `Duplicated` function arguments, as well as `Const` and various other `Annotation` types.

[mcabbott]

Suggested change

	julia> autodiff(ForwardWithPrimal, rosenbrock_inp, Duplicated(x, dx))
	julia> autodiff(ForwardWithPrimal, rosenbrock, Duplicated(x, dx))