Bundles and differentiation

[inakleinbottle]

So the time has come to start thinking about how differentiation should be implemented in RoughPy. It is fairly clear that that we need to have an object that associates to a value the accumulated derivative information as computations are performed. This takes the form of a bundle which has a base algebra (the value) and a fibre (the derivative information). For exposition:

template <typename Base, typename Fibre>
class Bundle {
    Base base;
    Fibre fibre;
};

Most of the time, the Fibre type will actually be the same as the Base type, but this isn't strictly necessary. All we need is that Base has left and right actions on Fibre. Multiplication is the only operation that does not act on both base and fibre in the same way. Here, the derivative is updated using the Leibniz identity lhs.base*rhs.fibre + lhs.fibre*rhs.base. Everything so far is fairly trivial to implement.
Tensor bundles also have exp, log, etc, which are easy to implement using the bundle operations.

Now we start to run into problems. The first is that the RoughPy streams only deal with Lie and FreeTensor objects (at least for the time being). This means that if the the user wants to track derivatives in their workflow will have to manually update after each interaction with the stream (if and when any such updates are necessary). This is obviously not ideal. This probably means that we should actually implement all algebra types along with its derivative information, rather than having separate bundle types. (This makes a lot more sense in the revised algebra design described in #63.) This idea of tagging all algebra types with their derivative information is nothing new, and probably is the best idea for how to proceed in any case.

The second issue is rather more subtle. It concerns the different components against which derivatives can be computed in the workflow. For instance, in some workflows one might wish to differentiate with respect to the driving stream, whereas in others we might want to differentiate with respect to some other function parameters. We need to support both. At the moment, I don't fully understand all of the implications of this and how this should be accounted for.