Understanding how AffineScalarFunc and Variable work

[jagerber48]

I'm trying to understand the discussion in #189. For this I need to better understand how AffineScalarFunc, Variable and LinearCombination work. I'm working through the code, but I think some quick discussion might get me there a little faster if folks are willing to teach. I think the other maintainers already have a much better idea than me.

I'm especially trying to understand how uncertainties is able to handle correlations between variables. I guess the idea is that if a set of variables are correlated then they are stored together within the same AffineScalarFunc (within a LinearCombination) object.

Looking through the code I think I'm seeing some documentation issues, I'm also wondering if there are opportunities for refactoring to make the code a bit easier to follow. But at this point I don't think I understand enough to make a judgement there.

[jagerber48]

For example, if there are N correlated values x_1, ... x_N why not store, as the core underlying data model, a vector of these values along with their NxN covariance or correlation matrix and use these two things to do calculations? Why are the layers of abstraction such as Variable, AffineScalarFunc, LinearCombination necessary?

[wshanks]

What is the context in which you have N correlated values? The model of Uncertainties is that you have independent (uncorrelated) variables (the Variable class, often generated by the ufloat function) holding a value and standard deviation and then modified versions of standard math operations which propagate errors for them. These math functions return AffineScalarFunc instances (and in the class hierarchy Variable is a subclass of AffineScalarFunc). AffineScalarFunc holds a nominal value and a mapping of Variables to derivatives of the nominal value with respect to each of those variables (stored in the LinearCombination). Correlation is managed by properly identifying that two inputs to a math operation have the same Variable in their derivative mapping and so adding those derivatives (so for x = ufloat(1, 1); y = 2* x; y - y == 0 # True because the derivative with respect to x becomes 0).

[jagerber48]

For example a fit with N parameters. The fit results will be correlated. But in the context of uncertainties I'm looking at the output of the correlated_values function.

[lebigot]

@wshanks is perfectly correct, except that more precisely, the derivatives stored in an AffineScalarFunc are those of the variable that it represents. For example, z = 1 + 2y is represented as an AffineScalarFunc with nominal value 1 (it's a constant, so technically has a null derivative), and a derivative of 2 with respect to Variable y.

@jagerber48: your question about why not storing covariances between variables is a fair one. The answer is that it would be extremely wasteful. For example, imagine that you have an array of 1000 nominal values, to which you add the same variable (error) δ: the covariance matrix has 1 million values, whereas the procedure described above stores only 1000 derivatives with respect to δ (with value 1), which is much smaller and therefore runs much faster.

[wshanks]

#189 is the context in which I have looked most closely at the code as well. I was really wishing for type hints while trying to understand LinearCombination 🙂, which is essentially a dict mapping Variables to floats (the derivatives) but supports a different internal representation for performance, only expanding to the dict when needed. I also felt like there was opportunity to refactor and break up some of the larger functions for readability, though to be fair that is just a first impression to reading a lot of code closely for the first time, so I don't want to make too strong of a statement.

[jagerber48]

Yup, I was wishing for type hints there also. Though it could have technically been clarified with a more clear/direct traditional docstring as well.

[lebigot]

@jagerber48: what's not traditional about the doctoring? As far as I remember, I essentially followed PEP 257, no?

[newville]

FWIW, ufloats can be correlated, using uncertaintes.correlated_values. Lmfit uses/needs this.

import numpy as np
from uncertainties import correlated_values

vals = np.array([20.0, 18.0])
std = np.array([2.50, 2.50])

for correl in (-0.8, -0.4, 0.0, 0.4, 0.8):
    covar = np.array([[std[0]**2,         std[0]*std[1]*correl],
                      [std[0]*std[1]*correl,  std[1]**2]])

    cvals = correlated_values(vals, covar)
    print(f"# With correlation {correl}")
    print("   v1, v2 = " ,cvals)
    print("   v1 + v2 = " ,cvals[0]+cvals[1])

prints

# With correlation -0.8
   v1, v2 =  (20.0+/-2.5, 18.0+/-2.5)
   v1 + v2 =  38.0+/-1.6
# With correlation -0.4
   v1, v2 =  (20.0+/-2.4999999999999996, 18.0+/-2.4999999999999996)
   v1 + v2 =  38.0+/-2.7
# With correlation 0.0
   v1, v2 =  (20.0+/-2.5, 18.0+/-2.5)
   v1 + v2 =  38.0+/-3.5
# With correlation 0.4
   v1, v2 =  (20.0+/-2.4999999999999996, 18.0+/-2.4999999999999996)
   v1 + v2 =  38+/-4
# With correlation 0.8
   v1, v2 =  (20.0+/-2.5, 18.0+/-2.5)
   v1 + v2 =  38+/-5

and then

print(cvals[0].derivatives)

gives

{0.0+/-0.44721359549995787: -1.7677669529663687, 0.0+/-1.3416407864998738: 1.7677669529663687}

I am not at all saying that I fully understand this, and the levels of indirection are somewhat (or more!) confusing.

[lebigot]

If your last comments refers to the last part about derivatives, then:

1. v1 and v2 are not independent variables, so uncertainties created two independent variables that can be linearly combined and give v1 and v2.
2. The last result shows this linear combination for v1 (v1 = 20 - 1.76va + 1.76vb, where va and vb where automatically created, with the values listed as keys).

I'm not sure what "the levels of indirection" refer to, here, so if you need any clarification, let me know!

[newville]

Yeah, I think that can be described as both "understandable" and "needs to be understood". And I think part of the issue that is being struggled with here is whether/when those temporary Variables (your va and vb) need to be maintained as part of cvals[0]. I think that we all just need to wrap our brains around that and consider it in the discussions of hash-ability, pickling, etc.

[lebigot]

Within the approach followed by uncertainties, where each dependent (and independent!) variable tracks its derivatives with respect to all the independent variables that it depends on, va and vb must be kept around in order to define v1 (cvals[0]).

In an approach where the covariances between variables are instead stored, this is not necessary. However, such an approach is very costly in terms of memory and execution time (see above).

I am not aware of any other option, but I'd be curious!

[jagerber48]

I guess if we're all a little confused I don't feel bad shouting out to @lebigot for a little primer on how these objects work together to give uncertainties the functionality it has.

[lebigot]

And you're right to not feel bad: I am still officially willing to be "on call" when @lebigot is mentioned. 😀

While I haven't looked at recent threads and can't look at this one right now, I'll have a look this week-end at the latest. I'll check this thread in priority.

[wshanks]

ufloats can be correlated

One confusion that we might want to clear up in the package's terminology is that there is a ufloat() helper function that returns a Variable and a UFloat class which is an alias for AffineScalarFunc. AffineScalarFunc instances can be correlated. Variables are always independent like I described above. The correlated_values function takes a covariance matrix and constructs AffineScalarFuncs with those correlations by creating hidden independent Variables and combining them with the appropriate weights.

[andrewgsavage]

Perhaps Variable could be renamed to IndependentVariable ? and AffineScalarFunc to DependentVariable?

[lebigot]

@wshanks: you are perfectly right again!

@andrewgsavage: I think that Variable could be renamed IndependentVariable. It might just make the code a bit more verbose.

Now, while most AffineScalarFunc objects are dependent variables (like z in z = x + y), technically Variable is a subclass of AffineScalarFunc. So if AffineScalarFunc is renamed DependentVariable, we will have IndependentVariable be a subclass of DependentVariable, which can be confusing. What happens is that an independent variable is essentially represented as $x = x_0 + (x-x_0)$, where $x_0$ is the nominal value. This makes a Variable an AffineScalarFunc with nominal value $x_0$ and derivative 1 with respect to itself.

Granted, this is more like creating a 0-centered independent variable $\delta x = x - x0$ and using an optimized short-circuit where we don't create $\delta x$ and replace this by the true fact that the derivative of $x$ with respect to $x$ is 1, like the derivative of $x$ with respect to $\delta x$. To tell you the truth, I don't see perfectly clearly why this works anymore, but if needed I can think about it again (just ping me).

The reason for doing this is that the code transparently handles mixes of dependent and independent variables (they all have a nominal value and derivatives with respect to independent variables).

Also, the current implementation doesn't introduce variables that the user didn't create (unless required, as when inputting correlated values), which could introduce their own confusing situations.

If you guys can think of another approach, I'd be happy to give feedback (just ping me!).

[newville]

I might suggest using Value or UValue for a single value with uncertainty, and Expression or UExpression for a quantity calculated from those -- the current AffineScalarFunc.

[lebigot]

My 2 cents:

1. The term variable is used for the very well established independent and dependent variables, so using Value instead would imply that it's something else. So if I can vote, I not a big fan. :)
2. A problem with Expression for something like z = sin(y) is that it suggests that uncertainties keeps track of the full expression, and so of the sin() function, which is not the case (instead z is approximated by… an affine function (AffineScalarFunc)). So I'm not a big fan of this either. :)

[andrewgsavage]

Variable could be renamed IndependentVariable. It might just make the code a bit more verbose.
I think that's a worthwhile cost as it's much clearer to new users, users can alisa it to IV if they wish

if AffineScalarFunc is renamed DependentVariable, we will have IndependentVariable be a subclass of DependentVariable, which can be confusing.

We could use subclassing to avoid this, by having a new class DependentVariable that's a subclass of AffineScalarFunc
Move the methods/properties of AffineScalarFunc that are overwritten by Variable currently to DependentVariable

I had similar thoughts on renaming to @newville when I first looked into these classes. #185

I think Expression is much clearer to a new user. When I google "Affine function" or "Affine scalar function" I get to https://en.wikipedia.org/wiki/Affine_transformation, read a few paragraphs but do not see the relation to probability or uncertainties. I think covering that it is an approximation in the doc string + documentation would be sufficient.

[jagerber48]

It would be good to have the circular dependency of: Variable is a AffineScalarFunc which is built from a LinearCombination which is built from Variable clarified or eliminated.

[newville]

Yes, I agree. It seems to be that

>>> a = uncertainties.ufloat(10, 2)
>>> b = uncertainties.ufloat(10, 2)
>>> hash(a)
6273767424
>>> a.std_dev = 3.1
>>> hash(a)
6273767424
>>> a == b
False
>>> a == a * 1.0
True
>>> a.derivatives
{10.0+/-3.1: 1.0}

demonstrates the things we are struggling with:
a) Variables are mutable. This could be changed, but that would be a big change.
b) Variables can be mutated and their hash does not change.
c) == and related operators make sense and have clear meanings for Variables.
d) Variables can be used as dictionary keys, and are used that way internally.
e) A Variable constructed as a plain Variable (ie, with ufloat) will have self as a key in self.derivatives.

I think b and c technically violate the recommended "data model", but that is a recommendation and apparently not enforced ;).

It might be that e is the bigger problem. It might be reasonable to allow None as a key to derivatives which signals to use self without actually using self as a key.

[lebigot]

Thanks for the discussion! I can comment on these points:

a) Why would that be a big change? From the point of view of users, I'm not sure that this would be a big change: I have yet to see anybody using the mutability of Variables. In terms of implementation, you can just remove the standard deviation setter (the nominal value cannot be modified).

b) This is an interesting point, but I'm not sure that this violates the Python data model.

The documentation indeed states that it does, but I think it's only a simplified approximation of reality:

If a class defines mutable objects and implements an eq() method, it should not implement hash(), since the implementation of hashable collections requires that a key’s hash value is immutable

In the case of uncertainties, the hash value is indeed immutable. I guess that they had in mind the case that they suggest, where the hash value depends on the value of the object, which is not the case for Variables in uncertainties.

Interestingly, the documentation describes a very common case that does not follow the prescription above:

User-defined classes have eq() and hash() methods by default; with them, all objects compare unequal (except with themselves) and x.hash() returns an appropriate value such that x == y implies both that x is y and hash(x) == hash(y).

This is exactly the situation of uncertainties: an independent variable is only exactly equal to itself (this is because it's a random variable independent of all the others). This holds even if the Variable is mutated (the previous random distribution just disappears, and the new random variable is still the only Variable equal to itself).

To summarize: Variables in uncertainties behave like user-defined classes that do not override __eq__ nor __hash__, and which can in general be mutable. This should guarantee that Variables can be both mutable and washable.

c) How is the data model violated by this? Nothing is enforced, indeed, as is often the case in Python. But, as per the reasoning above, I believe uncertainties to abide by the recommendations—even if it is done in some uncommon way.

d) This point is very useful, with the current implementation where each variable keeps track of the independent variables it depends on.

e) Why is this point a "problem"? It makes the calculation code simpler and faster by treating both dependent and independent variables transparently. Furthermore, virtually all users are shielded from this implementation detail. Those who want to access derivatives will similarly find it useful that a dependent variable $x$ knows that $\mathrm{d}x/\mathrm{d}x = 1$ (as indicated in the derivatives of an (independent) Variable.

I am not trying to defend the current implementation (as it's not in my hands anymore), but I am sharing part of the thought process that guided the choices made in the code (and which are sometimes subtle, as in the discussion about the hashability of mutable Variables), so as to help you guys decide what the best course of action is.

In conclusion:

1. I would say that from the point of view of users, making independent variables (Variable) immutable is only really useful if there is another need behind it (because, again, I believe this is a "luxury" that the implementation can afford). Now, I am not convinced that many users need the mutability, so making independent variables immutable would alleviate the need for a subtle discussion on hashability like the one above.

2. As for removing the fact that $\mathrm{d}x/\mathrm{d}x = 1$ from the implementation (in the form of a Variable storing this fact), this can only make the code of both uncertainties and of more advanced usage more cumbersome to write and maintain, in my opinion (every time an access to the derivatives of a variable is needed).

My 2 cents. :)

[wshanks]

Here I am mostly responding to:

a) Why would that be a big change? From the point of view of users, I'm not sure that this would be a big change: I have yet to see anybody using the mutability of Variables.
c) How is the data model violated by this?

If Variable could become immutable, the edge cases regarding the Python data model in #189 could be avoided.

I agree that Variable on its own satisfies the Python data model, but the __eq__ implementation supports other types by casting arguments to AffineScalarFunc before making the comparison. So for example

a = 1.0
v = Variable(value=a, std_dev=0)
print(a == v)  # True
print(hash(a) == hash(v))  # False, data model violation

If Variable is immutable, the case of std_dev=0 can be handled specially like,

def __hash__(self):
    if self.std_dev == 0:
        return hash(self.nominal_value)
    return id(self)

so that hash(a) == hash(v) above is True.

Making Variable immutable with respect to std_dev would also allow AffineScalarFunc to be hashable (which was desired for #186) because the hash could be hash((self.nominal_value, ((var, coef) for var, coef in self.derivatives if coef != 0))). With Variable.std_dev being immutable, AffineScalarFunc runs into the same issue as float in that whether or not it is equal to another Variable/AffineScalarFunc/float can depend on whether a Variable's std_dev has been set to 0 or not.

These edge cases do not come up for how uncertainties uses Variable.__hash__ internally to hold Variables in AffineScalarFunc.derivatives (really AffineScalarFunc._linear_part) since in that limited case only objects of type Variable are used (and because the available memory address space used for id() is smaller than the Python hash modulus so there are no hash collisions for id(); I mentioned this in my updated description in #218). So an alternate response to the edge cases could be to say that Variable.__hash__ exists for the internal implementation but users should not put Variables into dicts/sets themselves if mixing them with other Python types.

e) Why is this point a "problem"? It makes the calculation code simpler and faster by treating both dependent and independent variables transparently. Furthermore, virtually all users are shielded from this implementation detail.

Self-referential objects do make some things harder to reason about (like the stumbles with implementing AffineScalarFunc.__hash__ in #189 where pickle needs to hash a Variable to put it into the LinearCombination before it finishes initializing the Variable), but I think it is possible to work around those cases with care. One thing I wondered about in #218 (reply in thread) is if the ufloat() helper function should act like 1.0 * ufloat() -- so return an AffineScalarFunc. Then Variable could be treated as a bookkeeping object but not a mathematical operand and so would not need to hold its own derivative. I just suggest that as an idea for consideration though. I don't know if it is better than the current implementation or worth the disruption of changing the current behavior.

[andrewgsavage]

Might be worth looking other libraries in other languages to understand how they deal with this.