4.x Release plans

[andrewgsavage]

Most of the topics in #201 are completed so I'm starting this for the next release, which I presume will be 4.0.0

Skimming through the previous topic and development plans, it looks like the following weren't resolved and could be in the next release:

• Linting, precommit
• Renaming/aliasing of UFloat, AffineScalarFunc

I've been working on:

• Some reorganisation into smaller files Reorganise core #187
• improved numpy support (https://github.com/andrewgsavage/uncertainties/tree/numpy will submit a PR after Reorganise core #187 is merged)

Is there anything else you'd like to see in a 4.0.0 release?

[newville]

@andrewgsavage This all sounds great to me. I would still like to have something done about the self-referential
LinearCombination thing of:

class Variable(AffineScalarFunc):
     __slots__ = ('_std_dev', 'tag')
    def __init__(self, value, std_dev, tag=None):
        value = float(value)
        super(Variable, self).__init__(value, LinearCombination({self: 1.}))

with the lengthy discussion at #218 (somewhat noisy, as we are still coming to grips with what this means).

I think the simplest solution might be to replace LinearCombination({self: 1.}) with LinearCombination({None: 1.}) and then have AffineScalerFunc pass in its own self to LinearCombination.expand()as the "value to use in place ofself". I think that will make Variablebe easier to serialize, and not need the wholeid` business. I suppose that means I just volunteered to make a PR...

[wshanks]

I would be opposed to making std_dev un-settable (n and nominal_value are).

For what reason do you oppose making std_dev un-settable like nominal_value? Is it to preserve the feature of being able to modify an independent variable's uncertainty and calculate the new uncertainty for a dependent AffineScalarFunc? Or to keep backwards compatibility?

I would also be okay with #219 or something like it if we could add a warning to std_dev.setter of Variable that, if the user tried to change the value to or from 0, would complain that previous tests of equality and hashing might no longer be valid. That was the main issue for me -- a hash should never change and items that hash to the same value need to be equal, but changing a finite std_dev to 0 would make an AffineScalarFunc equal to a float with the same nominal value. Tweaking the value in any other way should be fine and with that warning the user could even set the value to 0 knowing they should try to use it as a dict key afterwards.

[newville]

@wshanks Very sorry, my mistake. I would NOT be opposed to making std_dev an un-settable property, just as n and nominal_value are unsettable. I do worry a bit that making _std_dev and _nomimal_value static might make implementation more complicated -- perhaps making these __std_dev__ etc would make them more obviously "private".

[jagerber48]

@newville @wshanks See #251

[andrewgsavage]

I have a hunch that core is architecture the way it is to support a late-evaluation of chains of dependencies between variables with uncertainty in a way that allegedly improves performance. I haven't seen any performance data anywhere, it's just a hunch I have based on the comments.

I think it would be worth seeing either trying an instant evaluation or using a covariance matrix and seeing how much slower it would be to understand the performance cost of a simpler implementation

[jagerber48]

@andrewgsavage after re-implementing the "linear combination of uncertainty atoms" and finally understanding how it works, I less have the feeling that it is unintuitive compared to a covariance matrix approach. And I think it's a bit nicer programmatically because there isn't shared information between different objects. In the linear-combination-of-uncertainty-atoms approach each variable keeps track of the atoms it depends on. Then when two variables come in contact their local atoms are compared.

In the covariance matrix approach there is a sort of global state that needs to be kept track of which variables exist and how are they correlated. There could be one huge covariance matrix that keeps track of all variables, but I think this would quickly become untenable if large calculations are done. One could imagine only storing the covariance between variables X and Y locally on those objects, but this would need to be synchronized, and, depending on the connectivity of calculations, we could also end up with huge covariance matrices.

In the linear combination of atomic uncertainties we write X = X_0 + dX, Y = Y_0 + dY Z = X + Y = (X_0 + Y_0) + (dX + dY). In the covariance approach we think of only Z = X+Y` but retain memory of the covariances between X, Y, and Z.

I have ideas how this can be more clearly documented.

---

However, you also mentioned testing early vs. late evaluation. I do think early evaluation could simplify the implementation, so I agree that this would be interesting.

[newville]

@andrewgsavage @wshanks @jagerber48 But also: is it OK if we view #245 as something we should merge now and then push out version 3.2.2 for Numpy2 compatibility, and then switch to more refactoring for 4.0?

[wshanks]

That sounds good to me!

[andrewgsavage]

Yes please, numpy2 support warrants a release as soon as it's ready

[wshanks]

One thing I think would be good to tackle for a 4.x release is the numpy.matrix deprecation warning (#114). I think the reason numpy wants to deprecate numpy.matrix is that the main motivation for it was so that * could be overloaded as matrix multiplication, but after pep425 added the @ symbol for matrix multiplication that motivation went away and it has just been a very slow deprecation in numpy as it waits for scipy, etc. to stop using numpy.matrix (it passed on the opportunity to complete the deprecation with numpy 2.0).

The problem for Uncertainties is that we have umatrix which uses numpy.matrix. We should figure out if we want to keep or deprecate umatrix. We could switch to keep the current umatrix interface without using numpy.matrix or we could deprecate it and just keep the inverse function and let users call that on a plain uarray when necessary. I think the inverse function is the main value added by umatrix. We don't need to make many changes for 4.0, but I think it would be good to add a deprecation warning on umatrix with that release if that is the direction we want to go.

[newville]

@wshanks I agree. Numpy encourages abandoning matrix. We should listen to them. I am +1 on deprecating umatrix either now or with 4.0, and hopefully removing it in a year or so.