-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sum!, prod!, any!, and all! may silently return incorrect results #39385
Comments
So what is the correct approach here? Add an alias check similar to that in |
Maybe worth noting that
This is |
Can @mbauman's |
I mean, in all these cases it needs to collapse to a no-op. Can we avoid worrying about aliasing at all with a simple |
I'm not sure what should happen, but a tricker example is:
The perhaps-comparable |
@mbauman You're right that calling At any rate, docstrings should mention this. |
should this bug be release blocking at any point? I feel that it is
|
at the very least, we could document that this is a case where weirdness can occur |
Please, add this to the 1.10 milestone... This is a correctness issue, not so difficult to fix, and correctness should have priority for any function in Base or the standard library. |
I beg to differ. Mutable functions are prone to weird issues like that, and that's one reason to avoid them or be careful with their use. Good thing we name them with a scary warning Completely agree it should be documented. But in my opinion this is a bit like expecting a specific defined behavior from |
Why is this on the milestone? The milestone is not something you just slap on issues and they get magically fixed. Is this a regression vs 1.9? |
Well, there is even a pull request already that might fix this issue... Not good for the reputation of Julia to delay this any further... |
forgive the naivete, but what even causes this? why does aliasing matter in the output array? that is, what makes
so different from
|
Any reducing function such as In this example, the non-mutating call to |
I do not agree. Functions in base should be as safe and correct as possible. If there would be a large performance penalty to add a check I would accept your argument, but if the average performance penalty is less than 10% we should add a check. Not all users of Julia have a PhD. Better safe than sorry. Julia should try to be as fast and safe as Rust and not as fast and unsafe as C. Unsafe high performance versions of standard functions can live in packages, or an enabling them via a macro might also be an option. |
I don't see this as remotely easy to "fix" in code. Note that our
BLAS is clearly not checking for aliasing internally. We have a courtesy check (that threw the initial error, based on checking for I say
Note that fixing these problems, in general, requires that we either throw an error or silently copy inputs (but some functions also need to observably mutate "inputs", so that solution could not work everywhere). Neither of those is what the user wanted, although I expect they would prefer the error to unspecified behavior. Even when it didn't cause deeper correctness problems, I can't promise the user would be happy we started copying stuff for them in a function they specifically called to avoid allocations. We can do people the favor of courtesy checks, but it seems that these will inevitably throw on legal uses and/or miss illegal ones. We can try to catch "easy and obvious" violations (like the To my point on reliability, I would say
Documenting the no-alias assumption appears to be the only complete solution. It would be worth auditing all of our |
Well, documenting is always good, but adding an incomplete solution is better than implementing no solution. If an incomplete solution makes it less likely that a newcomer hits this issue we should do it. |
Again, the milestone is not for getting random things fixed faster. I'll remove it since it doesn't seem to be a release blocker. |
This is bad and shows the attitude of leading team members to ignore correctness issues... |
Started PR #50824 to amend the docs because it doesn't cost much, even if we later decide to check for some forms of aliasing. Personally, I agree with @mikmoore that some edge cases will necessarily fall through the cracks. And I also don't think that comments like
will lead anywhere. If anything, the fact that we're here discussing this shows that we all take it to heart, although opinions may diverge. |
I agree, and potentially we could add those checks and even make them smarter (we potentially could and maybe should check for derived arrays like ranges or transposes because we can look for the parent) but the user is always able to get around those so the docs should reflect that. |
Maybe it's the case that these mutable reductions could be implemented in a way that is efficient and guarantee to behave like the identity function when the inputs have the same shape, even in the unreasonable case where the same array is the input and the output. In fact, the assignment to I find it pretty upsetting that so much attention is being given to this, while issue #45000, that I still consider very serious and relevant, was dismissed out of hand. |
There you were directed to |
|
I don't really think that issue is particularly relevant to this one, but linking it here did call my attention to it. |
At least sum! isn't used by Julia itself so ideally should not/never have been part of Julia Base, but at least can then be "fixed" without slowing Julia down.
Maybe, but all of those functions could be duplicated in an external package as is (right now, they could even be deprecated here), and people could do
I'm sympathetic to that, so should a check be added to at least the most important functions ASAP? |
I noticed that
sum!
andprod!
do not check for aliasing, leading to incorrect results when the same array is passed in both arguments. The incorrect behavior under aliasing is not documented and the result is a silent unexpected wrong result.Not all mutating functions have this problem and
cumsum!
can be used this way without error:Other functions correctly flag the aliasing in the case of
===
arguments and throw an error:The examples above were generated on Julia
1.6.0-beta1
and I see the same behavior in version1.5.3
.Edit: I noticed this issue also applies to
any!
andall!
:The text was updated successfully, but these errors were encountered: