-
Notifications
You must be signed in to change notification settings - Fork 10.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document, test, and null-annotate for toImmutableMap
's behavior when mergeFunction
returns null
#6824
Comments
Nice catch, thanks! As you say:
That does suggest that your proposed For public static <T extends @Nullable Object, K, V extends @Nullable Object>
Collector<T, ?, ImmutableMap<K, @NonNull V>> toImmutableMap(
Function<? super T, ? extends K> keyFunction,
Function<? super T, ? extends @NonNull V> valueFunction,
BinaryOperator<V> mergeFunction) { That would let existing users stay unchanged (I hope! I'd want to test to see what happens in practice) while allowing anyone who wants to return I think that would at least be preferable to the alternative of... public static <T extends @Nullable Object, K, V>
Collector<T, ?, ImmutableMap<K, V>> toImmutableMap(
Function<? super T, ? extends K> keyFunction,
Function<? super T, ? extends V> valueFunction,
BinaryOperator<@Nullable V> mergeFunction) { ...which would foist the nullable Even if we don't change the types, we should document the behavior. (I've been assuming in all this that the current behavior is what we want. I think it probably is, if only to ease migration from Thought on any of that? |
Function<? super T, Range<K>> keyFunction,
Function<? super T, ? extends V> valueFunction,
BiFunction<? super V, ? super V, ? super V> mergeFunction
(EDIT: See next comment.)
Please do!
I guess that's all :-) |
I've made stupid mistake. Function<? super T, Range<K>> keyFunction,
Function<? super T, ? extends V> valueFunction,
BiFunction<? super V, ? super V, ? extends V> mergeFunction ... which is fine and causes no compilation errors (which previously were caused by using |
Thanks, I'm glad to hear that I wonder if the sweet spot might be Still, this matters relatively rarely, so it's not necessarily worth going more complex than And I can give some data about how rarely! I ran some tests overnight, and I have a couple different things to report:
So I think my plan now is:
|
I have a change out for review for those first two bullets. I'm going to generalize this issue to also cover potentially changing the nullness annotations at some future point. Thanks again for the report. |
toImmutableMap
's behavior when mergeFunction
returns null
…`mergeFunction` returns `null`. (The test is Google-internal for now because we're in the process of reshuffling our `Collector` tests to make them run under Android as part of #6567.) This addresses the main part of #6824, but I'm keeping the bug open as a prompt to recognize our nullness annotations in the future. RELNOTES=n/a PiperOrigin-RevId: 580635720
…`mergeFunction` returns `null`. (The test is Google-internal for now because we're in the process of reshuffling our `Collector` tests to make them run under Android as part of #6567. Also, we skip the test under GWT+J2CL because of a bug in their implementation of `Collectors.toMap`.) This addresses the main part of #6824, but I'm keeping the bug open as a prompt to recognize our nullness annotations in the future. RELNOTES=n/a PiperOrigin-RevId: 580635720
…`mergeFunction` returns `null`. (The test is Google-internal for now because we're in the process of reshuffling our `Collector` tests to make them run under Android as part of #6567. Also, we skip the test under GWT+J2CL because of a bug in their implementation of `Collectors.toMap`.) This addresses the main part of #6824, but I'm keeping the bug open as a prompt to recognize our nullness annotations in the future. RELNOTES=n/a PiperOrigin-RevId: 580661189
Unfortunately there are still some problems I missed before (sorry for that!). You will likely have to generalize at lest once more.
|
Here's current, undocumented behavior. @Test
void toImmutableSortedMap() {
ImmutableSortedMap<Integer, Integer> map = Stream.of(0, 0).collect(ImmutableSortedMap
.toImmutableSortedMap(Comparator.<Integer>naturalOrder(), e -> e, e -> e, (a, b) -> null));
assertThat(map).isEmpty();
}
@Test
void toImmutableEnumMap() {
ImmutableMap<RoundingMode, RoundingMode> map = Stream
.of(RoundingMode.UP, RoundingMode.UP, RoundingMode.DOWN, RoundingMode.DOWN)
.collect(Maps.toImmutableEnumMap(e -> e, e -> e, (a, b) -> null));
assertThat(map).isEmpty();
} |
That's a tad embarrassing, given that I've been actively working on the various immutable-collection Thanks also for catching I'm inclined to think that a change to That still gives us the option to change And that brings us to the fact that we were reviewing I think it would be a step forward to test and document the current |
If nulls are used to eliminate duplicates then yes, it is going to be error prone. However, I can imagine some exotic use cases, involving
I would love to see consistency between |
It's also about consistency between |
True, I have focused entirely on the "remove duplicates" use case, neglecting legitimate use cases like the one in the new test that we'll be releasing of these days: toImmutableMap(
Map.Entry::getKey,
Map.Entry::getValue,
(a, b) -> {
int result = a + b;
return result == 0 ? null : result;
}); In fairness, I have done that because "remove duplicates" was the only use case I encountered in practice with But consistency is still a good argument. And while there's always danger in changing behavior, the danger is minimal when the old behavior was "throw an exception." |
Sadly, it probably will lead to less resilient programs. It's almost too tricky to use. Note that even in the example you gave in your last post, things may go wrong, depending on what happens before. It's easy to assume that the resulting map won't contain 0 values, but that's guaranteed only if there are no mappings to 0 in the collected entries. This fact is easy to miss when writing code and even easier to miss when reading. Maybe it would be best to have |
Yikes. That does sour me a little more on the idea. On the other side: One other lesson that I could stand to remind myself of is that, as much as we hate for our libraries to promote fragile coding practices (like "removing duplicates" that works only with an even number of copies), we've gone too far the other direction at least once, leading to (e.g.) crashes in Android apps because we threw an exception. I'm thinking here of (Not that I'm advocating for having two methods here. Without a lot more users who might benefit, I don't see enough justification for a If only we had universal nullness checking, in which case we could have made compilers reject any attempt to add [edit: It might be interesting to see whether the handful of "duplicate-removing" users I found inserted the duplicate-removal logic after seeing an exception in prod. It might also be interesting to see whether other people have seen the exception in prod, filed bugs, and resolved them in some other way. Maybe we're breaking people's apps when they really just want deduplication, or maybe we're catching real issues.] |
(It turns out that we have a |
@cpovirk You seem to be hoping to publish a release soon. I'd like to ask if you are absolutely sure that You probably have more important things to do and I understand that. I just wanted to remind you that publishing a release including #6826 will make it's changes permanent (behavior will change from unspecified to guaranteed). |
Thanks. I still find everything about this kind of gross... :\ Mainly, though, I think that we have clear evidence that some users depend on the until-recently undocumented behavior. A change to that behavior is probably more likely to ruin someone's day than to improve it. If anything, I could see reconsidering |
RELNOTES=n/a PiperOrigin-RevId: 610746438
RELNOTES=n/a PiperOrigin-RevId: 610828206
…fix `Collectors.toMap` null-handling. - Restrict `Collections.toMap` value-type arguments to non-nullable types. - ...in J2KT, following what [we'd found in JSpecify research](jspecify/jdk@15eda89) - Fix `Collections.toMap` to remove the key in question when `mergeFunction` returns `null`. - ...in J2KT - ...in J2CL - Use `@NonNull` / `& Any` in a few places in `Map.merge` and `Map.computeIfPresent`. - ...in J2KT - ...in Guava `Map` implementations, even though we don't yet include `@NonNull` annotations in the JDK APIs that we build Guava against. (See post-submit discussion on cl/559605577.) - Use `@Nullable` (to match the existing Kotlin `?` types) in the return types of `Map.computeIfPresent` and `Map.compute`. - ...in J2KT - Test a bunch of this. Note that the test for `mergeFunction` has to work around an overly restricted `toMap` signature that J2KT inherited from JSpecify. As discussed in a code comment there, this is fundamentally the same issue as we have in Guava with `ImmutableMap.toImmutableMap`, which is discussed as part of #6824. RELNOTES=n/a PiperOrigin-RevId: 580659517
…fix `Collectors.toMap` null-handling. - Restrict `Collections.toMap` value-type arguments to non-nullable types. - ...in J2KT, following what [we'd found in JSpecify research](jspecify/jdk@15eda89) - Fix `Collections.toMap` to remove the key in question when `mergeFunction` returns `null`. - ...in J2KT - ...in J2CL - Use `@NonNull` / `& Any` in a few places in `Map.merge` and `Map.computeIfPresent`. - ...in J2KT - ...in Guava `Map` implementations, even though we don't yet include `@NonNull` annotations in the JDK APIs that we build Guava against. (See post-submit discussion on cl/559605577.) - Use `@Nullable` (to match the existing Kotlin `?` types) in the return types of `Map.computeIfPresent` and `Map.compute`. - ...in J2KT - Test a bunch of this. Note that the test for `mergeFunction` has to work around an overly restricted `toMap` signature that J2KT inherited from JSpecify. As discussed in a code comment there, this is fundamentally the same issue as we have in Guava with `ImmutableMap.toImmutableMap`, which is discussed as part of #6824. RELNOTES=n/a PiperOrigin-RevId: 580659517
…fix `Collectors.toMap` null-handling. - Restrict `Collections.toMap` value-type arguments to non-nullable types. - ...in J2KT, following what [we'd found in JSpecify research](jspecify/jdk@15eda89) - Fix `Collections.toMap` to remove the key in question when `mergeFunction` returns `null`. - ...in J2KT - ...in J2CL - Use `@NonNull` / `& Any` in a few places in `Map.merge` and `Map.computeIfPresent`. - ...in J2KT - ...in Guava `Map` implementations, even though we don't yet include `@NonNull` annotations in the JDK APIs that we build Guava against. (See post-submit discussion on cl/559605577. But I've taken the shortcut of not waiting for the JDK APIs.) - Use `@Nullable` (to match the existing Kotlin `?` types) in the return types of `Map.computeIfPresent` and `Map.compute`. - ...in J2KT - Test a bunch of this. Note that the test for `mergeFunction` has to work around an overly restricted `toMap` signature that J2KT inherited from JSpecify. As discussed in a code comment there, this is fundamentally the same issue as we have in Guava with `ImmutableMap.toImmutableMap`, which is discussed as part of #6824. RELNOTES=n/a PiperOrigin-RevId: 580659517
…fix `Collectors.toMap` null-handling. - Restrict `Collections.toMap` value-type arguments to non-nullable types. - ...in J2KT, following what [we'd found in JSpecify research](jspecify/jdk@15eda89) - Fix `Collections.toMap` to remove the key in question when `mergeFunction` returns `null`. - ...in J2KT - ...in J2CL - Use `@NonNull` / `& Any` in a few places in `Map.merge` and `Map.computeIfPresent`. - ...in J2KT - ...in Guava `Map` implementations, even though we don't yet include `@NonNull` annotations in the JDK APIs that we build Guava against. (See post-submit discussion on cl/559605577. But I've taken the shortcut of not waiting for the JDK APIs.) - Use `@Nullable` (to match the existing Kotlin `?` types) in the return types of `Map.computeIfPresent` and `Map.compute`. - ...in J2KT - Test a bunch of this. Note that the test for `mergeFunction` has to work around an overly restricted `toMap` signature that J2KT inherited from JSpecify. As discussed in a code comment there, this is fundamentally the same issue as we have in Guava with `ImmutableMap.toImmutableMap`, which is discussed as part of google/guava#6824. PiperOrigin-RevId: 611445633
…fix `Collectors.toMap` null-handling. - Restrict `Collections.toMap` value-type arguments to non-nullable types. - ...in J2KT, following what [we'd found in JSpecify research](jspecify/jdk@15eda89) - Fix `Collections.toMap` to remove the key in question when `mergeFunction` returns `null`. - ...in J2KT - ...in J2CL - Use `@NonNull` / `& Any` in a few places in `Map.merge` and `Map.computeIfPresent`. - ...in J2KT - ...in Guava `Map` implementations, even though we don't yet include `@NonNull` annotations in the JDK APIs that we build Guava against. (See post-submit discussion on <unknown commit>. But I've taken the shortcut of not waiting for the JDK APIs.) - Use `@Nullable` (to match the existing Kotlin `?` types) in the return types of `Map.computeIfPresent` and `Map.compute`. - ...in J2KT - Test a bunch of this. Note that the test for `mergeFunction` has to work around an overly restricted `toMap` signature that J2KT inherited from JSpecify. As discussed in a code comment there, this is fundamentally the same issue as we have in Guava with `ImmutableMap.toImmutableMap`, which is discussed as part of google/guava#6824. PiperOrigin-RevId: 611445633
…fix `Collectors.toMap` null-handling. - Restrict `Collections.toMap` value-type arguments to non-nullable types. - ...in J2KT, following what [we'd found in JSpecify research](jspecify/jdk@15eda89) - Fix `Collections.toMap` to remove the key in question when `mergeFunction` returns `null`. - ...in J2KT - ...in J2CL - Use `@NonNull` / `& Any` in a few places in `Map.merge` and `Map.computeIfPresent`. - ...in J2KT - ...in Guava `Map` implementations, even though we don't yet include `@NonNull` annotations in the JDK APIs that we build Guava against. (See post-submit discussion on cl/559605577. But I've taken the shortcut of not waiting for the JDK APIs.) - Use `@Nullable` (to match the existing Kotlin `?` types) in the return types of `Map.computeIfPresent` and `Map.compute`. - ...in J2KT - Test a bunch of this. Note that the test for `mergeFunction` has to work around an overly restricted `toMap` signature that J2KT inherited from JSpecify. As discussed in a code comment there, this is fundamentally the same issue as we have in Guava with `ImmutableMap.toImmutableMap`, which is discussed as part of #6824. RELNOTES=n/a PiperOrigin-RevId: 611445633
Looking at this again after a teammate flagged it in a followup to #7077, I think the problem with the Kotlin experiment is that I'm changing Currently, the blocker to doing anything here remains that we're on an older version of the Checker Framework with type-inference problems. If that problem goes away, then I should try again with a corresponding change to the |
…precondition checks. It's not that we're not going to make such calls illegal, I promise :) I mean, we certainly aren't going to _in general_, but I am tempted for `com.google.common`, as discussed on cl/372346107 :) (It would have caught the problem of cl/612591080!) I'm testing what would happen if we did do it for `com.google.common` in case it shakes out any more bugs. It does reveal that I didn't complete the cleanup of cl/612591080. And it reveals a few places where we'd normally use `requireNonNull`, since the checks aren't "preconditions" in the sense of "the caller did something wrong" (from cl/15376243 and cl/526930990). I've made those changes. (I would have made some more changes if I had tried to address more of `com.google.common`. But I stuck to the "main" packages, and I didn't even fix enough errors to see full results.) Honestly, the more interesting thing that this exercise revealed was that there are more cases in which I'm especially sympathetic to calling `checkNotNull` on nullable values: - `DummyProxy` is making an `InvocationHandler` perform automatic precondition tests based on annotations on the interface it's implementing. - `EqualsTester` and Truth have permissive signatures because they're test utilities, as documented in cl/578260904 and discussed during the Truth CLs. And the yet more interesting thing that it revealed is that we may want to use `@NonNull` here in the future, similar to what we've discussed in #6824. RELNOTES=n/a PiperOrigin-RevId: 612937549
…precondition checks. It's not that we're not going to make such calls illegal, I promise :) I mean, we certainly aren't going to _in general_, but I am tempted for `com.google.common`, as discussed on cl/372346107 :) (It would have caught the problem of cl/612591080!) I'm testing what would happen if we did do it for `com.google.common` in case it shakes out any more bugs. It does reveal that I didn't complete the cleanup of cl/612591080. And it reveals a few places where we'd normally use `requireNonNull`, since the checks aren't "preconditions" in the sense of "the caller did something wrong" (from cl/15376243 and cl/526930990). I've made those changes. (I would have made some more changes if I had tried to address more of `com.google.common`. But I stuck to the "main" packages, and I didn't even fix enough errors to see full results.) Honestly, the more interesting thing that this exercise revealed was that there are more cases in which I'm especially sympathetic to calling `checkNotNull` on nullable values: - `DummyProxy` is making an `InvocationHandler` perform automatic precondition tests based on annotations on the interface it's implementing. - `EqualsTester` and Truth have permissive signatures because they're test utilities, as documented in cl/578260904 and discussed during the Truth CLs. And the yet more interesting thing that it revealed is that we may want to use `@NonNull` here in the future, similar to what we've discussed in #6824. RELNOTES=n/a PiperOrigin-RevId: 614074533
…precondition checks. It's not that we're not going to make such calls illegal, I promise :) I mean, we certainly aren't going to _in general_, but I am tempted for `com.google.common`, as discussed on cl/372346107 :) (It would have caught the problem of cl/612591080!) I'm testing what would happen if we did do it for `com.google.common` in case it shakes out any more bugs. It does reveal that I didn't complete the cleanup of cl/612591080. And it reveals a few places where we'd normally use `requireNonNull`, since the checks aren't "preconditions" in the sense of "the caller did something wrong" (from cl/15376243 and cl/526930990). I've made those changes. (I would have made some more changes if I had tried to address more of `com.google.common`. But I stuck to the "main" packages, and I didn't even fix enough errors to see full results.) Honestly, the more interesting thing that this exercise revealed was that there are more cases in which I'm especially sympathetic to calling `checkNotNull` on nullable values: - `DummyProxy` is making an `InvocationHandler` perform automatic precondition tests based on annotations on the interface it's implementing. - `EqualsTester` and Truth have permissive signatures because they're test utilities, as documented in cl/578260904 and discussed during the Truth CLs. And the yet more interesting thing that it revealed is that we may want to use `@NonNull` here in the future, similar to what we've discussed in google#6824. RELNOTES=n/a PiperOrigin-RevId: 614074533
API(s)
How do you want it to be improved?
It's not entirely clear whether
mergeFunction
is permitted to returnnull
or not.According to type annotations it's not (
T
extends@Nullable Object
, butK
andV
don't). Current implementation however (com.google.common.collect.CollectCollectors.toImmutableMap(Function, Function, BinaryOperator)
complies withMap.merge(K, V, BiFunction)
's description, that is, it removes value from temporalLinkedHashMap
if the merge result isnull
.(BTW note that
Map.merge
takesBiFunction
instead ofBinaryOperator
; in case of Guava, having "just one"V
forces both parameters and return type to be annotated together).Why do we need it to be improved?
Please either explicitly allow nulls or change implementation to throw NPE.
Example
Current Behavior
This results in empty
map
and no exceptions, which may, or may not, be fine.Desired Behavior
Maybe such code should lead to NPE, although allowing nulls appears to be less surprising for people assuming that merging should behave exactly as described in
Map.merge
orCollectors.toMap(Function, Function, BinaryOperator)
.Concrete Use Cases
I'm unable to provide concrete use cases. I was implementing #6822 for my own purposes and it struck me that I can't really choose if I should ban nulls entirely or not. I was willing to align with
ImmutableMap.toImmutableMap(Function, Function, BinaryOperator)
but found that what I'm looking for is unspecified.Checklist
I agree to follow the code of conduct.
I have read and understood the contribution guidelines.
I have read and understood Guava's philosophy, and I strongly believe that this proposal aligns with it.
The text was updated successfully, but these errors were encountered: