Can we have result value convention for fast error handling?

[tkf]

The current error handling landscape of Julia is very far from how good it can be. On one hand, we have throw and try-catch at the language level but the compiler does not optimize it well. On the other hand, there are various packages ¹ that provide a framework for reporting errors by returning a special type (like Rust's Result and Haskell's Either). However, doing this in an external package has the downside that we can't provide efficient error handling for methods defined in Base that require touching the internals.

The main question is: Can we have this "throw"-by-returning convention in Base?

More specifically, can we have a minimal set of types such as

struct Ok{T}
    value::T
end

struct Err{E}
    value::E
end

and minimal APIs around them in Base? For example, it let us implement an API like tryget(dict::Dict{K,V}, key) -> result::Union{Ok{V},Err{KeyError}}.

If we have minimal types for result value convention in Base that is used across the entire ecosystem, people can independently experiment with tooling (e.g., macros and functions) around it. Once these toolings have matured, we can consider integrating it in Base or in the language.

Concern: two modes of error handlings

Clearly, the main concern with this approach is that having two modes of error handling is rather ugly. It is reasonable to argue that it would be more beneficial to improve the error handling in a way more integrated to current exception handling (e.g., julep: "chain of custody" error handling) and improve the compiler to optimize the error handling. However, it may be beneficial to have two modes of error handling clearly distinguished by the entity to which the error is reported. The idea is to separate the kinds of errors into two cases:

Unrecoverable error: This is an error due to a mistake made by the programmer. It is unrecoverable by the caller program of the API since the invariance expected inside the callee API has been violated. It should be reported to the human programmer.

Recoverable error: This is an error that is clearly documented that it can happen. The caller may choose to handle this situation; i.e., it should be reported to the caller program.

The requirements for the mechanism of these kinds of errors are different. Since an unrecoverable error (unsatisfied precondition, violation of internal invariance, etc.) is a bug, it is important to record as much information as possible for reporting it. The performance impact is negligible because the program is likely to wait for human intervention after the error happens or invoke an expensive error logging routine. Thus, throwing an exception for this is adequate. Since handling this type of error typically happens at the level of UI applications such as REPL and IDE, the catch-all syntax we currently have is consistent with this notion of error.

On the other hand, the recoverable error needs to be as efficient as possible since it can occur in tight loops where the erroneous situation is somewhat expected (e.g., singular matrix detected in a linear solve with a small static matrix; contention detected in a nonblocking algorithm). Given excellent support of small Union in the compiler, it seems that the "throw"-by-returning convention suggested above is a good candidate for this kind of error handling. Furthermore, since the caller needs to know what errors are possible, the possible set of errors is captured in the API description. It simplifies our practice in API specification since we can simply document what is returned and not what can be thrown.

Multiple error handlings in other languages

It is not uncommon to have two modes of error handling. For example, Rust has panic for the (mostly) unrecoverable mode of error handling. This can occur, for example, when there is a bug when the programmer used Result::unwarp to unwrap a value that is assumed to be an Ok while in fact there is an error in the logic. A similar mode of error reporting of unrecoverable logic errors has been suggested for C++ ² as well.

Go also has panic besides its result, err = Api(...) return value convention. Like Rust, it can be used for unrecoverable errors. But Go's panic is subtlety different from Rust in that it could be used as a recoverable/expected error within a package. However, the error reporting across API boundaries does not typically use panic. Thus, it's another example where the error handling at API boundary is distinct from unrecoverable and internal errors.

These are some relevant examples that I remember but there are likely more examples like them. It's also sometimes unclear what it means to have "multiple modes." For example, Java and Python use exception type hierarchy to distinguish unrecoverable and recoverable errors. But they are more uniformly handled at the syntax level.

I don't want to say "it's Ok since others do it." But, just as a sanity check, it is good to know that having multiple modes of error handling is an accepted solution in other languages.

Other properties and implementation strategies of Ok/Err approach

I discussed other upsides and the implementation approaches of the "Ok/Err approach" in https://github.com/tkf/Try.jl such as

1. Should the result type be a Union or a struct? Or a hybrid?
2. Opt-in stack trace for Err
3. How it opens a more robust and minimalistic alternative approach to the trait-based feature detection

Julia needs a fresh take on error handling

Julia provides the performance of static language with a highly interactive approach to programming. This human-in-the-loop approach to programming requires a careful design in exception handling and it explains why it takes time to arrive at a good solution. However, now that Julia is in 1.x phase, I think it is time to provide some working solution. I think the approach based on Union{Ok,Err} supports both the performance requirement and high-level programming style of Julia.

Footnotes

1. There are packages such as jakobnissen/ErrorTypes.jl, iamed2/ResultTypes.jl, KristofferC/Expect.jl, and tkf/Try.jl. There are recent discussion in [ANN] ErrorTypes.jl - Rust-like safe errors in Julia - Package Announcements / Package announcements - JuliaLang and Try.jl - JuliaLang - Zulip. ↩

2. See "4.2 Proposed cleanup: Don’t report logic errors using exceptions" in Zero-overhead deterministic exceptions: Throwing values (PDF) or CppCon 2019 talk (44:45). It is based on Joe Duffy - The Error Model. ↩

[nalimilan]

This is appealing, but concretely how would you suggest implementing this in Base? We cannot change functions to return Union{Ok,Err} and we don't want to create a try* variant of each function either. Maybe some kind of macro like @try f(x) could allow turning exceptions into an Err return value (with some mechanism to avoid any performance impact)?

[ericphanson]

Hm, I don’t really find that convincing but I also don’t use a debugger much. They seem basically identical from a semantics perspective. And it seems much more ergonomic to compose functions rather than trying to have a convention of composition-via-first-argument.

If the concern is that just receiving the error value and generating a stacktrace doesn’t provide enough info and one wants access to local variables within the function too when in a debugging context, maybe the error value should also include whatever info is useful for debugging?

Or you could @inline the function at the call site to get access to them (I think?).

[tkf]

First of all, the stack trace is the most important point. The comment on debugger is a secondary point and pretty much implied by that CPS (in Julia) retains stack information.

maybe the error value should also include whatever info is useful for debugging?

A bug occurs because the programmer doesn't understand something. It seems contradictory to hope that the programmer understands the program enough for enumerating the information sufficient for debugging the bug caused by the misunderstanding of the program in question.

Furthermore, the set of variables is not enough to identify the state of the program. You need to know the location of the program where these variables are defined. Note that an error can happen at multiple locations.

Or you could @inline the function at the call site to get access to them (I think?).

Why? @inline is a compiler hint that is not supposed to change the semantics of the program. I think it'd be problematic if it changes the scope and lifetime of variables.

[ericphanson]

Ok, I think I get it; I had misunderstood before. Not really used to thinking about function boundaries as meaningful besides as function barriers and APIs/code organization. As an example,

julia> myget(obj, key) = haskey(obj, key) ? obj[key] : KeyError(key)
myget (generic function with 1 method)

julia> myget2(continuation, obj, key) = haskey(obj, key) ? continuation(obj[key]) : continuation(KeyError(key))
myget2 (generic function with 1 method)

julia> mycontinuation(x::Exception) = throw(x)
mycontinuation (generic function with 1 method)

julia> mycontinuation(x) = x
mycontinuation (generic function with 2 methods)

julia> function accesses_dict1(d)
           x = 1+1 # do some stuff
           y = mycontinuation(myget(d, "c"))
           return y + x
       end
accesses_dict1 (generic function with 1 method)

julia> function accesses_dict2(d)
           x = 1+1 # do some stuff
           y = myget2(mycontinuation, d, "c")
           return y + x
       end
accesses_dict2 (generic function with 1 method)

julia> d = Dict("a" => 1, "b" => 2)
Dict{String, Int64} with 2 entries:
  "b" => 2
  "a" => 1

julia> accesses_dict1(d)
ERROR: KeyError: key "c" not found
Stacktrace:
 [1] mycontinuation(x::KeyError)
   @ Main ./REPL[17]:1
 [2] accesses_dict1(d::Dict{String, Int64})
   @ Main ./REPL[19]:3
 [3] top-level scope
   @ REPL[22]:1

julia> accesses_dict2(d)
ERROR: KeyError: key "c" not found
Stacktrace:
 [1] mycontinuation(x::KeyError)
   @ Main ./REPL[17]:1
 [2] myget2
   @ ./REPL[16]:1 [inlined]
 [3] accesses_dict2(d::Dict{String, Int64})
   @ Main ./REPL[20]:3
 [4] top-level scope
   @ REPL[23]:1

You're saying that accesses_dict2 is better because the error occurs in myget2 instead of just in the outer function, so if you were in a debugger and hit the error, you'd have access to the local variables inside myget2, which is a consequence of the stacktrace including myget2 instead of that function not being involved in the error at all.

Ok, I think I see the benefit, and if we passed that Exception around through many layers of handlers before throwing it, we'd want to be able to have that whole sequence of layers in our stacktrace. I still don't really like it though 😄.

[tkf]

Thanks for the illustrative example. Yes, it's tricky because designing an error handling mechanism is not all about APIs but also about how to show the implementation details. You need to look at implementation details to fix bugs.

I also agree that this is ugly. It is straightforward to add a syntax sugar using macros to automate this (i.e., pretty much like async/await keywords in other languages) but this is tricky to cover all cases because of closures. Furthermore, if we become slightly eager (why not) and make it composable with other kinds of effects, that's significantly more challenging.

[tkf]

Playing with the (semi-)CPS error handling a bit, I noticed that it requires nested closures for composing functions (unless all failable functions are called at tail positions... which is obvious in retrospect).

Consider

f(x) = g(h(x))

where h may fail.

With CPS error handling, it would be lowered to

function f(k, x)
    function k2(ans)
        if iserr(ans)
            Err(k(ans))
        else
            ans
        end
    end
    ans = h(k2, x)
    if iserr(ans)
        return unwarp_err(ans)
    end
    return g(k, unwrap(ans))
end

When failable functions appear at non-tail ¹ positions like h above in a call stack, the continuation will accumulate the nested closure like k2. So, it can have a harsh performance cliff since the optimizable case does have a very nice property (call-site specific specialization of the continuation).

Aside: We also have the same issue even if in the "true" CPS:

function f(k, x)
    function k2(ans)
        if iserr(ans)
            k(ans)
        else
            g(k, unwrap(ans))
        end
    end
    h(k2, x)
end

Footnotes

1. On the other hand, if h is known to not fail and only g is failable, we can lower f(x) = g(h(x)) to f(k, x) = g(k, h(x)) and thus no nesting is required. ↩

[jakobnissen]

Some general remarks on this topic:

Two kinds of error is not a problem

The error handling system proposed here is quite simple. In fact, it is so simple that there is barely any system that needs to be learned:
No extra syntax to learn, no extra compiler magic to implement, no extra runtime to slow code down. The system consists only of returning values and relying on Julia to handle the resulting union types effectively. All Julia programmers already know how to return and accept Union types.

It is true that the proposal will create a choice for the programmer as to whether a given function should throw or return error values. However, programmers already have to make the conceptual distinction between recoverable and unrecoverable errors when writing code, no matter the language. In my experience, the choice is not too hard, and in the rare cases the user wants both, it's trivially easy to implement a throwing version given an error-returning version.

To prove the point: Is anyone confused by the fact that Base already returns error values for find*, tryparse and match? Would Base have been better if these functions threw errors to be consistent with the rest of Base's error model? Of course not!

Differences between Base's approach and this proposal

Okay, so e.g. findfirst already uses error types. What are the differences between Base's approach of returning Union{T, Nothing} and this proposal of returning Union{Ok{O}, Err{E}}?

First, this proposal allows arbitrary error types. I think it's obvious that this is preferrable - after all, some times there is actual error value that needs to be considered. We can always later add better ergonomics for Err{Nothing} for the cases where there is no particular error information to be returned: ErrorTypes.jl does this, and Rust does something similar (namely, distinguish between Result and Option).

Second, Nothing is, unfortunately, not much of an error type. Using methodswith and apropos("=nothing") reveals approx 40 methods in Base that takes nothing as a meaningful value. The implication is that a user cannot actually be confident that an unhandled error value is not mistakenly accepted as a legitimate value. One takeaway from this unfortunate situation is that we should be adamant in the documentation that users should avoid creating methods that take Err, if possible.

Third, there is the issue of safety:

Safety versis convenience

One major advantage of Union{T, Nothing} is that the user may ignore the possibility of getting a non-T. When errors are rare, or known to not happen, this allows the user to avoid having to deal with errors at all, which cuts down on boilerplate and keeps Julia code lean.
In my opinion, the terseness and expressiveness of Julia is a strength of the language we should not undermine. Think of the pain it would be having to unwrap Ok values would be when coding directly in the REPL.

On the other hand, the very fact that the user is able to ignore possible errors also makes it bug-prone, which leads to unstable code in the wild. Let's be real: Users often don't remember edge cases, or even understand how foreign APIs can fail. Having your code return the expected T in testing, only to return nothing in production is not fun - if the code returned Ok{T} instead, it would be impossible to miss the fact that the code could fail at that point.

So we have a direct conflict: Wrapping T in Ok makes it less expressive, but safer. Not wrapping makes it more convenient, but bug-prone.

Can we have the best of both worlds?

The only possibility I can think of is to implement both for a large set of failable Base functions. We should bikeshed a little over the signature of these failable functions - here are some ideas on how that could look for foo(a, b, c):

• tryfoo(a, b, c)
• Base.try(foo, a, b, c)
• foo(Err, a, b, c)
• SafeBase.foo(a, b, c)

Having two methods for many of Bases methods requires a LOT of extra methods in Base. However, it's not nearly as bad as it sounds: The failable function can be the only implementation, and the throwing versions can simply call the failable one and unwrap. See also #43149.

Having both versions would enable prototyping and exploring while simply ignoring the possibility of errors, like we can do now, while also being able to transition the existing prototype code to more safe, well behaved code. This plays into Julia's existing and well-known strength of being a language good for both prototyping and performance sensitive production code.

In essence, I hope that on the issue of correctness, Julia could bridge the "two language problem" between error prone but expressive scripting languages and rigid static languages. Julia is already in a particularly good position to do this, since it somewhat uniquely is both dynamic and expressive, and also have an optimizing compiler that enables static analysis.

Struct type vs union type

I made a list of pros and cons of the two representations of error values here: jakobnissen/ErrorTypes.jl#5
There may be more I have missed - but I'm of the opinion that union types are the superior choice. Despite most error type packages using struct types, I think the superiority of union types is not actually that controversial in Julia. After all, unions are what Base already uses.

Where to go from here?

Here is a possible plan of action to implement this in Base without breaking backwards compatibility, Comments are welcome.

• Base should implement Ok and Err in a (experimental?) module SafeBase
• For a set of common recoverable error functions, begin implementing them in SafeBase
  • e.g. SafeBase.getindex, SafeBase.maximum, ...
• Derive the Base implementations from SafeBase by simply calling and unwrapping
  • Note: This will lead to some inconvenience when reading the source code, as the user will be directed first to the wrapper function which simply calls the real implementation
  • The blog post linked by Takafumi claimed that error types are inferior performance wise to throwing, because it increases the size of the types returned by non-inlined functions by one byte. The blog claims the effect is small, but ubiquitous and measurable, but I'm somewhat sceptical that this really matters, practically speaking.

TL;DR

• We want Union{Ok{T}, Err{E}}, not Union{T, Err{E}}.
• We should implement Ok and Err in a new module under Base: SafeBase
• We should discourage people from implementing methods with Err except when explicitly developing error handling code.
• We should then derive a small set of failable Base functions from their underlying implementations in SafeBase

[tkf]

• The blog post linked by Takafumi claimed that error types are inferior performance wise to throwing

I think that part of the blog post can be safely ignored here since I'd assume the penalty of throw and catch is much larger in Julia because inference does not handle them.

[vchuravy]

One thing to keep in mind is that we used to have Optional{T} and one of the reasons we removed it was that generally the ergonomics were pretty poor. Ok{T} feels similar. Now of course Union{T, Errror} makes checking a bit weirder, and there is no way to "force" handling. LLVM Expected<T> has nice'ish semantics since you must handle it, e.g. you need to discard the Expected explicitly with cantFail since otherwise in debug mode it will complain that the error was never checked.

auto value = ...;
if (auto err = value.takeError()) {
   // handle error
}

// Use value by `*value`

I don't think we can emulate that API in Julia without unacceptable performance penalty. Go's approach is rather interesting since

value, err = f()

is already something "normal" in Julia.

[tkf]

One thing to keep in mind is that we used to have Optional{T}

Can you point to the old code? I tried git log -G'Optional\{' and git log -S'Optional{' don't show me anything. Also, how was it different from Union{Nothing,Some{T}}?

LLVM Expected<T> has nice'ish semantics since you must handle

IIRC there was discussion on this in CppCon 2018: Andrei Alexandrescu “Expect the expected” - YouTube but this design is not ported to std::expected. Indeed, it's not mentioned in the destructor specification http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p0323r11.html#expected.object.dtor

I agree it's kinda nice design but it looks like that even (because of?) within C++ community there's an agreement. Given that C++ is more leaning towards low-level engineering than Julia, I don't think that such strict enforcement works well in Julia.

I don't think we can emulate that API in Julia without unacceptable performance penalty.

If we only had linear types...

Go's approach

I think it'd have a return type of Union{Pair{T,Nothing},Pair{Nothing,E}} ¹ in the general case where you can't come up with the default T value. (Go does not have this problem since IIRC everything is nil/zero-able and possibly relying on the strong convention that "zero value" is useful.) So, it seems roughly isomorphic to Union{Ok{T},Err{E}}. Why is Go's approach better? Is it because it's easier to destructure? If so, we can add result[] for unwrapping Ok and throwing Err. It'd be actually terser as a syntax: tryget(dict, key)[]. Maybe this is not the best API but the point is that there are a lot flexibility in the caller-side API.

Also, in general, I'd consider that it's more idiomatic to have a specific type for a specific usage so that users can dispatch on it if they want to.

Footnotes

1. Using Pair instead of Tuple for simulating invariant tuple ↩

[adkabo]

Can you point to the old code? I tried git log -G'Optional{' and git log -S'Optional{' don't show me anything. Also, how was it different from Union{Nothing,Some{T}}?

This sounds like the Nullable{T} removed in #23642.

[tkf]

Thanks! Linking the old docs: https://docs.julialang.org/en/v0.6/stdlib/base/#Nullables-1

So, I discussed something similar in https://github.com/tkf/Try.jl#discussion but I think Nullable seemed to have the problem of other "static" result type libraries have. Instead, what I'm suggesting here is friendlier to Julia's "dynamic language" semantics; i.e., it's closer to Union{Some{T},Nothing} than Nullable{T}.

[tkf]

I opened a concrete RFC #45080

[ti-s]

I don't have deep knowledge about the compiler or the type system so I might miss something obvious, but was it already considered to make Err not a subtype of Any? That seems to have some advantages.

1. Errors would not be propagated by accident because almost no function would accept error values with a few exceptions like isa. Therefore, programs that don't handle errors would fail fast, close to where the error occurred.
2. As setindex! and similar would not accept error values, collections would not return errors on success and wrapping return values in Ok would not be necessary. Therefore, the happy path stays as convenient as with exceptions. If a caller does want to propagate an error or put it into a collection, it would need to wrap it first with, e.g.,

struct WrappedError{T<:Err}
    err::T
end
wrap(x) = x
wrap(x::Err) = WrappedError(x)

3. Static linters should be able to warn about unhandled errors in many cases. Maybe it would be even possible to always infer at least Union{Any, Err} when a function could return an error.

Unfortunately, one couldn't experiment with this approach in a package.

[tkf]

All types are subtype of Any. So, we can't do this.

But, for the sake of argument, we can refine the idea; i.e., introduce "ReallyAny" s.t. Any <: ReallyAny and Err <: ReallyAny. Then any type parameter T has T <: Any as an implicit bound unless opt-in T <: ReallyAny or T <: Err. Furthermore, a method definition without type bound is equivalent to ::Any bound; i.e., f(x) = x is equivalent to f(x::Any) = x.

However, you would quickly realize that something like Ok is required if you try writing a program generic over "success" and "failure" values. For example, it's not possible to write getfirst(xs::Tuple) -> ans::Union{Any,Err} with

• getfirst((getfirst(()),)) === getfirst(())
• getfirst((x,)) !== getfirst(()) for any x::ReallyAny

Note also that "don't allow Err values in a tuple" is not a solution since a tuple is a language construct. For example, f(args::ReallyAny...) = args creates a tuple of ReallyAny values.

[ti-s]

Feel free to ignore my armchair design comments, but why is it necessary to allow Err values in tuples? Before putting an Err into a tuple, one would need to wrap it first, so getfirst((x,)) !== getfirst(()) would hold. For example, the following function would throw an exception if getfirst actually returns an Err:

function foo(x)
    return 1, getfirst(x)

Instead, one would need to write

function foo(x)
    # This might eventually cause a `MethodError` if not checked for `WrappedError` somewhere else
    return 1, wrap(getfirst(x))

or

function foo(x)
    first = getfirst(x)
    if first isa Err:
        # Force the direct caller to handle the error
        return first
    return 1, first

Similarly, f(args::ReallyAny...) = args would either be disallowed or error if called with an Err value.

What did you mean with the first property? It doesn't seem to hold in Try.jl as well:

julia> getfirst(x) = trygetindex(x, 1)
getfirst (generic function with 1 method)

julia> getfirst(())
Try.Err: BoundsError: attempt to access Tuple{} at index [1]

julia> getfirst((getfirst(()),))
Try.Ok: Try.Err: BoundsError: attempt to access Tuple{} at index [1]

julia> getfirst((getfirst(()),)) === getfirst(())
false