Keyword arguments affect methods dispatch

[yuyichao]

So this is an issue that I find after figuring out how the keyword arguments are currently implemented in Julia. It will probably not be an issue anymore if #2773 is implemented.

The document on methods says

Methods are dispatched based only on positional arguments, with keyword arguments processed after the matching method is identified.

However, method dispatch actually behaves differently when doing a function call with or without keyword arguments. i.e.

julia> function f(::Integer)
       2
       end
f (generic function with 1 method)

julia> function f(::Number; kw...)
       1
       end
f (generic function with 2 methods)

julia> f(1)
2

julia> f(1; a = 2)
1

What happens here is that f.env.kwsorter only has one method defined and therefore when calling with keyword argument, f(::Integer) does not participate in method dispatch.

IMHO, there are several possible ways to fix it,

1. Fix the document to include this behavior. This should be the easiest fix but will probably make the whole keyword argument/optional argument/multiple dispatch more confusing especially for someone who does not know how it all works behind the scene. (It's already quite confusing/surprising that anonymous function does not support keyword argument for someone (like me) that expects python-like keyword argument implementation.)

2. Having an entry (that just throw an error) in env.kwsorter even for methods that does not take keyword arguments. This can also avoid the following confusing abuse of overriding method

   julia> function f(::Number; kw...)
          1
          end
   f (generic function with 1 method)
   
   julia> function f(::Number)
          2
          end
   f (generic function with 1 method)
   
   julia> f(1)
   2
   
   julia> f(1; a = 2)
   1

   This is probably the easiest way to fix the code and is consistent with the best long term behavior.

3. Fix Keyword arguments for anonymous functions #2773 and let the method themselves handle keyword argument dispatch. Since Keyword arguments for anonymous functions #2773 is on 1.0 milestone, hopefully this will be eventually be implemented.

[vtjnash]

see also #4469 (comment)

[JeffBezanson]

I think we should add keyword arguments to the calling convention, replacing the existing kwsorter-based implementation.

Currently, defining a keyword method actually defines three methods:

1. A method with a hidden name accepting all positional and keyword arguments, containing the full method code.
2. A method for the function actually being defined, accepting only positional arguments and passing default keyword arg values to method (1).
3. A method of the function's kwsorter function that does sorting and calls method (1).

My proposal merges methods (2) and (3):

1. Keep method (1).
2. Add a method like the existing method (2), but that also contains an Expr(:kwargs) to access a keyword argument container passed as a pointer via the calling convention, and then does sorting.
3. jl_call_method_internal and its codegen throw an error if no kwarg-accepting entry point is available for a given kwarg call.
4. Add Expr(:kwcall, ...) that has a slot for a keyword arg container.
5. When inlining runs on such an expression, it substitutes the passed container for Expr(:kwargs) and will then be able to specialize the sorting code.

It would be possible to get the performance aspect of this just by passing a different kind of container to our existing kwsorter, and changing its lowering to make it easier to specialize. However our current implementation is also very complex, and we'd still need a separate fix for the semantic issue described here. Having 2 methods instead of 3 will also make reflection easier. So I think this redesign should be considered.

[martinholters]

This shouldn't have been closed, yet, I guess. Or did I miss something?

[JeffBezanson]

Will have to punt on this for 1.0.

[sbromberger]

Ref: https://discourse.julialang.org/t/fun-with-kwargs-methods/15711

[cossio]

Can this behavior be seen as a bug? Then fixing it is not a breaking change.

[Moelf]

I'm pretty sure this is a bug, where should one starts to fix this?

[LilithHafner]

A path to fixing this would be making a PR that fixes it and running nanosoldier to see how much folks depend on this strange behavior.

[Moelf]

A path to fixing this would be making a PR that fixes it

"in order to fix it you just fix it"? I think myself from 12 months ago was asking which file should I change to actually fix it, not the logistics after fixing it. There's a fair chance this is way above my ability grade, if anyone knows how and doesn't have time to elaborate, please just fix it, this seems pretty bad we didn't treat it with urgency for 9 years.

[LilithHafner]

Oh, lol, my bad. I was thinking about the social and ecosystem factors, not technical.

[Tortar]

This should be considered a correctness bug

[matthias314]

There are actually functions in Base that depend on this incorrect behavior: For example,

julia/base/bitarray.jl

Line 1724 in bee75f7

function all(B::BitArray)

does not support keyword arguments. That all(trues(8); dims=:) doesn't throw an error is because the generic method

julia/base/reducedim.jl

Line 991 in bee75f7

all(a::AbstractArray; dims=:) = _all(a, dims)

kicks in. Same for any, maximum and minimum withBitArray argument.