Map type annotations to source text

[timholy]

This is an early-stage effort to make Cthulhu "newbie-friendly" by printing type-annotations against the raw source code. Timing-wise, there are several factors that led me to tackle & submit this now:

• I'm submitting very early because I suspect @aviatesk has given this some thought and I didn't want to spend too much time on it before providing opportunities for discussion
• JuliaSyntax appears to be converging and partially solves Julep: Expr provenance when lowering JuliaLang/julia#31162
• Julia 1.9 seems likely to increase enthusiasm for hunting invalidations, which is currently tricky if you don't read lowered/type-inferred code
• (most proximally) a course I'm teaching will hit type-inference in a couple of weeks and it would be lovely to have something for the class to play with.
• overall, I regard this as one of the (relatively few) significant usability issues we have to cross before Julia becomes ready for "everyone"

First a small (and very imperfect) demo is in order (EDIT: the demo below is outdated, the most up-to-date demo is in this branch's README):

(After this there is "cruft" from the fact that this source view has not been fully integrated so that it supersedes the current infrastructure.)

To hint at the option to suppress type-stable types,

Here it's only being applied to the signature, since very little else is type-stable in this example.

One really big caveat: I'm not sure this can be done properly until JuliaSyntax also handles lowering. The ugliest part of this by far is matching parsed expressions to their type-inferred equivalents. I'd particularly appreciate hearing thoughts on this issue.

In the long term future, it might be nice to allow the cursor to traverse the source text and enter into particular calls from there, without needing to print the list of calls.

CC @c42f

[timholy]

Still very crude, but perhaps already starting to look promising. From

@descend optimize=false iswarn=true annotate_source=true hide_type_stable=true Base.var"#mapslices#193"(2,mapslices, sum,A)

(which on my version of Julia is the "main" call from mapslices(sum, A; dims=2) where A::Array{Int,3}),

[timholy]

This is looking far more successful than I expected. I believe it's working well enough that this PR is now heading for completion:

• for lines with unmatched calls (e.g., a line containing d -> d in dims, where the in call is visible in the parsed source but gets hidden in a different CodeInfo), print a comment indicating the number of unmatched calls on that line (prob use yellow warning color). I think this is good enough that the benefits this provides outweigh the risk that we'll confuse users by failing to mark calls that are the actual root of the type-instability.
• needs tests
• needs proper integration with Cthulhu. I'm planning to make this the default mode of Cthulhu (so get your complaints in soon 😉). People who use Cthulhu primarily to better understand the compiler can of course use Preferences to pick a different default mode.
• needs documentation

The one part that I think won't happen soon is in-source call navigation, as that involves fancy REPL features we currently lack. For now I think we'll have to live with the current SSA-based call list. We might at least be able to strip the SSAs and print line numbers which could be matched against line numbers printed alongside the source.

CC @pfitzseb in case there's any of this that should be done mindfully of VS Code.

[vchuravy]

I think this is a fabolous idea and effort. There are a few other low hanging UX efforts that could go well with this (like using Term.jl or something similar to implement paging for source code).

The bigger question here for me is. Should this be the default mode? I am also feel that this transform Cthulhu to something else entirely and it might be better called Cuddulu or FluffyCthulhu. I love the name Cthulhu but it was also in part choosen to convey the notion of "all hope is lost, but I will persist in my quest for knowledge", that surrender of sanity in pursuit of elder wisdom xD

[vchuravy]

The other question is, can we do a medium thing? I don't find it all to useful to just show the Any I would certainly want to know the types of the call that led to Any

[timholy]

We could certainly split it out. We'd need a common core that implements all the wonderful work that's been done to ensure that inference is handled correctly. Would Cthulhu-proper then be much more than a UI package?

I don't find it all to useful to just show the Any I would certainly want to know the types of the call that led to Any

At present, the source will show ::Any but the list of calls below (ones you can descend into) include the argtypes. Or you can dive in and see more detail. Long arg lists and complicated types of course are a bit troublesome (there may not be room on a single line to print all of them), in those cases descending would probably be the only option.

In the long run, though, I had imagined a "cursor" over the source code, and you can dive into calls simply by moving the cursor to the right call and hitting Enter. That would obscure information about the argtypes. Perhaps <space> could be configured to display the argtypes of the call?

[timholy]

I love the name Cthulhu but it was also in part choosen to convey the notion of "all hope is lost, but I will persist in my quest for knowledge", that surrender of sanity in pursuit of elder wisdom

At the risk of ruining the fun of a lovely joke, I wonder how many beginning coders might feel that way even about the "fluffy" mode this implements. "Why does Julia have types at all, it's so annoying?!?" That is, to sophisticated developers like you, the two/three/four/five-language problem that you need to be able to read and understand Julia+lowered Julia+type-inferred Julia+IRCode+LLVM+native code seems complex, but for complete newbies even the notion of tracing types through a program may be a level of complexity that just about breaks them, even when we try to help them as much as we can. If there's more than a grain of truth in that, then perhaps the name Cthulhu is still appropriate, considering the audiences.

"Cthulhu: think it's easy? Wait until you try to gain the next level of insight!"

[pfitzseb]

Since VS Code was brought up: julia-vscode/LanguageServer.jl#1077 is somewhat relevant.

As always, with the current design of the language server, there's a bit of a tension between wanting to get type info statically from the source code, but having a full REPL context getting us much further. JET.jl does have integration for "runtime diagnostics"; we could conceivably do something similar for Cthulhu as well.

[vchuravy]

Yeah I sometimes wonder if Cthulhu as a name is not just another barrier to entry. Hard to spell, hard to pronounce, comes from a fantasy canon with questionable pedigree... It captured very well my emotional state at the time of first draft, but it might not be the best name,

[timholy]

I do see your point. Splitting the fluffy version out would be very easy and provide an opportunity for a rename, reserving the scarier mode for deep analysis.

I think it is fitting & best to have you make the call, so just let me know when you've decided.

[c42f]

This looks cool! So great to see JuliaSyntax in use :)

Some of the parts you're trying to use here are very under-documented in JuliaSyntax, and the APIs are still under-developed (SyntaxNode and SourceFile at least). Glad (not surprised!) to see you figured it out :)

The highlighting you're doing here looks very similar to the way diagnostics are show in JuliaSyntax and it looks like you might have rewritten some of the same tools? Check out how diagnostics are printed at https://github.com/JuliaLang/JuliaSyntax.jl/blob/e57b80044e3a863924909ef51e1370c381109e58/src/diagnostics.jl#L44

We should share tools for pretty printing upstream in JuliaSyntax ❤️

[c42f]

This string indexing might be problematic for unicode? Especially the firstidx-1 which may not work for multi byte chars? JuliaSyntax.SourceFile might help here in particular getindex(source::SourceFile, rng::AbstractRange) since that's careful with string indices.

[c42f]

node.source is a SourceFile which might help here?

[c42f]

If you want to stringify the node exactly as it was in the original source, you can just use JuliaSyntax.sourcetext(calltok) ?

[timholy]

Thanks @c42f! Your timing is excellent; just this morning I was thinking that I needed to rework all the printing here. The idea of upstreaming anything possible to JuliaSyntax sounds like the right design.

In some of my local work (not yet pushed because it's a little too ugly still), I've started on the first point in my TODO list. Here's a screenshot of the current output:

As you can see, I've started adding some warnings that some calls couldn't be found in the type-inferred code. The printing of the line number is unlikely to be permanent, it's mostly for me to check whether I've gotten the line number right. (In case you're curious, for the cases here, some of them can't be found because type-inference determines that they will never run (given the input types the condition is false), others because the code is actually split out into an anonymous function (which is a separate CodeInfo), etc.) I added these warnings because there are going to be cases where we can't map a source expression to a type-inferred result, and it would be concerning if we thus missed the "true source" of a type-instability and users wasted time trying to fix a problem in the wrong location. I think these warnings at least ensure that we're confessing that there's some stuff we aren't fully reporting to them.

So one issue is that I'll need not only to print type-annotations after some SyntaxNodes but also detect when I've reached the end of the line, and then do some additional printing. Thus the printing tools in JuliaSyntax might need to allow some kind of check on whether you've encountered a \n while printing text, and if so interrupt with some different output. A character-stream iterator that would leave this code "in charge" of the output would accomplish that very nicely, I think.

[c42f]

warnings that some calls couldn't be found in the type-inferred code

Cool. Interspersing diagnostics is tricky. I so want to preserve the "shape of the original code" in pretty printing. But in plain text that conflicts with providing this extra diagnostic info. We discussed this back and forth a little in JuliaLang/JuliaSyntax.jl#150

I can't really see a way around some kind of annotations system which will break the shape of the original code. I had some ideas for fancier browser-based UI in JuliaLang/JuliaSyntax.jl#150 (still somewhat integrated with the terminal experience via ANSI escape code hyperlinks) but we'll still need something which works well in a plain terminal by default.

[timholy]

Thanks for the many improvements, folks!

[timholy]

OK, I think I've belatedly come up with a much better design. The idea is to orthogonalize the analysis (the matching between text & types) and the printing:

• create a variant of SyntaxNode that stores the inferred type of the return value of the node: this gives us the analog of type-inferred code but for source-text. Advantage: many different "consumers" could take advantage of the result and do whatever they want with it.
• define printing for such type-annotated SyntaxNodes primarily in JuliaSyntax or extensible functions with methods defined in a different package. Advantage: it becomes a minor extension of JuliaSyntax and does not need any back-and-forth.

Note, the "unmached calls" could in fact be handled by defining a special type Unmatched and assigning it for any type-annotation where the match was uncertain. We could, e.g., print it as a yellow ::U to indicate unmatched; it would be shorter and more informative than what I display above.

I welcome additional suggestions, so please share any that occur. But to me this seems obviously better than the approach I've taken here, so I'll probably get started on this tomorrow morning.

[timholy]

I think I am ready to move forward with a more "final" implementation. I think the key question is how to organize the codebases. @vchuravy, have you decided whether you'd prefer the UI part to be separate from Cthulhu?

If it should be separate, then I propose to create a package ("TypeBrowser"? "CodeTyped"? "CodeSurfer"?) in the JuliaDebug org that would have the Cthulhu-like "descending" UI but oriented towards source text; this repo could also have a subdir package ("TypedSyntax"?) that handles the matching between the trees produced by JuliaSyntax and type-inferred code. The output of "TypedSyntax" would be a TreeNode{TypedSyntaxData} and thus usable in other applications without loading the UI components.

If instead you want it to live under the Cthulhu umbrella, then I'd propose the "TypedSyntax" module could either be a standalone package (JuliaDebug? or other org?) or could live in this repo as a subdir package. The UI elements would be added to Cthulhu itself.

[timholy]

Locally all tests pass; the only failure (on 1.9) is the fact that TypedSyntax can't be registered until after this merges. The tests of the new UI mode are minimal but probably good enough to merge. But I will hold off in case someone wants to review this.

[aviatesk]

Some issues I found during playing with the latest commit:

• How about falling back to the previous lowered code view when we can't retrieve a source code?
  For example, I often use anonymous function as an entry, but it now fails since it's hard for CodeTracking to find a source of method created by do block:

  julia> descend((Int,)) do x
             [x]
         end
  ┌ Warning: couldn't retrieve source of (::var"#7#8")(x) @ Main REPL[14]:2
  └ @ Cthulhu ~/julia/packages/Cthulhu/src/reflection.jl:344
  ERROR: MethodError: no method matching getindex(::Nothing, ::Int64)
  Stacktrace:
   [1] find_callsites(interp::Cthulhu.CthulhuInterpreter, CI::Core.CodeInfo, stmt_infos::Vector{Core.Compiler.CallInfo}, mi::Core.MethodInstance, slottypes::Vector{Any}, optimize::Bool, annotate_source::Bool)
     @ Cthulhu ~/julia/packages/Cthulhu/src/reflection.jl:98
   [2] _descend(term::REPL.Terminals.TTYTerminal, interp::Cthulhu.CthulhuInterpreter, curs::Cthulhu.CthulhuCursor; override::Nothing, debuginfo::Symbol, optimize::Bool, interruptexc::Bool, iswarn::Bool, hide_type_stable::Bool, verbose::Nothing, remarks::Bool, with_effects::Bool, inline_cost::Bool, type_annotations::Bool, annotate_source::Bool)
     @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:496
   [3] _descend(term::REPL.Terminals.TTYTerminal, interp::Cthulhu.CthulhuInterpreter, mi::Core.MethodInstance; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:iswarn,), Tuple{Bool}}})
     @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:760
   [4] _descend(term::REPL.Terminals.TTYTerminal, args::Any; interp::Core.Compiler.NativeInterpreter, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:iswarn,), Tuple{Bool}}})
     @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:776
   [5] __descend_with_error_handling(args::Any; terminal::Any, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
     @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:239
   [6] _descend_with_error_handling(f::Any, argtypes::Any; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:iswarn,), Tuple{Bool}}})
     @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:228
   [7] descend_code_typed(::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
     @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:173
   [8] descend_code_typed(::Any, ::Any)
     @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:173
   [9] top-level scope
     @ REPL[14]:1

• And there is a bug with descending into [x]?

  julia> foo(x) = [x]
  foo (generic function with 1 method)
  
  julia> descend(foo, (Int,))
  foo(x) @ Main REPL[18]:1
  1 foo(x::Int64)::Vector{Int64} = [x::Int64]
  Select a call to descend into or ↩ to ascend. [q]uit. [b]ookmark.
  Toggles: [w]arn, [h]ide type-stable statements, [t]ype annotations, [s]yntax highlight for Source/LLVM/Native.
  Show: [S]ource code, [A]ST, [T]yped code, [L]LVM IR, [N]ative code
  Actions: [E]dit source code, [R]evise and redisplay
   • 1 [x]
     ↩
  ERROR: KeyError: key Symbol("") not found
  Stacktrace:
    [1] getindex
      @ ./dict.jl:485 [inlined]
    [2] map_ssas_to_source(src::Core.CodeInfo, rootnode::JuliaSyntax.TreeNode{JuliaSyntax.SyntaxData}, Δline::Int32)
      @ TypedSyntax ~/julia/packages/Cthulhu/TypedSyntax/src/node.jl:405
    [3] get_typed_sourcetext(mi::Core.MethodInstance, src::Core.CodeInfo, rt::Nothing)
      @ Cthulhu ~/julia/packages/Cthulhu/src/reflection.jl:350
    [4] find_callsites(interp::Cthulhu.CthulhuInterpreter, CI::Core.CodeInfo, stmt_infos::Vector{Core.Compiler.CallInfo}, mi::Core.MethodInstance, slottypes::Vector{Any}, optimize::Bool, annotate_source::Bool)
      @ Cthulhu ~/julia/packages/Cthulhu/src/reflection.jl:28
    [5] _descend(term::REPL.Terminals.TTYTerminal, interp::Cthulhu.CthulhuInterpreter, curs::Cthulhu.CthulhuCursor; override::Nothing, debuginfo::Cthulhu.DInfo.DebugInfo, optimize::Bool, interruptexc::Bool, iswarn::Bool, hide_type_stable::Bool, verbose::Nothing, remarks::Bool, with_effects::Bool, inline_cost::Bool, type_annotations::Bool, annotate_source::Bool)
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:496
    [6] _descend(term::REPL.Terminals.TTYTerminal, interp::Cthulhu.CthulhuInterpreter, curs::Cthulhu.CthulhuCursor; override::Nothing, debuginfo::Symbol, optimize::Bool, interruptexc::Bool, iswarn::Bool, hide_type_stable::Bool, verbose::Nothing, remarks::Bool, with_effects::Bool, inline_cost::Bool, type_annotations::Bool, annotate_source::Bool)
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:624
    [7] _descend(term::REPL.Terminals.TTYTerminal, interp::Cthulhu.CthulhuInterpreter, mi::Core.MethodInstance; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:iswarn,), Tuple{Bool}}})
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:760
    [8] _descend(term::REPL.Terminals.TTYTerminal, args::Any; interp::Core.Compiler.NativeInterpreter, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:iswarn,), Tuple{Bool}}})
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:776
    [9] __descend_with_error_handling(args::Any; terminal::Any, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:239
   [10] _descend_with_error_handling(f::Any, argtypes::Any; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:iswarn,), Tuple{Bool}}})
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:228
   [11] descend_code_typed(::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:173
   [12] descend_code_typed(::Any, ::Any)
      @ Cthulhu ~/julia/packages/Cthulhu/src/Cthulhu.jl:173
   [13] top-level scope
      @ REPL[19]:1

[aviatesk]

It seems like we need to remove this semicolon otherwise we can't construct TypedSyntaxNode:

julia> function summer(list)
           s = 0
           for x in list
               s += x
           end
           return s
       end;

julia> TypedSyntaxNode(summer, (Vector{Float64},))
ERROR: MethodError: no method matching iterate(::Nothing)

Closest candidates are:
  iterate(::Union{LinRange, StepRangeLen})
   @ Base range.jl:887
  iterate(::Union{LinRange, StepRangeLen}, ::Integer)
   @ Base range.jl:887
  iterate(::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}}
   @ Base dict.jl:716
  ...

Stacktrace:
 [1] indexed_iterate(I::Nothing, i::Int64)
   @ Base ./tuple.jl:93
 [2] JuliaSyntax.TreeNode{TypedSyntax.TypedSyntaxData}(f::Any, t::Any; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ TypedSyntax ~/julia/packages/Cthulhu/TypedSyntax/src/node.jl:20
 [3] JuliaSyntax.TreeNode{TypedSyntax.TypedSyntaxData}(f::Any, t::Any)
   @ TypedSyntax ~/julia/packages/Cthulhu/TypedSyntax/src/node.jl:18
 [4] top-level scope
   @ REPL[33]:1

[timholy]

Update your CodeTracking

[timholy]

Thanks for the review!

I fixed the issues you noticed. It now falls back to the typed view if the source can't be found. (That's a very important change, thanks!)

[timholy]

Update your CodeTracking

[timholy]

One thought I had: should I split this into two PRs?

• one PR is to add TypedSyntax (assuming we still want it to be a subdir package here, or we can move it to a separate repo). We could register it before moving on to the next PR.
• another PR is to wire it into Cthulhu and make this the default mode.

[aviatesk]

One thought I had: should I split this into two PRs?

• one PR is to add TypedSyntax (assuming we still want it to be a subdir package here, or we can move it to a separate repo). We could register it before moving on to the next PR.
• another PR is to wire it into Cthulhu and make this the default mode.

I agree with it :)

[timholy]

See #353. Future reviews & development of TypedSyntax should go there.