Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unicode display errors #63

Closed
MilesCranmer opened this issue Jun 19, 2023 · 1 comment · Fixed by #64
Closed

Unicode display errors #63

MilesCranmer opened this issue Jun 19, 2023 · 1 comment · Fixed by #64

Comments

@MilesCranmer
Copy link
Contributor

MilesCranmer commented Jun 19, 2023 •
edited
Loading

I think this is unrelated to #13 but please close if so.

If I open up julia with VimBindings.jl set up and write:

julia> x = "ρ"
I see the following error (click to expand): 
ERROR: TaskFailedException
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/lib/julia/sys.dylib:-1
  [2] wait()
    @ Base ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/lib/julia/sys.dylib:-1
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/lib/julia/sys.dylib:-1
  [4] take_buffered(c::Channel{Any})
    @ Base ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/lib/julia/sys.dylib:-1
  [5] repl_backend_loop(backend::REPL.REPLBackend, get_module::Function)
    @ REPL ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/lib/julia/sys.dylib:-1
  [6] start_repl_backend(backend::REPL.REPLBackend, consumer::Any; get_module::Function)
    @ REPL ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/lib/julia/sys.dylib:-1
  [7] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool, backend::Any)
    @ REPL ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:379
  [8] run_repl(repl::REPL.AbstractREPL, consumer::Any)
    @ REPL ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:365
  [9] (::Base.var"#1017#1019"{Bool, Bool, Bool})(REPL::Module)
    @ Base ./client.jl:421
 [10] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [11] invokelatest
    @ ./essentials.jl:813 [inlined]
 [12] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
    @ Base ./client.jl:405
 [13] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:322
 [14] _start()
    @ Base ./client.jl:522

    nested task error: Unhandled Task ERROR: StringIndexError: invalid index [7], valid nearby indices [6]=>'ρ', [8]=>'\"'
Stacktrace:
  [1] string_index_err(s::String, i::Int64)
    @ Base ./strings/string.jl:12
  [2] getindex
    @ ./strings/string.jl:278 [inlined]
  [3] show(io::IOBuffer, rec::VimBindings.Changes.BufferRecord)
    @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:16
  [4] show(io::IOBuffer, entry::VimBindings.Changes.Entry)
    @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:194
  [5] show_full_history(selected::VimBindings.Changes.Entry)
    @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:139
  [6] record(buf::IOBuffer; cursor_index::Int64)
    @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:90
  [7] record
    @ ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:78 [inlined]
  [8] prompt!(term::REPL.Terminals.TTYTerminal, prompt::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ VimBindings ~/.julia/packages/VimBindings/SP6m7/src/lineeditalt.jl:48
  [9] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2642
 [10] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1300
 [11] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514
StringIndexError: invalid index [7], valid nearby indices [6]=>'ρ', [8]=>'\"'
    Stacktrace:
      [1] string_index_err(s::String, i::Int64)
        @ Base ./strings/string.jl:12
      [2] getindex
        @ ./strings/string.jl:278 [inlined]
      [3] show(io::IOBuffer, rec::VimBindings.Changes.BufferRecord)
        @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:16
      [4] show(io::IOBuffer, entry::VimBindings.Changes.Entry)
        @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:194
      [5] show_full_history(selected::VimBindings.Changes.Entry)
        @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:139
      [6] record(buf::IOBuffer; cursor_index::Int64)
        @ VimBindings.Changes ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:90
      [7] record
        @ ~/.julia/packages/VimBindings/SP6m7/src/changes.jl:78 [inlined]
      [8] prompt!(term::REPL.Terminals.TTYTerminal, prompt::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
        @ VimBindings ~/.julia/packages/VimBindings/SP6m7/src/lineeditalt.jl:48
      [9] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
        @ REPL.LineEdit ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2642
     [10] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
        @ REPL ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1300
     [11] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
        @ REPL ./task.jl:514

I think this is unrelated to #13 because it is about indexing strings, rather than moving in buffers (which does not face this issue). I recently had a similar issue on SymbolicRegression.jl when trying to split a string: MilesCranmer/SymbolicRegression.jl#223. Basically, the string "ρρ" has indices at 1 and 4, but nowhere in between. This is a quirk of how unicode is stored I guess.

The solution is to use the more generic operations, like eachindex, nextind, prevind, and iterate, rather than indexing with linear integers. One thing I've found easiest is to make an index array with I = collect(eachindex(s)), and then index the string with s[I[i]], where i goes from 1 to length(s).

Fixed with #64
@MilesCranmer MilesCranmer mentioned this issue Jun 19, 2023
Merged
caleb-allen pushed a commit that referenced this issue Jun 19, 2023
@MilesCranmer
Fix recording of unicode characters in record (#64)
302d5de 
Fixes #63. (as in, I no longer see the error)

The fix is to replace

```julia
    i = min(length(rec.text), rec.cursor_index)
    a = rec.text[begin:i]
    b = rec.text[i+1:end]
```
with
```julia
    i = min(length(rec.text), rec.cursor_index)
    a = ""
    b = ""
    if i > 0
        a = rec.text[begin:nextind(rec.text, i-1)]
        b = rec.text[nextind(rec.text, i):end]
    end
```
which accounts for strings which are not uniformly indexed, like "aαbβcγ" which has its indices at `1, 2, 4, 5, 7, 8`.
@caleb-allen
Copy link
Owner

caleb-allen commented Jun 19, 2023 •
edited
Loading

I think you're right, not exactly the same #13 but somewhat related. The solution to #13 needs to essentially do what your solution is in #64 and apply it to buffer operations (in other words, don't assume 1 byte = 1 code point); the package is pretty naive with characters right now.

Anyways, thanks for the report and the fix!

@caleb-allen caleb-allen mentioned this issue Jun 27, 2023
Closed
caleb-allen pushed a commit that referenced this issue Jul 14, 2023
@MilesCranmer @caleb-allen
Fix recording of unicode characters in record (#64)
4c623b3 
Fixes #63. (as in, I no longer see the error)

The fix is to replace

```julia
    i = min(length(rec.text), rec.cursor_index)
    a = rec.text[begin:i]
    b = rec.text[i+1:end]
```
with
```julia
    i = min(length(rec.text), rec.cursor_index)
    a = ""
    b = ""
    if i > 0
        a = rec.text[begin:nextind(rec.text, i-1)]
        b = rec.text[nextind(rec.text, i):end]
    end
```
which accounts for strings which are not uniformly indexed, like "aαbβcγ" which has its indices at `1, 2, 4, 5, 7, 8`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix recording of unicode characters MilesCranmer/VimBindings.jl
2 participants
@caleb-allen @MilesCranmer