modal prefix searching in REPL

[blakejohnson]

From the discussions in #6377, #8577, and #8468, it is clear that we need a better solution for the interaction between prefix history searching and multi-line history entries. This PR implements one such solution which is a modal prefix searching behavior. The idea is that pressing up/down arrow will start a prefix search, and then will continue to do prefix searching on subsequent presses until some key other than up/down is pressed. In this way, history searching is not "interrupted" by multi-line history entries. It also allows us to put the cursor at the end of the line, like the old history traversal behavior (i.e. this fixes #8577).

Reasons that this is WIP:

1) I seem to have re-broken down-arrow behavior in terms of continuing from a previously accepted history item. i.e. #8468 is back with this PR.
2) I need to figure out how to 'forward' a keypress to the parent REPL mode. At present, except for the arrow keys, enter/return, and ctrl+C, all other keys simply exit prefix searching mode without performing any other action. I am exploring alternatives to simply pulling in all the key bindings from the default keymap.
3) I need to rebase into a smaller number of commits.

[blakejohnson]

I've found some cases where the keypress passthru is not working, yet. Time to investigate.

[blakejohnson]

Travis failure is unrelated.

[blakejohnson]

Bump.

[blakejohnson]

@vtjnash does this adequately address your concerns?

[ivarne]

Finally managed to try this, and it is great!

Seems like there is still left some work to make it work with the shell> and help> modes. When editing history entries in one of the special modes, the mode is forgotten.

[blakejohnson]

@ivarne good catch. I'll take a look.

[blakejohnson]

@ivarne that should fix it.

[blakejohnson]

bump

[tkelman]

needs a rebase?

[blakejohnson]

@tkelman Indeed. Merge conflicts should be fixed now.

[mbauman]

Seems to be a very nice improvement. I really appreciate how you can get back down to your originally typed (incomplete) line. It's a little tricky with multi-line inputs, but I think this is as good as it can possibly get, and it works fairly well in practice.

I've not dug into it, but I can trigger an EOF error with the sequence up, down, page-down:

  | | |_| | | | (_| |  |  Version 0.4.0-dev+1792 (2014-11-24 15:44 UTC)
 _/ |\__'_|_|_|\__'_|  |  blakejohnson-modal-prefix/1b1d6ba (fork: 8 commits, 0 days)
|__/                   |  x86_64-apple-darwin13.4.0

julia> ERROR: read: end of file
 in read at ./iobuffer.jl:86
 in read at ./io.jl:134
 in match_input at LineEdit.jl:740
 in match_input at LineEdit.jl:744 (repeats 3 times)
 in match_input at LineEdit.jl:740
 in anonymous at LineEdit.jl:1315
 in anonymous at LineEdit.jl:750
 in prompt! at ./LineEdit.jl:1440
 in run_interface at ./LineEdit.jl:1414
 in run_interface at /Users/mbauman/Code/julia-0.4/usr/lib/julia/sys.dylib
 in run_frontend at ./REPL.jl:825
 in run_repl at ./REPL.jl:170
 in _start at ./client.jl:427
 in _start at /Users/mbauman/Code/julia-0.4/usr/lib/julia/sys.dylib

[blakejohnson]

@mbauman I cannot seem to reproduce the error. Is that with any input currently in the prompt? Also maybe relevant... what is your most recent history item?

[blakejohnson]

Nevermind... I can reproduce it.

[blakejohnson]

@mbauman that issue should now be fixed.

[tkelman]

The appveyor timeout on win64 (https://ci.appveyor.com/project/StefanKarpinski/julia/build/1.0.46/job/xvjollvk9d5j67cx) was most likely #7942, I triggered a rebuild of the PR

[mbauman]

Looks great. I can no longer trigger the error.

[blakejohnson]

@rfourquet the cursor movement thing was responding to #8577. After reading that issue, I did a few tests and thought I found that @PythonNut was correct, that moving the cursor to the end was "standard behavior". Since then I have found that several REPL's choose to do something else. For instance, ipython moves the cursor to the end when navigating history with no prefix. With a prefix, it leaves the cursor in its current location. I see similar behavior in bash. So, is that a better choice? I guess I don't know. It certainly would be straightforward to implement.

In any case, I've squashed some commits. I can squash it a little further if you prefer.

[PythonNut]

iPython leaves the cursor be in prefix history search, because it's more important to remind the user what the prefix is. I.e. the text before the cursor is your prefix, and the rest is subject to change. This seems like a sound UI choice.

I was actually unaware that Julia supported prefix search at the time of #8577.

If you adopt this then the cursor will stay put on prefix search, and go to the end in normal history search (C-p style)?

[blakejohnson]

@PythonNut that's the idea. Not implemented yet, but perhaps it would be better. On the other hand, there seemed to be fair number of people who disabled prefix searching in their REPL keymap after it landed, precisely because it changed the cursor movement behavior.

[rfourquet]

I think your idea of unifying prefix/non-prefix search is great; my prefered UI would be best explained with adding a conceptual (and invisible) end+1 position to history entries, representing normal history searching:

1. key up when the cursor is on an empty entry: this is interpreted as position end+1: normal history searching (which is equivalent to prefix search with empty prefix), and the cursor moves to the end of searched lines (to address Move cursor to end of line on history recall in REPL #8577)
2. if the cursor is not at position end+1 when key-up is pressed, then this is prefix-searching: keep the cursor at that position so that the searched prefix is clearly visible.
3. if the cursor is moved left during a normal history search, the cursor moves from position end+1 to position end: this is not a visible move, but this initiates prefix searching; this addresses this case: I want a prefix-search with the whole previous entry as prefix: "up" (brings previous entry) then "left" (activate prefix searching).
4. "rigth" when at end position deactivate prefix searching, and "End" key puts unconditionally the cursor at "end+1" position.

Is it possible that the cursor is changed (shape or color) to visually indicate that it is in prefix-search mode?

Sorry if this sounds complicated. Anyway, that's just me and not a requirement; as @mbauman said, I see the current state of the PR as an improvement over previous functioning and would like to see it merged with or without these modifications.

[blakejohnson]

Try out this last commit and let me know what you think (its implementation is quite a bit simpler than what you describe). It doesn't give 3) exactly as you describe, but you can get there with up, left, right, up (in most cases, up, left, up would be just as good).

[blakejohnson]

It also doesn't do your 4), but what you describe seems somewhat unintuitive to me. We have separate keybindings that always do a normal history search: ^P, ^N, and PageUp/Down. I think it would be better for people to learn those than to have some non-visible cursor position after the last input character.

[rfourquet]

Yes for 3) I first proposed the exact thing you said as an alternative, and then deleted this part to not overload too much the text. My thinking with returning in normal-mode with "End" is that by default "up" from empty line starts in normal-mode, so there need a way to get back from prefix-mode to normal-mode again usable with "up"/"down", but "Home" key does exactly that, so that's now perfect for me!
My next wish will be to optionnally show the history number in the prompt while navigating in history, but that's a separate issue.
@mbauman what do you think about merging this new version? I'm looking forward to have it.

[mbauman]

I just played with it, and I agree, I think this latest change reduces the cognitive load when moving around. I like it. I also tried as hard as I could to break it, to no avail. Merge it!

[rfourquet]

I tried to dive in the code but am too unfamiliar with it to really review... but if no-one opposes, I won't resist the merge button by tomorrow.

[ivarne]

Too late @rfourquet 😄

[timholy]

I haven't been paying attention here, but just yesterday I was noticing some issues with history. Looking forward to experiencing this---thanks @blakejohnson!

[rfourquet]

😄

[Keno]

I like this a lot better than what we had before. The one thing I just now noticed that I don't like is the following:

• Up to get to the last history line
• Left
• Down (back to the empty line)
• Up will now search modally with the content of the last line up to the end

I would have expected the empty line to cancel prefix searching.

[rfourquet]

I think this appeared with this PR: "up" then "Ctrl-Left" (or right, or...) prints "5D" first, and if this is repeated, goes one word left as expected. Adding the entry "\e[1;5*" => "*" in prefix_history_keymap seems to fix the problem, hence #9388: could you confirm this is an OK fix @blakejohnson ?

[carlobaldassi]

This change has broken the old history_prev_prefix behaviour, leaving the documentation in an outdated state. Following the docs, the result is now that the REPL crashes:

ERROR: `history_prev_prefix` has no method matching history_prev_prefix(::Base.LineEdit.MIState, ::Base.REPL.REPLHistoryProvider)

[carlobaldassi]

Sorry, disregard the previous message, I misread the docs, they don't actually mention hist_prev_prefix. It's still true that history_prev_prefix is no longer accessible as it used to be though, meaning my keybindings are currently broken... any suggestion about what this line should be changed to:

"\e[5~" => (s,o...)->(LineEdit.history_prev_prefix(s, LineEdit.mode(s).hist)),

?

[blakejohnson]

Unfortunately, the new prefix method enter_prefix_search is no longer bindable in the user keymap since it depends on having access to an instance of PrefixHistoryPrompt. I'm not sure what to do about this at the moment. It is also the case that you cannot rebind the search prompt keys for similar reasons.

[carlobaldassi]

I suspected as much, thanks for looking into this. For the time being, I solved my problem by grabbing the old version of history_prex_prefix and putting it in my juliarc.