[grep-] add loader for output of grep/ripgrep

[midichef]

• [X ] If contributing a core loader, the loader checklist was referenced.

This is a loader for working with output of ripgrep, when run as rg --sort path --json pattern |vd -f grep
(--sort path makes ripgrep's search result order be deterministic, but slower due to loss of parallelism in searching files). This is a great use case for visidata. It's very helpful to filter grep results within visidata, and then use Enter to jump into editing the file, and quit out of the editor, and jump into a different file with Enter.

The ripgrep output format is quite advanced and implements much more than this loader takes advantage of. For example, ripgrep says which column a match starts at, not just which line.

This commit is only a draft. One aspect that definitely needs to be changed is the command to open a file at a specific line. Right now it uses commands of the form $EDITOR +5, which works with vim and emacsclient, but not other editors. How should we let the user customize their command to open a file in an editor at a given line number?

One use case to keep in mind is annotating grep results by adding columns. For example, if I'm debugging visidata and I want to audit the calls to a particular function like openSource(), I'll run rg --sort path --json openSource |vd -f grep. Then I'll add a column for notes on whether that particular line is safe, and edit it for each line as I inspect each use. When I'm pausing my audit for the day, I'll save the file as notes.grep, and the next day I'll resume by loading: vd notes.grep.

And one feature to consider, for power users, would be to add lines of context to each row. These would come from rg -A and rg -B, which work just the same as grep -A and grep -B, showing lines after and before the match. If each row had these lines of context it would allow very powerful filtering of results. I think it'd be quite helpful as a shell tool, and could draw more users to visidata. But it's not a feature I can work on at the moment.

[midichef]

For using ripgrep to search data that is not UTF-8, like binary data, the use of the 'text' field will have to be changed, to handle the field 'bytes':

visidata/visidata/loaders/grep.py

Line 41 in 4721764

'text': match_data['lines']['text'].rstrip(linesep)

The grep-printer docs on encoding describe when this happens:

Therefore, this printer will emit valid UTF-8 encoded bytes as normal JSON strings and otherwise base64 encode data that isn’t valid UTF-8. To communicate whether this process occurs or not, strings are keyed by the name text where as arbitrary bytes are keyed by bytes.

[midichef]

I added the ability to parse the output of standard grep and grep -n (which shows line numbers). Those are strings of the form file:text and file:line_no:text.

[saulpw]

Never use a bare except: in Python. At minimum, use except Exception:, as almost all exceptions inherit from Exception. This is a general rule for Python, but in fact is important for VisiData, as its EscapeException inherits from Python's BaseException (which is the true base class for all exceptions), and is used when the user cancels an operation with Ctrl+C. So anywhere a bare except: is used in VisiData, may lead to a situation where the user cannot cancel an operation.

[midichef]

Oh, interesting! Okay, I've fixed it to be more specific.

[saulpw]

This is really awesome, @midichef! I think this is possibly close to being merge-able as is. The main thing that I'd want, is to create a page of documentation which describes how to use it (perhaps as a Guide). Some of the internal comments, like the shell functions at the top, would then be moved there.

One aspect that definitely needs to be changed is the command to open a file at a specific line. Right now it uses commands of the form $EDITOR +5, which works with vim and emacsclient, but not other editors. How should we let the user customize their command to open a file in an editor at a given line number?

I don't think this is strictly necessary. Someone who needs to open it in a different editor can create a small wrapper script which parses the +5 arg and passes it to their editor as needed. Being compatible with vim and emacs is totally sufficient for a first pass at a feature like this.

[midichef]

Okay I've added a guide. Let me know if I should change the guide in any way, to improve its helpfulness.

I think using vd for grep is a powerful combination, and I'd like to make sure the guide explains it well for new users.

[anjakefala]

Hi @midichef!

Thank you so much for writing a guide! I made some edits, that I summarised in the commit. I wanted to write an explanation for some of them:

• For the ripvd snippet: because {} is used for code formatting, there isn't yet support for using it in guides. I have a TODO list item to add that support. I think that snippet is really useful, I'm going to add it myself, and it'd be a good idea for us to have a place where we can collect snippets like that. A guide? A wikipage?
• We have a /formats page, and I moved the info about the format there.
• VisiData strips the leading tabs when it visually represents the guide. I changed the tabs to spaces. I will have a fix for it to not strip leading spaces.
• I set the sheettype, so that the help.commands macro can find the appropriate sysopen-row. =)

[midichef]

Hi @midichef!

• We have a /formats page, and I moved the info about the format there.
• VisiData strips the leading tabs when it visually represents the guide. I changed the tabs to spaces. I will have a fix for it to not strip leading spaces.
• I set the sheettype, so that the help.commands macro can find the appropriate sysopen-row. =)

Thanks for explaining the chanegs. I'll keep these in mind for the future.