[Batch File] Fix escape sequences, word boundaries and redirections (#…

…3755)

* [Batch File] Fix redirections

This commit modifies behavior of Batch File.sublime-syntax to treat
redirections as normal command arguments. That's what Bash also does.

Up to this commit, redirections have been expected, with some exceptions,
to appear at the end of a command only. Thus they terminated statements.

This commit therefore applies following changes:

1. `<` and `>` are removed from command terminator patterns

2. a new `redirection_or_eoc` variable is introduced to terminate tokens,
   which are known not to contain redirections.

3. the `metachar` variable and some of its children are tweaked, mostly
   to reduce duplicated characters and to clarify dependencies.

4. `redirections` context is included into all relevant contexts

5. various remaining issues related to unquoted vs. quoted escape sequences
   and interpolations are fixed by including clearly named contexts:

   - quoted-escapes-and-interpolations
   - unquoted-escapes-and-interpolations
   
   into related quoted and unquoted token contexts.

   That's a follow-up of #3743.

* [Batch File] Fix word boundary issues

This commit...

1. introduces `unquoted-words` context which is pushed on stack if an unquoted
   word starts and is popped by `{{metachar}}` or quotation mark to ensure not
   to match e.g. numeric redirection file handles in the middle of normal words.

2. `cmd-args-quoted` is moved and renamed to `quoted-words` as counter part of
   `unquoted-words` context.

3. introduces `cmd-set-variable-[inner|outer]-common` and
   `cmd-set-value-[inner|outer]-common` contexts with all common content and
   terminator patterns of quoted and unquoted parts of a `set` statement.

4. fixes an issue which caused `set "variable & no command` statement from being
   terminated at `&`, which is wrong as ampersand is treated literal in quoted
   regions. As a result distinction between `unquoted-eoc-pop` and
   `quoted-eoc-pop` got obsolete. The latter one does not exist. Quoted regions
   are not terminated before end of line. As a result the former one is renamed
   to `eoc-pop`.

* [Batch File] Rename redirection_or_eoc variable

This commit...

1. renames `{{redirection_or_eoc}}` to `{{redir_or_eoc_char}}` in order to
   clarify it consumes characters.
2. a lookahead variable `{{redir_or_eoc}}` is added, also.

* [Batch File] Add syntax tests

This commit...

1. adds tests for line continuation in `if a == b` statements
2. adds tests to verify `meta.statement.conditional` to cover expected tokens

Line continuations have broken syntax highlighting in if statements before.

* [Batch File] Remove obsolete line continuation after `rem`

Line continuations are handled by `unquoted-eol-pop` already.

* [Batch File] Fix redirections in ignored tails

* [Batch File] Fix label word boundaries

Label names...

1. may start with any non-meta-char except `:` or `+`.
2. are terminated by meta-chars
3. don't support quoted identifier parts
4. may however contain escaped characters
5. support variable interpolation/expansion only in CALL/GOTO statements
6. support line continuation in variable names only in CALL/GOTO statements

Labels may be followed by commands separated with `&` or `|`

* [Batch File] Tweak label symbol lists

This commit...

1. removes escape `^` punctuation from label symbols, so labels are resolved
   without regards to whether they contain escaped chars or not.
2. adds indexed references for `GOTO <label>` to enable "Goto Reference"

* [Batch File] Simplify command argument contexts

This commit merges `cmd-args-common` and `cmd-args-meta` into `cmd-args-body`.

* [Batch File] Fix /option word boundaries

This commit updates boundaries of command line arguments.

1. `(^|\s*)` is obsolete due to former word boundary fixes
2. options are terminated by normal meta chars only.
3. `meta.string` and `punctuation` scopes are removed from quoted words as
   various commands treat them literal. In general it's the called command,
   which interprets passed arguments and decides, how quoted parts are treated
   or how the whole argument is parsed/interpreted.

* [Batch File] Fix option value word boundaries

Values are words and thus are terminated by meta chars, only.

* [Batch File] Fix redirection after `echo off`

This commit...

1. ensures to include redirection after `@echo off` into command's arguments
2. ensures to scope `on` and `off` constant, if redirection is following them.

* [Batch File] Tweak command argument patterns

This commit...

1. removes obsolete capture groups
2. improves illegal `for` argument pattern.

* [Batch File] Add support for /? in all builtin commands

* [Batch File] Fix path wildcard scope

This commit follows the suggestion of #3766 and scopes wildcards `?` and `*`
`keyword.operator.wildcard`.

Batch File/Batch File (Compound).sublime-syntax

@@ -21,4 +21,4 @@ hidden: true
 extends: Packages/Batch File/Batch File.sublime-syntax
 
 variables:
-  command_terminator_chars: '[|&<>)]'
+  eoc_char: '[\n|&)]'