Take a look at documentation/setup.md
for details on installing dependencies and such.
If you believe you've successfully made a change.
- Create a
your_feature.cpp
file in thelanguage_examples/
folder. Once it is created, add C++ code to it that demonstrates your feature (more demonstration the better). - Then use
project test
to generate specs for all the examples. - If there were no side effects, then
your_feature.spec.yaml
should be the only new/changed file. However, if there were side effects then some of the other.spec.yaml
files will be changed. Sometimes those side effects are good, sometimes they're irrelevent, and often times they're a regression. - Once that is ready, make a pull request!
- Everything really begins in
main/main.rb
- The TLDR is
- we create a grammar object
- we create patterns using
Pattern.new
andPatternRange.new
- we decide which patterns "go first" by putting them in the
grammar[:$initial_context]
- then we compile the grammar to a .tmLanguage.json file
- Sadly the C++ repo is a bit of spaghetti, due in large part to the language complexity
(So if you happen to be one of the approximately 200 people on earth that have used textmate grammars) Something like this in a tmLanguage.json file
{
"match": "blah/blah/blah",
"name": "punctuation.separator.attribute.cpp",
"patterns": [
{
"include": "#evaluation_context"
},
]
}
Becomes this inside main.rb
Pattern.new(
match: /blah\/blah\/blah/,
tag_as: "punctuation.separator.attribute",
includes: [
:evaluation_context,
],
)
And things like this
{
"begin": "\\[\\[",
"end": "\\]\\]",
"beginCaptures": {
"0": {
"name": "punctuation.section.attribute.begin.cpp"
}
},
"endCaptures": {
"0": {
"name": "punctuation.section.attribute.end.cpp"
}
},
"name": "support.other.attribute.cpp",
"patterns": [
{
"include": "#attributes_context"
},
]
}
Become this
PatternRange.new(
start_pattern: Pattern.new(
match: /\[\[/,
tag_as: "punctuation.section.attribute.begin"
),
end_pattern: Pattern.new(
match: /\]\]/,
tag_as: "punctuation.section.attribute.end",
),
tag_as: "support.other.attribute",
# tag_content_as: "support.other.attribute", # <- alternative that doesnt double-tag the start/end
includes: [
:attributes_context,
]
)
To add something to the grammar's repository just do
grammar[:the_pattern_name] = Pattern.new(/blahblahblah/)
Where this gets really powerful is that you can nest/reuse patterns.
smalltalk = Pattern.new(
match: /blah\/blah\/blah/,
tag_as: "punctuation.separator.attribute",
includes: [
:evaluation_context,
],
)
quote = Pattern.new(
match: /"/,
tag_as: "quote",
)
phrase = Pattern.new(
match: Pattern.new(/the man said: /).then(quote).then(smalltalk).then(quote),
tag_as: "other.phrase",
)
Regex is pretty hard to read, so this repo uses a library to help.
Pattern.new(*attributes)
or.then(*attributes)
creates a new "shy" group- example:
Pattern.new(/foo/)
=> `/(?:foo)/
- example:
.or(*attributes)
adds an alternation (|
)- example:
Pattern.new(/foo/).or(/bar/)
=>/foo|(?:bar)/
- please note you may need more shy groups depending on order
Pattern.new(/foo/).or(/bar/).maybe(@spaces)
becomes (simplified)/(?:foo|bar)\s*/
- example:
maybe(*attributes)
or.maybe(*attributes)
causes the pattern to match zero or one times (?
)- example
maybe(/foo/)
=>/(?:foo)?/
- example
zeroOrMoreTimes(*attributes)
or.zeroOrMoreTimes(*attributes)
causes the pattern to be matched zero or more times (*
)- example
zeroOrMoreTimes(/foo/)
=> `/(?:foo)*/
- example
oneOrMoreTimes(*attributes)
or.oneOrMoreTimes(*attributes)
causes the pattern to be matched one or more times (+
)- example
oneOrMoreTimes(/foo/)
=> `/(?:foo)+/
- example
lookBehindFor(regex)
or.lookBehindFor(regex)
add a positive lookbehind- example
lookBehindFor(/foo/)
=> `/(?<=foo)/
- example
lookBehindToAvoid(regex)
or.lookBehindToAvoid(regex)
add a negative lookbehind- example
lookBehindToAvoid(/foo/)
=> `/(?<!foo)/
- example
lookAheadFor(regex)
or.lookAheadFor(regex)
add a positive lookahead- example
lookAheadFor(/foo/)
=> `/(?=foo)/
- example
lookAheadToAvoid(regex)
or.lookAheadToAvoid(regex)
add a negative lookahead- example
lookAheadToAvoid(/foo/)
=> `/(?!foo)/
- example
backreference(reference)
or.backreference(reference)
adds a backreference- example
Pattern.new(match: /foo|bar/, reference: "foobar").backreference("foobar")
=>/(foo|bar)\1/
- example
helpers that are marked as accepting *attributes
can accept either a regular expression, a hash that provides more info, or a variable that is either of those.
the hash provided to the helper patterns can have the following keys:
match:
the regular expression that should be matchedtag_as:
the scope-name to give this sub-expressionreference:
a name used to refer to this sub-expression in atag_as
orback_reference
comment:
unused, use regular ruby commentsshould_partial_match
,should_not_partial_match
,should_fully_match
, andshould_not_fully_match
see unit testing
look{Ahead,Behind}{ToAvoid,For} helpers only accept regular expressions use .without_numbered_capture_groups
to convert a pattern to a regular expression
PatternRange.new
is used to create a begin/end pattern rule.
By supplying one of the unit testing keys to the pattern, you can ensure that pattern only matches what you want it to.
should_partial_match
asserts that the pattern matches anywhere in the test stringsshould_not_partial_match
asserts that the pattern does not match at all in the test strings.should_fully_match
asserts that the pattern matches all the characters in the test strings.should_not_fully_match
asserts that the pattern does not match all the characters in the test strings.- note:
should_not_fully_match
does not implyshould_partial_match
, that is a failure to match satisfiesshould_not_fully_match
- note:
Integration testing uses vscode-textmate
to confirm that that the fixture files (found at test/fixtures
) have the same tokens and scopes as described in the spec files (found at test/specs
). Each issue should have a fixture and spec file or an entry in test/fixtures/issues/skipped issues
explaining why it was skipped.
npm test [path/to/fixture]
runs the specified test or all if no test specified.
npm run generate-specs [-- --generate-all] [-- path/to/fixture]
used to generate spec files for newly added fixtures, or, if needed, the specified fixture, or all of them.
npm run sort-specs
used to resort the spec keys. This command has no effect on the passing/not passing status of a test
- https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide Visual Studio Code's guide to language grammar
- https://macromates.com/manual/en/language_grammars Textmate's guide to language grammars.
- https://www.sublimetext.com/docs/3/scope_naming.html Sublime Text's guide to textmate scope selection.
- https://www.apeth.com/nonblog/stories/textmatebundle.html More explanation on how grammars are structured.
- https://rubular.com/ Ruby compatible regular expression tester
- https://github.com/stedolan/jq/wiki/Docs-for-Oniguruma-Regular-Expressions-(RE.txt) Documentation for Oniguruma flavored (textmate) regular expressions
- https://github.com/Microsoft/vscode-textmate the Textmate grammar parser for Visual Studio Code