Try new Rust grammar #108254

alexr00 · 2020-10-07T14:48:38Z

Related to #64488

alexr00 · 2020-10-07T14:50:44Z

extensions/rust/test/colorize-results/test_rs.json

@@ -210,7 +342,29 @@
 	},
 	{
 		"c": "let",


In other languages, let is usually scoped as storage.type. Should that be the case for Rust?

I think so. Even if not, keyword.control seems wrong as it doesn't affect control flow.

The new grammer has an issue I am seeing right now:

fn and use should not be highlighted as keyword.control. In my opinion fn does not not necessarily signify control-flow. The same applies to use, there is no control-flow.

alexr00 · 2020-10-07T14:53:26Z

extensions/rust/test/colorize-results/test_rs.json

@@ -154,19 +242,30 @@
 		}
 	},
 	{
-		"c": "println!",
-		"t": "source.rust support.function.std.rust",
+		"c": "println",


In other languages (and in the old grammar for Rust), print statements are support.function should that be the case for Rust too?

In Rust println! is a macro and not a function. I also think that println! maybe should be considered a single token and not two tokens (println and !)

alexr00 · 2020-10-07T14:55:15Z

extensions/rust/test/colorize-results/test_rs.json

+		}
+	},
+	{
+		"c": "mut",


In the old grammar mut was storage.modifier, is keyword more correct?

https://macromates.com/manual/en/language_grammars

storage — things relating to “storage”.

modifier — a storage modifier like static, final, abstract, etc.

Should be storage.modifier I think.

alexr00 · 2020-10-07T15:01:26Z

extensions/rust/test/colorize-results/test-6611_rs.json

@@ -1,13 +1,13 @@
 [
 	{
 		"c": "impl",
-		"t": "source.rust storage.type.rust",
+		"t": "source.rust keyword.control.rust",


Not sure what scope this should have, but the new grammar does change the scope it gets.

alexr00 · 2020-10-07T15:23:05Z

extensions/rust/test/colorize-results/test-6611_rs.json

+	},
+	{
+		"c": "struct",
+		"t": "source.rust keyword.control.rust",


In other languages, struct is scoped as storage.type, and in the old grammar it was also storate.type. Should it change?

https://macromates.com/manual/en/language_grammars

storage — things relating to “storage”.

type — the type of something, class, function, int, var, etc.

Yeah, it should be storage.type I think.

alexr00 · 2020-10-07T15:23:54Z

extensions/rust/test/colorize-results/test-6611_rs.json

+		}
+	},
+	{
+		"c": "trait",


Should this stay storage.type?

alexr00 · 2020-10-07T15:33:17Z

@dustypomerleau I have tried adopting your rust grammar in this PR. We have colorization tests that compare the color and scope of tokens in sample language files. The language files for rust are here:
https://github.com/Microsoft/vscode/blob/5bb9d0b0987c9db4335535e7b339a26d20dda473/extensions/rust/test/colorize-fixtures/test-6611.rs#L1-L1
https://github.com/Microsoft/vscode/blob/8509d4e47a8e85355353500d0b0636d3fcbd0462/extensions/rust/test/colorize-fixtures/test.rs#L1-L1

Your grammar definitely colors more tokens and colors some tokens more specifically, which is great. However, there are some tokens that I'm not confident have the correct scope. I left some comments in the output of the colorization tests where the scope doesn't match what I'd expect.

As you probably know already, the reason we have some standard scopes that we expect various tokens to be is so that one theme can look good on any language. I am not a rust user, so I'm not 100% certain that the comments I have are relevant, but please take a look and convince me that the scopes are correct (or update your grammar and I'll pull the change into this PR).

Once we've addressed the scope differences, I'll bring this grammar into insiders for a month or two. Given the low user count for this grammar currently, I want to make sure that it has plenty of time in insiders so users and find any issues.

dustypomerleau · 2020-10-08T13:08:47Z

@alexr00 The reason I chose keyword for many of these scopes is because that's how they are classified by the Rust lexer. Strict keywords listed in The Rust Reference include most of the tokens caught by your test:

impl
let
mut
struct
trait

I'm happy to move some of the keywords into storage, but which ones? Anything that allocates on the heap?

As for println!, @bjorn3 has already mentioned that this is a macro in rust. It is one of the macros provided by the standard library, so I can see how it might be viewed as support. My only opinion here is that it should be possible to theme macros that use the macro_bang! syntax differently from plain functions. I personally don't feel a need to have different scopes for standard library macros versus user-defined macros, but if others felt strongly I could separate them. As for the ! itself, I can provide a separate scope for that (distinct from it's use as NOT), and let theme designers decide whether to match it to the rest of the macro or let it stand out.

alexr00 · 2020-10-08T13:29:07Z

@dustypomerleau since many of the tokens that get scoped as storage in other languages are also keywords in their respective languages, I think something like the guideline that @bjorn3 quoted is a good rule of thumb for when to use storage: #108254 (comment)

What you said about macros makes sense to me.

Please ping here when you've made the changes you want to make and I'll update again!

dustypomerleau · 2020-10-10T01:28:54Z

@alexr00 Changes have been pushed. Appreciate your help.

dustypomerleau · 2020-10-16T00:26:53Z

@alexr00 Pushed an additional commit to fix a highlighting error for raw strings rust-lang/rust-analyzer#6042.

alexr00 · 2020-10-20T09:23:08Z

@dustypomerleau I'll try to get to this this week. Thanks for the updates!

alexr00 · 2020-10-30T14:07:29Z

I was not able to review this again for October. Will address in November.

alexr00 · 2020-11-02T14:52:04Z

@dustypomerleau I am happy with the changes, thanks! I will merge the grammar from https://github.com/dustypomerleau/rust-syntax/ into VS Code and we will see what users think for a month.

dustypomerleau · 2020-11-03T00:30:23Z

Thanks for your help, @alexr00! For patch updates, should I ping someone or submit my own PR with updated commit hash? We are still finding occasional edge cases since I merged into Rust Analyzer.

alexr00 · 2020-11-03T10:43:54Z

@dustypomerleau if there's an urgent fix in the grammar, please just open an issue and ping me on it. Otherwise, I pick up grammar changes with a script about once per month at the beginning of our milestones.

alexr00 · 2020-11-03T10:44:31Z

and @bjorn3 thanks for the additional feedback!

cynecx · 2020-11-07T17:53:08Z

The grammar in its current state has some issues, I'd like to get it fixed:

The new grammer has an issue I am seeing right now:

fn and use as should not be highlighted as keyword.control. In my opinion fn does not not necessarily signify control-flow. The same applies to use, there is no control-flow.

Is it preferred to open a PR here or upstream (https://github.com/dustypomerleau/rust-syntax/)? Also the current situation is quite weird, because we've got this same grammar in multiple places vscode, upstream extension and in rust-analyzer.

bjorn3 · 2020-11-07T18:02:47Z

I think vscode doesn't accept any changes to grammars. I think they only periodically pull in updates from the upstream version, which is https://github.com/dustypomerleau/rust-syntax/ in this case.

alexr00 · 2020-11-09T10:28:15Z

@bjorn3 is correct.

alexr00 · 2020-12-07T16:18:10Z

There has been no feedback about this grammar having problems. I will leave it in for the release of 1.52.

Try new Rust grammar

568976e

alexr00 commented Oct 7, 2020

View reviewed changes

alexr00 self-assigned this Oct 7, 2020

alexr00 commented Oct 7, 2020

View reviewed changes

extensions/rust/test/colorize-results/test-6611_rs.json Outdated

}

},

{

"c": "trait",

Copy link

Member Author

alexr00 Oct 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this stay storage.type?

alexr00 mentioned this pull request Oct 7, 2020

Need a better grammar support for Rust lang #64488

Closed

dustypomerleau mentioned this pull request Oct 9, 2020

add a new TextMate grammar rust-lang/rust-analyzer#6137

Merged

alexr00 added this to the November 2020 milestone Oct 30, 2020

alexr00 added 2 commits November 2, 2020 15:24

Merge branch 'master' into alexr00/tryRustGrammar

8e3f4e9

Update rust grammar

7b952c3

alexr00 merged commit a5d719c into master Nov 2, 2020

alexr00 deleted the alexr00/tryRustGrammar branch November 2, 2020 14:52

benesch mentioned this pull request Nov 2, 2020

Latest nightly seems to ignore semantic highlighting settings rust-lang/rust-analyzer#6267

Closed

cynecx mentioned this pull request Nov 7, 2020

Non control-flow keywords dustypomerleau/rust-syntax#5

Closed

github-actions bot locked and limited conversation to collaborators Dec 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try new Rust grammar #108254

Try new Rust grammar #108254

alexr00 commented Oct 7, 2020 •

edited

Loading

alexr00 Oct 7, 2020

bjorn3 Oct 8, 2020

cynecx Nov 7, 2020 •

edited

Loading

alexr00 Oct 7, 2020

bjorn3 Oct 8, 2020

alexr00 Oct 7, 2020

bjorn3 Oct 8, 2020 •

edited

Loading

alexr00 Oct 7, 2020

alexr00 Oct 7, 2020

bjorn3 Oct 8, 2020 •

edited

Loading

alexr00 Oct 7, 2020

alexr00 commented Oct 7, 2020

dustypomerleau commented Oct 8, 2020

alexr00 commented Oct 8, 2020

dustypomerleau commented Oct 10, 2020

dustypomerleau commented Oct 16, 2020 •

edited

Loading

alexr00 commented Oct 20, 2020

alexr00 commented Oct 30, 2020

alexr00 commented Nov 2, 2020

dustypomerleau commented Nov 3, 2020

alexr00 commented Nov 3, 2020

alexr00 commented Nov 3, 2020

cynecx commented Nov 7, 2020 •

edited

Loading

bjorn3 commented Nov 7, 2020

alexr00 commented Nov 9, 2020

alexr00 commented Dec 7, 2020

Try new Rust grammar #108254

Try new Rust grammar #108254

Conversation

alexr00 commented Oct 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cynecx Nov 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjorn3 Oct 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjorn3 Oct 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexr00 commented Oct 7, 2020

dustypomerleau commented Oct 8, 2020

alexr00 commented Oct 8, 2020

dustypomerleau commented Oct 10, 2020

dustypomerleau commented Oct 16, 2020 • edited Loading

alexr00 commented Oct 20, 2020

alexr00 commented Oct 30, 2020

alexr00 commented Nov 2, 2020

dustypomerleau commented Nov 3, 2020

alexr00 commented Nov 3, 2020

alexr00 commented Nov 3, 2020

cynecx commented Nov 7, 2020 • edited Loading

bjorn3 commented Nov 7, 2020

alexr00 commented Nov 9, 2020

alexr00 commented Dec 7, 2020

alexr00 commented Oct 7, 2020 •

edited

Loading

cynecx Nov 7, 2020 •

edited

Loading

bjorn3 Oct 8, 2020 •

edited

Loading

bjorn3 Oct 8, 2020 •

edited

Loading

dustypomerleau commented Oct 16, 2020 •

edited

Loading

cynecx commented Nov 7, 2020 •

edited

Loading