Added rust-style raw string syntax support for the Rhai Tokenizer

[cellomath]

Added support for Rust-style raw-string literal syntax in Rhai.

Example:

let example_raw_string: string = r##"I can use quotes and / and even one "# without escaping this string
How about we put a newline in it too!"##;

[schungx]

Looks great, except that whether r### is the best syntax. Rhai already looks more like JavaScript than Rust, so best to keep to a consistent style.

Hopefully JS formatters can successfully format most of Rhai...

Do you know how other languages handle the same?

[cellomath]

Python uses r"raw string" syntax to use string with no escapes in it, but I think Rust is pretty unique in allowing the programmer to specify, in place, the number of "#"'s they need to represent the data without any distracting escape characters.

I see what you're saying about JavaScript though. I think raw strings could supplement the verbatim ones while adding negligible overhead, and aligning the language more with standard Rust features.

[schungx]

I do agree that the r### solution is quite elegant, just that the syntax doesn't jive very much with the rest of Rhai's syntax...

I agree that this can be very useful -- I have immediate uses for this myself in my projects!

Would you mind taking care of the review questions as well?

[cellomath]

Could you clarify what/where the review questions are?

[schungx]

Can't you see it?

[cellomath]

Your review says "Pending", probably why I can't see it. Is there a "submit review" button?

[schungx]

Your review says "Pending", probably why I can't see it. Is there a "submit review" button?

Ah OK! Pressed it!

[schungx]

I'll need to fix it up a bit because you cannot just return LexError on EOL... that would cause errors for editors which may be parsing a script line by line. However, this is more intricate so I'd do it.

[schungx]

@cellomath Come to think of it, since # is only used to start an object map literal... can we just use #" ... "# as raw string literals instead? This way, we do away with the r and we'd always have hashes...

[cellomath]

My objective was to match Rust's syntax as much as possible, and prefixing raw strings with an "r" is pretty common in other languages too. Changing the syntax would undermine this, but if you like, I wouldn't be opposed for your #"..."# syntax to alias r#"..."#.

[schungx]

Rhai already deviates from Rust quite significantly, such as using switch instead of match. To me, r" ... " always seems incomplete because of the lack of symmetry, and it is not easy for the user to spot immediately whether something is a raw string. The small r is quite easy to miss.

One more benefit is that the code is cleaner because we no longer have any raw strings without hashes...

EDIT: Caught a position bug

[schungx]

I have an implementation out and it seems to make the code quite a bit simpler. So we'd have:

#"hello world!"#

#"Multiple lines:
Second line
             Final line
"#

##"Lines starting with #'s are comments while "'s are quoted"##

####"You can do ###"hello"### and it'll work"####

[cellomath]

Okay, I see your point. Maybe keeping the languages separated makes it more distinguishable anyways!
¯_(ツ)_/¯

[schungx]

Now we just have to figure out a way to make syntax highlighting work... Not sure how the Rust grammar does it...

Do you know anything about this?

[cellomath]

Not sure what you mean. Can you link to Rhai's existing syntax highlighting? I'm not too familiar with language servers, but I figured they used something similar to regular expressions.

If the syntax highlighter uses the Rhai parser, it should automatically appear as a string literal, since that's what the parser yields from the raw string literal syntax