Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not lex .. as a single DotDotToken #75549

Merged
merged 51 commits into from
Oct 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
634330c
Add failing test
Rekkonnect Oct 16, 2024
44645f7
WorkItem :)
Rekkonnect Oct 16, 2024
9818686
Merge remote-tracking branch 'upstream/main' into fix/74456-increment…
CyrusNajmabadi Oct 16, 2024
d6e55b4
Renames and docs
CyrusNajmabadi Oct 16, 2024
80b11ee
Update tests
CyrusNajmabadi Oct 16, 2024
5a32172
Do not have lexer produce .. token
CyrusNajmabadi Oct 16, 2024
ade242c
Move error
CyrusNajmabadi Oct 16, 2024
98016cc
Docs
CyrusNajmabadi Oct 16, 2024
3631afb
Advance
CyrusNajmabadi Oct 16, 2024
24d5ef7
Place at the same locaiton
CyrusNajmabadi Oct 16, 2024
756efc0
Cleanup and consistency
CyrusNajmabadi Oct 16, 2024
1123f41
Break apart numerics
CyrusNajmabadi Oct 16, 2024
18692ca
in progress
CyrusNajmabadi Oct 16, 2024
5181f47
Lexer work
CyrusNajmabadi Oct 17, 2024
3588567
restore
CyrusNajmabadi Oct 17, 2024
f202cab
Update src/Compilers/CSharp/Test/Syntax/Parsing/ParsingTests.cs
CyrusNajmabadi Oct 17, 2024
853e317
Simplify
CyrusNajmabadi Oct 17, 2024
206f67d
Merge branch 'dotDotToken' of https://github.com/CyrusNajmabadi/rosly…
CyrusNajmabadi Oct 17, 2024
82d2c50
Update quick scanner as well
CyrusNajmabadi Oct 17, 2024
1cdc99b
More cases
CyrusNajmabadi Oct 17, 2024
f5557ec
Consequence parsing
CyrusNajmabadi Oct 17, 2024
46c2a9b
Fixup test
CyrusNajmabadi Oct 17, 2024
2b2ab95
More precise errors
CyrusNajmabadi Oct 17, 2024
28acd3b
Specialized error message
CyrusNajmabadi Oct 17, 2024
e57e967
Simplify expression parsing
CyrusNajmabadi Oct 17, 2024
e6eb643
Extract helper
CyrusNajmabadi Oct 17, 2024
4f5e6fc
simplify
CyrusNajmabadi Oct 17, 2024
b8c14af
simplify
CyrusNajmabadi Oct 17, 2024
cc0efdb
simplify
CyrusNajmabadi Oct 17, 2024
28fad18
inline
CyrusNajmabadi Oct 17, 2024
16497d4
Simplify token merging
CyrusNajmabadi Oct 18, 2024
c5f6fb7
Fix
CyrusNajmabadi Oct 18, 2024
c19fd49
Simplify
CyrusNajmabadi Oct 18, 2024
6dcfc98
Simplify
CyrusNajmabadi Oct 18, 2024
1976c00
Simplify
CyrusNajmabadi Oct 18, 2024
7ff8920
Update src/Compilers/CSharp/Portable/Parser/LanguageParser.cs
CyrusNajmabadi Oct 22, 2024
b2145be
Merge remote-tracking branch 'upstream/main' into expressionParsing
CyrusNajmabadi Oct 23, 2024
7874ffb
Merge branch 'expressionParsing' of https://github.com/CyrusNajmabadi…
CyrusNajmabadi Oct 23, 2024
fe0c172
Merge remote-tracking branch 'upstream/main' into expressionParsing
CyrusNajmabadi Oct 23, 2024
804f7a6
Tweak comment
CyrusNajmabadi Oct 23, 2024
caf82c5
Grammar
CyrusNajmabadi Oct 23, 2024
52190a9
Update src/Compilers/CSharp/Portable/Parser/LanguageParser.cs
CyrusNajmabadi Oct 23, 2024
cc4ce54
Lower value slightly
CyrusNajmabadi Oct 23, 2024
78be939
Merge branch 'expressionParsing' into dotDotToken
CyrusNajmabadi Oct 23, 2024
7bb8097
Fix
CyrusNajmabadi Oct 23, 2024
cf6235b
Clean up usings
CyrusNajmabadi Oct 23, 2024
6dc3f54
Add tests
CyrusNajmabadi Oct 23, 2024
a0b9c62
Merge remote-tracking branch 'upstream/main' into dotDotToken
CyrusNajmabadi Oct 23, 2024
a9b9710
Simplify
CyrusNajmabadi Oct 25, 2024
d86ef0d
Add tests
CyrusNajmabadi Oct 25, 2024
ff0e6e4
Add tests
CyrusNajmabadi Oct 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions src/Compilers/CSharp/Portable/Parser/Blender.Reader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,7 @@

using System.Collections.Immutable;
using System.Diagnostics;
using Microsoft.CodeAnalysis.CSharp.Symbols;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.CodeAnalysis.Text;
using Roslyn.Utilities;

namespace Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax
{
Expand Down Expand Up @@ -307,6 +304,7 @@ internal static bool IsFabricatedToken(SyntaxKind kind)
case SyntaxKind.GreaterThanGreaterThanEqualsToken:
case SyntaxKind.GreaterThanGreaterThanGreaterThanToken:
case SyntaxKind.GreaterThanGreaterThanGreaterThanEqualsToken:
case SyntaxKind.DotDotToken:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is what fixes incremental parsing. .. is now always a 'fabricated' (like << is), causing the incremental parser to not reuse those. Instead, we always just produce single dot tokens, and allow the parser to decie what to do with them.

return true;
default:
return SyntaxFacts.IsContextualKeyword(kind);
Expand Down
3 changes: 0 additions & 3 deletions src/Compilers/CSharp/Portable/Parser/Blender.cs
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,7 @@
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Diagnostics;
using Microsoft.CodeAnalysis.CSharp.Symbols;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.CodeAnalysis.Text;
using Roslyn.Utilities;

namespace Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax
{
Expand Down
161 changes: 91 additions & 70 deletions src/Compilers/CSharp/Portable/Parser/LanguageParser.cs

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -193,9 +193,12 @@ private PatternSyntax ParsePrimaryPattern(Precedence precedence, bool afterIs, b
{
case SyntaxKind.OpenBracketToken:
return this.ParseListPattern(inSwitchArmPattern);
case SyntaxKind.DotDotToken:
return _syntaxFactory.SlicePattern(EatToken(),
IsPossibleSubpatternElement() ? ParsePattern(precedence, afterIs: false, inSwitchArmPattern) : null);
case SyntaxKind.DotToken when IsAtDotDotToken():
return _syntaxFactory.SlicePattern(
EatDotDotToken(),
IsPossibleSubpatternElement()
? ParsePattern(precedence, afterIs: false, inSwitchArmPattern)
: null);
case SyntaxKind.LessThanToken:
case SyntaxKind.LessThanEqualsToken:
case SyntaxKind.GreaterThanToken:
Expand Down
43 changes: 29 additions & 14 deletions src/Compilers/CSharp/Portable/Parser/Lexer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ internal enum XmlDocCommentStyle
Delimited = 1
}

internal partial class Lexer : AbstractLexer
internal sealed partial class Lexer : AbstractLexer
{
private const int TriviaListInitialCapacity = 8;

Expand Down Expand Up @@ -459,26 +459,41 @@ private void ScanSyntaxToken(ref TokenInfo info)
break;

case '.':
if (!this.ScanNumericLiteral(ref info))
if (this.TextWindow.PeekChar(1) is >= '0' and <= '9')
{
TextWindow.AdvanceChar();
if (TextWindow.TryAdvance('.'))
var atDotPosition = this.TextWindow.Position;
if (atDotPosition >= 1 &&
atDotPosition == this.TextWindow.LexemeStartPosition)
{
if (TextWindow.PeekChar() == '.')
// We have something like: .0
//
// This could be a fp number *except* the case where we have `..0` in that case, we want two
// dots followed by an integer (which will be treated as a range expression).
//
// Move back one space to see what's before this dot and adjust accordingly.

this.TextWindow.Reset(atDotPosition - 1);
var priorCharacterIsDot = this.TextWindow.PeekChar() is '.';
this.TextWindow.Reset(atDotPosition);

if (priorCharacterIsDot)
{
// Triple-dot: explicitly reject this, to allow triple-dot
// to be added to the language without a breaking change.
// (without this, 0...2 would parse as (0)..(.2), i.e. a range from 0 to 0.2)
this.AddError(ErrorCode.ERR_TripleDotNotAllowed);
// We have two dots in a row. Treat the second dot as a dot, not the start of a number literal.
TextWindow.AdvanceChar();
info.Kind = SyntaxKind.DotToken;
break;
}

info.Kind = SyntaxKind.DotDotToken;
}
else
{
info.Kind = SyntaxKind.DotToken;
// Fall through naturally and scan the number out as a floating point number.
}
}

if (!this.ScanNumericLiteral(ref info))
{
TextWindow.AdvanceChar();
info.Kind = SyntaxKind.DotToken;
}

break;

case ',':
Expand Down
5 changes: 1 addition & 4 deletions src/Compilers/CSharp/Portable/Parser/QuickScanner.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@

using System;
using System.Diagnostics;
using Microsoft.CodeAnalysis.CSharp.Symbols;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.CodeAnalysis.Text;
using Roslyn.Utilities;

namespace Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax
Expand Down Expand Up @@ -152,7 +149,7 @@ private enum CharFlags : byte
(byte)QuickScanState.FollowingCR, // CR
(byte)QuickScanState.DoneAfterNext, // LF
(byte)QuickScanState.Done, // Letter
(byte)QuickScanState.Number, // Digit
(byte)QuickScanState.Bad, // Dot followed by number. Could be a fp `.0` or could be a range + num `..0`. Can't tell here.
CyrusNajmabadi marked this conversation as resolved.
Show resolved Hide resolved
(byte)QuickScanState.Done, // Punct
(byte)QuickScanState.Bad, // Dot (DotDot range token, exit so that we handle it in subsequent scanning code)
(byte)QuickScanState.Done, // Compound
Expand Down
2 changes: 1 addition & 1 deletion src/Compilers/CSharp/Portable/Parser/SyntaxParser.cs
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,7 @@ protected SyntaxToken CurrentToken
{
get
{
return _currentToken ?? (_currentToken = this.FetchCurrentToken());
return _currentToken ??= this.FetchCurrentToken();
}
}

Expand Down
Loading