Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix indentation tokens lengths #1708

Merged
merged 3 commits into from
Oct 14, 2024
Merged

Fix indentation tokens lengths #1708

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3994c08
Fix indentation duplication as a whitespace token
aabounegm Oct 2, 2024
1919efa
Fix dedent tokens being nonempty
aabounegm Oct 2, 2024
048d76d
Place the dedent after first new line
aabounegm Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 6 additions & 14 deletions packages/langium/src/parser/indentation-aware.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,16 +272,7 @@ export class IndentationAwareTokenBuilder<Terminals extends string = string, Key

this.indentationStack.push(currIndentLevel);

const indentToken = this.createIndentationTokenInstance(
this.indentTokenType,
text,
match?.[0] ?? '',
offset,
);
tokens.push(indentToken);

// Token already added, let the indentation now be consumed as whitespace and ignored
return null;
return match;
}

/**
Expand Down Expand Up @@ -321,19 +312,20 @@ export class IndentationAwareTokenBuilder<Terminals extends string = string, Key
}

const numberOfDedents = this.indentationStack.length - matchIndentIndex - 1;
const newlinesBeforeDedent = text.substring(0, offset).match(/[\r\n]+$/)?.[0].length ?? 1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should not only backtrack through all the newline characters, but also whitespace. For example, if we dedent later:

{
    value
    <-- whitespace
    <-- whitespace
} <-- Dedent token appears at start of this line

This seems pretty weird, as if we remove the whitespace, the dedent already appears in the 3rd line. Since this PR is there to fix behavior such as the folding, I would've thought it should keep the DEDENT token in the 3rd line.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if emit the DEDENT token on the beginning of the 3rd line, then it means that the whitespace after it will be detected as INDENT, and naturally followed by another DEDENT on the next line (and once more, for this example). These extra INDENT/DEDENT tokens are unexpected and will cause parsing errors.

I generally think that trailing whitespace is problematic and should be marked as a parsing error, especially in indentation-sensitive languages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I guess that makes sense. Thanks for the explanation 👍


for (let i = 0; i < numberOfDedents; i++) {
const token = this.createIndentationTokenInstance(
this.dedentTokenType,
text,
match?.[0] ?? '',
offset,
'', // Dedents are 0-width tokens
offset - (newlinesBeforeDedent - 1), // Place the dedent after the first new line character
);
tokens.push(token);
this.indentationStack.pop();
}

// Token already added, let the dedentation now be consumed as whitespace and ignored
// Token already added, let the dedentation now be consumed as whitespace (if any) and ignored
return null;
}

Expand Down Expand Up @@ -365,7 +357,7 @@ export class IndentationAwareTokenBuilder<Terminals extends string = string, Key
const remainingDedents: IToken[] = [];
while (this.indentationStack.length > 1) {
remainingDedents.push(
this.createIndentationTokenInstance(this.dedentTokenType, text, this.options.dedentTokenName, text.length)
this.createIndentationTokenInstance(this.dedentTokenType, text, '', text.length)
);
this.indentationStack.pop();
}
Expand Down
Loading