gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer

[pablogsal]

• Issue: Consider emitting buffered DEDENT tokens on the last line #104976

[lysnikolaou]

@pablogsal Can you provide a very quick explanation of how this works and changes the location of the DEDENT token? Nothing too detailed, just a quick overview of the though process, cause I'm having difficulties understanding this.

[pablogsal]

Can you provide a very quick explanation of how this works and changes the location of the DEDENT token?

Yep: the objective is that inplicit dedents like these:

if X:
  if Y:
     ...

are reported in the same line as the ENDMARKER. For doing this we note that trailing dedent tokens are emitted in a situation where the tokenizer has reached EOF so tok->done is E_EOF for all of them, which is great because this allows to identify them easily. We add now a branch for these to put the line number that is expected as the previous tokenizer.

Unfortunately, this forces us to handle ENDMARKER in the extension because otherwise we don't have enough information to get correctly the line for it because checking the last token may now have these "artificial" new lines.

[pablogsal]

Tell me if something else is not clear:)

[pablogsal]

@pablogsal Can you provide a very quick explanation of how this works and changes the location of the DEDENT token? Nothing too detailed, just a quick overview of the though process, cause I'm having difficulties understanding this.

Ah wait, I forgot to push some commits 😓

[lysnikolaou]

Ah wait, I forgot to push some commits 😓

Aha! Ok, makes sense now. Thanks for the explanation though, man!

[pablogsal]

Aha! Ok, makes sense now. Thanks for the explanation though, man!

Sorry for the confusion! 😅 Check again when you have time!

[lysnikolaou]

Yup! Looks good!

[miss-islington]

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[bedevere-bot]

GH-105000 is a backport of this pull request to the 3.12 branch.