Update parser API to merge lexing and parsing

[dhruvmanila]

Summary

This PR updates the parser API within the ruff_python_parser crate. It doesn't change any of the references in this PR.

The final API looks like:

pub fn parse_module(source: &str) -> Result<Program<ModModule>, ParseError> {}

pub fn parse_expression(source: &str) -> Result<Program<ModExpression>, ParseError> {}

pub fn parse_expression_range(
    source: &str,
    range: TextRange,
) -> Result<Program<ModExpression>, ParseError> {}

pub fn parse(source: &str, mode: Mode) -> Result<Program<Mod>, ParseError> {}

// Temporary. The `parse` will replace this function once we update the downstream
// tools to work with programs containing syntax error.
pub fn parse_unchecked(source: &str, mode: Mode) -> Program<Mod> {}

// Same as `parse_unchecked` but using `PySourceType` instead of the `Mode`
pub fn parsed_unchecked_source(source: &str, source_type: PySourceType) -> Program<ModModule> {}

Following is a detailed list of changes:

• Make Program generic over T which can be either Mod (enum), ModModule or ModExpression
  • Add helper methods to cast Mod into ModModule or ModExpression
  • Add helper method Program::into_result which converts a Program<T> into a Result<Program<T>, ParseError> where the Err variant contains the first ParseError
• Update TokenSource to store the comment ranges
  • Parser crate depends on ruff_python_trivia because of CommentRanges. This struct could possibly be moved in the parser crate itself at the end
• Move from parse_expression_starts_at to parse_expression_range which parses the source code at the given range using Mode::Expression. Unlike the starts_at variant, this accepts the entire source code
• Remove all access to the Lexer
• Remove all parse_* functions which works on the tokens provided by the caller

Test Plan

The good news is that the tests in ruff_python_parser can be run. So,

cargo insta test --package ruff_python_parser

[dhruvmanila]

The Tokens struct will gain new methods as I understand how the lexer APIs are being used by the downstream tools but two APIs have been added:

1. up_to_first_unknown - Slice of tokens up to (and excluding) the first unknown token. This is what the linter sees.
2. tokens_in_range - This will replace most usages of lex_starts_at

[MichaReiser]

What's the sizse of a TokenSourceCheckpoint and ParserCheckpoint at this point? It seems to becoming somewhat big ;)

[dhruvmanila]

Yeah, it is getting quite big:

• ParserCheckpoint - 176 bytes
• TokenSourceCheckpoint - 152 bytes
• LexerCheckpoint - 136 bytes

[MichaReiser]

I'm not entirely sure if we should collect the comments in the parse phase or if it is actually faster to extract them after by simply looping over the tokens where needed.

Doint it later has the advantage that it doesn't require any rewinding. I'm okay going with this solution but it might be worth testing that the performance difference is between doing it inside of the parser and doing it as a separate pass. I somewhat suspect that both are equally fast, in which case I would probably prefer doing it outside of the parser.

[dhruvmanila]

Yeah, I can add it to my todo list to look at once the code starts to compile :)

It makes sense to keep it outside the parser.

[MichaReiser]

I suspect that this will fail if the string has any prefixes. While uncommon, it is valid python code to e.g. have r"test"

[dhruvmanila]

I think the name of the function is misleading, it does match against both prefixes and quotes:

ruff/crates/ruff_python_ast/src/str.rs

Lines 207 to 221 in c30d34e

	/// An [`AhoCorasick`] matcher for string and byte literal prefixes.
	static PREFIX_MATCHER: Lazy<AhoCorasick> = Lazy::new(|| {
	AhoCorasick::builder()
	.start_kind(StartKind::Anchored)
	.match_kind(MatchKind::LeftmostLongest)
	.kind(Some(AhoCorasickKind::DFA))
	.build(
	TRIPLE_QUOTE_STR_PREFIXES
	.iter()
	.chain(TRIPLE_QUOTE_BYTE_PREFIXES)
	.chain(SINGLE_QUOTE_STR_PREFIXES)
	.chain(SINGLE_QUOTE_BYTE_PREFIXES),
	)
	.unwrap()
	});

[MichaReiser]

Nit: Maybe parse_with_recovery to make it clear that this parsing mode performs error recovery.

[MichaReiser]

You brought it up in Discord that we'll have a conflict with red-knot.

How about Parsed or SyntaxTree?

[dhruvmanila]

Yeah, I like Parsed. Thanks!

[dhruvmanila]

I'll change this in a follow-up PR.