Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libsyntax/parse/parser.rs became too big for GitHub #60015

Closed
petrochenkov opened this issue Apr 16, 2019 · 15 comments · Fixed by #63469
Closed

libsyntax/parse/parser.rs became too big for GitHub #60015

petrochenkov opened this issue Apr 16, 2019 · 15 comments · Fixed by #63469
Assignees
Labels
A-diagnostics Area: Messages for errors, warnings, and lints A-frontend Area: Compiler frontend (errors, parsing and HIR) E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. P-high High priority

Comments

@petrochenkov
Copy link
Contributor

Very convenient "Blame" functionality doesn't work in particular.

The solution is to move something from that file.
The obvious candidate for moving is diagnostics-related code, that can be moved into libsyntax/parse/diagnostics.rs.

@petrochenkov petrochenkov added A-diagnostics Area: Messages for errors, warnings, and lints A-frontend Area: Compiler frontend (errors, parsing and HIR) E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. labels Apr 16, 2019
@Centril
Copy link
Contributor

Centril commented Apr 16, 2019

After the diagnostics extraction, ...

...I think we should split the parser up into semantic categories "items", "expressions", "types", "patterns", and so on and so forth since that makes understanding the parser much easier. E.g. you could use some Parse trait which these each of these semantic categories implement.

Aside: rustc has some ridiculously large files; we should also seriously consider making tidy warn on too large files at minimum and eventually setting some hard boundaries (e.g. > 3000 LOC => build failure) to avoid files getting so large in the future. We'd need to lower the limit incrementally of course.

@agnxy
Copy link
Contributor

agnxy commented Apr 17, 2019

@petrochenkov I'd like to work on this issue 😃 I might need some guidance as I'm a beginner to Rust and syntax crate. I'm reading parser.rs to see what can be extracted to diagnostics.rs. Thanks.

@petrochenkov
Copy link
Contributor Author

@agnxy
Moving maybe_report_ambiguous_plus, maybe_recover_from_bad_type_plus, maybe_recover_from_bad_qpath, maybe_recover_from_bad_qpath_stage_2, maybe_consume_incorrect_semicolon, trait RecoverQPath and its impls would be a good start.

@petrochenkov
Copy link
Contributor Author

In general, if any function does only error reporting or error recovery, it can be moved.

@matklad
Copy link
Member

matklad commented Apr 22, 2019

Heh, if I am not mistaking, parser.rs is the largest non-generated Rust file out there :) I use it for benchmarking my parsers and will be sad to see it go /s

On a more serious note, I quite like the grammar layout I use for rust-analyzer.

There's also a vague desire to make a parsing library, which will work inside rustc, syn and IDE. I don't think we should actively plan for this right now, but it might be a good idea to keep in mind. Specifically, it might be interesting to take a look at how swift's parser is organized: IIUC, it produces both the traditional AST, like rustc, and a Swift libsyntanx concrete syntax tree. This seems like a setup we might want to do as well, it is consistent with current RLS-2.0 approach.

bors added a commit that referenced this issue Apr 26, 2019
Add a tidy check for files with over 3,000 lines

Files with a large number of lines can cause issues in GitHub (e.g. #60015) and also tend to be indicative of opportunities to refactor into less monolithic structures.

This adds a new check to tidy to warn against files that have more than 3,000 lines, as suggested in #60015 (comment). (This number was chosen fairly arbitrarily as a reasonable indicator of size.) This check can be ignored with `// ignore-tidy-filelength`.

Existing files with greater than 3,000 lines currently ignore the check, but this helps us spot when files are getting too large. (We might try to split up all files larger than this in the future, as in #60015).
Centril added a commit to Centril/rust that referenced this issue Apr 28, 2019
move some functions from parser.rs to diagostics.rs

Starting with a few functions mentioned in rust-lang#60015 (comment). We might refactor parser.rs further in subsequent changes.
r? @petrochenkov
Centril added a commit to Centril/rust that referenced this issue May 1, 2019
move some functions from parser.rs to diagostics.rs

Starting with a few functions mentioned in rust-lang#60015 (comment). We might refactor parser.rs further in subsequent changes.
r? @petrochenkov
@agnxy
Copy link
Contributor

agnxy commented May 4, 2019

Hi,I have a noob question about the workflow. Since I may make more changes for this issue and #60348 is merged, should I create a new PR on a new branch? Thanks.

@petrochenkov
Copy link
Contributor Author

@agnxy
For already merged PRs the same branch can be reused, but it doesn't make any difference, there are no requirements on this or anything.

@agnxy
Copy link
Contributor

agnxy commented May 4, 2019

Thanks @petrochenkov . I'll try to figure out more items for diagnostics.rs

@agnxy
Copy link
Contributor

agnxy commented May 13, 2019

@petrochenkov , shall I also move recover_closing_delimiter, recover_seq_parse_error, report_invalid_macro_expansion_item, expected_item_err, err_dotdotdot_syntax and missing_assoc_item_kind_err to diagnostics.rs? I'm not sure if that's ok. Thanks a lot.

@estebank estebank added the P-high High priority label May 25, 2019
@estebank estebank self-assigned this May 25, 2019
Centril added a commit to Centril/rust that referenced this issue May 25, 2019
Tweak `self` arg not as first argument of a method diagnostic

Mention that `self` is only valid on "associated functions"
```
error: unexpected `self` argument in function
  --> $DIR/self-in-function-arg.rs:1:15
   |
LL | fn foo(x:i32, self: i32) -> i32 { self }
   |               ^^^^ not valid as function argument
   |
   = note: `self` is only valid as the first argument of an associated function
```

When it is a method, mention it must be first
```
error: unexpected `self` argument in function
  --> $DIR/trait-fn.rs:4:20
   |
LL |     fn c(foo: u32, self) {}
   |                    ^^^^ must be the first associated function argument
```

Move a bunch of error recovery methods to `diagnostics.rs` away from `parser.rs`.

Fix rust-lang#51547. CC rust-lang#60015.
Centril added a commit to Centril/rust that referenced this issue May 26, 2019
Tweak `self` arg not as first argument of a method diagnostic

Mention that `self` is only valid on "associated functions"
```
error: unexpected `self` argument in function
  --> $DIR/self-in-function-arg.rs:1:15
   |
LL | fn foo(x:i32, self: i32) -> i32 { self }
   |               ^^^^ not valid as function argument
   |
   = note: `self` is only valid as the first argument of an associated function
```

When it is a method, mention it must be first
```
error: unexpected `self` argument in function
  --> $DIR/trait-fn.rs:4:20
   |
LL |     fn c(foo: u32, self) {}
   |                    ^^^^ must be the first associated function argument
```

Move a bunch of error recovery methods to `diagnostics.rs` away from `parser.rs`.

Fix rust-lang#51547. CC rust-lang#60015.
@estebank
Copy link
Contributor

I believe this can be closed now. We should continue on the diet regardless.

@ehuss
Copy link
Contributor

ehuss commented Jun 26, 2019

I still can't load https://github.com/rust-lang/rust/blame/master/src/libsyntax/parse/parser.rs

@estebank
Copy link
Contributor

Weird, when I did it a week ago it loaded fine.

@ehuss
Copy link
Contributor

ehuss commented Jun 26, 2019

It's a bit random. In the past, sometimes after getting the unicorn, hitting reload it would pop up immediately. Presumably there's some caching going on where subsequent requests finish within the time limit. In fact, I just now got it to load after reloading a few times. 😆

@crlf0710
Copy link
Member

Current state: file load fine, but blame still doesn't work.

@Centril
Copy link
Contributor

Centril commented Aug 11, 2019

I'm fixing this permanently today.

@Centril Centril assigned Centril and unassigned estebank Aug 11, 2019
bors added a commit that referenced this issue Aug 12, 2019
libsyntax: Refactor `parser.rs` into reasonably sized logical units

Here we split `parser.rs` (~7.9 KLOC) into more reasonably sized files (all < 1.8 KLOC):

- `./src/libsyntax/parse/`
   - `parser.rs`
   - `parser/`
      - `pat.rs`
      - `expr.rs`
      - `stmt.rs`
      - `ty.rs`
      - `path.rs`
      - `generics.rs`
      - `item.rs`
      - `module.rs`

Closes #60015.

r? @petrochenkov
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-diagnostics Area: Messages for errors, warnings, and lints A-frontend Area: Compiler frontend (errors, parsing and HIR) E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. P-high High priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants