Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove special code-path for handing unknown tokens #63017

Merged
merged 3 commits into from
Aug 6, 2019

Conversation

matklad
Copy link
Member

@matklad matklad commented Jul 26, 2019

In StringReader, we have a buffer of fatal errors, which is used only in a single case: when we see something which is not a reasonable token at all, like 🦀. I think a more straightforward thing to do here is to produce an explicit error token in this case, and let the next layer (the parser), deal with it.

However currently this leads to duplicated error messages. What should we do with this? Naively, I would think that emitting (just emitting, not raising) FatalError should stop other errors, but looks like this is not the case? We can also probably tweak parser on the case-by-case basis, to avoid emitting "expected" errors if the current token is an Err. I personally also fine with cascading errors in this case: it's quite unlikely that you actually type a fully invalid token.

@petrochenkov, which approach should we take to fight cascading errors?

@petrochenkov petrochenkov self-assigned this Jul 26, 2019
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-07-26T17:54:12.0959196Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-07-26T17:54:12.1162368Z ##[command]git config gc.auto 0
2019-07-26T17:54:12.1284747Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-07-26T17:54:12.1307444Z ##[command]git config --get-all http.proxy
2019-07-26T17:54:12.1464262Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63017/merge:refs/remotes/pull/63017/merge
---
2019-07-26T17:54:48.3686286Z do so (now or later) by using -b with the checkout command again. Example:
2019-07-26T17:54:48.3687783Z 
2019-07-26T17:54:48.3688717Z   git checkout -b <new-branch-name>
2019-07-26T17:54:48.3689509Z 
2019-07-26T17:54:48.3689678Z HEAD is now at dbad794e2 Merge b756c5d163752c8684015f9280d8f703b93d90ce into 1a563362865e6051d4c350544131228e8eff5138
2019-07-26T17:54:48.3830949Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-07-26T17:54:48.3834097Z ==============================================================================
2019-07-26T17:54:48.3834156Z Task         : Bash
2019-07-26T17:54:48.3834204Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-07-26T18:56:29.8663306Z .................................................................................................... 200/5862
2019-07-26T18:56:34.6825664Z .................................................................................................... 300/5862
2019-07-26T18:56:39.0426869Z .................................................................................................... 400/5862
2019-07-26T18:56:42.9342096Z .................................................................................................... 500/5862
2019-07-26T18:56:46.8966031Z ........................................................................i........................... 600/5862
2019-07-26T18:56:56.6235173Z .................................................................................................... 800/5862
2019-07-26T18:57:02.9964269Z .................................................................................................... 900/5862
2019-07-26T18:57:08.8927469Z .................................................................................................... 1000/5862
2019-07-26T18:57:08.8927469Z .................................................................................................... 1000/5862
2019-07-26T18:57:15.3382241Z i...........i....................................................................................... 1100/5862
2019-07-26T18:57:19.8797058Z ...................F..........iiiii................................................................. 1200/5862
2019-07-26T18:57:26.8416080Z .................................................................................................... 1400/5862
2019-07-26T18:57:29.9885551Z .................................................................................................... 1500/5862
2019-07-26T18:57:34.4173422Z .................................................................................................... 1600/5862
2019-07-26T18:57:37.5566395Z .................................................................................................... 1700/5862
2019-07-26T18:57:37.5566395Z .................................................................................................... 1700/5862
2019-07-26T18:57:41.8119512Z ......................................................................i............................. 1800/5862
2019-07-26T18:57:51.3193752Z .................................................................................................... 2000/5862
2019-07-26T18:57:56.3542287Z .................................................................................................... 2100/5862
2019-07-26T18:58:00.6152862Z .................................................................................................... 2200/5862
2019-07-26T18:58:00.6152862Z .................................................................................................... 2200/5862
2019-07-26T18:58:05.0848428Z ......................................................i............................................. 2300/5862
2019-07-26T18:58:16.1382036Z .................................................................................................... 2500/5862
2019-07-26T18:58:20.8821514Z .................................................................................................... 2600/5862
2019-07-26T18:58:26.8425313Z .................................................................................................... 2700/5862
2019-07-26T18:58:31.8551656Z .................................................................................................... 2800/5862
2019-07-26T18:58:31.8551656Z .................................................................................................... 2800/5862
2019-07-26T18:58:36.6786722Z .................................................................................................... 2900/5862
2019-07-26T18:58:42.4483199Z .................................................................................................... 3000/5862
2019-07-26T18:58:47.0701663Z .................................................................................................... 3100/5862
2019-07-26T18:58:52.9647607Z .................................................................................................... 3200/5862
2019-07-26T18:58:57.0079095Z .................................................................................................... 3300/5862
2019-07-26T18:59:01.5398152Z .................................................................................................... 3400/5862
2019-07-26T18:59:07.2985361Z .................................................................................................... 3500/5862
2019-07-26T18:59:11.2358065Z .....................i.............................................................................. 3600/5862
2019-07-26T18:59:15.6843492Z .................................................................................................ii. 3700/5862
2019-07-26T18:59:19.4753167Z ..i..ii............................................................................................. 3800/5862
2019-07-26T18:59:29.1066666Z .................................................................................................... 4000/5862
2019-07-26T18:59:29.1066666Z .................................................................................................... 4000/5862
2019-07-26T18:59:33.8664727Z ................ii.................................................................................. 4100/5862
2019-07-26T18:59:36.2248799Z .....................................i.............................................................. 4200/5862
2019-07-26T18:59:38.7647995Z ...............................................................................F.F.................. 4300/5862
2019-07-26T18:59:41.2971168Z .......i............................................................................................ 4400/5862
2019-07-26T18:59:45.3967055Z .........F.F........................................................................................ 4500/5862
2019-07-26T19:00:08.5830156Z .................................................................................................... 4700/5862
2019-07-26T19:00:12.5909439Z .................................................................................................... 4800/5862
2019-07-26T19:00:17.6503947Z .................................................................................................... 4900/5862
2019-07-26T19:00:24.6695494Z .................................................................................................... 5000/5862
---
2019-07-26T19:00:53.9504142Z .................................................................................................... 5500/5862
2019-07-26T19:00:58.8456524Z .................................................................................................... 5600/5862
2019-07-26T19:01:02.4198361Z .................................................................................................... 5700/5862
2019-07-26T19:01:06.0708677Z .................................................................................................... 5800/5862
2019-07-26T19:01:08.5577909Z ..i...........................................................
2019-07-26T19:01:08.5624484Z 
2019-07-26T19:01:08.5625077Z ---- [ui] ui/did_you_mean/issue-49746-unicode-confusable-in-float-literal-expt.rs stdout ----
2019-07-26T19:01:08.5625946Z 
2019-07-26T19:01:08.5625946Z 
2019-07-26T19:01:08.5667146Z error: /checkout/src/test/ui/did_you_mean/issue-49746-unicode-confusable-in-float-literal-expt.rs:1: unexpected error: '1:53: 1:54: expected one of `.`, `;`, `?`, or an operator, found `−`'
2019-07-26T19:01:08.5669724Z 
2019-07-26T19:01:08.5669829Z error: 1 unexpected errors found, 0 expected errors not found
2019-07-26T19:01:08.5669899Z status: exit code: 1
2019-07-26T19:01:08.5671185Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/did_you_mean/issue-49746-unicode-confusable-in-float-literal-expt.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/did_you_mean/issue-49746-unicode-confusable-in-float-literal-expt" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/did_you_mean/issue-49746-unicode-confusable-in-float-literal-expt/auxiliary" "-A" "unused"
2019-07-26T19:01:08.5671591Z unexpected errors (from JSON output): [
2019-07-26T19:01:08.5671640Z     Error {
2019-07-26T19:01:08.5671706Z         line_num: 1,
2019-07-26T19:01:08.5671750Z         kind: Some(
2019-07-26T19:01:08.5671835Z         ),
2019-07-26T19:01:08.5671835Z         ),
2019-07-26T19:01:08.5672204Z         msg: "1:53: 1:54: expected one of `.`, `;`, `?`, or an operator, found `−`",
2019-07-26T19:01:08.5672313Z ]
2019-07-26T19:01:08.5672363Z 
2019-07-26T19:01:08.5672714Z thread '[ui] ui/did_you_mean/issue-49746-unicode-confusable-in-float-literal-expt.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:1505:13
2019-07-26T19:01:08.5672783Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-07-26T19:01:08.5672783Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-07-26T19:01:08.5672838Z 
2019-07-26T19:01:08.5673077Z ---- [ui] ui/parser/lex-bad-token.rs stdout ----
2019-07-26T19:01:08.5673123Z 
2019-07-26T19:01:08.5673480Z error: /checkout/src/test/ui/parser/lex-bad-token.rs:1: unexpected error: '1:1: 1:2: expected item, found `●`'
2019-07-26T19:01:08.5673538Z 
2019-07-26T19:01:08.5673588Z error: 1 unexpected errors found, 0 expected errors not found
2019-07-26T19:01:08.5673635Z status: exit code: 1
2019-07-26T19:01:08.5674653Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/parser/lex-bad-token.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/lex-bad-token" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/lex-bad-token/auxiliary" "-A" "unused"
2019-07-26T19:01:08.5674758Z unexpected errors (from JSON output): [
2019-07-26T19:01:08.5674826Z     Error {
2019-07-26T19:01:08.5674884Z         line_num: 1,
2019-07-26T19:01:08.5674927Z         kind: Some(
2019-07-26T19:01:08.5675031Z         ),
2019-07-26T19:01:08.5675031Z         ),
2019-07-26T19:01:08.5675285Z         msg: "1:1: 1:2: expected item, found `●`",
2019-07-26T19:01:08.5675395Z ]
2019-07-26T19:01:08.5675422Z 
2019-07-26T19:01:08.5675715Z thread '[ui] ui/parser/lex-bad-token.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:1505:13
2019-07-26T19:01:08.5675758Z 
2019-07-26T19:01:08.5675758Z 
2019-07-26T19:01:08.5676184Z ---- [ui] ui/parser/lex-stray-backslash.rs stdout ----
2019-07-26T19:01:08.5676232Z 
2019-07-26T19:01:08.5676555Z error: /checkout/src/test/ui/parser/lex-stray-backslash.rs:1: unexpected error: '1:1: 1:2: expected item, found `\`'
2019-07-26T19:01:08.5676616Z 
2019-07-26T19:01:08.5676897Z error: 1 unexpected errors found, 0 expected errors not found
2019-07-26T19:01:08.5676944Z status: exit code: 1
2019-07-26T19:01:08.5678402Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/parser/lex-stray-backslash.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/lex-stray-backslash" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/lex-stray-backslash/auxiliary" "-A" "unused"
2019-07-26T19:01:08.5678687Z unexpected errors (from JSON output): [
2019-07-26T19:01:08.5678755Z     Error {
2019-07-26T19:01:08.5678803Z         line_num: 1,
2019-07-26T19:01:08.5678847Z         kind: Some(
2019-07-26T19:01:08.5678949Z         ),
2019-07-26T19:01:08.5678949Z         ),
2019-07-26T19:01:08.5678996Z         msg: "1:1: 1:2: expected item, found `\\`",
2019-07-26T19:01:08.5679099Z ]
2019-07-26T19:01:08.5679135Z 
2019-07-26T19:01:08.5679484Z thread '[ui] ui/parser/lex-stray-backslash.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:1505:13
2019-07-26T19:01:08.5679526Z 
2019-07-26T19:01:08.5679526Z 
2019-07-26T19:01:08.5679781Z ---- [ui] ui/parser/unicode-chars.rs stdout ----
2019-07-26T19:01:08.5679815Z 
2019-07-26T19:01:08.5680141Z error: /checkout/src/test/ui/parser/unicode-chars.rs:2: unexpected error: '2:14: 2:15: expected one of `.`, `;`, `?`, or an operator, found `;`'
2019-07-26T19:01:08.5680203Z 
2019-07-26T19:01:08.5680262Z error: 1 unexpected errors found, 0 expected errors not found
2019-07-26T19:01:08.5680309Z status: exit code: 1
2019-07-26T19:01:08.5681609Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/parser/unicode-chars.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/unicode-chars" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/unicode-chars/auxiliary" "-A" "unused"
2019-07-26T19:01:08.5681725Z unexpected errors (from JSON output): [
2019-07-26T19:01:08.5681792Z     Error {
2019-07-26T19:01:08.5681837Z         line_num: 2,
2019-07-26T19:01:08.5681880Z         kind: Some(
2019-07-26T19:01:08.5681982Z         ),
2019-07-26T19:01:08.5681982Z         ),
2019-07-26T19:01:08.5682268Z         msg: "2:14: 2:15: expected one of `.`, `;`, `?`, or an operator, found `;`",
2019-07-26T19:01:08.5682384Z ]
2019-07-26T19:01:08.5682412Z 
2019-07-26T19:01:08.5682708Z thread '[ui] ui/parser/unicode-chars.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:1505:13
2019-07-26T19:01:08.5682769Z 
2019-07-26T19:01:08.5682769Z 
2019-07-26T19:01:08.5683007Z ---- [ui] ui/parser/unicode-quote-chars.rs stdout ----
2019-07-26T19:01:08.5683041Z 
2019-07-26T19:01:08.5683356Z error: /checkout/src/test/ui/parser/unicode-quote-chars.rs:4: unexpected error: '4:26: 4:27: unknown start of token: \u{201d}'
2019-07-26T19:01:08.5683416Z 
2019-07-26T19:01:08.5683798Z error: /checkout/src/test/ui/parser/unicode-quote-chars.rs:4: unexpected help message: '4:26: 4:27: Unicode character '”' (Right Double Quotation Mark) looks like '"' (Quotation Mark), but it is not'
2019-07-26T19:01:08.5683848Z 
2019-07-26T19:01:08.5684164Z error: /checkout/src/test/ui/parser/unicode-quote-chars.rs:4: unexpected error: '4:15: 4:20: expected token: `,`'
2019-07-26T19:01:08.5684316Z 
2019-07-26T19:01:08.5684373Z error: 3 unexpected errors found, 0 expected errors not found
2019-07-26T19:01:08.5684439Z status: exit code: 1
2019-07-26T19:01:08.5685854Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/parser/unicode-quote-chars.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/unicode-quote-chars" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/unicode-quote-chars/auxiliary" "-A" "unused"
2019-07-26T19:01:08.5686130Z unexpected errors (from JSON output): [
2019-07-26T19:01:08.5686177Z     Error {
2019-07-26T19:01:08.5686224Z         line_num: 4,
2019-07-26T19:01:08.5686285Z         kind: Some(
2019-07-26T19:01:08.5686599Z         ),
2019-07-26T19:01:08.5686599Z         ),
2019-07-26T19:01:08.5686956Z         msg: "4:26: 4:27: unknown start of token: \\u{201d}",
2019-07-26T19:01:08.5687074Z     Error {
2019-07-26T19:01:08.5687116Z         line_num: 4,
2019-07-26T19:01:08.5687177Z         kind: Some(
2019-07-26T19:01:08.5687220Z             Help,
2019-07-26T19:01:08.5687220Z             Help,
2019-07-26T19:01:08.5687262Z         ),
2019-07-26T19:01:08.5687704Z         msg: "4:26: 4:27: Unicode character \'”\' (Right Double Quotation Mark) looks like \'\"\' (Quotation Mark), but it is not",
2019-07-26T19:01:08.5687821Z     Error {
2019-07-26T19:01:08.5687882Z         line_num: 4,
2019-07-26T19:01:08.5687925Z         kind: Some(
2019-07-26T19:01:08.5687966Z             Error,
2019-07-26T19:01:08.5687966Z             Error,
2019-07-26T19:01:08.5688008Z         ),
2019-07-26T19:01:08.5688074Z         msg: "4:15: 4:20: expected token: `,`",
2019-07-26T19:01:08.5688157Z ]
2019-07-26T19:01:08.5688203Z 
2019-07-26T19:01:08.5688527Z thread '[ui] ui/parser/unicode-quote-chars.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:1505:13
2019-07-26T19:01:08.5688568Z 
---
2019-07-26T19:01:08.5690207Z 
2019-07-26T19:01:08.5690473Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:535:22
2019-07-26T19:01:08.5690510Z 
2019-07-26T19:01:08.5690536Z 
2019-07-26T19:01:08.5692754Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-07-26T19:01:08.5693062Z 
2019-07-26T19:01:08.5693092Z 
2019-07-26T19:01:08.5693649Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-07-26T19:01:08.5693854Z Build completed unsuccessfully in 0:59:27
2019-07-26T19:01:08.5693854Z Build completed unsuccessfully in 0:59:27
2019-07-26T19:01:09.5574151Z ##[error]Bash exited with code '1'.
2019-07-26T19:01:09.5620129Z ##[section]Starting: Checkout
2019-07-26T19:01:09.5622724Z ==============================================================================
2019-07-26T19:01:09.5622799Z Task         : Get sources
2019-07-26T19:01:09.5622848Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented Jul 26, 2019

☔ The latest upstream changes (presumably #63015) made this pull request unmergeable. Please resolve the merge conflicts.

@bors bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jul 26, 2019
@petrochenkov petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 27, 2019
@petrochenkov
Copy link
Contributor

which approach should we take to fight cascading errors?

It seems like unknown tokens are usually some weird unicode punctuation (at least from tests).
In this case would probably not be very helpful to e.g. parse and unknown token as ExprKind::Err in expression position.

#62963 provided the best solution to this, IMO - treating unknown punctuation as some similar known punctuation.
If punctuation doesn't have anything similar, it's probably most reasonable to pass it to parser as a whitespace (i.e. skip it).

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 28, 2019
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-07-29T18:11:02.0155323Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-07-29T18:11:02.0326737Z ##[command]git config gc.auto 0
2019-07-29T18:11:02.0389517Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-07-29T18:11:02.0442078Z ##[command]git config --get-all http.proxy
2019-07-29T18:11:02.0563355Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63017/merge:refs/remotes/pull/63017/merge
---
2019-07-29T18:11:38.1266617Z do so (now or later) by using -b with the checkout command again. Example:
2019-07-29T18:11:38.1267570Z 
2019-07-29T18:11:38.1269436Z   git checkout -b <new-branch-name>
2019-07-29T18:11:38.1270799Z 
2019-07-29T18:11:38.1271586Z HEAD is now at 76a98f320 Merge a0ca45d10c513f5da4f2bbf0e99b688cebdbe7b4 into 04b88a9eba8abbac87eddcb2998beea09589c2c9
2019-07-29T18:11:38.1390257Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-07-29T18:11:38.1392864Z ==============================================================================
2019-07-29T18:11:38.1392926Z Task         : Bash
2019-07-29T18:11:38.1392959Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-07-29T19:06:50.3197566Z .................................................................................................... 1400/8804
2019-07-29T19:06:55.8917797Z .................................................................................................... 1500/8804
2019-07-29T19:07:08.2373641Z ................................................................i...............i................... 1600/8804
2019-07-29T19:07:15.8204417Z .................................................................................................... 1700/8804
2019-07-29T19:07:29.6259466Z ..................................................iiiii............................................. 1800/8804
2019-07-29T19:07:39.7470313Z .................................................................................................... 2000/8804
2019-07-29T19:07:42.4925033Z .................................................................................................... 2100/8804
2019-07-29T19:07:45.6209852Z .................................................................................................... 2200/8804
2019-07-29T19:07:51.9281965Z .................................................................................................... 2300/8804
---
2019-07-29T19:11:13.9860773Z .................................................................................................... 5200/8804
2019-07-29T19:11:24.5232170Z .................................................................................................... 5300/8804
2019-07-29T19:11:31.6764425Z ..i................................................................................................. 5400/8804
2019-07-29T19:11:36.4801017Z .................................................................................................... 5500/8804
2019-07-29T19:11:47.4359754Z ................................................................................................ii.. 5600/8804
2019-07-29T19:12:01.2861768Z .i..ii...........i.................................................................................. 5700/8804
2019-07-29T19:12:14.3704576Z .................................................................................................... 5900/8804
2019-07-29T19:12:18.8934431Z ................................................................................................i..i 6000/8804
2019-07-29T19:12:32.2722885Z i................................................................................................... 6100/8804
2019-07-29T19:12:47.7012955Z .................................................................................................... 6200/8804
---
2019-07-29T19:17:04.3445432Z  finished in 18.339
2019-07-29T19:17:04.3603156Z Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-29T19:17:05.1260085Z 
2019-07-29T19:17:05.1327753Z running 146 tests
2019-07-29T19:17:07.3336160Z i....iii......iii..iiii....i............................i..i................i....i.........ii.i.i..i 100/146
2019-07-29T19:17:09.1269648Z iii..............i.........iii.i......ii......
2019-07-29T19:17:09.1270951Z 
2019-07-29T19:17:09.1307624Z  finished in 4.564
2019-07-29T19:17:09.1319163Z Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-29T19:17:09.1338375Z 
---
2019-07-29T19:17:10.9036924Z  finished in 1.963
2019-07-29T19:17:10.9189805Z Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-29T19:17:11.0590963Z 
2019-07-29T19:17:11.0591563Z running 9 tests
2019-07-29T19:17:11.0592321Z iiiiiiiii
2019-07-29T19:17:11.0593824Z 
2019-07-29T19:17:11.0593986Z  finished in 0.140
2019-07-29T19:17:11.0748076Z Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-29T19:17:11.2169465Z 
---
2019-07-29T19:17:27.2832930Z  finished in 16.208
2019-07-29T19:17:27.2986710Z Check compiletest suite=debuginfo mode=debuginfo-gdb+lldb (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-29T19:17:27.4475726Z 
2019-07-29T19:17:27.4476134Z running 122 tests
2019-07-29T19:17:48.7537124Z .iiiii...i.....i..i...i..i.i.i..i.ii..i.i.....i..i....i..........iiii..........i...ii...i.......ii.i 100/122
2019-07-29T19:17:52.7181016Z .i.i......iii.i.....ii
2019-07-29T19:17:52.7181913Z 
2019-07-29T19:17:52.7186875Z  finished in 25.420
2019-07-29T19:17:52.7192945Z Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-29T19:17:52.7193244Z Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
2019-07-29T19:24:20.5410430Z failures:
2019-07-29T19:24:20.5414232Z 
2019-07-29T19:24:20.5416406Z ---- [rustdoc] rustdoc/bad-codeblock-syntax.rs stdout ----
2019-07-29T19:24:20.5416460Z 
2019-07-29T19:24:20.5416648Z error: htmldocck failed!
2019-07-29T19:24:20.5417121Z status: exit code: 1
2019-07-29T19:24:20.5417682Z command: "/usr/bin/python2.7" "/checkout/src/etc/htmldocck.py" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc/bad-codeblock-syntax" "/checkout/src/test/rustdoc/bad-codeblock-syntax.rs"
2019-07-29T19:24:20.5417976Z ------------------------------------------
2019-07-29T19:24:20.5418001Z 
2019-07-29T19:24:20.5418184Z ------------------------------------------
2019-07-29T19:24:20.5418378Z stderr:
2019-07-29T19:24:20.5418378Z stderr:
2019-07-29T19:24:20.5418584Z ------------------------------------------
2019-07-29T19:24:20.5418620Z 2: @has check failed
2019-07-29T19:24:20.5418668Z  `XPATH PATTERN` did not match
2019-07-29T19:24:20.5418853Z  // @has - '//*[@class="docblock"]/pre/code' '\_'
2019-07-29T19:24:20.5418889Z 9: @has check failed
2019-07-29T19:24:20.5418936Z  `XPATH PATTERN` did not match
2019-07-29T19:24:20.5419129Z  // @has - '//*[@class="docblock"]/pre/code' '`baz::foobar`'
2019-07-29T19:24:20.5419166Z 16: @has check failed
2019-07-29T19:24:20.5419206Z  `XPATH PATTERN` did not match
2019-07-29T19:24:20.5419407Z  // @has - '//*[@class="docblock"]/pre/code' '\_'
2019-07-29T19:24:20.5419468Z Encountered 3 errors
2019-07-29T19:24:20.5419490Z 
2019-07-29T19:24:20.5419711Z ------------------------------------------
2019-07-29T19:24:20.5419736Z 
---
2019-07-29T19:24:20.5420545Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:533:22
2019-07-29T19:24:20.5420588Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-07-29T19:24:20.5420613Z 
2019-07-29T19:24:20.5420648Z 
2019-07-29T19:24:20.5422832Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/rustdoc" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "rustdoc" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-07-29T19:24:20.5423148Z 
2019-07-29T19:24:20.5423184Z 
2019-07-29T19:24:20.5429881Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-07-29T19:24:20.5458454Z Build completed unsuccessfully in 1:06:16
2019-07-29T19:24:20.5458454Z Build completed unsuccessfully in 1:06:16
2019-07-29T19:24:22.8561653Z ##[error]Bash exited with code '1'.
2019-07-29T19:24:22.8598146Z ##[section]Starting: Checkout
2019-07-29T19:24:22.8599662Z ==============================================================================
2019-07-29T19:24:22.8599707Z Task         : Get sources
2019-07-29T19:24:22.8599744Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@matklad
Copy link
Member Author

matklad commented Jul 30, 2019

If punctuation doesn't have anything similar, it's probably most reasonable to pass it to parser as a whitespace (i.e. skip it).

Yeah, just skipping it on the parser level seems the right behavior. I am not sure that using whitespace token is best here: it seems like a hack, and it also erases information about the original token. I've added a dedicated TokenKind::Unknown for this, which is explicitly skiped together with comments. It seems better than just using whitespace, but I worry that, between Unknown and LiteralKind::Err, we now have two error tokens.

@matklad matklad changed the title WIP: remove special code-path for handing unknown tokens Remove special code-path for handing unknown tokens Jul 30, 2019
@matklad matklad force-pushed the no-fatal branch 2 times, most recently from d43949e to e6ce091 Compare July 30, 2019 10:45
@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 30, 2019
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-07-30T10:46:11.6594496Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-07-30T10:46:11.6782313Z ##[command]git config gc.auto 0
2019-07-30T10:46:11.6843361Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-07-30T10:46:11.6901171Z ##[command]git config --get-all http.proxy
2019-07-30T10:46:11.7035675Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63017/merge:refs/remotes/pull/63017/merge
---
2019-07-30T10:46:46.4494261Z do so (now or later) by using -b with the checkout command again. Example:
2019-07-30T10:46:46.4494301Z 
2019-07-30T10:46:46.4494719Z   git checkout -b <new-branch-name>
2019-07-30T10:46:46.4494755Z 
2019-07-30T10:46:46.4494793Z HEAD is now at 2a86c9731 Merge e6ce0917ffe5abd185b838799af9c781f5a1a4ad into 4eeaaa722d6ac6d24de6e4d3faefb7c44e674b37
2019-07-30T10:46:46.4633628Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-07-30T10:46:46.4636183Z ==============================================================================
2019-07-30T10:46:46.4636247Z Task         : Bash
2019-07-30T10:46:46.4636282Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-07-30T11:45:29.2724931Z .................................................................................................... 1400/8813
2019-07-30T11:45:34.9980466Z .................................................................................................... 1500/8813
2019-07-30T11:45:47.1516665Z ................................................................i...............i................... 1600/8813
2019-07-30T11:45:54.5687573Z .................................................................................................... 1700/8813
2019-07-30T11:46:09.2446285Z ..................................................iiiii............................................. 1800/8813
2019-07-30T11:46:20.0910055Z .................................................................................................... 2000/8813
2019-07-30T11:46:22.4857503Z .................................................................................................... 2100/8813
2019-07-30T11:46:25.8928639Z .................................................................................................... 2200/8813
2019-07-30T11:46:32.3153374Z .................................................................................................... 2300/8813
---
2019-07-30T11:50:20.8184031Z .................................................................................................... 5300/8813
2019-07-30T11:50:27.7152889Z ........i........................................................................................... 5400/8813
2019-07-30T11:50:32.8243822Z .................................................................................................... 5500/8813
2019-07-30T11:50:44.8663159Z .................................................................................................... 5600/8813
2019-07-30T11:50:58.3857774Z ..ii...i..ii...........i............................................................................ 5700/8813
2019-07-30T11:51:14.6151793Z .................................................................................................... 5900/8813
2019-07-30T11:51:19.0023155Z .................................................................................................... 6000/8813
2019-07-30T11:51:19.0023155Z .................................................................................................... 6000/8813
2019-07-30T11:51:32.6843440Z ..i..ii............................................................................................. 6100/8813
2019-07-30T11:51:50.5666477Z .............................................i...................................................... 6300/8813
2019-07-30T11:51:52.5859423Z .................................................................................................... 6400/8813
2019-07-30T11:51:54.9126159Z ...............i.................................................................................... 6500/8813
2019-07-30T11:51:59.2317824Z .................................................................................................... 6600/8813
---
2019-07-30T11:56:20.8929205Z  finished in 19.695
2019-07-30T11:56:20.9099943Z Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T11:56:21.0679540Z 
2019-07-30T11:56:21.0681296Z running 146 tests
2019-07-30T11:56:24.2004335Z i....iii......iii..iiii....i............................i..i................i....i.........ii.i.i..i 100/146
2019-07-30T11:56:26.0177829Z iii..............i.........iii.i......ii......
2019-07-30T11:56:26.0178657Z 
2019-07-30T11:56:26.0181935Z  finished in 5.108
2019-07-30T11:56:26.0348910Z Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T11:56:26.1921078Z 
---
2019-07-30T11:56:28.1463010Z  finished in 2.111
2019-07-30T11:56:28.1619748Z Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T11:56:28.3067958Z 
2019-07-30T11:56:28.3068975Z running 9 tests
2019-07-30T11:56:28.3071076Z iiiiiiiii
2019-07-30T11:56:28.3071709Z 
2019-07-30T11:56:28.3073946Z  finished in 0.145
2019-07-30T11:56:28.3249331Z Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T11:56:28.4870488Z 
---
2019-07-30T11:56:45.6742304Z  finished in 17.349
2019-07-30T11:56:45.6909264Z Check compiletest suite=debuginfo mode=debuginfo-gdb+lldb (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T11:56:45.8383844Z 
2019-07-30T11:56:45.8384266Z running 122 tests
2019-07-30T11:57:07.8132368Z .iiiii...i.....i..i...i..i.i.i..i.ii..i.i.....i..i....i..........iiii..........i...ii...i.......ii.i 100/122
2019-07-30T11:57:12.1664690Z .i.i......iii.i.....ii
2019-07-30T11:57:12.1666274Z 
2019-07-30T11:57:12.1670244Z  finished in 26.476
2019-07-30T11:57:12.1677992Z Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T11:57:12.1678603Z Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
2019-07-30T12:10:48.7954002Z 
2019-07-30T12:10:48.7954870Z    Doc-tests core
2019-07-30T12:10:52.7034330Z 
2019-07-30T12:10:52.7036080Z running 2379 tests
2019-07-30T12:11:04.5383705Z ......iiiii......................................................................................... 100/2379
2019-07-30T12:11:16.3249932Z .........................................................................ii......................... 200/2379
2019-07-30T12:11:44.3102509Z .................................................................................................... 400/2379
2019-07-30T12:11:44.3102509Z .................................................................................................... 400/2379
2019-07-30T12:11:54.3695115Z ..............................i..i.................iiii............................................. 500/2379
2019-07-30T12:12:16.9197723Z .................................................................................................... 700/2379
2019-07-30T12:12:28.1497108Z .................................................................................................... 800/2379
2019-07-30T12:12:39.2600353Z .................................................................................................... 900/2379
2019-07-30T12:12:50.7396370Z .................................................................................................... 1000/2379
---
2019-07-30T12:17:55.5512097Z 
2019-07-30T12:17:55.5512331Z running 989 tests
2019-07-30T12:18:19.4331392Z i................................................................................................... 100/989
2019-07-30T12:18:32.2318297Z .................................................................................................... 200/989
2019-07-30T12:18:41.0298052Z .................iii......i......i...i......i....................................................... 300/989
2019-07-30T12:18:45.7662356Z .................................................................................................... 400/989
2019-07-30T12:18:53.8367136Z ................................i..i.................................ii............................. 500/989
2019-07-30T12:19:08.2299785Z .................................................................................................... 700/989
2019-07-30T12:19:08.2299785Z .................................................................................................... 700/989
2019-07-30T12:19:16.4420654Z ...............iiii................................................................................. 800/989
2019-07-30T12:19:30.7216846Z .................................................................................................... 900/989
2019-07-30T12:19:38.1454698Z .....................................iiii................................................
2019-07-30T12:19:38.1454863Z 
2019-07-30T12:19:38.1676870Z  finished in 237.247
2019-07-30T12:19:38.1696710Z Testing unwind stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T12:19:38.3629717Z     Finished release [optimized] target(s) in 0.19s
---
2019-07-30T12:39:07.2859884Z  finished in 33.634
2019-07-30T12:39:07.3178929Z Check compiletest suite=run-make-fulldeps mode=run-make (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-07-30T12:39:07.4799252Z 
2019-07-30T12:39:07.4800039Z running 199 tests
2019-07-30T12:39:42.8334661Z ....................i...ii................................................................i......... 100/199
2019-07-30T12:40:27.6826268Z ..............................iiii.......i..........iiii.iii....................................i..
2019-07-30T12:40:27.6831585Z 
2019-07-30T12:40:27.6834771Z  finished in 80.365
2019-07-30T12:40:27.6840501Z doc tests for: /checkout/src/doc/rustdoc/src/command-line-arguments.md
2019-07-30T12:40:27.6939510Z doc tests for: /checkout/src/doc/rustdoc/src/documentation-tests.md
---
2019-07-30T12:40:57.4053840Z 
2019-07-30T12:40:57.4053976Z + error: unknown start of token: \
2019-07-30T12:40:57.4054305Z +  --> <doctest>:1:1
2019-07-30T12:40:57.4054492Z +   |
2019-07-30T12:40:57.4054868Z + 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4055804Z + 
2019-07-30T12:40:57.4056002Z + error: unknown start of token: \
2019-07-30T12:40:57.4056431Z +  --> <doctest>:1:43
2019-07-30T12:40:57.4056640Z +   |
2019-07-30T12:40:57.4056640Z +   |
2019-07-30T12:40:57.4057047Z + 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4057404Z + 
2019-07-30T12:40:57.4057578Z + error: unknown start of token: \
2019-07-30T12:40:57.4058130Z +  --> <doctest>:1:60
2019-07-30T12:40:57.4058371Z +   |
2019-07-30T12:40:57.4058371Z +   |
2019-07-30T12:40:57.4058805Z + 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4059319Z + 
2019-07-30T12:40:57.4059480Z 1 warning: could not parse code block as Rust code
2019-07-30T12:40:57.4059839Z 2   --> $DIR/invalid-syntax.rs:3:5
2019-07-30T12:40:57.4060016Z 3    |
2019-07-30T12:40:57.4060016Z 3    |
2019-07-30T12:40:57.4060137Z 
2019-07-30T12:40:57.4060546Z 6 LL | | /// \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4061423Z 8    | |_______^
2019-07-30T12:40:57.4061776Z -    |
2019-07-30T12:40:57.4062166Z -    = note: error from rustc: unknown start of token: \
2019-07-30T12:40:57.4062362Z 11 help: mark blocks that do not contain Rust code as text
---
2019-07-30T12:40:57.4063439Z +  --> <doctest>:3:30
2019-07-30T12:40:57.4063619Z +   |
2019-07-30T12:40:57.4063747Z + 3 |    |     ^^^^^^ did you mean `baz::foobar`?
2019-07-30T12:40:57.4063875Z +   |                              ^
2019-07-30T12:40:57.4064256Z + help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4064444Z +   |
2019-07-30T12:40:57.4064795Z + 3 |    |     ^^^^^^ did you mean 'baz::foobar`?
2019-07-30T12:40:57.4065103Z + 
2019-07-30T12:40:57.4065860Z + error: unknown start of token: `
2019-07-30T12:40:57.4066282Z +  --> <doctest>:3:42
2019-07-30T12:40:57.4066460Z +   |
2019-07-30T12:40:57.4066460Z +   |
2019-07-30T12:40:57.4066607Z + 3 |    |     ^^^^^^ did you mean `baz::foobar`?
2019-07-30T12:40:57.4066775Z +   |                                          ^
2019-07-30T12:40:57.4067207Z + help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4067760Z + 3 |    |     ^^^^^^ did you mean `baz::foobar'?
2019-07-30T12:40:57.4067946Z +   |                                          ^
2019-07-30T12:40:57.4068103Z + 
2019-07-30T12:40:57.4068260Z 16 warning: could not parse code block as Rust code
---
2019-07-30T12:40:57.4082681Z 68 
2019-07-30T12:40:57.4082806Z + error: unknown start of token: `
2019-07-30T12:40:57.4083109Z +  --> <doctest>:1:1
2019-07-30T12:40:57.4083263Z +   |
2019-07-30T12:40:57.4083404Z + 1 | ```
2019-07-30T12:40:57.4083540Z +   | ^
2019-07-30T12:40:57.4083897Z + help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4084061Z +   |
2019-07-30T12:40:57.4084346Z + 1 | '``
2019-07-30T12:40:57.4084639Z + 
2019-07-30T12:40:57.4084778Z + error: unknown start of token: `
2019-07-30T12:40:57.4085079Z +  --> <doctest>:1:2
2019-07-30T12:40:57.4085776Z +   |
2019-07-30T12:40:57.4085776Z +   |
2019-07-30T12:40:57.4085927Z + 1 | ```
2019-07-30T12:40:57.4086077Z +   |  ^
2019-07-30T12:40:57.4086519Z + help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4086706Z +   |
2019-07-30T12:40:57.4087065Z + 1 | `'`
2019-07-30T12:40:57.4087243Z +   |  ^
2019-07-30T12:40:57.4087712Z + error: unknown start of token: `
2019-07-30T12:40:57.4088125Z +  --> <doctest>:1:3
2019-07-30T12:40:57.4088442Z +   |
2019-07-30T12:40:57.4088442Z +   |
2019-07-30T12:40:57.4088579Z + 1 | ```
2019-07-30T12:40:57.4088732Z +   |   ^
2019-07-30T12:40:57.4089277Z + help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4089441Z +   |
2019-07-30T12:40:57.4089734Z + 1 | ``'
2019-07-30T12:40:57.4089890Z +   |   ^
2019-07-30T12:40:57.4090171Z 69 warning: could not parse code block as Rust code
2019-07-30T12:40:57.4090484Z 70   --> $DIR/invalid-syntax.rs:55:9
2019-07-30T12:40:57.4090645Z 71    |
2019-07-30T12:40:57.4090752Z 
---
2019-07-30T12:40:57.4100660Z + 
2019-07-30T12:40:57.4100778Z + error: unknown start of token: `
2019-07-30T12:40:57.4101099Z +  --> <rustdoc-highlighting>:1:1
2019-07-30T12:40:57.4101257Z +   |
2019-07-30T12:40:57.4101375Z + 1 | ```
2019-07-30T12:40:57.4101519Z +   | ^
2019-07-30T12:40:57.4101866Z + help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4102029Z +   |
2019-07-30T12:40:57.4102336Z + 1 | '``
2019-07-30T12:40:57.4107146Z + 
2019-07-30T12:40:57.4107205Z + error: unknown start of token: \
2019-07-30T12:40:57.4107606Z +  --> <rustdoc-highlighting>:2:1
2019-07-30T12:40:57.4107655Z +   |
---
2019-07-30T12:40:57.4109720Z +  --> <rustdoc-highlighting>:3:30
2019-07-30T12:40:57.4109760Z +   |
2019-07-30T12:40:57.4109814Z + 3 |    |     ^^^^^^ did you mean `baz::foobar`?
2019-07-30T12:40:57.4109856Z +   |                              ^
2019-07-30T12:40:57.4110095Z + help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4110155Z +   |
2019-07-30T12:40:57.4110357Z + 3 |    |     ^^^^^^ did you mean 'baz::foobar`?
2019-07-30T12:40:57.4110436Z + 
2019-07-30T12:40:57.4110489Z + error: unknown start of token: \
2019-07-30T12:40:57.4110678Z +  --> <rustdoc-highlighting>:1:1
2019-07-30T12:40:57.4110716Z +   |
2019-07-30T12:40:57.4110716Z +   |
2019-07-30T12:40:57.4110969Z + 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4111054Z 97 
2019-07-30T12:40:57.4111088Z 98 
2019-07-30T12:40:57.4111130Z 
2019-07-30T12:40:57.4111153Z 
2019-07-30T12:40:57.4111153Z 
2019-07-30T12:40:57.4111193Z The actual stderr differed from the expected stderr.
2019-07-30T12:40:57.4111469Z Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/invalid-syntax/invalid-syntax.stderr
2019-07-30T12:40:57.4111929Z To update references, rerun the tests and pass the `--bless` flag
2019-07-30T12:40:57.4112159Z To only update this specific test, also pass `--test-args invalid-syntax.rs`
2019-07-30T12:40:57.4112243Z error: 1 errors occurred comparing output.
2019-07-30T12:40:57.4112281Z status: exit code: 0
2019-07-30T12:40:57.4112281Z status: exit code: 0
2019-07-30T12:40:57.4112886Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/src/test/rustdoc-ui/invalid-syntax.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/invalid-syntax" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/invalid-syntax/auxiliary"
2019-07-30T12:40:57.4113163Z ------------------------------------------
2019-07-30T12:40:57.4113215Z 
2019-07-30T12:40:57.4113405Z ------------------------------------------
2019-07-30T12:40:57.4113444Z stderr:
2019-07-30T12:40:57.4113444Z stderr:
2019-07-30T12:40:57.4113639Z ------------------------------------------
2019-07-30T12:40:57.4113681Z error: unknown start of token: \
2019-07-30T12:40:57.4136600Z  --> <doctest>:1:1
2019-07-30T12:40:57.4136663Z   |
2019-07-30T12:40:57.4136947Z 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4137051Z 
2019-07-30T12:40:57.4137112Z error: unknown start of token: \
2019-07-30T12:40:57.4137334Z  --> <doctest>:1:43
2019-07-30T12:40:57.4137390Z   |
2019-07-30T12:40:57.4137390Z   |
2019-07-30T12:40:57.4137662Z 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4137759Z 
2019-07-30T12:40:57.4137802Z error: unknown start of token: \
2019-07-30T12:40:57.4138368Z  --> <doctest>:1:60
2019-07-30T12:40:57.4138420Z   |
2019-07-30T12:40:57.4138420Z   |
2019-07-30T12:40:57.4138872Z 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4138949Z 
2019-07-30T12:40:57.4139000Z warning: could not parse code block as Rust code
2019-07-30T12:40:57.4139265Z   --> /checkout/src/test/rustdoc-ui/invalid-syntax.rs:3:5
2019-07-30T12:40:57.4139308Z    |
2019-07-30T12:40:57.4139308Z    |
2019-07-30T12:40:57.4139345Z LL |   /// ```
2019-07-30T12:40:57.4139395Z    |  _____^
2019-07-30T12:40:57.4139818Z LL | | /// \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4139988Z    | |_______^
2019-07-30T12:40:57.4140029Z help: mark blocks that do not contain Rust code as text
2019-07-30T12:40:57.4140067Z    |
2019-07-30T12:40:57.4140114Z LL | /// ```text
2019-07-30T12:40:57.4140114Z LL | /// ```text
2019-07-30T12:40:57.4140150Z    |     ^^^^^^^
2019-07-30T12:40:57.4140175Z 
2019-07-30T12:40:57.4140211Z error: unknown start of token: `
2019-07-30T12:40:57.4140463Z  --> <doctest>:3:30
2019-07-30T12:40:57.4140536Z   |
2019-07-30T12:40:57.4140589Z 3 |    |     ^^^^^^ did you mean `baz::foobar`?
2019-07-30T12:40:57.4140642Z   |                              ^
2019-07-30T12:40:57.4140891Z help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4140934Z   |
2019-07-30T12:40:57.4141137Z 3 |    |     ^^^^^^ did you mean 'baz::foobar`?
2019-07-30T12:40:57.4141216Z 
2019-07-30T12:40:57.4141261Z error: unknown start of token: `
2019-07-30T12:40:57.4141455Z  --> <doctest>:3:42
2019-07-30T12:40:57.4141494Z   |
2019-07-30T12:40:57.4141494Z   |
2019-07-30T12:40:57.4141540Z 3 |    |     ^^^^^^ did you mean `baz::foobar`?
2019-07-30T12:40:57.4141582Z   |                                          ^
2019-07-30T12:40:57.4141828Z help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4142234Z 3 |    |     ^^^^^^ did you mean `baz::foobar'?
2019-07-30T12:40:57.4142283Z   |                                          ^
2019-07-30T12:40:57.4142309Z 
2019-07-30T12:40:57.4142346Z warning: could not parse code block as Rust code
---
2019-07-30T12:40:57.4146011Z 
2019-07-30T12:40:57.4146068Z warning: could not parse code block as Rust code
2019-07-30T12:40:57.4146355Z   --> /checkout/src/test/rustdoc-ui/invalid-syntax.rs:41:9
2019-07-30T12:40:57.4146404Z    |
2019-07-30T12:40:57.4146463Z LL |   ///     code with bad syntax
2019-07-30T12:40:57.4146549Z LL | | ///     \_
2019-07-30T12:40:57.4146591Z    | |__________^
2019-07-30T12:40:57.4146626Z 
2019-07-30T12:40:57.4146668Z error: unknown start of token: `
2019-07-30T12:40:57.4146668Z error: unknown start of token: `
2019-07-30T12:40:57.4146876Z  --> <doctest>:1:1
2019-07-30T12:40:57.4146932Z   |
2019-07-30T12:40:57.4146972Z 1 | ```
2019-07-30T12:40:57.4147012Z   | ^
2019-07-30T12:40:57.4147278Z help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4147526Z 1 | '``
2019-07-30T12:40:57.4147579Z   | ^
2019-07-30T12:40:57.4147606Z 
2019-07-30T12:40:57.4147655Z error: unknown start of token: `
2019-07-30T12:40:57.4147655Z error: unknown start of token: `
2019-07-30T12:40:57.4147869Z  --> <doctest>:1:2
2019-07-30T12:40:57.4147913Z   |
2019-07-30T12:40:57.4147963Z 1 | ```
2019-07-30T12:40:57.4148004Z   |  ^
2019-07-30T12:40:57.4148272Z help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4148521Z 1 | `'`
2019-07-30T12:40:57.4148564Z   |  ^
2019-07-30T12:40:57.4148591Z 
2019-07-30T12:40:57.4148640Z error: unknown start of token: `
2019-07-30T12:40:57.4148640Z error: unknown start of token: `
2019-07-30T12:40:57.4148986Z  --> <doctest>:1:3
2019-07-30T12:40:57.4149023Z   |
2019-07-30T12:40:57.4149057Z 1 | ```
2019-07-30T12:40:57.4149098Z   |   ^
2019-07-30T12:40:57.4149321Z help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4149530Z 1 | ``'
2019-07-30T12:40:57.4149565Z   |   ^
2019-07-30T12:40:57.4149587Z 
2019-07-30T12:40:57.4149631Z warning: could not parse code block as Rust code
---
2019-07-30T12:40:57.4151543Z 
2019-07-30T12:40:57.4151580Z warning: doc comment contains an invalid Rust code block
2019-07-30T12:40:57.4151789Z   --> /checkout/src/test/rustdoc-ui/invalid-syntax.rs:63:1
2019-07-30T12:40:57.4151829Z    |
2019-07-30T12:40:57.4151874Z LL | / #[doc = "```"]
2019-07-30T12:40:57.4151908Z LL | | /// \_
2019-07-30T12:40:57.4151944Z LL | | #[doc = "```"]
2019-07-30T12:40:57.4152025Z    |
2019-07-30T12:40:57.4152064Z    = help: mark blocks that do not contain Rust code as text: ```text
2019-07-30T12:40:57.4152091Z 
2019-07-30T12:40:57.4152136Z error: unknown start of token: \
---
2019-07-30T12:40:57.4153198Z  --> <rustdoc-highlighting>:1:1
2019-07-30T12:40:57.4153241Z   |
2019-07-30T12:40:57.4153275Z 1 | ```
2019-07-30T12:40:57.4153308Z   | ^
2019-07-30T12:40:57.4153717Z help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4154080Z 1 | '``
2019-07-30T12:40:57.4154113Z   | ^
2019-07-30T12:40:57.4154139Z 
2019-07-30T12:40:57.4154171Z error: unknown start of token: \
---
2019-07-30T12:40:57.4155811Z  --> <rustdoc-highlighting>:3:30
2019-07-30T12:40:57.4155856Z   |
2019-07-30T12:40:57.4155913Z 3 |    |     ^^^^^^ did you mean `baz::foobar`?
2019-07-30T12:40:57.4155960Z   |                              ^
2019-07-30T12:40:57.4156232Z help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
2019-07-30T12:40:57.4156288Z   |
2019-07-30T12:40:57.4156530Z 3 |    |     ^^^^^^ did you mean 'baz::foobar`?
2019-07-30T12:40:57.4156609Z 
2019-07-30T12:40:57.4156666Z error: unknown start of token: \
2019-07-30T12:40:57.4156884Z  --> <rustdoc-highlighting>:1:1
2019-07-30T12:40:57.4156928Z   |
2019-07-30T12:40:57.4156928Z   |
2019-07-30T12:40:57.4157192Z 1 | \__________pkt->size___________/          \_result->size_/ \__pkt->size__/
2019-07-30T12:40:57.4157268Z 
2019-07-30T12:40:57.4157293Z 
2019-07-30T12:40:57.4157517Z ------------------------------------------
2019-07-30T12:40:57.4157556Z 
---
2019-07-30T12:40:57.4158497Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:533:22
2019-07-30T12:40:57.4158555Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-07-30T12:40:57.4158596Z 
2019-07-30T12:40:57.4158621Z 
2019-07-30T12:40:57.4160285Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/rustdoc-ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-07-30T12:40:57.4160575Z 
2019-07-30T12:40:57.4160774Z 
2019-07-30T12:40:57.4160814Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-07-30T12:40:57.4161016Z Build completed unsuccessfully in 1:48:13
2019-07-30T12:40:57.4161016Z Build completed unsuccessfully in 1:48:13
2019-07-30T12:40:58.4342962Z ##[error]Bash exited with code '1'.
2019-07-30T12:40:58.4394792Z ##[section]Starting: Checkout
2019-07-30T12:40:58.4397192Z ==============================================================================
2019-07-30T12:40:58.4397261Z Task         : Get sources
2019-07-30T12:40:58.4397326Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

self.lexer.try_next_token().map_err(|()| HighlightError::LexError)
let token = self.lexer.next_token();
if let token::Unknown(..) = &token.kind {
return Err(HighlightError::LexError);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is HighlightError::LexError necessary?
Can't the highlighter treat the token as a whitespace and continue, similarly to parser?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly to minimize the diff in code and tests. I think rustdoc side needs a different approach altogether, to avoid duplicating lexer errors in two passes. But rusdoc can be improved separatelly

@petrochenkov
Copy link
Contributor

I worry that, between Unknown and LiteralKind::Err, we now have two error token

Well, since their recovery strategy is different ("skip" vs "parse as {Expr,Ty,Whatever}Kind::Err"), we'll need two different error tokens anyway (or at least a flag in one token).

I'm mildly skeptical about introducing Unknown since its behavior is identical to Whitespace, but I guess it's not bad either.

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 31, 2019
@bors
Copy link
Contributor

bors commented Aug 1, 2019

☔ The latest upstream changes (presumably #63194) made this pull request unmergeable. Please resolve the merge conflicts.

@matklad
Copy link
Member Author

matklad commented Aug 5, 2019

Filed #63284 for rustdoc fix (which we probably should do regardless of the current PR) and updated the tests to squash irrelevant errors.

I'm mildly skeptical about introducing Unknown since its behavior is identical to Whitespace, but I guess it's not bad either.

Yeah, I get roughly the same feelings. However it seems that merging whitespace and unknown later would be easier than splitting them.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 5, 2019
@petrochenkov
Copy link
Contributor

@bors r+

@bors
Copy link
Contributor

bors commented Aug 5, 2019

📌 Commit b3e8c8b has been approved by petrochenkov

@bors bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Aug 5, 2019
Centril added a commit to Centril/rust that referenced this pull request Aug 5, 2019
Remove special code-path for handing unknown tokens

In `StringReader`, we have a buffer of fatal errors, which is used only in a single case: when we see something which is not a reasonable token at all, like `🦀`. I think a more straightforward thing to do here is to produce an explicit error token in this case, and let the next layer (the parser), deal with it.

However currently this leads to duplicated error messages. What should we do with this? Naively, I would think that emitting (just emitting, not raising) `FatalError` should stop other errors, but looks like this is not the case? We can also probably tweak parser on the case-by-case basis, to avoid emitting "expected" errors if the current token is an `Err`. I personally also fine with cascading errors in this case: it's quite unlikely that you actually type a fully invalid token.

@petrochenkov, which approach should we take to fight cascading errors?
Centril added a commit to Centril/rust that referenced this pull request Aug 6, 2019
Remove special code-path for handing unknown tokens

In `StringReader`, we have a buffer of fatal errors, which is used only in a single case: when we see something which is not a reasonable token at all, like `🦀`. I think a more straightforward thing to do here is to produce an explicit error token in this case, and let the next layer (the parser), deal with it.

However currently this leads to duplicated error messages. What should we do with this? Naively, I would think that emitting (just emitting, not raising) `FatalError` should stop other errors, but looks like this is not the case? We can also probably tweak parser on the case-by-case basis, to avoid emitting "expected" errors if the current token is an `Err`. I personally also fine with cascading errors in this case: it's quite unlikely that you actually type a fully invalid token.

@petrochenkov, which approach should we take to fight cascading errors?
Centril added a commit to Centril/rust that referenced this pull request Aug 6, 2019
Remove special code-path for handing unknown tokens

In `StringReader`, we have a buffer of fatal errors, which is used only in a single case: when we see something which is not a reasonable token at all, like `🦀`. I think a more straightforward thing to do here is to produce an explicit error token in this case, and let the next layer (the parser), deal with it.

However currently this leads to duplicated error messages. What should we do with this? Naively, I would think that emitting (just emitting, not raising) `FatalError` should stop other errors, but looks like this is not the case? We can also probably tweak parser on the case-by-case basis, to avoid emitting "expected" errors if the current token is an `Err`. I personally also fine with cascading errors in this case: it's quite unlikely that you actually type a fully invalid token.

@petrochenkov, which approach should we take to fight cascading errors?
Centril added a commit to Centril/rust that referenced this pull request Aug 6, 2019
Remove special code-path for handing unknown tokens

In `StringReader`, we have a buffer of fatal errors, which is used only in a single case: when we see something which is not a reasonable token at all, like `🦀`. I think a more straightforward thing to do here is to produce an explicit error token in this case, and let the next layer (the parser), deal with it.

However currently this leads to duplicated error messages. What should we do with this? Naively, I would think that emitting (just emitting, not raising) `FatalError` should stop other errors, but looks like this is not the case? We can also probably tweak parser on the case-by-case basis, to avoid emitting "expected" errors if the current token is an `Err`. I personally also fine with cascading errors in this case: it's quite unlikely that you actually type a fully invalid token.

@petrochenkov, which approach should we take to fight cascading errors?
bors added a commit that referenced this pull request Aug 6, 2019
Rollup of 14 pull requests

Successful merges:

 - #61457 (Implement DoubleEndedIterator for iter::{StepBy, Peekable, Take})
 - #63017 (Remove special code-path for handing unknown tokens)
 - #63184 (Explaining the reason why validation is performed in to_str of path.rs)
 - #63230 (Make use of possibly uninitialized data [E0381] a hard error)
 - #63260 (fix UB in a test)
 - #63264 (Revert "Rollup merge of #62696 - chocol4te:fix_#62194, r=estebank")
 - #63272 (Some more libsyntax::attr cleanup)
 - #63285 (Remove leftover AwaitOrigin)
 - #63287 (Don't store &Span)
 - #63293 (Clarify align_to's requirements and obligations)
 - #63295 (improve align_offset docs)
 - #63299 (Make qualify consts in_projection use PlaceRef)
 - #63312 (doc: fix broken sentence)
 - #63315 (Fix #63313)

Failed merges:

r? @ghost
@bors bors merged commit b3e8c8b into rust-lang:master Aug 6, 2019
@matklad matklad deleted the no-fatal branch August 12, 2019 09:30
bors added a commit that referenced this pull request Jan 6, 2020
buffer lexer errors in rustdoc syntax checking

The code isn't ideal (I really would like to display the errors inline), but this at least gets us to where we were before #63017.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants