Skip to content

Commit

Permalink
Shrink Token.
Browse files Browse the repository at this point in the history
From 72 bytes to 12 bytes (on x86-64).

There are two parts to this:
- Changing various source code offsets from 64-bit to 32-bit. This is
  not a problem because the rest of rustc also uses 32-bit source code
  offsets. This means `Token` is no longer `Copy` but this causes no
  problems.
- Removing the `RawStrError` from `LiteralKind`. Raw string literal
  invalidity is now indicated by a `None` value within
  `RawStr`/`RawByteStr`, and the new `validate_raw_str` function can be
  used to re-lex an invalid raw string literal to get the `RawStrError`.

There is one very small change in behaviour. Previously, if a raw string
literal matched both the `InvalidStarter` and `TooManyHashes` cases,
the latter would override the former. This has now changed, because
`raw_double_quoted_string` now uses `?` and so returns immediately upon
detecting the `InvalidStarter` case. I think this is a slight
improvement to report the earlier-detected error, and it explains the
change in the `test_too_many_hashes` test.

The commit also removes a couple of comments that refer to rust-lang#77629 and
say that the size of these types don't affect performance. These
comments are wrong, though the performance effect is small.
  • Loading branch information
nnethercote committed Jul 31, 2022
1 parent 3641b0c commit 09f9ace
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 4 deletions.
2 changes: 1 addition & 1 deletion clippy_lints/src/matches/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1112,7 +1112,7 @@ fn span_contains_cfg(cx: &LateContext<'_>, s: Span) -> bool {
let mut pos = 0usize;
let mut iter = tokenize(&snip).map(|t| {
let start = pos;
pos += t.len;
pos += t.len as usize;
(t.kind, start..pos)
});

Expand Down
2 changes: 1 addition & 1 deletion clippy_lints/src/undocumented_unsafe_blocks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ fn text_has_safety_comment(src: &str, line_starts: &[BytePos], offset: usize) ->
if line.starts_with("/*") {
let src = src[line_start..line_starts.last().unwrap().to_usize() - offset].trim_start();
let mut tokens = tokenize(src);
return src[..tokens.next().unwrap().len]
return src[..tokens.next().unwrap().len as usize]
.to_ascii_uppercase()
.contains("SAFETY:")
&& tokens.all(|t| t.kind == TokenKind::Whitespace);
Expand Down
4 changes: 2 additions & 2 deletions clippy_utils/src/hir_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ impl HirEqInterExpr<'_, '_, '_> {
let mut left_pos = 0;
let left = tokenize(&left)
.map(|t| {
let end = left_pos + t.len;
let end = left_pos + t.len as usize;
let s = &left[left_pos..end];
left_pos = end;
(t, s)
Expand All @@ -156,7 +156,7 @@ impl HirEqInterExpr<'_, '_, '_> {
let mut right_pos = 0;
let right = tokenize(&right)
.map(|t| {
let end = right_pos + t.len;
let end = right_pos + t.len as usize;
let s = &right[right_pos..end];
right_pos = end;
(t, s)
Expand Down

0 comments on commit 09f9ace

Please sign in to comment.