Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Break tokens before checking if they are 'probably equal' #72306

Merged
merged 3 commits into from
May 22, 2020

Commits on May 19, 2020

  1. Break tokens before checking if they are 'probably equal'

    Fixes rust-lang#68489
    
    When checking two `TokenStreams` to see if they are 'probably equal',
    we ignore the `IsJoint` information associated with each `TokenTree`.
    However, the `IsJoint` information determines whether adjacent tokens
    will be 'glued' (if possible) when construction the `TokenStream` - e.g.
    `[Gt Gt]` can be 'glued' to `BinOp(Shr)`.
    
    Since we are ignoring the `IsJoint` information, 'glued' and 'unglued'
    tokens are equivalent for determining if two `TokenStreams` are
    'probably equal'. Therefore, we need to 'unglue' all tokens in the
    stream to avoid false negatives (which cause us to throw out the cached
    tokens, losing span information).
    Aaron1011 committed May 19, 2020
    Configuration menu
    Copy the full SHA
    9b2b8a5 View commit details
    Browse the repository at this point in the history
  2. Use a fixed-point iteration when breaking tokens

    Some tokens need to be broken in a loop until we reach
    'unbreakable' tokens.
    Aaron1011 committed May 19, 2020
    Configuration menu
    Copy the full SHA
    4a8ccdc View commit details
    Browse the repository at this point in the history

Commits on May 20, 2020

  1. Fix tests

    Aaron1011 committed May 20, 2020
    Configuration menu
    Copy the full SHA
    633293f View commit details
    Browse the repository at this point in the history