Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## Summary This PR implements the `TokenFlags` which will be stored on each `Token` and certain flags will be set depending on the token kind. Currently, it's equivalent to `AnyStringFlags` but it will help in the future to provide additional information regarding certain tokens like unterminated string, number kinds, etc. The main motivation to add a `TokenFlags` is to store certain information related to the token which will then be used by downstream tools. Currently, the information is only related to string tokens. The downstream tools should not be allowed access to the flags directly, it's an implementation detail. Instead, methods will be provided on `Token` to query certain information. An example can be seen in the follow-up PR (#11592). For example, the `Stylist` and `Indexer` uses the string flags stored on `String`/`FStringStart` token to get certain information. They will be updated to use these flags instead, thus removing the need for `Tok` completely. Prior art in TypeScript: https://github.com/microsoft/TypeScript/blob/16beff101ae1dae0600820ebf22632ac8a40cfc8/src/compiler/types.ts#L2788-L2827
- Loading branch information