refactor[parser]: remove ASTTokens

[charles-cooper]

What I did

remove asttokens from the parse machinery, since the method is buggy (see below bugs) and slow (#4272). brings down parse time (time spent in ast generation) between 40-70%.

fixes multiple tokenizer-related bugs, including

• bug: wrong column offset in error message #3430
• cannot log event which contains Attribute expr #3059
• 'IndexError: list index out of range' raised calling vyper.compiler.compile_code with 54 double quotes #2258
• compiler crash due to ASTTokens instantiation #4139
• feat[ux]: improve hint for events kwarg upgrade #4275 (comment)
• feat[lang]: auto-export events in ABI #3808 (comment)

How I did it

another way to adjust tokens

How to verify it

Commit message

this commit removes `asttokens` from the parse machinery, since the
method is buggy (see below bugs) and slow. this commit brings down
parse time (time spent in ast generation) between 40-70%.

the `mark_tokens()` machinery is replaced with a modified version of
`python.ast`'s `fix_missing_locations()` function, which recurses
through the AST and adds missing line info based on the parent node.

it also changes to a more consistent method for updating source
offsets that are modified by the `pre_parse` step, which fixes several
outstanding bugs with source location reporting.

there were some exceptions to the line info fixup working, the issues
and corresponding workarounds are described as follows:

- some python AST nodes returned by `ast.parse()` are singletons, which
  we work around by deepcopying the AST before operating on it.

- notably, there is an interaction between our AST annotation and
  `coverage.py` in the case of `USub`. in this commit we paper over the
  issue by simply always overriding line info for `USub` nodes. in the
  future, we should refactor `VyperNode` generation by bypassing the
  python AST annotation step entirely, which is a more proper fix to the
  problems encountered in this PR.

the `asttokens` package is not removed entirely since it still has a
limited usage inside of the natspec parser. we could remove it in a
future PR; for now it is out-of-scope.

referenced bugs:
- https://github.com/vyperlang/vyper/issues/2258
- https://github.com/vyperlang/vyper/issues/3059
- https://github.com/vyperlang/vyper/issues/3430
- https://github.com/vyperlang/vyper/issues/4139

Description for the changelog

Cute Animal Picture

[cyberthirst]

i find the name modification_offset confusing, it blends with adjustments and imo doesn't capture the essence. can't we name smth like original_types?

[cyberthirst]

can we generalize this code? it's pretty much the same thing 3 times

vyper/vyper/ast/pre_parser.py

Lines 289 to 324 in 5c1fb8b

	if string in VYPER_CLASS_TYPES and start[1] == 0:
	new_keyword = "class"
	toks = [TokenInfo(NAME, new_keyword, start, end, line)]
	adjustment = len(string) - len(new_keyword)
	# adjustments for following tokens
	lineno, col = start
	_col_adjustments[lineno] += adjustment
	modification_offsets[newstart] = VYPER_CLASS_TYPES[string]
	elif string in CUSTOM_STATEMENT_TYPES:
	new_keyword = "yield"
	adjustment = len(string) - len(new_keyword)
	# adjustments for following tokens
	lineno, col = start
	_col_adjustments[lineno] += adjustment
	toks = [TokenInfo(NAME, new_keyword, start, end, line)]
	modification_offsets[newstart] = CUSTOM_STATEMENT_TYPES[string]
	elif string in CUSTOM_EXPRESSION_TYPES:
	# a bit cursed technique to get untokenize to put
	# the new tokens in the right place so that modification_offsets
	# will work correctly.
	# (recommend comparing the result of parse with the
	# source code side by side to visualize the whitespace)
	new_keyword = "await"
	vyper_type = CUSTOM_EXPRESSION_TYPES[string]
	adjustment = len(string) - len(new_keyword)
	# adjustments for following tokens
	lineno, col = start
	_col_adjustments[lineno] += adjustment
	# fixup for when `extcall/staticcall` follows `log`
	modification_offsets[newstart] = vyper_type

[cyberthirst]

let's make a better comment hehe