Reduce memory usage of DartScanner

[DartBot]

This issue was originally filed by zundel@google.com

---

When running some unit tests with heap space constrained to 32M, 2 tests run out of memory:

=== debugia32 dartc co19/LibTest/core/List/sort/List/sort/A01/t06 ===
=== debugia32 dartc co19/LibTest/core/List/sort/List/sort/A01/t05 ===

Top on the heap histogram are DartScanner.Location, DartScanner.Position and DartScanner.TokenData, with over 200K instances each adding up to 75M of the 32M heap.

1. The scanner currently tokenizes the entire file into memory, which may be over aggressive. If we do not keep references to these objects throughout the parse, we may be able to GC them if we only tokenize the file in chunks.

2. In DartScanner.Location, the code currently stores Position objects for start,end of each token. Each Position object contains 3 integers, line # ,column #, and offset.

We could reduce memory usage by storing only 2 integers in Location as byte offsets for start/end from the start of the file. Then, for that source file keep an index to indicate what offset corresponds with each line number. Since the column and line position is rarely accessed, we could use something as simple as an array and use binary search to find the right line number for a given character offset.

[DartBot]

This comment was originally written by zundel@google.com

---

We looked a bit at #­1 and it is tricky due to the scanner's ability to do rollback. We need to keep all parsed tokens in memory in case of rollback (or cleverly rewind and re-tokenize to a particular place in that case.)

As an alternative to save some memory, we noticed that the entire source is kept around. We tried to eliminate first all references to the source in DartScanner by making new String() instances for each TokenData.value field. That helped there, but the DartSource object also keeps around a full copy of the source.

[kasperl]

Removed Area-Compiler label.
Added Area-Analyzer label.

[bwilkerson]

Added this to the Later milestone.

[bwilkerson]

Removed this from the Later milestone.
Added this to the M3 milestone.

[scheglov]

Removed this from the M3 milestone.
Added this to the Later milestone.
Added AnalysisEngine label.

[DartBot]

This comment was originally written by amouravski@google.com

---

Added Editor-AnalysisEngine label.

[bwilkerson]

dartc has been deprecated in favor of the new analysis engine.

---

Removed AnalysisEngine, Editor-AnalysisEngine labels.
Added WontFix label.

[clayberg]

Removed this from the Later milestone.
Added this to the M5 milestone.