Frog tokenizer doesn't issue good error messages

[jmesserly]

an example of how things are handled now:

  Token finishNumberExtra(int kind) {
    ...
    if (...) {
      _nextChar();
      // numbers cannot contain identifiers
      return _errorToken();
    }

The string is there in the comment :), but it needs to end up in an error message. The tricky part is, we don't want to issue world.error directly from the tokenizer. So we need to track the message another way--probably in the error token? Or alternatively, just have different TokenKinds for different types of errors, and then map that to the appropriate message.

There aren't a lot of tokenizer errors, but when you hit one, it's not fun :)

Better parser error recovery in http://code.google.com/p/dart/issues/detail?id=661 would also help.

[jmesserly]

(edit title)

---

Changed the title to: "Frog tokenizer doesn't issue good error messages".

[sigmundch]

I'm implementing first alternative above (save the error message in the error token).

We might eventually want to introduce different token kinds for the various types of errors so that we can more easily recover from errors in the parser, but I don't want to overgeneralize the solution until we get there.

---

Set owner to @sigmundch.

[sigmundch]

implementation submitted in r2201

---

Added Fixed label.