dataform cli format issue when columns contain "--"

[kevinzhou-izivia]

config {
  type: "table",
}

SELECT
  source_data_json.timestamp,
  source_data_json.`TEMPERATURE--BODY`,
  source_data_json.`ENERGY_ACTIVE_IMPORT_REGISTER--BODY`,

   FROM
  `XXX.YYY.ZZZ`

is formatted as

config {
  type: "table",
}

SELECT
  source_data_json.timestamp,
  source_data_json.`TEMPERATURE
  --BODY`,
  source_data_json.` ENERGY_ACTIVE_IMPORT_REGISTER
  --BODY`,
FROM
  `XXX.YYY.ZZZ`

expected:

config {
  type: "table",
}

SELECT
  source_data_json.timestamp,
  source_data_json.`TEMPERATURE--BODY`,
  source_data_json.`ENERGY_ACTIVE_IMPORT_REGISTER--BODY`,
FROM
  `XXX.YYY.ZZZ`

dataform --version returns 2.9.0

[Ekrekr]

@Ekrekr test that this is still reproducible after #1741 is merged.

[Ekrekr]

Issue is not fixed. I think this is caused by the lexing, where we still treat inner SQL literal strings as comments even when we shouldn't be

dataform/sqlx/lexer.ts

Line 374 in 2531b12

innerSqlBlockLexer[INNER_SQL_BLOCK_LEXER_TOKEN_NAMES.SINGLE_LINE_COMMENT] = /--.*?$/;

[kevin-zhou-dev]

Another maybe linked error ?
Executed with dataform cli version : 3.0.2

Example: definitions/test.sqlx

config { type: "view"} 
WITH int_table AS (
SELECT id
FROM `my_dataset.my_table`) -- test
SELECT id
FROM int_table

What we get
When running dataform format --actions="definitions/test.sqlx", an error is returned :
Errors encountered during formatting: definitions/test.sqlx: Formatter unable to determine final formatted form.

While we expect:

config {
  type: "view"
}

WITH
  int_table AS (
  SELECT
    id
  FROM
    `my_dataset.my_table`) -- test
SELECT
  id
FROM
  int_table