[BUG] import fails with latest antlr4-python3-runtime

[ayushdg]

Minimal Code To Reproduce

import fugue_sql

Error message:

Error Trace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "miniconda3/envs/terr/lib/python3.8/site-packages/fugue_sql/__init__.py", line 4, in <module>
    from fugue_sql.workflow import FugueSQLWorkflow, fsql
  File "miniconda3/envs/terr/lib/python3.8/site-packages/fugue_sql/workflow.py", line 4, in <module>
    from fugue import (
  File "miniconda3/envs/terr/lib/python3.8/site-packages/fugue/__init__.py", line 17, in <module>
    from fugue.execution.execution_engine import ExecutionEngine, SQLEngine
  File "miniconda3/envs/terr/lib/python3.8/site-packages/fugue/execution/__init__.py", line 3, in <module>
    from fugue.execution.factory import (
  File "miniconda3/envs/terr/lib/python3.8/site-packages/fugue/execution/factory.py", line 4, in <module>
    from fugue.execution.native_execution_engine import NativeExecutionEngine
  File "miniconda3/envs/terr/lib/python3.8/site-packages/fugue/execution/native_execution_engine.py", line 34, in <module>
    from qpd_pandas import run_sql_on_pandas
  File "miniconda3/envs/terr/lib/python3.8/site-packages/qpd_pandas/__init__.py", line 3, in <module>
    from qpd_pandas.engine import QPDPandasEngine, run_sql_on_pandas
  File "miniconda3/envs/terr/lib/python3.8/site-packages/qpd_pandas/engine.py", line 6, in <module>
    from qpd import PandasLikeUtils, QPDEngine, run_sql
  File "miniconda3/envs/terr/lib/python3.8/site-packages/qpd/__init__.py", line 7, in <module>
    from qpd.run import run_sql
  File "miniconda3/envs/terr/lib/python3.8/site-packages/qpd/run.py", line 3, in <module>
    from qpd._parser.sql import QPDSql
  File "miniconda3/envs/terr/lib/python3.8/site-packages/qpd/_parser/sql.py", line 6, in <module>
    from _qpd_antlr import QPDLexer, QPDParser
  File "miniconda3/envs/terr/lib/python3.8/site-packages/_qpd_antlr/__init__.py", line 3, in <module>
    from _qpd_antlr.sqlLexer import sqlLexer as QPDLexer
  File "miniconda3/envs/terr/lib/python3.8/site-packages/_qpd_antlr/sqlLexer.py", line 1313, in <module>
    class sqlLexer(Lexer):
  File "miniconda3/envs/terr/lib/python3.8/site-packages/_qpd_antlr/sqlLexer.py", line 1315, in sqlLexer
    atn = ATNDeserializer().deserialize(serializedATN())
  File "miniconda3/envs/terr/lib/python3.8/site-packages/antlr4/atn/ATNDeserializer.py", line 28, in deserialize
    self.checkVersion()
  File "miniconda3/envs/terr/lib/python3.8/site-packages/antlr4/atn/ATNDeserializer.py", line 50, in checkVersion
    raise Exception("Could not deserialize ATN with version " + str(version) + " (expected " + str(SERIALIZED_VERSION) + ").")
Exception: Could not deserialize ATN with version  (expected 4).

Describe the bug
On creating a new environment either via pip pip install fugue[sql] or conda conda install -c conda-forge fugue, importing fugue_sql fails with errors pointing to antlr4. Caught by upstream CI in dask-sql: dask-contrib/dask-sql#455.

There was a new 4.10 antlr4 (& antlr4-python-runtime) release recently which now gets picked up by the solvers that seems to be the cause of incompatibility, confirmed that things work with the older 4.9.3.

Expected behavior
Working import either via pinning to an older version of antlr or updating to be compatible with the latest version.

Environment (please complete the following information):

• Backend: N/A
• Backend version: N/A
• Python version: 3.8
• OS: linux (ubuntu 20.04)

[goodwanghan]

@ayushdg thanks for reporting. The incompatibility between antlr 4.10 and 4.9 is unexpected. It will take some effort to make it work. So I just made a quick fix: adding the version cap for antlr4-python-runtime. And in the later versions I will remove the cap when we fix the root cause.

[goodwanghan]

And DuckDB also just got a breaking change released today at 0.3.3.... what a day...

[goodwanghan]

@ayushdg I have released Fugue 0.6.6, it should have solved this issue

[ayushdg]

Thanks @goodwanghan, appreciate the quick fix. Can confirm that the 0.6.6 release does fix this 😄 .

Just adding a quick note about conda though it's not directly related to fugue: People getting fugue from conda will still see issues in imports since the conda solver picks up the newer antlr version. The fugue package doesn't explicitly specify antlr in it's conda recipe but qpd one of the packages fugue depends on, has the antlr dependency and faces the same import issues.