PyDBEngine for OmniSci engine and pyarrow.gandiva for pyarrow storage format are incompatible

[dchigarev]

PyDBEngine that is being imported when using Modin with STORAGE_FORMAT=Omnisci and pyarrow.gandiva - when using modin with STORAGE_FORMAT=Pyarrow are incompatible:
If ganvida is imported first everything is alright:

import pyarrow.gandiva
from modin.experimental.core.execution.native.implementations.omnisci_on_native.utils import (
    PyDbEngine,
)

But vice-versa it fails with segfault:

from modin.experimental.core.execution.native.implementations.omnisci_on_native.utils import (
    PyDbEngine,
)
import pyarrow.gandiva
# Output:
# : CommandLine Error: Option 'enable-vfe' registered more than once!
# LLVM ERROR: inconsistency in registered CommandLine options
# Aborted (core dumped)

Packages versions:

omniscidb-common          5.8.0                h1234567_1    conda-forge
omniscidbe                5.8.0            h1234567_1_cpu    conda-forge
pyomniscidbe              5.8.0           py38h1234567_1_cpu    conda-forge

arrow-cpp                 5.0.0           py38h43b8b71_5_cpu    conda-forge
pyarrow                   5.0.0           py38h1bc9799_5_cpu    conda-forge

Ideally, there shouldn't be scenarios when both of these modules are imported, however, as the recent #3783 showed, it still could happen sometimes.

The role of this issue is to track progress (if any) of resolving the incompatibility of these packages.

[vnlitvinov]

I wonder if #3783 fixed that, could someone from @modin-project/modin-omnisci please re-check?

[anmyachev]

Looks very outdated. The issue could not be reproduced as follows:

from modin.experimental.core.execution.native.implementations.hdk_on_native.db_worker import DbWorker
import pyarrow.gandiva

If I have not missed anything, then I suggest merging #6343 to close this issue.

UPD: I tested on windows so it worked, but on linux the error is still there

[YarShev]

HDK engine is deprecated and will be removed in a future version.