remove duplicated integration tests

[edublancas]

we have some duplicate integration tests, instead we should only write them once and parameterized them, just like we are already doing with many of our integration tests:

https://github.com/search?q=repo%3Aploomber%2Fjupysql%20test_query_count&type=code

[bryannho]

To clarify this issue, I should remove the duplicate test_query_count tests from test_mssql.py and test_oracle.py. Then I should modify test_query_count in test_generic_db_operations.py to include MSSQL and Oracle via parameterization?

[edublancas]

yes.

but those are some that I found. you might encounter other duplicated tests, so take look at the testing files. see if you spot anything duplicated

[bryannho]

I have attempted to run the integration tests using pytest src/tests/integration/test_generic_db_operations.py with my changes however am running into connection errors with all DB's. For example, clickhouse, oracle, and mssql return errors:

FAILED src/tests/integration/test_generic_db_operations.py::test_run_query[ip_with_clickhouse--LIMIT 3] - requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=N...

FAILED src/tests/integration/test_generic_db_operations.py::test_run_query[ip_with_oracle--FETCH FIRST 3 ROWS ONLY] - ModuleNotFoundError: No module named 'oracledb'

FAILED src/tests/integration/test_generic_db_operations.py::test_run_query[ip_with_MSSQL-TOP 3-] - ImportError: dlopen(/Users/bryanho/opt/anaconda3/envs/jupysql/lib/python3.10/sit...

It seems I am missing some setup but I cannot find a guide to running integration tests specifically. Is there anywhere else I can refer to troubleshoot this issue? Must I spin up an instance of each DB individually to ensure tests run correctly?

[edublancas]

we have an integration testing guide here: https://jupysql.ploomber.io/en/latest/community/developer-guide.html#integration-tests there's a command there that shows you how to run tests for specific dbs

looks like you're missing some dependencies, so you need to install them. the integration tests automatically spin up a database and shut it down (except for snowflake and redshift). the guide might be inaccurate since we haven't updated it so if you find any inaccuracies, open a PR.

[bryannho]

we have an integration testing guide here: https://jupysql.ploomber.io/en/latest/community/developer-guide.html#integration-tests there's a command there that shows you how to run tests for specific dbs

Thank you, I was able to solve most of the dependency issues with the guide. Will continue working

[bryannho]

After following the developer guide I have fixed a few issues however I am still not able to get integration tests for Oracle or MSSQL working. To run the tests, I ensure that Docker is running and execute pytest src/tests/integration -k oracle or pytest src/tests/integration -k mssql where I run into these errors:

Oracle Tests: sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:oracle.oracledb

MSSQL Tests: ImportError: dlopen(/Users/my_user/opt/anaconda3/envs/jupysql/lib/python3.10/site-packages/pyodbc.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (_SQLAllocHandle)

Thus far, I have tried:

1. Installing all dependencies with: pip install -e '.[integration]'
2. Installing a new version of Docker Desktop
3. For Oracle: Installing Colima and running colima start --cpu 4 --memory 4 --disk 30 --arch x86_64
4. For Oracle: Experimenting with different versions of SQLAlchemy, although this breaks other tests
5. For Oracle: Adding this code to conftest.py in accordance with this guide:
   import sys
import oracledb
oracledb.version = "8.3.0"
sys.modules["cx_Oracle"] = oracledb
6. For MSSQL: Reinstalling mssql and pyodbc with:
   brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
brew update
brew upgrade
HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql18 mssql-tools18
pip3 install --pre --no-binary :all: pyodbc --force-reinstall

The issue may be related to Apple Silicon, as I am running everything on Apple M1 Macbook Pro. Please let me know if these are known issues. I am aware that running integration tests locally is tricky, so I'll continue looking for workarounds.

[edublancas]

at this point, it might be better to just run them in the CI, so you don't have to find workarounds