Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Create Fugue pytest fixtures and plugins #504

Closed
7 tasks done
goodwanghan opened this issue Aug 16, 2023 · 0 comments · Fixed by #502 or #528
Closed
7 tasks done

[FEATURE] Create Fugue pytest fixtures and plugins #504

goodwanghan opened this issue Aug 16, 2023 · 0 comments · Fixed by #502 or #528
Labels
enhancement New feature or request unit test
Milestone

Comments

@goodwanghan
Copy link
Collaborator

goodwanghan commented Aug 16, 2023

Projects including Fugue itself needs to test with different backends. The current way to test is to manually create Spark, Dask, Ray and Duckdb sessions, test and stop them. In order to get good unit testing performance, it's actually tricky to setup those backends. Another issues is, if there is session level pytest fixture, then some sessions will be started and stopped in each single test, adding significant overhead.

So, we will create fixtures for Fugue and related projects. As the first release, we will have these hard coded fixtures (with the best possible performance for unit tests)

  • Spark support, fugue_spark_session
  • Dask support: fugue_dask_client
  • Ray support: fugue_ray_session
  • Duckdb support: fugue_duckdb_connection
  • Make them configurable in the common pytest config
  • Make them available automatically as long as fugue is installed
  • Create enumerated engine (parameterized), so it will automatically test all available backend

To use them, we just need to pip install fugue with the specific extra, then here is an example:

import fugue.api as fa

def test_my_app(fugue_dask_client):
    with fa.engine_context(fugue_dask_client):
        """test"""

Currently, for spark backend, please keep using pytest-spark and spark_session provided by the package.

@goodwanghan goodwanghan modified the milestone: 0.8.7 Aug 16, 2023
@goodwanghan goodwanghan added enhancement New feature or request unit test labels Aug 16, 2023
@goodwanghan goodwanghan changed the title [FEATURE] Fugue pytest fixtures [FEATURE] Create Fugue pytest fixtures Aug 16, 2023
@goodwanghan goodwanghan reopened this Aug 16, 2023
@goodwanghan goodwanghan added this to the 0.9.0 milestone Dec 1, 2023
@goodwanghan goodwanghan changed the title [FEATURE] Create Fugue pytest fixtures [FEATURE] Create Fugue pytest fixtures and plugins Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment