Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Remove execution from FugueWorkflow context manager, remove engine from FugueWorkflow #369

Closed
goodwanghan opened this issue Oct 10, 2022 · 0 comments · Fixed by #370

Comments

@goodwanghan
Copy link
Collaborator

goodwanghan commented Oct 10, 2022

Currently, we can use FugueWorkflow + execution engine in this way:

with FugueWorkflow(engine) as dag:
    dag.df(..).show()

This is not an ideal design. Workflow definition shouldn't be related with execution engine. So in this change, we no longer allow FugueWorkflow to take engine as input, and the with statement becomes a pure cosmetic syntax to make boundary of the dag definition. Instead, you can always do:

dag = FugueWorkflow({"compile_configs":"..."}):
dag.df(..).show()

dag.run(engine)

It's a bit more coding, but it separates compile time and run time definitions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant