[FEA] Separate Pipeline Type Inference from Node Creation
#229
Comments
3 tasks
rapids-bot bot
pushed a commit
that referenced
this issue
Oct 19, 2023
…#1233) * Builds upon changes in PR #1175 * Type inference/checking moved to `Pipeline.pre_build` and doesn't require MRC * Wraps input/output types between stages into a `StageSchema` class which can later be expanded to describe tensors and dataframe columns. * `StreamWrapper` renamed to `BaseStage`. * `BaseStage` defines a new abstract method `compute_schema` * Fix unrelated bug in `generate_viz_frames_stage.py` where the `overwrite` argument was ignored leading to a failed assert. * Calling `add_edge`, `add_segment_edge`, or `add_stage` after calling `build` will trigger a failed assert. fixes #229 fixes #230 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1233
github-project-automation
bot
moved this from Review - Ready for Review
to Done
in Morpheus Boards
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Before integrating SRF, the
Pipeline.build()function would perform type inference and create the stage nodes synchronously. After integrating SRF, both type inference and node creation happen asynchronously in a single step. This means anything that depends on the type inference (for example,visualize()) cannot be run until after the pipeline has been completed.Describe the solution you'd like
Ideally, the type inference would be separated out and run synchronously in
build()leaving node creation to be completed in a separate step, asynchronously.Describe alternatives you've considered
Its possible that type inference could happen on demand, but this would be difficult with cyclic and non-linear pipelines.
The text was updated successfully, but these errors were encountered: