Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[draft] morpheus-llm conda package #1912

Closed
wants to merge 17 commits into from
Closed

[draft] morpheus-llm conda package #1912

wants to merge 17 commits into from

Conversation

AnuradhaKaruppiah
Copy link
Contributor

@AnuradhaKaruppiah AnuradhaKaruppiah commented Sep 24, 2024 •
edited
Loading

  • Create morpheus-llm conda package
  • Update github workflow to upload the morpheus-llm conda package to anaconda

@AnuradhaKaruppiah
Maintain a todo.md for DFP refactoring.
6a75379 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Move production dfp stages/modules, messages and utils to morpheus_dfp
d238af7 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update import paths within the morpheus_dfp module
1c18b17 
Style used for purposes of consistency -
1. Imports in __init__ and pipeline are absolute
2. All other imports (withing morpheus_dfp) are relative

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Setup files for morpheus_dfp python package
89ee982 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Build morpheus_dfp python package via cmake
6c06ddf 
The morpheus_dfp package is entirely python and setup.py can be used directly to
create this package. However for the sake of consistency cmake with
mopheus_utils_build_python_package is used.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update dfp imports in examples and tests
cdda885 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
morpheus_dfp conda recipe and CI updates
c464c5a 
1. Add a build flag to disable compiling morpheus-core
2. Add a build flag for morpheus_dfp
3. Add morpheus_dfp conda recipe

Cleanup:
1. Avoid enabling morpheus extended package flags in the runner container.
Instead set them via the build scripts. This is mainly to avoid the
confusion in managing the flags.
2. Removed some of the build flags passed to the conda recipes to avoid
confusion.
@AnuradhaKaruppiah
Update github workflow to build/upload the morpheus_dfp conda package
46258e5 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah AnuradhaKaruppiah requested review from a team as code owners September 24, 2024 22:55
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Sep 24, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@AnuradhaKaruppiah AnuradhaKaruppiah changed the title morpheus-llm conda package [draft] morpheus-llm conda package Sep 24, 2024
@AnuradhaKaruppiah AnuradhaKaruppiah marked this pull request as draft September 24, 2024 22:55
@AnuradhaKaruppiah AnuradhaKaruppiah added non-breaking Non-breaking change improvement Improvement to existing functionality conda-build Enables running the conda-build step on a PR conda-lib-build labels Sep 24, 2024
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Build morpheus_llm without building morpheus_core
0cc36c7 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Conda recipe for generating the morpheus-llm package
e001ce9 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update github workflow to build/upload the morpheus-llm conda package
93c5ca3 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Rename morpheus-core to morpheus-libs
dbb34a8 
1. ci/conda/recipes/morpheus will be removed once the morpheus runtime
   container is updated to install the morpheus libraries
2. morpheus-libs/meta.yaml will generate conda packages for all the morpheus
   libraries

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah AnuradhaKaruppiah force-pushed the package-llm branch 4 times, most recently from 53734d0 to 03ce1cf Compare September 30, 2024 21:08
@AnuradhaKaruppiah AnuradhaKaruppiah force-pushed the package-llm branch 3 times, most recently from 0fd41fc to 3950005 Compare October 1, 2024 02:31
@AnuradhaKaruppiah
Combine conda recipes into one one meta.yaml generating multiple outputs
93a8d7e 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
moprheus_llm cmake fixups
ef7dce3 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Revert "Update github workflow to build/upload the morpheus-llm conda…
f274e78 
… package"

This reverts commit 93c5ca3.
@AnuradhaKaruppiah
Drop conda-lib-build label
ed331eb 
1. conda-build is used for building conda pacakge for the morpheus libraries.
2. Dropped the conda package with "all components". This will be replaced with
the individual conda packages.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update the github workflow to use the conda split package script
5517e88 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

Moved these commits to PR #1897 . It was challenging to fixup the conda recipe across two PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv left review comments

Assignees
No one assigned
Labels
conda-build Enables running the conda-build step on a PR improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AnuradhaKaruppiah @mdemoret-nv