diff --git a/CHANGELOG.md b/CHANGELOG.md index 21ab786dbf..016ac721ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,139 @@ See the License for the specific language governing permissions and limitations under the License. --> +# Morpheus 24.10.00 (01 Nov 2024) + +## 🚨 Breaking Changes + +- Support LLM pipelines in CPU-only mode ([#1906](https://github.com/nv-morpheus/Morpheus/pull/1906)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove Starter Digital Fingerprinting (DFP) ([#1903](https://github.com/nv-morpheus/Morpheus/pull/1903)) [@efajardo-nv](https://github.com/efajardo-nv) +- Finalize removing `MultiMessage` from Morpheus ([#1886](https://github.com/nv-morpheus/Morpheus/pull/1886)) [@yczhang-nv](https://github.com/yczhang-nv) +- Add support for a CPU-only Mode ([#1851](https://github.com/nv-morpheus/Morpheus/pull/1851)) [@dagardner-nv](https://github.com/dagardner-nv) +- Removing support for `MultiMessage` from stages ([#1803](https://github.com/nv-morpheus/Morpheus/pull/1803)) [@yczhang-nv](https://github.com/yczhang-nv) + +## 🐛 Bug Fixes + +- Pin boto3 and s3fs to compatible versions to resolve access denied errors ([#2039](https://github.com/nv-morpheus/Morpheus/pull/2039)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix phishing Python API example to match CLI example ([#2037](https://github.com/nv-morpheus/Morpheus/pull/2037)) [@dagardner-nv](https://github.com/dagardner-nv) +- Model updates and cleanup following upgrade to to triton 24.09 ([#2036](https://github.com/nv-morpheus/Morpheus/pull/2036)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Copy data files needed by root_cause_analysis to examples/data ([#2032](https://github.com/nv-morpheus/Morpheus/pull/2032)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Fix for duplicate row IDs in `log_parsing` output ([#2031](https://github.com/nv-morpheus/Morpheus/pull/2031)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix `log_parsing` example pipeline null output issue ([#2024](https://github.com/nv-morpheus/Morpheus/pull/2024)) [@yczhang-nv](https://github.com/yczhang-nv) +- Fixup file paths in the modular digital fingerprinting documentation. ([#2016](https://github.com/nv-morpheus/Morpheus/pull/2016)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Fix `DeserializeStage` to ensure output messages correctly contain the correct rows for each batch ([#2015](https://github.com/nv-morpheus/Morpheus/pull/2015)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix offset calculation when taking a slice of a `SlicedMessageMeta` ([#2006](https://github.com/nv-morpheus/Morpheus/pull/2006)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix CUDF's `Column.from_column_view` by copying it and adjusting. ([#2004](https://github.com/nv-morpheus/Morpheus/pull/2004)) [@cwharris](https://github.com/cwharris) +- Fix up file paths in the DFP README ([#2003](https://github.com/nv-morpheus/Morpheus/pull/2003)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Fix AttributeError: 'int' object has no attribute 'item' ([#1995](https://github.com/nv-morpheus/Morpheus/pull/1995)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix utilities submodule commit ([#1987](https://github.com/nv-morpheus/Morpheus/pull/1987)) [@cwharris](https://github.com/cwharris) +- Update `val-run-all.sh` to run cpp pipeline only ([#1986](https://github.com/nv-morpheus/Morpheus/pull/1986)) [@yczhang-nv](https://github.com/yczhang-nv) +- Fix `onnx-to-trt` utility ([#1984](https://github.com/nv-morpheus/Morpheus/pull/1984)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update Utilities submodule and fix compilation with latest build of MRC ([#1981](https://github.com/nv-morpheus/Morpheus/pull/1981)) [@mdemoret-nv](https://github.com/mdemoret-nv) +- Fix missing dependency in DFP Grafana example ([#1977](https://github.com/nv-morpheus/Morpheus/pull/1977)) [@efajardo-nv](https://github.com/efajardo-nv) +- Populate all the LFS data needed for running examples within the release container ([#1976](https://github.com/nv-morpheus/Morpheus/pull/1976)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Ensure timestamps are copied in `LLMEngineStage` ([#1975](https://github.com/nv-morpheus/Morpheus/pull/1975)) [@dagardner-nv](https://github.com/dagardner-nv) +- Install sentence-transformers via pip to avoid CPU-torch conda dependencies ([#1974](https://github.com/nv-morpheus/Morpheus/pull/1974)) [@efajardo-nv](https://github.com/efajardo-nv) +- Add `**kwargs` back to `NVFoundationLLMClient.generate_batch()` and `generate_batch_async()` ([#1967](https://github.com/nv-morpheus/Morpheus/pull/1967)) [@ashsong-nv](https://github.com/ashsong-nv) +- Benchmark updates/fixes ([#1958](https://github.com/nv-morpheus/Morpheus/pull/1958)) [@efajardo-nv](https://github.com/efajardo-nv) +- Improve test performance ([#1953](https://github.com/nv-morpheus/Morpheus/pull/1953)) [@dagardner-nv](https://github.com/dagardner-nv) +- Adopt updated utilities fix in-place Python installs ([#1952](https://github.com/nv-morpheus/Morpheus/pull/1952)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update cuda version for docker containers ([#1941](https://github.com/nv-morpheus/Morpheus/pull/1941)) [@dagardner-nv](https://github.com/dagardner-nv) +- Multiple fixes related to `SharedProcessPool` & `MultiProcessingStage` ([#1940](https://github.com/nv-morpheus/Morpheus/pull/1940)) [@yczhang-nv](https://github.com/yczhang-nv) +- Fix dask error in DFP Integrated training pipeline ([#1931](https://github.com/nv-morpheus/Morpheus/pull/1931)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove `SharedProcessPool.terminate()` related tests to avoid stack traces and blocking remote-ci ([#1929](https://github.com/nv-morpheus/Morpheus/pull/1929)) [@yczhang-nv](https://github.com/yczhang-nv) +- Provide a timeout to the queue.get call in `HttpServerSourceStage` to avoid spinlocking ([#1928](https://github.com/nv-morpheus/Morpheus/pull/1928)) [@dagardner-nv](https://github.com/dagardner-nv) +- Ensure that `pytest` is able to run without optional dependencies ([#1927](https://github.com/nv-morpheus/Morpheus/pull/1927)) [@dagardner-nv](https://github.com/dagardner-nv) +- Better handle exceptions generated in the `LLMEngine` to not show the `stoul` error ([#1922](https://github.com/nv-morpheus/Morpheus/pull/1922)) [@mdemoret-nv](https://github.com/mdemoret-nv) +- Fixing the docker build when Morpheus is a submodule ([#1914](https://github.com/nv-morpheus/Morpheus/pull/1914)) [@mdemoret-nv](https://github.com/mdemoret-nv) +- Build morpheus_llm by default ([#1911](https://github.com/nv-morpheus/Morpheus/pull/1911)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Fix conda path for missing llm packages ([#1907](https://github.com/nv-morpheus/Morpheus/pull/1907)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update `WriteToVectorDBStage` to re-raise errors from the underlying database ([#1905](https://github.com/nv-morpheus/Morpheus/pull/1905)) [@dagardner-nv](https://github.com/dagardner-nv) +- Avoid memory leak warnings from `pypdfium2` ([#1902](https://github.com/nv-morpheus/Morpheus/pull/1902)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove redundant copy of the `load_labels_file` method ([#1901](https://github.com/nv-morpheus/Morpheus/pull/1901)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix `Can't find 'action.yml'` CI error ([#1896](https://github.com/nv-morpheus/Morpheus/pull/1896)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix DFP integrated training Azure pipeline ([#1894](https://github.com/nv-morpheus/Morpheus/pull/1894)) [@yczhang-nv](https://github.com/yczhang-nv) +- Drop 'CI Pipeline / Check' dependency from the 'package-core' job ([#1885](https://github.com/nv-morpheus/Morpheus/pull/1885)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Python source stages now optionally receive a reference to `mrc.Subscription` ([#1881](https://github.com/nv-morpheus/Morpheus/pull/1881)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix `Unregistered type : mrc::pymrc::coro::BoostFibersMainPyAwaitable` error ([#1869](https://github.com/nv-morpheus/Morpheus/pull/1869)) [@dagardner-nv](https://github.com/dagardner-nv) +- Revert PR_1736 "Always run the PR builder step even if others are cancelled" ([#1860](https://github.com/nv-morpheus/Morpheus/pull/1860)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- ensure columns are strings before concatenation ([#1857](https://github.com/nv-morpheus/Morpheus/pull/1857)) [@cwharris](https://github.com/cwharris) +- Update Kafka DL script to `2.13-3.8.0` ([#1856](https://github.com/nv-morpheus/Morpheus/pull/1856)) [@cwharris](https://github.com/cwharris) +- Update `isort` settings file path in `fix_all.sh` ([#1855](https://github.com/nv-morpheus/Morpheus/pull/1855)) [@yczhang-nv](https://github.com/yczhang-nv) +- Move isort settings into pyproject.toml ([#1854](https://github.com/nv-morpheus/Morpheus/pull/1854)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update location of morpheus setup and data files in VS settings ([#1843](https://github.com/nv-morpheus/Morpheus/pull/1843)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Fix isort config marking `_utils` as known first party ([#1842](https://github.com/nv-morpheus/Morpheus/pull/1842)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix usage of the C++ impl of `write_df_to_file` ([#1840](https://github.com/nv-morpheus/Morpheus/pull/1840)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix shutdown on Ctrl+C for Python source stages ([#1839](https://github.com/nv-morpheus/Morpheus/pull/1839)) [@dagardner-nv](https://github.com/dagardner-nv) +- Improved type-hints for stage and source decorators ([#1831](https://github.com/nv-morpheus/Morpheus/pull/1831)) [@dagardner-nv](https://github.com/dagardner-nv) +- Add tests to confirm that a mis-configured MultiPortModulesStage will raise an exception rather than segfaulting ([#1829](https://github.com/nv-morpheus/Morpheus/pull/1829)) [@dagardner-nv](https://github.com/dagardner-nv) +- Ensure proper initialization of `CMAKE_INSTALL_PREFIX` if needed ([#1815](https://github.com/nv-morpheus/Morpheus/pull/1815)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix merging of CLI args and Yaml configs in `vdb_upload` example ([#1813](https://github.com/nv-morpheus/Morpheus/pull/1813)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix log parsing undefined variable and duplicate sequence id errors ([#1809](https://github.com/nv-morpheus/Morpheus/pull/1809)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove obsolete version string from compose yamls ([#1808](https://github.com/nv-morpheus/Morpheus/pull/1808)) [@dagardner-nv](https://github.com/dagardner-nv) +- Ensure the release container does not contain any unintended files ([#1807](https://github.com/nv-morpheus/Morpheus/pull/1807)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update `ci/release/update-version.sh` to include missed files ([#1801](https://github.com/nv-morpheus/Morpheus/pull/1801)) [@dagardner-nv](https://github.com/dagardner-nv) + +## 📖 Documentation + +- Add known issue for dask shutdown ([#2027](https://github.com/nv-morpheus/Morpheus/pull/2027)) [@cwharris](https://github.com/cwharris) +- Set the version in the conda packages docs ([#2017](https://github.com/nv-morpheus/Morpheus/pull/2017)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Fix mis-leading deserialize stage comments ([#2009](https://github.com/nv-morpheus/Morpheus/pull/2009)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update Morpheus docs to use cuda 12.5 ([#2008](https://github.com/nv-morpheus/Morpheus/pull/2008)) [@yczhang-nv](https://github.com/yczhang-nv) +- Fix minor issues with LLM example documentation ([#1992](https://github.com/nv-morpheus/Morpheus/pull/1992)) [@dagardner-nv](https://github.com/dagardner-nv) +- Incorporate review comments in the conda packages documentation ([#1982](https://github.com/nv-morpheus/Morpheus/pull/1982)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Add CPU-only documentation ([#1969](https://github.com/nv-morpheus/Morpheus/pull/1969)) [@dagardner-nv](https://github.com/dagardner-nv) +- Document each of the Conda environment files ([#1932](https://github.com/nv-morpheus/Morpheus/pull/1932)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update documentation to reflect CPU-only execution mode ([#1924](https://github.com/nv-morpheus/Morpheus/pull/1924)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove `TODO` statements from documentation ([#1879](https://github.com/nv-morpheus/Morpheus/pull/1879)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove automock for merlin as we no longer have/use merlin ([#1830](https://github.com/nv-morpheus/Morpheus/pull/1830)) [@dagardner-nv](https://github.com/dagardner-nv) +- Add documentation checks to CI ([#1821](https://github.com/nv-morpheus/Morpheus/pull/1821)) [@dagardner-nv](https://github.com/dagardner-nv) +- Fix documentation links to work in both source repo and documentation builds ([#1814](https://github.com/nv-morpheus/Morpheus/pull/1814)) [@dagardner-nv](https://github.com/dagardner-nv) +- Update documentation for `vdb_upload` to use realistic source data with the `--file_source` flag ([#1800](https://github.com/nv-morpheus/Morpheus/pull/1800)) [@dagardner-nv](https://github.com/dagardner-nv) + +## 🚀 New Features + +- Install morpheus-dfp conda package in the DFP container ([#1971](https://github.com/nv-morpheus/Morpheus/pull/1971)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Morpheus docs update post compartmentalization ([#1964](https://github.com/nv-morpheus/Morpheus/pull/1964)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Adding implementation of Router Nodes ([#1963](https://github.com/nv-morpheus/Morpheus/pull/1963)) [@mdemoret-nv](https://github.com/mdemoret-nv) +- Include requirements files in the morpheus packages ([#1957](https://github.com/nv-morpheus/Morpheus/pull/1957)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Unit tests for the namespace update script ([#1954](https://github.com/nv-morpheus/Morpheus/pull/1954)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Script for updating the namespace due to compartmentalization changes ([#1946](https://github.com/nv-morpheus/Morpheus/pull/1946)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Move tests/common to tests/morpheus/common ([#1942](https://github.com/nv-morpheus/Morpheus/pull/1942)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing ([#1933](https://github.com/nv-morpheus/Morpheus/pull/1933)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Remove debug log in `HttpServerSourceStage` when the queue is empty ([#1921](https://github.com/nv-morpheus/Morpheus/pull/1921)) [@dagardner-nv](https://github.com/dagardner-nv) +- Refactor digital_fingerprinting stages and add morpheus-split conda recipe (core, dfp, llm) ([#1897](https://github.com/nv-morpheus/Morpheus/pull/1897)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Move vector db stages to morpheus-llm ([#1889](https://github.com/nv-morpheus/Morpheus/pull/1889)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Scripts for building and uploading the morpheus-core conda package ([#1883](https://github.com/nv-morpheus/Morpheus/pull/1883)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Implement `MultiProcessingStage` ([#1878](https://github.com/nv-morpheus/Morpheus/pull/1878)) [@yczhang-nv](https://github.com/yczhang-nv) +- Update to RAPIDS 24.10 ([#1874](https://github.com/nv-morpheus/Morpheus/pull/1874)) [@cwharris](https://github.com/cwharris) +- Add support for a CPU-only Mode ([#1851](https://github.com/nv-morpheus/Morpheus/pull/1851)) [@dagardner-nv](https://github.com/dagardner-nv) +- [morpheus-refactor] Move morpheus source to python/morpheus ([#1836](https://github.com/nv-morpheus/Morpheus/pull/1836)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Support for `ControlMessage` as an output type for `HttpServerSourceStage` and `HttpClientSourceStage` ([#1834](https://github.com/nv-morpheus/Morpheus/pull/1834)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove NVTabular ([#1825](https://github.com/nv-morpheus/Morpheus/pull/1825)) [@cwharris](https://github.com/cwharris) +- Create a Docker image for Morpheus models ([#1804](https://github.com/nv-morpheus/Morpheus/pull/1804)) [@dagardner-nv](https://github.com/dagardner-nv) +- Add unique column to output of the `log_parsing` pipeline ([#1795](https://github.com/nv-morpheus/Morpheus/pull/1795)) [@dagardner-nv](https://github.com/dagardner-nv) + +## 🛠️ Improvements + +- Update to Triton Inference Server container version 24.09 ([#2001](https://github.com/nv-morpheus/Morpheus/pull/2001)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove temporary DFP todo list ([#1998](https://github.com/nv-morpheus/Morpheus/pull/1998)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- devcontainer: replace `VAULT_HOST` with `AWS_ROLE_ARN` ([#1962](https://github.com/nv-morpheus/Morpheus/pull/1962)) [@jjacobelli](https://github.com/jjacobelli) +- Reduce the number of warnings emitted ([#1947](https://github.com/nv-morpheus/Morpheus/pull/1947)) [@dagardner-nv](https://github.com/dagardner-nv) +- Set lower CPU usage for `test_shared_process_pool.py` to avoid slowing down the test ([#1935](https://github.com/nv-morpheus/Morpheus/pull/1935)) [@yczhang-nv](https://github.com/yczhang-nv) +- Remove unused pymysql dependency from DFP mlflow container ([#1930](https://github.com/nv-morpheus/Morpheus/pull/1930)) [@dagardner-nv](https://github.com/dagardner-nv) +- Support LLM pipelines in CPU-only mode ([#1906](https://github.com/nv-morpheus/Morpheus/pull/1906)) [@dagardner-nv](https://github.com/dagardner-nv) +- Remove Starter Digital Fingerprinting (DFP) ([#1903](https://github.com/nv-morpheus/Morpheus/pull/1903)) [@efajardo-nv](https://github.com/efajardo-nv) +- Finalize removing `MultiMessage` from Morpheus ([#1886](https://github.com/nv-morpheus/Morpheus/pull/1886)) [@yczhang-nv](https://github.com/yczhang-nv) +- Run pre-commit on all files, not just python ([#1880](https://github.com/nv-morpheus/Morpheus/pull/1880)) [@cwharris](https://github.com/cwharris) +- Prefer `len(os.sched_getaffinity(0))` over `os.cpu_count()` ([#1866](https://github.com/nv-morpheus/Morpheus/pull/1866)) [@cwharris](https://github.com/cwharris) +- Remove cloudtrail debug log from autoencoder source stage ([#1865](https://github.com/nv-morpheus/Morpheus/pull/1865)) [@cwharris](https://github.com/cwharris) +- Run yapf, flake8, isort as part of pre-commit ([#1859](https://github.com/nv-morpheus/Morpheus/pull/1859)) [@cwharris](https://github.com/cwharris) +- Warn when `Config`'s `pipeline_batch_size < model_max_batch_size` ([#1858](https://github.com/nv-morpheus/Morpheus/pull/1858)) [@cwharris](https://github.com/cwharris) +- Breakout morpheus_llm ([#1853](https://github.com/nv-morpheus/Morpheus/pull/1853)) [@AnuradhaKaruppiah](https://github.com/AnuradhaKaruppiah) +- Install built documentation into release container ([#1806](https://github.com/nv-morpheus/Morpheus/pull/1806)) [@dagardner-nv](https://github.com/dagardner-nv) +- Removing support for `MultiMessage` from stages ([#1803](https://github.com/nv-morpheus/Morpheus/pull/1803)) [@yczhang-nv](https://github.com/yczhang-nv) +- Batch incoming DOCA raw packet data ([#1731](https://github.com/nv-morpheus/Morpheus/pull/1731)) [@dagardner-nv](https://github.com/dagardner-nv) + # Morpheus 24.06.01 (23 Aug 2024) ## 🛠️ Improvements