Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[lmi][docs] add deepspeed deprecation notice to lmi docs #2024

Merged
merged 1 commit into from
Jun 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions serving/docs/lmi/announcements/deepspeed-deprecation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# DeepSpeed deprecation announcement

**Note: This only impacts users that are using the `deepspeed` container that are also using the `DeepSpeed` library for inference.
If you are using lmi-dist, vLLM, or HuggingFace accelerate in this container, you are not impacted.**

With LMI V10 (0.28.0), we are changing the name from LMI DeepSpeed DLC to LMI (LargeModelInference) DLC.
The `deepspeed` container has been renamed to the `lmi` container.
As part of this change, we have decided to discontinue integration with the DeepSpeed inference library.
You can continue to use vLLM or LMi-dist Library with the LMI container. If you plan to use DeepSpeed Library, please follow the steps below, or use LMI V9 (0.27.0).

## Migrating from DeepSpeed

If you are not using DeepSpeed library (through importing) or using DeepSpeed as your inference engine, you can stop reading from here.
You are leveraging DeepSpeed in the LMI container if you meet any of the following conditions:

You are using the DeepSpeed engine via:

* `engine=DeepSpeed` (serving.properties)
* `OPTION_ENGINE=DeepSpeed` (environment variable)

You are using the built-in DeepSpeed inference handler:

* `option.entryPoint=djl_python.deepspeed` (serving.properties)
* `OPTION_ENTRYPOINT=djl_pytyhon.deepspeed` (environment variable)

You are importing `deepspeed` in a custom inference handler (model.py)

```python
import deepspeed
```

If none of these conditions apply to your use-case, you can stop reading as you are not impacted.

### Custom inference script (model.py)

If you are using a custom handler (model.py) that is not coming from LMI and you import DeepSpeed like this:

```python
import deepspeed
```

You will need to install DeepSpeed manually into our container. Add `deepspeed` to your `requirements.txt` and the model server will install the dependency at runtime.
In a network isolated deployment environment, you will need to package the `deepspeed` wheel along with other necessary dependencies with the model.py.

### Built-In Handler

If you set any of the following configurations:

serving.properties file

```
engine=DeepSpeed
```

```
option.entryPoint=djl_python.deepspeed
```

environment variables

```
OPTION_ENGINE=DeepSpeed
```

```
OPTION_ENTRYPOINT=djl_python.deepspeed
```

Then you should migrate to:

serving.properties

```
engine=Python
option.mpi_mode=true
```

environment variables

```
HF_MODEL_ID=<hub model id or s3 uri> (container will auto-detect best engine to use for your model)
```



This updated configuration will leverage either `lmi-dist`, `vllm`, or `huggingface accelerate` depending on your model architecture.
Most model architectures supported by deepspeed are also supported by `lmi-dist` or `vllm`. These inference libraries should provide increased performance compared to `deepspeed`.

We rename DeepSpeed into more generic as Python + MPI way to better represent the use case.
In the meantime, we inherit most of the DeepSpeed feature with our LMI-Dist Engine, including all models DeepSpeed supported today. Please refer to [LMI-Dist Engine guide](../user_guides/lmi-dist_user_guide.md) to start with.
4 changes: 3 additions & 1 deletion serving/docs/lmi/user_guides/deepspeed_user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

> [!WARNING]
> DeepSpeed support has been removed starting with LMI container release 0.28.0.
> You should migrate to one of the other supported backends.
> You should migrate to one of the other supported backends.
> Please see the [migration document](../announcements/deepspeed-deprecation.md) for more details.


## Model Artifacts Structure

Expand Down