Plugin SDK: ModelServerRequest tries to access direct buffer

[marrodion]

🐛 Describe the bug

When using Request abstraction from serving SDK, attempt to call getInputStream() will throw an exception if underlying buffer is a direct buffer.

Error logs

direct buffer
java.lang.UnsupportedOperationException: direct buffer
at io.netty.buffer.UnpooledDirectByteBuf.array(UnpooledDirectByteBuf.java:181)
at org.pytorch.serve.servingsdk.impl.ModelServerRequest.getInputStream(ModelServerRequest.java:49)
at org.pytorch.serve.servingsdk.impl.ModelServerRequest.getInputStream(ModelServerRequest.java:13)

Installation instructions

Reproducible both in Docker and when installing from source

Model Packaing

Works with any model

config.properties

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
load_models=all
model_store=/home/model-server/model-store
plugins_path=/home/model-server/plugins
default_workers_per_model=4

Versions

---

Environment headers

Torchserve branch:

torchserve==0.8.1b20230724
torch-model-archiver==0.8.1b20230724

Python version: 3.10 (64-bit runtime)
Python executable: ./serve/venv/bin/python

Versions of relevant python libraries:
captum==0.6.0
numpy==1.24.3
psutil==5.9.5
pygit2==1.12.1
pylint==2.6.0
pytest==7.3.1
pytest-cov==4.1.0
pytest-mock==3.10.0
requests==2.31.0
requests-toolbelt==1.0.0
torch==2.0.1
torch-model-archiver==0.8.1b20230724
torch-workflow-archiver==0.2.9b20230724
torchaudio==2.0.2
torchdata==0.6.1
torchpippy==0.1.1
torchserve==0.8.1b20230724
torchtext==0.15.2
torchvision==0.15.2
transformers==4.30.0
wheel==0.41.0
torch==2.0.1
torchtext==0.15.2
torchvision==0.15.2
torchaudio==2.0.2

Java Version:

OS: Mac OSX 13.4.1 (arm64)
GCC version: N/A
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.26.4

Versions of npm installed packages:
markdown-link-check@3.11.2
UNMET PEER DEPENDENCY newman@5.3.2
newman-reporter-html@1.0.5

Repro instructions

A test that reproduces the issue:

public class UtilsTest {
  @Test
  public void testReadRequestBodyFromPlugin() throws IOException {
    ByteBuf buf = Unpooled.directBuffer(1);
    buf.writeByte(1);
    FullHttpRequest req = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "", buf);
    Request request = new ModelServerRequest(req, new QueryStringDecoder(""));
    // Will throw an exception: java.lang.UnsupportedOperationException: direct buffer
    request.getInputStream();
  }
}

Possible Solution

Use NettyUtils.getBytes() in relevant method

The method would look like this:

@Override
public ByteArrayInputStream getInputStream() {
    return new ByteArrayInputStream(NettyUtils.getBytes(req.content()));
}

The test above no longer throws an exception with this fix.

[agunapal]

@marrodion Thanks for reporting the problem and the solution @lxning Should this be submitted as a PR.

[marrodion]

@agunapal I've opened the PR with a fix, hope that's ok.
Please, let me know if its not or something needs to be changed