Support continuous batching in sequence batch streaming case

[lxning]

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

 pytest test_example_stateful_sequence_continuous_batching_http.py
============================== test session starts ===============================
platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/ubuntu/serve
plugins: cov-4.1.0, mock-3.12.0
collected 3 items

test_example_stateful_sequence_continuous_batching_http.py ...             [100%]

=============================== 3 passed in 19.81s ===============================

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

[mreso]

Left some comments. Please add continuous batching to the test to make sure this part works as well.

[mreso]

We should document how the jobs will be interleaved when they land in the handler and the batch_size > 1. Is is [Job_1_1, Job_1_2,Job_2_1, Job_2_2] or [Job_1_1, Job_2_1,Job_1_2, Job_2_2]? Or will it be scrambled and the handler needs to sort this out.

[lxning]

No, it is not necessary for backend handler to sort b/c backend handler preprocessing is able to mark the previous running request as "cancel"

[mreso]

@lxning So you mean the preprocessing in the handler will need to sort this out as there can be multiple requests from multiple sequences and in no specific pattern. In the end the preprocessing in the handler will need to go though the batch and see if besides the previous request another cancel request is in the batch as well and than cancel it/clean it up.

[lxning]

No, it is not necessary to sort. Frontend guarantee the order of the requests in a sequence, and pass them to backend. Currently the use case is synchronized b/w client and TS, and the continuous batch size is fixed as 2 for each sequence. This means a cancel cmd applies on all requests of this sequence at backend. So here is based on "num_requests"

[mreso]

Please see comments.

[mreso]

LGTM