add new feature for llava-next-video #417
Conversation
Add video support
| from io import BytesIO | ||
| from json import dumps | ||
|
|
||
| import cv2 |
There was a problem hiding this comment.
opencv-python should be added in pyproject.toml.
There was a problem hiding this comment.
There are many user-defined paths and some parameters in argparse are not being used.
README.md
Outdated
| @@ -1,411 +1,74 @@ | |||
| To refine and clean up the README you've provided for the SGLang project, I'll focus on improving clarity, organization, and conciseness. This includes providing clear installation instructions, simplifying steps where possible, and ensuring the document is easy to navigate. Here's a revised version: | |||
There was a problem hiding this comment.
Revert the change to README.md
python/pyproject.toml
Outdated
| @@ -20,7 +20,7 @@ dependencies = [ | |||
|
|
|||
| [project.optional-dependencies] | |||
| srt = ["aiohttp", "fastapi", "psutil", "rpyc", "torch", "uvloop", "uvicorn", | |||
| "zmq", "vllm>=0.3.3", "interegular", "pydantic", "pillow", "outlines>=0.0.27"] | |||
| "zmq", "vllm==0.3.3", "interegular", "pydantic", "pillow", "outlines>=0.0.27"] | |||
There was a problem hiding this comment.
Recently, we moved to vllm==0.4.2, can you update accordingly?
There are only minor changes. You can change similar to the changes in this PR #380
examples/demo/model_call.py
Outdated
| @@ -0,0 +1,198 @@ | |||
| # from flask import Flask, request, jsonify | |||
There was a problem hiding this comment.
Rename examples/demo to examples/usage/llava_video
| compile_and_cleanup_final_results(cur_chunk, num_batches, save_dir) | ||
|
|
||
|
|
||
| import argparse |
There was a problem hiding this comment.
move this to the beginning of the file.
| parser.add_argument('--chunk-idx', type=int, default=0, help='The index of the chunk to process.') | ||
| parser.add_argument('--num-chunks', type=int, default=8, help='The number of chunks to process.') | ||
| parser.add_argument('--save-dir', type=str, default="./work_dirs/llava_video", help='The directory to save the processed video files.') | ||
| parser.add_argument('--video-dir', type=str, default="/mnt/bn/vl-research/workspace/yhzhang/data/sora/", help='The directory to save the processed video files.') |
There was a problem hiding this comment.
It seems there are many personal paths. Can someone else easily run this script?
Maybe you can move all things under a single folder examples/llava_video and add a README.md with an example command that runs on your Q98Z4OTh8RwmDonc.mp4
python/sglang/utils.py
Outdated
| return video_base64 | ||
|
|
||
|
|
||
| # def encode_video_base64(video_path): |
| @@ -32,6 +32,9 @@ | |||
| ) | |||
| from vllm.logger import _default_handler as vllm_default_handler | |||
|
|
|||
| from typing import Any, Dict, List, Optional, Tuple, Union | |||
llava-next-video: https://llava-vl.github.io/blog/2024-04-30-llava-next-video/