model_override_args with server

[ValeKnappich]

When using a server, one currently cannot use the model_overide_args which could be very useful, e.g. for rope scaling.

This is currently the sglang.launch_server.py:

import argparse

from sglang.srt.server import launch_server
from sglang.srt.server_args import ServerArgs

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    ServerArgs.add_cli_args(parser)
    args = parser.parse_args()
    server_args = ServerArgs.from_cli_args(args)

    launch_server(server_args, None)

The model_overide_args would be the third argument to launch_server defaulting to None. Adding a small cli parser that allows arbitrary model args would be great, e.g.

python -m sglang.launch_server --model_overide_args.rope_scaling.factor 2 --model_overide_args.rope_scaling.type linear

[merrymercy]

@ValeKnappich Agree. Could you send a pull request to support this feature?

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.