Enable mypy type checking

[simon-mo]

Anything you want to discuss about vllm.

Even though vLLM is type annotated but we did not enable type checking. It would be useful to add it, even incrementally.

[rkooo567]

The current remaining

[rkooo567] mypy vllm/engine/*.py --follow-imports=skip --config-file pyproject.toml
[rkooo567] mypy vllm/worker/*.py --follow-imports=skip --config-file pyproject.toml
[rkooo567] mypy vllm/spec_decode/*.py --follow-imports=skip --config-file pyproject.toml
[rkooo567] mypy vllm/model_executor/*.py --follow-imports=skip --config-file pyproject.toml
mypy vllm/lora/*.py --follow-imports=skip --config-file pyproject.toml
mypy vlllm/entrypoints/openai/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/transformers_utils/configs/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/transformers_utils/tokenizers/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/distributed/device_communicators/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/attention/backends/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/transformers_utils/tokenizer_group/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/core/block/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/model_executor/layers/*.py --follow-imports=skip --config=file pyprojet.toml
mypy vllm/model_executor/layers/*.py --follow-imports=skip --config=file pyprojet.toml

and misc after

[DarkLight1337]

For those who are new to the vLLM repo, determining the correct type of each variable is a great way to enhance your understanding of how the code is connected together.

[DarkLight1337]

Quick update on the latest progress:

• @rkooo567 has enabled type checking for main vllm library ([mypy][7/N] Cover all directories #4555 being the last PR)
• I have enabled type checking for tests directory ([mypy] Enable type checking for test directory #5017)

However, the effectiveness of type checking is still quite minimal because imported types are currently treated as Any due to the setting --follow_imports=skip:

vllm/pyproject.toml

Lines 46 to 51 in 2be6955

	[tool.mypy]
	python_version = "3.8"
	ignore_missing_imports = true
	check_untyped_defs = true
	follow_imports = "skip"

My IDE, which uses Pyright basic mode, still flags a lot of errors, showing that there is much work to be done.

Moving forward, we should work on gradually applying --follow_imports=normal for each directory, fixing errors as we go. I recommend going through the directories following a similar order as @rkooo567 did (these search filters should be able to locate the relevant PRs).

[youkaichao]

Cool, looking forward to the progress.

[rkooo567]

This is a great issue to improve the code quality of the repo, so please feel free to work on it if anyone is interested! One example PR in the past #4043

[jberkhahn]

I'm taking a crack at vllm/core, just to get my feet wet with the codebase.

[pooyadavoodi]

There are errors indicating incompatibilities between AsyncEngineClient and AsyncLLMEngine. Those errors seem similar to what is discussed in the following links (still an open issue):

• @overload doesn't work with @asynccontextmanager python/mypy#17340
• https://discuss.python.org/t/overloads-of-async-generators-inconsistent-coroutine-wrapping/56665
• Propagate is_async_generator down to overloads python/mypy#17435

An example of the error:

vllm/entrypoints/openai/run_batch.py:132: error: Argument 1 to "OpenAIServingChat" has incompatible type "AsyncLLMEngine"; expected "AsyncEngineClient"  [arg-type]
vllm/entrypoints/openai/run_batch.py:132: note: Following member(s) of "AsyncLLMEngine" have conflicts:
vllm/entrypoints/openai/run_batch.py:132: note:     Expected:
vllm/entrypoints/openai/run_batch.py:132: note:         def encode(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], pooling_params: PoolingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ...) -> Coroutine[Any, Any, AsyncIterator[EmbeddingRequestOutput]]
vllm/entrypoints/openai/run_batch.py:132: note:     Got:
vllm/entrypoints/openai/run_batch.py:132: note:         def encode(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], pooling_params: PoolingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ...) -> AsyncIterator[EmbeddingRequestOutput]
vllm/entrypoints/openai/run_batch.py:132: note:     Expected:
vllm/entrypoints/openai/run_batch.py:132: note:         def generate(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], sampling_params: SamplingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ..., prompt_adapter_request: Optional[PromptAdapterRequest] = ...) -> Coroutine[Any, Any, AsyncIterator[RequestOutput]]
vllm/entrypoints/openai/run_batch.py:132: note:     Got:
vllm/entrypoints/openai/run_batch.py:132: note:         def generate(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], sampling_params: SamplingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ..., prompt_adapter_request: Optional[PromptAdapterRequest] = ...) -> AsyncIterator[RequestOutput]

[DarkLight1337]

There are errors indicating incompatibilities between AsyncEngineClient and AsyncLLMEngine. Those errors seem similar to what is discussed in the following links (still an open issue):

• @overload doesn't work with @asynccontextmanager python/mypy#17340
• https://discuss.python.org/t/overloads-of-async-generators-inconsistent-coroutine-wrapping/56665
• Propagate is_async_generator down to overloads python/mypy#17435

An example of the error:

vllm/entrypoints/openai/run_batch.py:132: error: Argument 1 to "OpenAIServingChat" has incompatible type "AsyncLLMEngine"; expected "AsyncEngineClient"  [arg-type]
vllm/entrypoints/openai/run_batch.py:132: note: Following member(s) of "AsyncLLMEngine" have conflicts:
vllm/entrypoints/openai/run_batch.py:132: note:     Expected:
vllm/entrypoints/openai/run_batch.py:132: note:         def encode(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], pooling_params: PoolingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ...) -> Coroutine[Any, Any, AsyncIterator[EmbeddingRequestOutput]]
vllm/entrypoints/openai/run_batch.py:132: note:     Got:
vllm/entrypoints/openai/run_batch.py:132: note:         def encode(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], pooling_params: PoolingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ...) -> AsyncIterator[EmbeddingRequestOutput]
vllm/entrypoints/openai/run_batch.py:132: note:     Expected:
vllm/entrypoints/openai/run_batch.py:132: note:         def generate(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], sampling_params: SamplingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ..., prompt_adapter_request: Optional[PromptAdapterRequest] = ...) -> Coroutine[Any, Any, AsyncIterator[RequestOutput]]
vllm/entrypoints/openai/run_batch.py:132: note:     Got:
vllm/entrypoints/openai/run_batch.py:132: note:         def generate(self, inputs: Union[Union[str, TextPrompt, TokensPrompt], ExplicitEncoderDecoderPrompt], sampling_params: SamplingParams, request_id: str, lora_request: Optional[LoRARequest] = ..., trace_headers: Optional[Mapping[str, str]] = ..., prompt_adapter_request: Optional[PromptAdapterRequest] = ...) -> AsyncIterator[RequestOutput]

I have been working on this for the past few days. Stay tuned for updates!

[DarkLight1337]

I have been working on this for the past few days. Stay tuned for updates!

Opened #7248.

[jberkhahn]

Was going to take a crack at vllm/executor next.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[DarkLight1337]

Bump

[lucas-tucker]

I noticed there is now mypy checking for a subset of directories under tools/mypy.sh. Is this still a relevant issue, and what additional directories or classes/functions need mypy checking?

[DarkLight1337]

We try to enforce mypy checking for new code, e.g. #11105. Efforts to add mypy checking for the existing code are still welcome!