Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Hardware][Neuron] Add on-device sampling support for Neuron #8746

Merged
merged 12 commits into from
Oct 4, 2024
Prev Previous commit
Next Next commit
remove deepcopy.
chongmni-aws committed Oct 2, 2024
commit 273ccfd2b82aa8e5cd5c3922a1f4b3fbd58151fd
3 changes: 0 additions & 3 deletions vllm/worker/neuron_model_runner.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import copy
from dataclasses import dataclass
from importlib.util import find_spec
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
@@ -270,8 +269,6 @@ def _update_neuron_generation_config(self, sampling_metadata):
current_generation_config.top_k = top_k
current_generation_config.top_p = top_p
current_generation_config.temperature = temperature
self.model_config.generation_config = copy.deepcopy(
current_generation_config)

self.model.model.update_generation_config(
current_generation_config)