You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A/B Model Traffic split is not working as expected after edit of Experiment CR.
Based on the Config Dump from envoy, the weighted routes keeps getting appended. It should replace the old weight with updated weight. Instead it maintains all the previous configuration.
The issue might be related to this, as the traffic split happens correctly when create Experiment with 100% and 0% first time itself.
Verify the request goes to both the models - SUCCESS
Modify the Experiment to 0% traffic split to 'iris' and 100% traffic split to iris2
Invoke the inference calls using REST/GRPC
Verify the request goes to only model with iris2(100%) and not to iris(0%) - FAIL
- > Traffic is routing to both iris and iris2 model
Expected behaviour
The traffic split should happen based on the edited configuration.
E.g., When weight is updated from 50% to 0%, traffic should not be routed to the model.
Expected Behaviour - Traffic to be only routed to iris2 model
Current Behaviour - Traffic is routing to both iris and iris2 model
Based on the Config Dump from envoy, the weighted routes keeps getting appended. It should replace the old weight with updated weight. Instead it maintains all the previous configuration.
The issue might be related to this, as the traffic split happens correctly when Experiment is created with 100% and 0% first time itself.
Environment
Kubernetes version : 1.27.7
Seldon Version : 2.6.0
Model Details
Images of your model: [Output of: kubectl get seldondeployment -n <yourmodelnamespace> <seldondepname> -o yaml | grep image: where <yourmodelnamespace>]
Logs of your model: [You can get the logs of your model by running kubectl logs -n <yourmodelnamespace> <seldonpodname> <container>]
The text was updated successfully, but these errors were encountered:
@DineshKrishnanG Thank you for raising this. I created the experiment as per the documentation using 2.7.0 and the traffic split seems to work fine after the editing Experiment CR. Could you expand on how you're modifying the experiment and provide more logs if possible?
Describe the bug
A/B Model Traffic split is not working as expected after edit of Experiment CR.
Based on the Config Dump from envoy, the weighted routes keeps getting appended. It should replace the old weight with updated weight. Instead it maintains all the previous configuration.
The issue might be related to this, as the traffic split happens correctly when create Experiment with 100% and 0% first time itself.
To reproduce
- > Traffic is routing to both iris and iris2 model
Expected behaviour
The traffic split should happen based on the edited configuration.
E.g., When weight is updated from 50% to 0%, traffic should not be routed to the model.
Expected Behaviour - Traffic to be only routed to iris2 model
Current Behaviour - Traffic is routing to both iris and iris2 model
Based on the Config Dump from envoy, the weighted routes keeps getting appended. It should replace the old weight with updated weight. Instead it maintains all the previous configuration.
The issue might be related to this, as the traffic split happens correctly when Experiment is created with 100% and 0% first time itself.
Environment
Kubernetes version : 1.27.7
Seldon Version : 2.6.0
Model Details
kubectl get seldondeployment -n <yourmodelnamespace> <seldondepname> -o yaml | grep image:
where<yourmodelnamespace>
]kubectl logs -n <yourmodelnamespace> <seldonpodname> <container>
]The text was updated successfully, but these errors were encountered: