Skip to content
This repository has been archived by the owner on May 3, 2022. It is now read-only.

Add model validation threshold for acceptance #321

Closed
srowen opened this issue Oct 24, 2016 · 1 comment
Closed

Add model validation threshold for acceptance #321

srowen opened this issue Oct 24, 2016 · 1 comment
Assignees
@srowen
Labels
BatchLayer enhancement MLTier
Milestone
2.2.2

Comments

@srowen
Copy link
Member

srowen commented Oct 24, 2016

Right now, when a model is built, several variations of the model can be built in order to select the hyperparameters that result in the best model according to some evaluation metric. However it might also be useful to further require that the metric meet some threshold before it's published. That way, if the 'best' model were quite bad, it would be rejected.
@srowen srowen added this to the 2.3.0 milestone Oct 24, 2016
@srowen srowen closed this as completed in 341182b Oct 25, 2016
srowen added a commit to srowen/oryx that referenced this issue Oct 25, 2016
@srowen
Closes OryxProject#321 : add oryx.ml.eval.threshold parameter for rej…
2e054cc 
…ecting models based on evaluation. And some related test refactoring
@srowen srowen modified the milestones: 2.2.2, 2.3.0 Oct 25, 2016
@srowen srowen self-assigned this Oct 25, 2016
@neerajBaji
Copy link

so this is another parameter that can be set from oryx.conf or is there more to it? Like the model coming up with a reasonable threshold that evolves with more training data.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@srowen srowen

Labels
BatchLayer enhancement MLTier
Projects
None yet
Milestone
2.2.2
Development

No branches or pull requests
2 participants
@srowen @neerajBaji