Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the evaluation metric configurable #13

Open
bkersbergen opened this issue Jun 23, 2022 · 0 comments
Open

Make the evaluation metric configurable #13

bkersbergen opened this issue Jun 23, 2022 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@bkersbergen
Copy link
Collaborator

bkersbergen commented Jun 23, 2022

The evaluation metric (MRR@20) that is used for hyperparameter search is currently hardcoded.
Serenade has supports for multiple evaluation metrics such as MRR, F1, HitRate, Ndcg, Popularity. Precision, Recall etc.

The choice for which evaluation metric to be used, should be configurable by the user via the configuration file.

Proposed solution:
in the config file introduce a new variable for the section 'hyperparam' used by tpe_hyperparameter_optm.rs:
metric_name = "MRR"
metric_length = 20

in the config file introduce a new section that evaluate_file.rs should use:
[evaluation]
metric_name = "MRR"
metric_length = 20
training_data_path = "train.txt"
test_data_path = "test.txt"

Both the tpe_hyperparameter_optm.rs and evaluate_file.rs should be included to support this feature.

The trait SessionMetric could be extended with a function fn is_larger_better(&self) -> bool
By default all metrics must return true for fn is_larger_better except for popularity.rs which should return false

The tpe_hyperparameter_optm.rs ( line 130 ) should use the 'max' or 'min' score depending on the is_larger_better value.

@bkersbergen bkersbergen added enhancement New feature or request good first issue Good for newcomers labels Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant