Allow multiple objectives in the optimizer config #725

motus · 2024-03-26T23:57:56Z

Update JSON schema to use "optimizer_target": {"score": "min"} format
Update base Optimizer class to use the new config format (throw NotImplementedError if > 1 target)
Modify all unit tests to use the new format

Part of #692

Merge after #723

… when loading the results

…lk_register()

…led trials into the optimizer

…eOptimizer

…to sergiym/opt/multiobjective

…tive_config

…se optimizer

mlos_bench/mlos_bench/config/schemas/cli/common-defs-subschemas.json

mlos_bench/mlos_bench/config/schemas/optimizers/optimizer-schema.json

mlos_bench/mlos_bench/config/experiments/README.md

mlos_bench/mlos_bench/optimizers/base_optimizer.py

bpkroth

Just a slight naming change, otherwise LGTM

...s_bench/tests/config/schemas/optimizers/test-cases/bad/invalid/bad_optimization_target.jsonc

…/opt/multiobjective_config

…MLOS into sergiym/opt/multiobjective_config

Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

…MLOS into sergiym/opt/multiobjective_config

…t/optimization_direction config

bpkroth

LGTM!

* [x] Store multiple optimization targets and directions in base `Optimizer` class * [x] Support multiple optimization targets in mock and grid search optimizers * [x] Check for single objective in `MlosCoreOptimizer` class (will add support for multiple objectives after implementing that feature in `mlos_core` in subsequent PRs) * [x] Update unit tests in mlos_bench Merge after ~#723~ and ~#725~ --------- Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

motus added 24 commits March 22, 2024 14:55

load ALL metrics from the DB when feeding the optimizer

2abf2c7

disable incorrect pylint warning

63a4e15

fix unit tests to handle the new data types returned from teh storage…

93d4161

… when loading the results

register multi-dimensional results in optimizer.bulk_register()

4adf433

update unit tests to validate multi-dimensional input to Optimizer.bu…

9c16ba6

…lk_register()

fix the key/value SELECT statement in SQL Storage implementaion

7874fec

fix the input to grid search optimizer .bulk_register call

6cb1fb6

minor fix in GridSearchOptimizer

e5adef5

remove unused import

a79e555

bugfix: take the optimization direction into account when loading fai…

b48c698

…led trials into the optimizer

handle -Inf in try_parse_val(). check it in unit tests

3c29bed

move try_parse_val fix to a separate PR; undo the sign fix in MlosCor…

5015f75

…eOptimizer

start updating the configs to include more than one optimization target

6b2804d

Merge branch 'main' into sergiym/opt/multiobjective

dc84cfa

Merge branch 'main' into sergiym/opt/multiobjective

e99c591

Merge branch 'sergiym/opt/multiobjective' of github.com:motus/MLOS in…

323fba8

…to sergiym/opt/multiobjective

add a comment re: Row._tuple() pylint warning

838ca37

Merge branch 'sergiym/opt/multiobjective' into sergiym/opt/multiobjec…

623f012

…tive_config

fix the optimization_target JSON schema; handle it properly in the ba…

6661ac0

…se optimizer

use NotImplementedError instead of NotImplemented

dd44568

add type annotations

f1e989a

fix the json schema with the new optimizer_type format

5f70f02

fix teh optimization_target schema

9c20101

typo fixed

7e3d1d2

motus requested a review from a team as a code owner March 26, 2024 23:57

use common optimization_target definition in all JSON schemas

70790ce

motus self-assigned this Mar 27, 2024

motus added ready for review Ready for review mlos-bench labels Mar 27, 2024

motus changed the title ~~Allow multiple objectives in optimizer config~~ Allow multiple objectives in the optimizer config Mar 28, 2024

motus mentioned this pull request Mar 29, 2024

Implement multi-objective optimization in mlos_bench base classes #726

Merged

4 tasks

Merge branch 'main' into sergiym/opt/multiobjective_config

e28feb3