Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiple objectives in the optimizer config #725

Merged
merged 33 commits into from
Apr 30, 2024
Merged

Allow multiple objectives in the optimizer config #725

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
2abf2c7
load ALL metrics from the DB when feeding the optimizer
motus Mar 22, 2024
63a4e15
disable incorrect pylint warning
motus Mar 22, 2024
93d4161
fix unit tests to handle the new data types returned from teh storage…
motus Mar 22, 2024
4adf433
register multi-dimensional results in optimizer.bulk_register()
motus Mar 22, 2024
9c16ba6
update unit tests to validate multi-dimensional input to Optimizer.bu…
motus Mar 22, 2024
7874fec
fix the key/value SELECT statement in SQL Storage implementaion
motus Mar 22, 2024
6cb1fb6
fix the input to grid search optimizer .bulk_register call
motus Mar 22, 2024
e5adef5
minor fix in GridSearchOptimizer
motus Mar 22, 2024
a79e555
remove unused import
motus Mar 22, 2024
b48c698
bugfix: take the optimization direction into account when loading fai…
motus Mar 22, 2024
3c29bed
handle -Inf in try_parse_val(). check it in unit tests
motus Mar 23, 2024
5015f75
move try_parse_val fix to a separate PR; undo the sign fix in MlosCor…
motus Mar 23, 2024
6b2804d
start updating the configs to include more than one optimization target
motus Mar 23, 2024
dc84cfa
Merge branch 'main' into sergiym/opt/multiobjective
motus Mar 25, 2024
e99c591
Merge branch 'main' into sergiym/opt/multiobjective
motus Mar 25, 2024
323fba8
Merge branch 'sergiym/opt/multiobjective' of github.com:motus/MLOS in…
motus Mar 25, 2024
838ca37
add a comment re: Row._tuple() pylint warning
motus Mar 25, 2024
623f012
Merge branch 'sergiym/opt/multiobjective' into sergiym/opt/multiobjec…
motus Mar 26, 2024
6661ac0
fix the optimization_target JSON schema; handle it properly in the ba…
motus Mar 26, 2024
dd44568
use NotImplementedError instead of NotImplemented
motus Mar 26, 2024
f1e989a
add type annotations
motus Mar 26, 2024
5f70f02
fix the json schema with the new optimizer_type format
motus Mar 26, 2024
9c20101
fix teh optimization_target schema
motus Mar 26, 2024
7e3d1d2
typo fixed
motus Mar 26, 2024
70790ce
use common optimization_target definition in all JSON schemas
motus Mar 26, 2024
e28feb3
Merge branch 'main' into sergiym/opt/multiobjective_config
bpkroth Apr 29, 2024
499da79
Merge branch 'main' of https://github.com/microsoft/MLOS into sergiym…
motus Apr 29, 2024
498635e
Merge branch 'sergiym/opt/multiobjective_config' of github.com:motus/…
motus Apr 29, 2024
b192c4a
Update mlos_bench/mlos_bench/config/experiments/README.md
motus Apr 29, 2024
a39cf70
optimization_target -> optimization_targets rename
motus Apr 29, 2024
eab796b
Merge branch 'sergiym/opt/multiobjective_config' of github.com:motus/…
motus Apr 29, 2024
13fe064
make sure the schema validation fails on old-style optimization_targe…
motus Apr 29, 2024
1e3c94f
Merge branch 'main' into sergiym/opt/multiobjective_config
bpkroth Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions mlos_bench/mlos_bench/config/experiments/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,9 @@ it looks like this:
"redis": []
},

"optimization_target": "score",
"optimization_direction": "min"
"optimization_target": {
"score": "min"
}
}
```

Expand All @@ -78,12 +79,13 @@ It has a mixture of parameters from different components of the framework. for e

At runtime, these values will be pushed down to the `AzureVMService` configuration, e.g., [`service-linux-vm-ops.jsonc`](../services/remote/azure/service-linux-vm-ops.jsonc).

Likewise, parameters
Likewise, parameter
motus marked this conversation as resolved.
Show resolved Hide resolved

```jsonc
{
"optimization_target": "score",
"optimization_direction": "min"
"optimization_target": {
"score": "min"
}
}
```

Expand Down
3 changes: 1 addition & 2 deletions mlos_bench/mlos_bench/config/experiments/experiment_RedisBench.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,5 @@
"redis": []
},

"optimization_target": "score",
"optimization_direction": "min"
"optimization_target": {"score": "min"}
}
3 changes: 1 addition & 2 deletions mlos_bench/mlos_bench/config/optimizers/mlos_core_default_opt.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
"class": "mlos_bench.optimizers.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_direction": "min",
"optimization_target": {"score": "min"},
"max_suggestions": 100
}
}
5 changes: 2 additions & 3 deletions mlos_bench/mlos_bench/config/optimizers/mlos_core_flaml.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@
"class": "mlos_bench.optimizers.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_direction": "min",
"max_suggestions": 100,
"optimizer_type": "FLAML"
"optimizer_type": "FLAML",
"optimization_target": {"score": "min"}
}
}
3 changes: 1 addition & 2 deletions mlos_bench/mlos_bench/config/optimizers/mlos_core_smac.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
"class": "mlos_bench.optimizers.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_direction": "min",
"optimization_target": {"score": "min"},
"max_suggestions": 100,
"optimizer_type": "SMAC",
"output_directory": null // Override to have a permanent output with SMAC history etc.
Expand Down
2 changes: 1 addition & 1 deletion mlos_bench/mlos_bench/config/optimizers/mock_opt.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"class": "mlos_bench.optimizers.MockOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 5,
"seed": 42
}
Expand Down
9 changes: 8 additions & 1 deletion mlos_bench/mlos_bench/config/schemas/cli/common-defs-subschemas.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
"uniqueItems": true
}
},

"experiment_id": {
"description": "The name of the experiment. Controls trial reuse and sometimes used to disambiguate certain resources definitions. Note: changes to this value will result in a new experiment being created, which may be necessary upon certain config or script changes.",
"type": "string",
Expand All @@ -31,6 +30,14 @@
"description": "Config ID to use (or resume) for the experiment.",
"type": "integer",
"minimum": 0
},
"optimization_target": {
motus marked this conversation as resolved.
Show resolved Hide resolved
"description": "Names of the optimization targets and their directions.",
"type": "object",
"additionalProperties": {
"enum": ["min", "max"]
},
"minItems": 1
}
}
}
5 changes: 3 additions & 2 deletions mlos_bench/mlos_bench/config/schemas/cli/globals-schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,9 @@
"type": "string",
"pattern": "/schemas/cli/globals-schema.json$"
},

"tunable_params_map": {
"$ref": "./common-defs-subschemas.json#/$defs/tunable_params_map"
},

"experiment_id": {
"$ref": "./common-defs-subschemas.json#/$defs/experiment_id"
},
Expand All @@ -23,6 +21,9 @@
},
"config_id": {
"$ref": "./common-defs-subschemas.json#/$defs/config_id"
},
"optimization_target": {
"$ref": "./common-defs-subschemas.json#/$defs/optimization_target"
}
},
"additionalProperties": {
Expand Down
9 changes: 1 addition & 8 deletions mlos_bench/mlos_bench/config/schemas/optimizers/optimizer-schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,7 @@
"type": "object",
"properties": {
"optimization_target": {
motus marked this conversation as resolved.
Show resolved Hide resolved
"description": "The name of the metric to optimize.",
"type": "string",
"example": "score"
},
"optimization_direction": {
"description": "Whether to minimize or maximize the optimization target.",
"enum": ["min", "max"],
"example": "min"
"$ref": "../cli/common-defs-subschemas.json#/$defs/optimization_target"
},
"max_suggestions": {
"description": "The maximum number of additional (in the case of merging experiment data or resuming experiments) config suggestions to run when we launch the app, or no limit if 0 is provided. Note: configs may be repeated in more than one trial.",
Expand Down
12 changes: 9 additions & 3 deletions mlos_bench/mlos_bench/optimizers/base_optimizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ class Optimizer(metaclass=ABCMeta): # pylint: disable=too-many-instance-attr
# See Also: mlos_bench/mlos_bench/config/schemas/optimizers/optimizer-schema.json
BASE_SUPPORTED_CONFIG_PROPS = {
"optimization_target",
"optimization_direction",
"max_suggestions",
"seed",
"start_with_defaults",
Expand Down Expand Up @@ -78,8 +77,15 @@ def __init__(self,
self._start_with_defaults: bool = bool(
strtobool(str(self._config.pop('start_with_defaults', True))))
self._max_iter = int(self._config.pop('max_suggestions', 100))
self._opt_target = str(self._config.pop('optimization_target', 'score'))
self._opt_sign = {"min": 1, "max": -1}[self._config.pop('optimization_direction', 'min')]

opt_targets: Dict[str, str] = self._config.pop('optimization_target', {'score': 'min'})
motus marked this conversation as resolved.
Show resolved Hide resolved
if not isinstance(opt_targets, dict):
raise ValueError(f"optimization_target should be a dict: {opt_targets}")
# TODO: Implement multi-target optimization.
if len(opt_targets) != 1:
raise NotImplementedError("Multi-target optimization is not implemented.")
(self._opt_target, opt_dir) = list(opt_targets.items())[0]
self._opt_sign = {"min": 1, "max": -1}[opt_dir]

def _validate_json_config(self, config: dict) -> None:
"""
Expand Down
3 changes: 1 addition & 2 deletions ...h/tests/config/schemas/optimizers/test-cases/bad/invalid/bad_optimization_direction.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_direction": "INVALID"
"optimization_target": {"score": "INVALID"}
}
}
3 changes: 1 addition & 2 deletions ...ench/tests/config/schemas/optimizers/test-cases/bad/invalid/bad_optimization_target.jsonc
View file
Open in desktop
motus marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": ["latency", "throughput"],
"optimization_direction": "min"
"optimization_target": ["latency", "throughput"]
}
}
2 changes: 1 addition & 1 deletion ...ch/mlos_bench/tests/config/schemas/optimizers/test-cases/bad/invalid/bad_schema_ref.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"class": "mlos_bench.optimizers.MockOptimizer",

"config": {
"optimization_target": "score"
"optimization_target": {"score": "min"}
}
}

2 changes: 1 addition & 1 deletion ...bench/tests/config/schemas/optimizers/test-cases/bad/invalid/bad_space_adapter_type.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
// space_adapter_type only accepts certain values
"space_adapter_type": "unknown"
}
Expand Down
2 changes: 1 addition & 1 deletion ...h/mlos_bench/tests/config/schemas/optimizers/test-cases/bad/invalid/config_bad_type.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"class": "mlos_bench.optimizers.MockOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"seed": "wrong type"
}
}
2 changes: 1 addition & 1 deletion ...los_bench/tests/config/schemas/optimizers/test-cases/bad/invalid/invalid_opt_config.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@
"class": "mlos_bench.optimizers.InvalidOptimizer",

"config": {
"optimization_target": "score"
"optimization_target": {"score": "min"}
}
}
2 changes: 1 addition & 1 deletion ...tests/config/schemas/optimizers/test-cases/bad/invalid/invalid_space_adapter_config.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"space_adapter_type": "IDENTITY",
"space_adapter_config": {
// IDENTITY space adapter doesn't accept any config values
Expand Down
2 changes: 1 addition & 1 deletion ...nch/tests/config/schemas/optimizers/test-cases/bad/invalid/missing_opt_class_config.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false
Expand Down
2 changes: 1 addition & 1 deletion .../tests/config/schemas/optimizers/test-cases/bad/invalid/mlos_opt_bad_optimizer_type.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
// optimizer_type only accepts certain values
"optimizer_type": "unknown"
}
Expand Down
2 changes: 1 addition & 1 deletion ...bench/tests/config/schemas/optimizers/test-cases/bad/invalid/mlos_opt_flaml_bad_arg.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"optimizer_type": "FLAML",
"base_estimator": "foo"
}
Expand Down
2 changes: 1 addition & 1 deletion ...nch/tests/config/schemas/optimizers/test-cases/bad/invalid/mlos_opt_random_bad_conf.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"space_adapter_type": null,
"optimizer_type": "RANDOM",
"random_seed": 0
Expand Down
2 changes: 1 addition & 1 deletion ...ests/config/schemas/optimizers/test-cases/bad/invalid/mlos_opt_smac_bad_probability.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",
"config": {
// Here we do our best to list the exhaustive set of full configs available for the base optimizer config.
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false,
Expand Down
2 changes: 1 addition & 1 deletion ..._bench/tests/config/schemas/optimizers/test-cases/bad/unhandled/mlos_opt_smac_extra.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"optimizer_type": "SMAC",
// base_estimator is only valid for optimizer_type == SKOPT
"base_estimator": "gp"
Expand Down
2 changes: 1 addition & 1 deletion ...ts/config/schemas/optimizers/test-cases/bad/unhandled/mlos_opt_smac_llamatune_extra.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false,
Expand Down
2 changes: 1 addition & 1 deletion ...bench/tests/config/schemas/optimizers/test-cases/bad/unhandled/mock_opt_extra_outer.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"extra_outer": "should not be here",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false
Expand Down
2 changes: 1 addition & 1 deletion ...s_bench/tests/config/schemas/optimizers/test-cases/bad/unhandled/one_shot_opt_extra.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.one_shot_optimizer.OneShotOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"extra_param": "should not be here"
}
}
5 changes: 2 additions & 3 deletions ...los_bench/tests/config/schemas/optimizers/test-cases/good/full/grid_search_opt_full.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,8 @@
],
"config": {
"max_suggestions": 100,
"optimization_direction": "max",
"optimization_target": "score",
"optimization_target": {"score": "max"},
"seed": 12345,
"start_with_defaults": true
}
}
}
2 changes: 1 addition & 1 deletion ...bench/tests/config/schemas/optimizers/test-cases/good/full/mlos_opt_flaml_null_full.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

"config": {
// Here we do our best to list the exhaustive set of full configs available for the base optimizer config.
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 10,
"seed": 12345,
"start_with_defaults": false,
Expand Down
2 changes: 1 addition & 1 deletion .../tests/config/schemas/optimizers/test-cases/good/full/mlos_opt_random_identity_full.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

"config": {
// Here we do our best to list the exhaustive set of full configs available for the base optimizer config.
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false,
Expand Down
2 changes: 1 addition & 1 deletion ...h/tests/config/schemas/optimizers/test-cases/good/full/mlos_opt_smac_llamatune_full.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

"config": {
// Here we do our best to list the exhaustive set of full configs available for the base optimizer config.
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false,
Expand Down
2 changes: 1 addition & 1 deletion ..._bench/tests/config/schemas/optimizers/test-cases/good/full/mlos_opt_smac_null_full.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",
"config": {
// Here we do our best to list the exhaustive set of full configs available for the base optimizer config.
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false,
Expand Down
2 changes: 1 addition & 1 deletion ...bench/mlos_bench/tests/config/schemas/optimizers/test-cases/good/full/mock_opt_full.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

"config": {
// Here we do our best to list the exhaustive set of full configs available for the base optimizer config.
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false
Expand Down
2 changes: 1 addition & 1 deletion ...h/mlos_bench/tests/config/schemas/optimizers/test-cases/good/full/one_shot_opt_full.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

"config": {
// Here we do our best to list the exhaustive set of full configs available for the base optimizer config.
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"start_with_defaults": false
Expand Down
2 changes: 1 addition & 1 deletion ...s/config/schemas/optimizers/test-cases/good/partial/mlos_opt_smac_llamatune_partial.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"class": "mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"max_suggestions": 20,
"seed": 12345,
"optimizer_type": "SMAC",
Expand Down
2 changes: 1 addition & 1 deletion ...ts/config/schemas/optimizers/test-cases/good/partial/mock_opt_schema_partial_config.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"class": "mlos_bench.optimizers.MockOptimizer",

"config": {
"optimization_target": "score",
"optimization_target": {"score": "min"},
"seed": 12345
}
}
Loading
Loading