Refactor mlos_bench.storage and add TunableConfigTrialGroup property for TrialData and ExperimentData
#648
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Work split out from microsoft#633 - Reorgs the unit test fixtures for reuse by mlos_viz (future PR) - Removes TunablesGroups from Storage classes. Currently unnecessary, belongs in Experiment, and causes inconsistencies with ExperimentData interactions. - Adjusts the initialization code to match.
…m-storage-classes
- Standardize on experiment_id instead of exp_id (but not the db schema for now) - Standardize on tunable_config_id instead of config_id in the API (but not the db schema for now) - Rework the idea of TunableConfigTrialGroup as an object. - Rework the idea of a TunableConfig as an object. - Rename results APIs to results_df (similar for others that return dataframes) - Refactor test fixtures to match other styles and for future use. - Expand tests
bpkroth
changed the title
config_trial_group_id property for TrialData and ExperimentDataTunableConfigTrialGroup property for TrialData and ExperimentData
bpkroth
commented
Jan 23, 2024
mlos_bench/mlos_bench/tests/storage/tunable_config_trial_group_data_test.py
Outdated
Show resolved
Hide resolved
bpkroth
commented
Jan 23, 2024
mlos_bench/mlos_bench/tests/storage/tunable_config_data_test.py
Outdated
Show resolved
Hide resolved
bpkroth
commented
Jan 23, 2024
bpkroth
commented
Jan 23, 2024
bpkroth
commented
Jan 23, 2024
bpkroth
changed the title
TunableConfigTrialGroup property for TrialData and ExperimentDatamlos_bench.storage and add TunableConfigTrialGroup property for TrialData and ExperimentData
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Useful for grouping trials by the config they used. In use by upcoming #633 for generating graphs with variance error bars for repeated configs.
Also refactors a number of other things:
experiment_idinstead ofexp_id(but not the db schema for now)
tunable_config_idinstead ofconfig_idin the API since we also call ittunable_configfor the object fetching property to distinguish from theconfigdict used internally.(but not the db schema for now)
TunableConfigTrialGroupas an object inaddition to an ID (further methods can be added later to move back and forth between types when doing interactive analysis).TunableConfigas an object for fetching tunable value assignments (similar justification - easier grouping in the future by fetching trial across experiments based on config - eventually could be used to house the experiment merge logic).resultsAPIs toresults_df(similar for others that returnpandas.DataFrame) to match theresults_dictthat returndictNOTE:
results-> toresults_dfandexp_id->experiment_id).Currently builds off of #644 and splits work out of #633