-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for storing the optimization targets and direction of an experiment #628
Add support for storing the optimization targets and direction of an experiment #628
Conversation
wait, we already store that data in the MLOS/mlos_bench/mlos_bench/run.py Lines 120 to 124 in 2032d79
you can add more key/value pairs to store with each trial there |
We had a conversation about this offline. I think the conclusion was that:
I think with that in mind, I will still pursue this PR and a future one can enforce the strictness aspects of not changing things, perhaps after we make it easier to merge other "related" TrialData (though capturing which those are might itself be tricky). |
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! I leave it up to you to keep the optimization direction nullable or roll back to the non-nullable variant
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and dabl/dabl#335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: Microsoft-CISL/sqlite-autotuning#41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
#628 mistakenly included an early attempt at adding `optimization_target` and `optimization_direction` to the `experiment` table in the `mlos_bench.storage.sql` backend. In that PR we later moved it to its own `objectives` table to eventually support multi-objectives. Nothing accesses those columns now, however including them in the metadata makes it impossible to load storage backends previously created with the old schema since adjusting columns with sqlalchemy's `create_all()` API only considers table existence. On the contrary, the latter means that we will automatically support old storage backends with the new code for the `objectives` table. Removing these two columns in the `metadata` schema description simply allows that to proceed without error. See Also: #649 Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
This PR is useful for mlos-viz and dabl wrapper (#624) to be able to automatically graph the results for a given optimization target, for instance via something like the following:
Since the prior efforts on capturing this data in the Trial metadata are somewhat problematic (allow conflicting changes between runs of an experiment, don't support multi-objective), we extend them to also store values directly as a part of the Experiment, which is a somewhat more appropriate location. Upon retrieval, an attempt is also made to merge the two data sources for backwards compatibility.
This PR does not enforce strictness on that metadata, but future versions could (e.g., disallow resuming an Experiment if it looks like the objective targets have changed. In that case the prior Trial results can potentially still be used to prewarm a new Experiment's optimizer).