Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LlamaTune inverse_trainsform() produces invalid configuration #805

Closed
motus opened this issue Jul 18, 2024 · 1 comment · Fixed by #812
Closed

LlamaTune inverse_trainsform() produces invalid configuration #805

motus opened this issue Jul 18, 2024 · 1 comment · Fixed by #812
Labels
bug Something isn't working mlos-core

Comments

@motus
Copy link
Member

motus commented Jul 18, 2024

LlamaTuneAdapter.inverse_transform() apparently produces small float values like 1.5 and 0.998 and assigns them to int hyperparameters with range [1, 10000].
If we add validation immediately after producing such config, it will fail, causing the corresponding unit test failure:

FAILED mlos_bench/mlos_bench/tests/optimizers/llamatune_opt_test.py::test_llamatune_optimizer - ConfigSpace.exceptions.IllegalValueError: Value 1.5: (<class 'float'>) is not allowed for hyperparameter dim_0, Type: UniformInteger, Range: [1, 10000], Default: 5000

Interestingly, in ConfigSpace 1.0 same float inputs get scaled up from [0, 1] to int range of the corresponding hyperparameter. That also produces invalid configuration because value of 1.5 got scaled to 15000, i.e., outside of the hyperparameter's [1, 10000] range.

@motus
Copy link
Member Author

motus commented Jul 18, 2024

@kkanellis can you please take a look at it? I am bit lost in LlamaTune code. Use #806 to reproduce the problem, then run

pytest -vv -n1 mlos_bench/mlos_bench/tests/optimizers/llamatune_opt_test.py::test_llamatune_optimizer

Thank you!

@motus motus mentioned this issue Jul 25, 2024
5 tasks
motus added a commit that referenced this issue Aug 2, 2024
Summary of changes:
* [x] Update dependencies to ConfigSpace >= 1.0.
* [x] Update dependencies to SMAC3 >= 2.2.
* [x] Migrate to new ConfigSpace API.
* [x] Better definition of ConfigSpace fixture in unit tests.
* [x] Fix mypy and pylint issues triggered by the migration.

Notes:
* Side effect: **Works with Python 3.12!**
* ~Supersedes #799~
* ~Depends on upcoming SMAC3 release with [SMAC3
#1124](automl/SMAC3#1124) merged in and a new
release cut with it.~
* Quantization will be added back later:
    * #803 
* ~Tests fail due to LlamaTune issue:~
    * #805 

Closes #727

---------

Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working mlos-core
Projects
None yet
1 participant