Support constraints for intermediate values plot #4851

adjeiv · 2023-08-01T20:42:56Z

Motivation

References #4830

Description of the changes

Support constraints with the intermediate values plots, displaying infeasible trials in #CCCCCC colour.
Tests added accordingly.

import optuna
from optuna.Trial import Trial, FrozenTrial
from optuna.Study import create_study
from optuna.visualization import plot_intermediate_values
from typing import Sequence

def objective_with_constraints(trial: Trial) -> float:
        trial.set_user_attr("constraint", [trial.number % 2])

        trial.report(trial.number, step=0)
        trial.report(trial.number + 1, step=1)
        return 0.0

def constraints(trial: FrozenTrial) -> Sequence[float]:
    return trial.user_attrs["constraint"]

study = create_study(sampler=optuna.samplers.NSGAIIISampler(constraints_func=constraints))
study.optimize(objective_with_constraints, n_trials=10)

plot_intermediate_values(study)

Plot is below:

adjeiv · 2023-08-01T22:09:22Z

Re. failing docs build, readthedocs/readthedocs.org#10279 seems relevant... Pinning the sphinx version to be less than 7 should fix this

not522 · 2023-08-02T05:18:48Z

Thank you for your PR and the investigation for CI fail. The Optuna maintainers are working to fix the CI. The fix is not necessary to merge this PR, so don't worry about it.

not522 · 2023-08-02T05:19:21Z

@Alnusjaponica Could you review this PR?

Alnusjaponica

Thanks for your contribution.
Could you make a similar change in optuna/visualization/matplotlib/_intermediate_values.py?

Alnusjaponica · 2023-08-03T04:35:07Z

optuna/visualization/_intermediate_values.py

-            marker={"maxdisplayed": 10},
+            marker=default_marker
+            if tinfo.feasible
+            else default_marker | {"color": "#CCCCCC"},  # type: ignore[operator]


Using | as a union operator for dict is only available since Python 3.9. Several ways to do it before Python 3.9 is mentioned in PEP584, so please use one of them.

Alnusjaponica · 2023-08-03T04:47:54Z

tests/visualization_tests/test_intermediate_plot.py

@@ -69,7 +103,9 @@ def objective(trial: Trial, report_intermediate_values: bool) -> float:
        _IntermediatePlotInfo(trial_infos=[]),


By adding the following example, you'll see the test fails when python<=3.8
ref. #4851 (comment)

_IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ), _TrialInfo( trial_number=1, sorted_intermediate_values=[(1, 2.0), (0, 1.0)], feasible=False ) ] )

Could you add this test case to confirm if test_plot_intermediate_values works when study has infeasible results?

Alnusjaponica

Thanks for your prompt response!
I have added some comments and would appreciate it if you could review them.

Alnusjaponica · 2023-08-03T06:49:53Z

optuna/visualization/_intermediate_values.py

-            marker={"maxdisplayed": 10},
+            marker=default_marker
+            if tinfo.feasible
+            else default_marker.update({"color": "#CCCCCC"}),  # type: ignore[dict-item]


Sorry for the confusion but dict.update() returns None like list.sort(). Please change this part like this.

Suggested change

else default_marker.update({"color": "#CCCCCC"}), # type: ignore[dict-item]

else {**default_marker, "color": "#CCCCCC"},

Of course - silly mistake. Thank you :)

Alnusjaponica · 2023-08-03T06:53:12Z

tests/visualization_tests/test_intermediate_plot.py

@@ -69,7 +103,9 @@ def objective(trial: Trial, report_intermediate_values: bool) -> float:
        _IntermediatePlotInfo(trial_infos=[]),


Could you add this test case to confirm if test_plot_intermediate_values works when study has infeasible results?

Alnusjaponica · 2023-08-03T07:53:19Z

Thanks! LGTM.
Let me approve this PR after we confirm CI success.

not522

Thank you! LGTM except the format. Please check the Checks CI.

codecov-commenter · 2023-08-03T09:09:38Z

Codecov Report

Merging #4851 (0da819e) into master (55ed020) will decrease coverage by 0.05%.
Report is 84 commits behind head on master.
The diff coverage is 100.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #4851      +/-   ##
==========================================
- Coverage   89.52%   89.48%   -0.05%     
==========================================
  Files         197      200       +3     
  Lines       14676    14785     +109     
==========================================
+ Hits        13139    13230      +91     
- Misses       1537     1555      +18

Files Changed	Coverage Δ
...a/visualization/matplotlib/_intermediate_values.py	`92.59% <ø> (+3.70%)`	⬆️
optuna/visualization/_intermediate_values.py	`94.87% <100.00%> (+1.12%)`	⬆️

... and 18 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

not522

LGTM!

Alnusjaponica

Thanks! LGTM.

Support constraints for intermediate values plot

0864314

github-actions bot added the optuna.visualization Related to the `optuna.visualization` submodule. This is automatically labeled by github-actions. label Aug 1, 2023

Add ignore for type and re-format

e17dc22

not522 assigned not522 and Alnusjaponica Aug 2, 2023

not522 added the feature Change that does not break compatibility, but affects the public interfaces. label Aug 2, 2023

Alnusjaponica requested changes Aug 3, 2023

View reviewed changes

Alnusjaponica reviewed Aug 3, 2023

View reviewed changes

Fix for Python versions lower than 3.9, and update matplotlib backend

ad6e96c

Alnusjaponica requested changes Aug 3, 2023

View reviewed changes

Fix dictionary, add test case

ac7bf18

not522 reviewed Aug 3, 2023

View reviewed changes

Fix formatting

0da819e

not522 approved these changes Aug 4, 2023

View reviewed changes

Alnusjaponica approved these changes Aug 4, 2023

View reviewed changes

Alnusjaponica removed their assignment Aug 4, 2023

not522 added this to the v3.4.0 milestone Aug 4, 2023

not522 merged commit 056dd2f into optuna:master Aug 4, 2023
30 checks passed

adjeiv deleted the intermediate_values_plot_constraints branch August 4, 2023 07:39

nabenabe0928 mentioned this pull request Oct 31, 2023

Support intermediate value plots for constrained optimization optuna/optuna-dashboard#676

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support constraints for intermediate values plot #4851

Support constraints for intermediate values plot #4851

adjeiv commented Aug 1, 2023 •

edited

Loading

adjeiv commented Aug 1, 2023

not522 commented Aug 2, 2023

not522 commented Aug 2, 2023

Alnusjaponica left a comment

Alnusjaponica Aug 3, 2023

Alnusjaponica Aug 3, 2023 •

edited

Loading

Alnusjaponica Aug 3, 2023

Alnusjaponica left a comment

Alnusjaponica Aug 3, 2023 •

edited

Loading

adjeiv Aug 3, 2023

Alnusjaponica Aug 3, 2023

Alnusjaponica commented Aug 3, 2023

not522 left a comment •

edited

Loading

codecov-commenter commented Aug 3, 2023 •

edited

Loading

not522 left a comment

Alnusjaponica left a comment

		@@ -69,7 +103,9 @@ def objective(trial: Trial, report_intermediate_values: bool) -> float:
		_IntermediatePlotInfo(trial_infos=[]),

	else default_marker.update({"color": "#CCCCCC"}), # type: ignore[dict-item]
	else {**default_marker, "color": "#CCCCCC"},

Support constraints for intermediate values plot #4851

Support constraints for intermediate values plot #4851

Conversation

adjeiv commented Aug 1, 2023 • edited Loading

Motivation

Description of the changes

adjeiv commented Aug 1, 2023

not522 commented Aug 2, 2023

not522 commented Aug 2, 2023

Alnusjaponica left a comment

Choose a reason for hiding this comment

Alnusjaponica Aug 3, 2023

Choose a reason for hiding this comment

Alnusjaponica Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Alnusjaponica Aug 3, 2023

Choose a reason for hiding this comment

Alnusjaponica left a comment

Choose a reason for hiding this comment

Alnusjaponica Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

adjeiv Aug 3, 2023

Choose a reason for hiding this comment

Alnusjaponica Aug 3, 2023

Choose a reason for hiding this comment

Alnusjaponica commented Aug 3, 2023

not522 left a comment • edited Loading

Choose a reason for hiding this comment

codecov-commenter commented Aug 3, 2023 • edited Loading

Codecov Report

not522 left a comment

Choose a reason for hiding this comment

Alnusjaponica left a comment

Choose a reason for hiding this comment

adjeiv commented Aug 1, 2023 •

edited

Loading

Alnusjaponica Aug 3, 2023 •

edited

Loading

Alnusjaponica Aug 3, 2023 •

edited

Loading

not522 left a comment •

edited

Loading

codecov-commenter commented Aug 3, 2023 •

edited

Loading