Create notebook for Auto and Tune

[Mr-Geekman]

Before submitting (must do checklist)

• Did you read the contribution guide?
• Did you update the docs? We use Numpy format for all the methods and classes.
• Did you write any new necessary tests?
• Did you update the CHANGELOG?

Proposed Changes

1. Skip duplicates during optimization.
2. Remove duplicates from top_k
3. Look Create notebook for Auto and Tune #1274.

Closing issues

Closes #1274

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (automl-2.0@c343ab8). Click here to learn what that means.
The diff coverage is n/a.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@              Coverage Diff              @@
##             automl-2.0    #1285   +/-   ##
=============================================
  Coverage              ?   88.44%           
=============================================
  Files                 ?      186           
  Lines                 ?    11264           
  Branches              ?        0           
=============================================
  Hits                  ?     9962           
  Misses                ?     1302           
  Partials              ?        0           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[github-actions]

🚀 Deployed on https://deploy-preview-1285--etna-docs.netlify.app

[Mr-Geekman]

There is a problem of big memory consumption of the notebook.

Setting gc_after_trial=True in a notebook doesn't solve a problem. Running script instead of the notebook doesn't solve it either.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

alex-hse-repository commented on 2023-06-13T11:50:23Z
----------------------------------------------------------------

Do we really want to keep the summary with duplicates?

Mr-Geekman commented on 2023-06-13T13:41:31Z
----------------------------------------------------------------

I think that summary should show all trials, it is strange to hide smth from the user.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

alex-hse-repository commented on 2023-06-13T11:50:24Z
----------------------------------------------------------------

Let's hide this output

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

martins0n commented on 2023-06-13T13:14:30Z
----------------------------------------------------------------

We should colaps these outputs

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

martins0n commented on 2023-06-13T13:14:31Z
----------------------------------------------------------------

May be we should show only important columns

Mr-Geekman commented on 2023-06-13T13:39:33Z
----------------------------------------------------------------

I tried to do this in the next block. What do you mean by showing only important columns? Change existing summary method or limit columns in this block and don't show full sammary ?

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

martins0n commented on 2023-06-13T13:14:32Z
----------------------------------------------------------------

It's not an easy task to find difference - we should show it.

Moreover if results ok we should add metrics on holdout to be more convincing

Mr-Geekman commented on 2023-06-13T13:48:02Z
----------------------------------------------------------------

It is easy to see at the summary by the column study. Top rows are with tune/ values, not with pool.I can emphasize it in the text.

Holdout is a good idea, I will try to add it.

[martins0n]

#1285 (comment)

Did you find reason? In what place does it happen?

[Mr-Geekman]

I tried to do this in the next block. What do you mean by showing only important columns? Change existing summary method or limit columns in this block and don't show full sammary ?

---

View entire conversation on ReviewNB

[Mr-Geekman]

I think that summary should show all trials, it is strange to hide smth from the user.

---

View entire conversation on ReviewNB

[Mr-Geekman]

It is easy to see at the summary by the column study. Top rows are with tune/ values, not with pool.I can emphasize it in the text.

Holdout is a good idea, I will try to add it.

---

View entire conversation on ReviewNB