Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[air][tune] Aim logger #32041

Merged
merged 77 commits into from
Mar 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
fd93d57
added a first working draft of the aim_logger alongside unit test
ju2ez Nov 26, 2022
7276a39
adjusted the logger/__init__.py. Now it imports the AimCallback
ju2ez Nov 26, 2022
61d1b6e
[feat] Add docs
tamohannes Jan 15, 2023
f71c41e
[fix] Sign-off
tamohannes Jan 15, 2023
1b0ba2a
param check is now using pop
ju2ez Jan 21, 2023
af613e5
metrics are now checked before they are logged.
ju2ez Jan 21, 2023
20b7f7d
[feat] Add tests
tamohannes Jan 21, 2023
63b646b
Removed comment
ju2ez Jan 22, 2023
99ffd41
[feat] Add as_multirun option. Update the docstring
tamohannes Dec 29, 2022
2533be5
[feat] Add docs
tamohannes Jan 15, 2023
2d80acd
Fixed the imports and added try except statement for third party libr…
ju2ez Jan 2, 2023
59e3538
Update python/ray/tune/logger/aim.py
ju2ez Jan 2, 2023
feb9f8a
Added assertion for aim import.
ju2ez Jan 2, 2023
b4de23f
Type hint for _create_run(..)
ju2ez Jan 2, 2023
62210e3
Renamed log_hparams function.
ju2ez Jan 2, 2023
080cf1f
Fixed function call.
ju2ez Jan 2, 2023
1c701c6
Adjusted docstring and added the possibility to give kwargs to aim.sd…
ju2ez Jan 2, 2023
d860cb8
[feat] Add docs
tamohannes Jan 15, 2023
7bacdb8
Small refinements
ju2ez Jan 30, 2023
a208e56
Update python/ray/tune/logger/aim.py
ju2ez Feb 3, 2023
de457eb
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
5b6a5f5
Update doc/source/tune/api_docs/logging.rst
ju2ez Feb 3, 2023
10ba378
Update doc/source/tune/api_docs/logging.rst
ju2ez Feb 3, 2023
3310e8a
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
7a36941
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
8baf62d
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
a6a2b0e
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
688c8bd
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
81ca4e0
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
1d91634
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
3164826
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 3, 2023
61e1c72
Update python/ray/tune/logger/aim.py
ju2ez Feb 3, 2023
d710082
Update python/ray/tune/logger/aim.py
ju2ez Feb 3, 2023
394370b
Update python/ray/tune/logger/aim.py
ju2ez Feb 3, 2023
78fa601
Update python/ray/tune/logger/aim.py
ju2ez Feb 3, 2023
0c2a80e
Merge branch 'ray-project:master' into aim
ju2ez Feb 3, 2023
b355e03
Minor adjustments to the docs
ju2ez Feb 3, 2023
0312a67
Update python/ray/tune/logger/aim.py
ju2ez Feb 6, 2023
a89ca3e
Update python/ray/tune/logger/aim.py
ju2ez Feb 6, 2023
7fe127e
Update python/ray/tune/logger/aim.py
ju2ez Feb 6, 2023
b855ff0
Update python/ray/tune/logger/aim.py
ju2ez Feb 6, 2023
143404f
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 6, 2023
6b08871
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 6, 2023
87ff75f
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 6, 2023
9632173
Update python/ray/tune/logger/aim.py
ju2ez Feb 6, 2023
5924080
Update python/ray/tune/logger/aim.py
ju2ez Feb 6, 2023
5374e29
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 6, 2023
7b749e2
Update doc/source/tune/examples/tune-aim.ipynb
ju2ez Feb 6, 2023
4006c93
Merge branch 'ray-project:master' into aim
ju2ez Feb 6, 2023
f390ef1
Fix code in aim.py
justinvyu Feb 6, 2023
d79a72d
Rename AimCallback -> AimLoggerCallback
justinvyu Feb 6, 2023
2cdca85
Various cleanups
justinvyu Feb 6, 2023
3b820d8
Support tuples, sets
justinvyu Feb 6, 2023
3fedfeb
Fix certain hparam types getting thrown away incorrectly
justinvyu Feb 7, 2023
b976e44
Improve testing to cover different configuration options
justinvyu Feb 7, 2023
9751b11
Fix notebook example
justinvyu Feb 7, 2023
210701a
Update python/ray/tune/logger/aim.py
ju2ez Feb 7, 2023
c1c7997
Update python/ray/tune/logger/aim.py
ju2ez Feb 7, 2023
a4d41e9
Remove as_multirun flag
justinvyu Feb 7, 2023
7f97cfa
Merge branch 'aim' of https://github.com/ju2ez/ray into aimlogger
justinvyu Feb 7, 2023
19ce007
Fix merge
justinvyu Feb 7, 2023
6b7295a
Add aim as a Tune dependency
justinvyu Feb 7, 2023
ae163bf
Check that aim callback can be imported from ray.tune.logger
justinvyu Feb 7, 2023
7175239
Merge branch 'master' of https://github.com/ray-project/ray into aiml…
justinvyu Feb 7, 2023
0c7fafe
Fix formatting for kwargs in docstring
justinvyu Feb 7, 2023
ebaf3bf
Merge branch 'master' of https://github.com/ray-project/ray into aiml…
justinvyu Feb 7, 2023
f59a78c
Merge branch 'master' of https://github.com/ray-project/ray into aiml…
justinvyu Feb 27, 2023
7a39431
Pin aim to 3.16.1 w/ versioning patch fix for CI to pass
justinvyu Feb 27, 2023
04d0b1e
Fix aim logger tests
justinvyu Mar 2, 2023
b17eb00
Merge branch 'master' of https://github.com/ray-project/ray into aiml…
justinvyu Mar 2, 2023
01a6541
Update docs to follow new format
justinvyu Mar 2, 2023
c2ecbd0
Fix text in user guide
justinvyu Mar 2, 2023
6c94f37
Don't import aim callback by default
justinvyu Mar 2, 2023
48b6341
Merge branch 'master' of https://github.com/ray-project/ray into aiml…
justinvyu Mar 2, 2023
29d1caa
Fix module path in API ref
justinvyu Mar 2, 2023
6b02036
Fix tests + example imports
justinvyu Mar 2, 2023
80be6f9
Merge branch 'master' of https://github.com/ray-project/ray into aiml…
justinvyu Mar 2, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions doc/source/_toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -201,12 +201,14 @@ parts:
title: "Huggingface Example"
- file: tune/examples/experiment-tracking
sections:
- file: tune/examples/tune-comet
title: "Comet Example"
- file: tune/examples/tune-wandb
title: "Weights & Biases Example"
- file: tune/examples/tune-mlflow
title: "MLflow Example"
- file: tune/examples/tune-aim
title: "Aim Example"
- file: tune/examples/tune-comet
title: "Comet Example"
- file: tune/examples/hpo-frameworks
sections:
- file: tune/examples/ax_example
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/source/images/aim_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/source/images/aim_logo_full.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
78 changes: 48 additions & 30 deletions doc/source/tune/api/logging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,33 @@ If you need to log something lower level like model weights or gradients,
see :ref:`Trainable Logging <trainable-logging>`.

.. note::

Tune's per-trial ``Logger`` classes have been deprecated. They can still be used, but we encourage you
to use our new interface with the ``LoggerCallback`` class instead.
to use the ``LoggerCallback`` interface instead.


Viskit
------
.. currentmodule:: ray

Tune automatically integrates with `Viskit <https://github.com/vitchyr/viskit>`_ via the ``CSVLoggerCallback`` outputs.
To use VisKit (you may have to install some dependencies), run:
.. _logger-interface:

.. code-block:: bash
LoggerCallback Interface (tune.logger.LoggerCallback)
-----------------------------------------------------

$ git clone https://github.com/rll/rllab.git
$ python rllab/rllab/viskit/frontend.py ~/ray_results/my_experiment
.. autosummary::
:toctree: doc/
:template: autosummary/class_with_autosummary.rst

The non-relevant metrics (like timing stats) can be disabled on the left to show only the
relevant ones (like accuracy, loss, etc.).
~tune.logger.LoggerCallback

.. image:: ../images/ray-tune-viskit.png
.. autosummary::
:toctree: doc/

~tune.logger.LoggerCallback.log_trial_start
~tune.logger.LoggerCallback.log_trial_restore
~tune.logger.LoggerCallback.log_trial_save
~tune.logger.LoggerCallback.log_trial_result
~tune.logger.LoggerCallback.log_trial_end

.. currentmodule:: ray

Tune Built-in Loggers
---------------------
Expand All @@ -44,46 +49,59 @@ Tune Built-in Loggers
tune.logger.TBXLoggerCallback


MLFlow Integration: MLFlowLoggerCallback
----------------------------------------
MLFlow Integration
------------------

Tune also provides a logger for `MLflow <https://mlflow.org>`_.
You can install MLflow via ``pip install mlflow``.
You can see the :doc:`tutorial here </tune/examples/tune-mlflow>`.
See the :doc:`tutorial here </tune/examples/tune-mlflow>`.

.. autosummary::
:toctree: doc/

air.integrations.mlflow.MLflowLoggerCallback

Wandb Integration: WandbLoggerCallback
--------------------------------------
Wandb Integration
-----------------

Tune also provides a logger for `Weights & Biases <https://www.wandb.ai/>`_.
You can install Wandb via ``pip install wandb``.
You can see the :doc:`tutorial here </tune/examples/tune-wandb>`.
See the :doc:`tutorial here </tune/examples/tune-wandb>`.

.. autosummary::
:toctree: doc/

air.integrations.wandb.WandbLoggerCallback

.. _logger-interface:
Aim Integration
---------------

LoggerCallback Interface (tune.logger.LoggerCallback)
-----------------------------------------------------
Tune also provides a logger for the `Aim <https://aimstack.io/>`_ experiment tracker.
You can install Aim via ``pip install aim``.
See the :doc:`tutorial here </tune/examples/tune-aim>`

.. autosummary::
:toctree: doc/
:template: autosummary/class_with_autosummary.rst

~tune.logger.LoggerCallback
~tune.logger.aim.AimLoggerCallback

.. autosummary::
:toctree: doc/

~tune.logger.LoggerCallback.log_trial_start
~tune.logger.LoggerCallback.log_trial_restore
~tune.logger.LoggerCallback.log_trial_save
~tune.logger.LoggerCallback.log_trial_result
~tune.logger.LoggerCallback.log_trial_end
Other Integrations
------------------

Viskit
~~~~~~

Tune automatically integrates with `Viskit <https://github.com/vitchyr/viskit>`_ via the ``CSVLoggerCallback`` outputs.
To use VisKit (you may have to install some dependencies), run:

.. code-block:: bash

$ git clone https://github.com/rll/rllab.git
$ python rllab/rllab/viskit/frontend.py ~/ray_results/my_experiment

The non-relevant metrics (like timing stats) can be disabled on the left to show only the
relevant ones (like accuracy, loss, etc.).

.. image:: ../images/ray-tune-viskit.png

8 changes: 8 additions & 0 deletions doc/source/tune/examples/experiment-tracking.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,14 @@ to use Ray Tune with Tensorboard, you can find more information in our
:column: col-md-4 px-2 py-2
:img-top-cls: pt-5 w-75 d-block mx-auto

---
:img-top: /images/aim_logo.png

+++
.. link-button:: tune-aim-ref
:type: ref
:text: Using Aim with Ray Tune For Experiment Management
:classes: btn-link btn-block stretched-link
---
:img-top: /images/comet_logo_full.png

Expand Down
Loading