Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Moved some content and added links to Kubeflow docs #990

Merged
merged 6 commits into from
Jan 6, 2020
Merged

Moved some content and added links to Kubeflow docs #990

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
aeaf422
WIP Added links to Kubeflow docs in Katib README.
sarahmaddox Dec 31, 2019
74010bf
Merge branch 'master' of https://github.com/kubeflow/katib into docs
sarahmaddox Dec 31, 2019
8a5e7e7
Moved content from user-guide in repo to Kubeflow website.
sarahmaddox Dec 31, 2019
f510550
Moved some content from workflow doc in repo to Kubeflow website.
sarahmaddox Dec 31, 2019
7afdd63
Added links to Kubeflow docs in the quickstart.
sarahmaddox Dec 31, 2019
00a974a
Moved content from algorithms doc in repo to Kubeflow website.
sarahmaddox Dec 31, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 22 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,22 @@ The system is inspired by [Google vizier][3] and supports multiple ML/DL framewo

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure where the source of the TOC is, but on line 29: Pytorch --> PyTorch

## Getting Started
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Update titles to sentence case


See the [getting-started
guide](https://www.kubeflow.org/docs/components/hyperparameter-tuning/hyperparameter/)
on the Kubeflow website.

## Name

Katib stands for `secretary` in Arabic. As `Vizier` stands for a high official or a prime minister in Arabic, this project Katib is named in the honor of Vizier.

## Concepts in Katib

For a detailed description of the concepts in Katib, hyperparameter tuning, and
neural architecture search, see the [Kubeflow
documentation](https://www.kubeflow.org/docs/components/hyperparameter-tuning/overview/).

Katib has the concepts of Experiment, Trial, Job and Suggestion.

### Experiment
Expand All @@ -50,7 +60,9 @@ Each `Experiment` contains a configuration
2. Search Space: Constraints for configurations describing the feasible space.
3. Search Algorithm: How to find the optimal configurations

`Experiment` is defined as a CRD. Refer to [here](docs/user-guide.md) about how to customize a `Experiment`.
`Experiment` is defined as a CRD. See the detailed guide to [configuring and running a Katib
experiment](https://kubeflow.org/docs/components/hyperparameter-tuning/experiment/)
in the Kubeflow docs.

### Suggestion

Expand Down Expand Up @@ -93,11 +105,6 @@ Each component communicates with others via GRPC and the API is defined at `pkg/
- katib-ui: User interface of katib.
- katib-controller: Controller for katib CRDs in Kubernetes.


## Getting Started

Please see [here](./docs/quick-start.md) for more details.

## Web UI

Katib provides a Web UI.
Expand All @@ -112,7 +119,15 @@ Please refer to [api.md](./pkg/apis/manager/v1alpha3/gen-doc/api.md).

## Installation

For standard installation of Katib with support for all job operators, refer to [Kubeflow Official Docs](http://kubeflow.org) and skip this section. Or if you want to install Katib manually, follow these steps
For standard installation of Katib with support for all job operators,
install Kubeflow. See the documentation:

* [Kubeflow installation
guide](https://www.kubeflow.org/docs/started/getting-started/)
* [Kubeflow hyperparameter tuning
guides](https://www.kubeflow.org/docs/components/hyperparameter-tuning/).

Alternatively, if you want to install Katib manually, follow these steps:

```
git clone git@github.com:kubeflow/manifests.git
Expand Down
66 changes: 6 additions & 60 deletions docs/algorithm-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# Hyperparameter Tuning Algorithms

This document describes the usage of hyperparameter tuning algorithms implemented or integrated in katib.

Table of Contents
=================

Expand All @@ -19,75 +17,23 @@ Table of Contents

Created by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)

## Grid Search

Grid sampling applies when all variables are discrete (Doubles and integers need to be quantized) and the number of possibilities is low. A grid search will perform the exhaustive combinatorial search over all possibilities making the search extremely long even for medium sized problems.

### [Chocolate][]

> Chocolate is a completely asynchronous optimisation framework relying solely on a database to share information between workers. Chocolate uses no master process for distributing tasks. Every task is completely independent and only gets its information from the database. Chocolate is thus ideal in controlled computing environments where it is hard to maintain a master process for the duration of the optimisation.

Algorithm name in katib is `grid`.

## Random Search

Random sampling is an alternative to grid search when the number of discrete parameters to optimize and the time required for each evaluation is high. When all parameters are discrete, random search will perform sampling without replacement making it an algorithm of choice when combinatorial exploration is not possible. With continuous parameters, it is preferable to use quasi random sampling.

### [Hyperopt][]

Algorithm name in katib is `random`, and there are some algortihm settings that we support:

| Setting Name | Description | Example |
|------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
| random_state | [int]: Set random state to something other than None for reproducible results. | 10 |


## TPE

### [Hyperopt][]

Algorithm name in katib is `tpe`.

<!-- ## Quasi Random Search

QuasiRandom sampling ensures a much more uniform exploration of the search space than traditional pseudo random. Thus, quasi random sampling is preferable when not all variables are discrete, the number of dimensions is high and the time required to evaluate a solution is high.

### [Chocolate][]
### [Chocolate](https://chocolate.readthedocs.io)

Algorithm name in katib is `chocolate-quasirandom`. -->

<!-- ## CMAES

CMAES search is one of the most powerful black-box optimization algorithm. However, it requires a significant number of model evaluation (in the order of 10 to 50 times the number of dimensions) to converge to an optimal solution. This search method is more suitable when the time required for a model evaluation is relatively low.

### [Chocolate][]
### [Chocolate](https://chocolate.readthedocs.io)

Algorithm name in katib is `chocolate-CMAES`. -->

## Bayesian Optimization

Bayes search models the search space using gaussian process regression, which allows to have an estimate of the loss function and the uncertainty on that estimate at every point of the search space. Modeling the search space suffers from the curse of dimensionality, which makes this method more suitable when the number of dimensions is low. Moreover, since it models both the expected loss and uncertainty, this search algorithm converges in few steps on superior configurations, making it a good choice when the time to complete the evaluation of a parameter configuration is high.

### [scikit-optimize][]

> Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It implements several methods for sequential model-based optimization. skopt aims to be accessible and easy to use in many contexts.

Algorithm name in katib is `skopt-bayesian-optimization`, and there are some algortihm settings that we support:

| Setting Name | Description | Example |
|------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
| base_estimator | ["GP", "RF", "ET", "GBRT" or sklearn regressor, default="GP"]: Should inherit from `sklearn.base.RegressorMixin`. In addition, the `predict` method, should have an optional `return_std` argument, which returns `std(Y | x)` along with `E[Y | x]`. If base_estimator is one of ["GP", "RF", "ET", "GBRT"], a default surrogate model of the corresponding type is used corresponding to what is used in the minimize functions. More in [skopt document](https://scikit-optimize.github.io/#skopt.Optimizer) | GP |
| n_initial_points | [int, default=10]: Number of evaluations of `func` with initialization points before approximating it with `base_estimator`. Points provided as `x0` count as initialization points. If len(x0) < n_initial_points additional points are sampled at random. More in [skopt document](https://scikit-optimize.github.io/#skopt.Optimizer) | 10 |
| acq_func | [string, default=`"gp_hedge"`]: Function to minimize over the posterior distribution. More in [skopt document](https://scikit-optimize.github.io/#skopt.Optimizer) | gp_hedge |
| acq_optimizer | [string, "sampling" or "lbfgs", default="auto"]: Method to minimize the acquistion function. The fit model is updated with the optimal value obtained by optimizing acq_func with acq_optimizer. More in [skopt document](https://scikit-optimize.github.io/#skopt.Optimizer) | auto |
| random_state | [int]: Set random state to something other than None for reproducible results. | 10 |

## References

- [Chocolate][]
- [scikit-optimize][]
- [Hyperopt][]

[Hyperopt]: http://hyperopt.github.io/hyperopt/
[scikit-optimize]: https://github.com/scikit-optimize/scikit-optimize
[Chocolate]: https://chocolate.readthedocs.io
For information about the hyperparameter tuning algorithms and neural
architecture search implemented or integrated in Katib, see the detailed guide
to [configuring and running a Katib
experiment](https://kubeflow.org/docs/components/hyperparameter-tuning/experiment/).
10 changes: 10 additions & 0 deletions docs/developer-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,16 @@ Created by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)

# Developer Guide

This developer guide is for people who want to contribute to the Katib project.
If you're interesting in using Katib in your machine learning project,
see the following user guides:

* [Concepts](https://www.kubeflow.org/docs/components/hyperparameter-tuning/overview/)
in Katib, hyperparameter tuning, and neural architecture search.
* [Getting started with Katib](https://kubeflow.org/docs/components/hyperparameter-tuning/hyperparameter/).
* Detailed guide to [configuring and running a Katib
experiment](https://kubeflow.org/docs/components/hyperparameter-tuning/experiment/).

## Requirements

- [Go](https://golang.org/)
Expand Down
10 changes: 10 additions & 0 deletions docs/quick-start.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,16 @@ Before you run the hyperparameter tuning experiment, you need to have:

- A Kubernetes cluster with Kubeflow 0.7

## Katib in Kubeflow

See the following guides in the Kubeflow documentation:

* [Concepts](https://www.kubeflow.org/docs/components/hyperparameter-tuning/overview/)
in Katib, hyperparameter tuning, and neural architecture search.
* [Getting started with Katib](https://kubeflow.org/docs/components/hyperparameter-tuning/hyperparameter/).
* Detailed guide to [configuring and running a Katib
experiment](https://kubeflow.org/docs/components/hyperparameter-tuning/experiment/).

## Hyperparameter Tuning on MNIST

Katib supports multiple [Machine Learning Frameworks](https://en.wikipedia.org/wiki/Comparison_of_deep-learning_software) (e.g. TensorFlow, PyTorch, MXNet, and XGBoost).
Expand Down
11 changes: 3 additions & 8 deletions docs/user-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
`Experiment` is user level CRD. When you want to tune hyperparameter for your machine learning model by Katib, you just need create an Experiment CR. There are several examples [here](../examples/v1alpha3/) for your reference. In this document, fields of `Experiment` will be introduced in detail.

## metadata
- name
- namespace
## spec
- (TODO)
- [spec.metricsCollectorSpec.source.filter.metricsFormat](https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1alpha3/common_types.go#L94-L98): Its type is string array and each element is regular expression (here it is required that the regular expression must have two top subexpressions, the first matched one will be taken as metric name and the second one as metric value). For now, when metrics collector kind is `File` and `StdOut`, this field can take effect and a default one `([\w|-]+)\s*=\s*((-?\d+)(\.\d+)?)` will be applied if not specified. [Here](https://github.com/kubeflow/katib/blob/master/examples/v1alpha3/file-metricscollector-example.yaml) is an example of customized metrics filter.
See the detailed guide to [configuring and running a Katib
experiment](https://kubeflow.org/docs/components/hyperparameter-tuning/experiment/)
in the Kubeflow docs.
Loading