Skip to content

Commit

Permalink
Add Katib metric strategies info (#2316)
Browse files Browse the repository at this point in the history
* Add metricStrategies information

* Add list

* Modify doc

* Remove article

* Address the comments
  • Loading branch information
andreyvelich authored Nov 13, 2020
1 parent 3b107a6 commit 14a9943
Showing 1 changed file with 34 additions and 3 deletions.
37 changes: 34 additions & 3 deletions content/en/docs/components/katib/experiment.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,14 +64,45 @@ These are the fields in the experiment configuration spec:
how the hyperparameters work with the model.
Katib records the value of the best `objectiveMetricName` metric (maximized
or minimized based on `type`) and the corresponding hyperparameter set
in Experiment's `.status.currentOptimalTrial.parameterAssignments`.
If the `objectiveMetricName` metric for a set of hyperparameters reaches the `goal`,
Katib stops trying more hyperparameter combinations.
in the experiment's `.status.currentOptimalTrial.parameterAssignments`.
If the `objectiveMetricName` metric for a set of hyperparameters reaches the
`goal`, Katib stops trying more hyperparameter combinations.

You can run the experiment without specifying the `goal`. In that case, Katib
runs the experiment until the corresponding successful trials reach `maxTrialCount`.
`maxTrialCount` parameter is described below.

The default way to calculate the experiment's objective is:

- When the objective `type` is `maximize`, Katib compares all maximum
metric values.

- When the objective `type` is `minimize`, Katib compares all minimum
metric values.

To change the default settings, define `metricStrategies` with various rules
(`min`, `max` or `latest`) to extract values for each metric from the
experiment's `objectiveMetricName` and `additionalMetricNames`.
The experiment's objective value is calculated in
accordance with the selected strategy.

For example, you can set the parameters in your experiment as follows:

```yaml
. . .
objectiveMetricName: accuracy
type: maximize
metricStrategies:
- name: accuracy
value: latest
. . .
```

where the Katib controller is searching for the best maximum from the all
latest reported `accuracy` metrics for each trial. Check the
[metrics strategies example](https://github.com/kubeflow/katib/blob/master/examples/v1beta1/metric-strategy-example.yaml).
The default strategy type for each metric is equal to the objective `type`.

Refer to the
[`ObjectiveSpec` type](https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1beta1/common_types.go#L93).

Expand Down

0 comments on commit 14a9943

Please sign in to comment.