General stateful metrics fixes #9446

pasky · 2018-02-21T18:03:22Z

This implements a variety of stateful metrics improvements I noted in #9253 .

Require stateful metrics layers to be actually stateful: It seems a bit confusing to me that the Layer of a stateful metric doesn't need to have the stateful attribute set - with the assumption that all Layer metrics will be stateful, and behaving statefully even without this attribute. Is that a good assumption to make for the future?
Prevent stateful metrics to leak np.floats to the History object: When logging normal metrics, they are added in _*_loop() to 0., making them float. This doesn't happen with stateful metrics (they are assigned directly), so they are np.float32. This is annoying if you e.g. want to serialize the history object to JSON after training, which used to work fine before.
Progbar: Format stateful metrics values as floats alike other metrics: In progress bar (verbose=1), stateful metric values aren't formatted as %.4f but %s. This is messy with many metrics, but also sometimes one \b too many is printed (didn't find out why) and the progress bar jumps a line upwards (overwriting earlier content)
Most importantly, fit_generator() and evaluate_generator() now support stateful metrics.
I wrote some documentation.

This makes sure the metric is reset before evaluating valset.

…rator()

It would be even better to have full-fledged stateful layers documentations, but I lack the knowledge and experience to explain that well.

briannemsick · 2018-02-22T01:25:18Z

docs/templates/metrics.md

@@ -19,7 +19,7 @@ model.compile(loss='mean_squared_error',

 A metric function is similar to a [loss function](/losses), except that the results from evaluating a metric are not used when training the model.

-You can either pass the name of an existing metric, or pass a Theano/TensorFlow symbolic function (see [Custom metrics](#custom-metrics)).
+You can either pass the name of an existing metric, or pass a function or layer (see [Custom metrics](#custom-metrics)).


Consider minor reword: You can either pass the name of an existing metric, a metric function, or a layer (see...)

"pass a layer" is not sound advice in the general case.

For now I would not be comfortable covering usage of stateful metrics in the public docs. This is an API that is still experimental, and subject to changes in the future (e.g. this PR changes it). We'll document it when it is more mature.

Please revert the edits on this page. You can keep them around and we may include them at a future date.

Makes sense! Will do.

fchollet · 2018-02-22T19:49:37Z

docs/templates/metrics.md

@@ -19,7 +19,7 @@ model.compile(loss='mean_squared_error',

 A metric function is similar to a [loss function](/losses), except that the results from evaluating a metric are not used when training the model.

-You can either pass the name of an existing metric, or pass a Theano/TensorFlow symbolic function (see [Custom metrics](#custom-metrics)).
+You can either pass the name of an existing metric, or pass a function or layer (see [Custom metrics](#custom-metrics)).


"pass a layer" is not sound advice in the general case.

For now I would not be comfortable covering usage of stateful metrics in the public docs. This is an API that is still experimental, and subject to changes in the future (e.g. this PR changes it). We'll document it when it is more mature.

Please revert the edits on this page. You can keep them around and we may include them at a future date.

fchollet · 2018-02-22T19:51:56Z

keras/utils/generic_utils.py

@@ -337,7 +337,7 @@ def update(self, current, values=None):
                    self._values[k][0] += v * (current - self._seen_so_far)
                    self._values[k][1] += (current - self._seen_so_far)
            else:
-                self._values[k] = v
+                self._values[k] = [v, 1]


To distinguish a numerical value (this means taking an average from a single-item list, i.e. just keeping the numerical value the same - I'll add a comment) from a non-numerical log value that should be copied verbatim to output. This way, the floats are properly formatted the same as other numerical metrics.

fchollet · 2018-02-22T19:52:23Z

keras/engine/training.py

@@ -2404,13 +2416,18 @@ def evaluate_generator(self, generator, steps=None,
                enqueuer.stop()

        if not isinstance(outs, list):
+            assert not stateful_metric_indices


Why? Please add a comment.

In case of loss-only models, obviously no stateful_metric_indices may exist.

I have now instead refactored the code to be consistent with all other Model methods in handling of loss-only models.

fchollet · 2018-02-22T19:53:02Z

keras/engine/training.py

-                averages.append(np.average([out[i] for out in all_outs],
-                                           weights=batch_sizes))
+                # [0] is the loss, the rest are metrics
+                if i == 0 or (i - 1) not in stateful_metric_indices:


Unclear what the underlying logic is... either implement in a simpler/clearer way (preferred), or comment clearly.

Oh - now I understand why other methods use Model.metrics_names rather than Model.metrics to determine the indices. Rewritten.

fchollet · 2018-02-22T19:56:48Z

Require stateful metrics layers to be actually stateful

This is an ok change.

Dref360 · 2018-02-24T02:56:20Z

tests/keras/metrics_test.py

            name: String, name for the metric.
        """

        def __init__(self, name='true_positives', **kwargs):
            super(BinaryTruePositives, self).__init__(name=name, **kwargs)
+            self.stateful = True


Would we have to specify this attribute for each metric?

That's the idea, indeed - as long as your metric behaves as a stateful layer.

briannemsick · 2018-03-07T17:49:20Z

keras/engine/training.py

@@ -2331,6 +2334,15 @@ def evaluate_generator(self, generator, steps=None,
        """
        self._make_test_function()

+        stateful_metric_indices = []


This reset is needed to make Stateful Metrics work for generators.

How do you feel about spinning this bug fix out into a separate PR? Should be a quick approval.

Some of the other changes, for instance m.stateful will likely have some discussion. I really want this bug fix to make it into the next release :)

Do you mind if I raise the reset in evaluate generator to get it through?

Don't want to steal your thunder.

No worries about that! But I'm going to take better care of the MR from now on and update it in a couple of minutes, sorry about the delays!

Looks like you got it, I just wanted to make sure this PR wasn't going to get stuck for another couple of releases.

Use metrics_names, rather than metrics + index juggling to skip loss. Make loss-only output handling consistent with other Model methods. all_outs -> outs_per_batch to avoid confusion, all_outs has swapped dimensions in predict_generator().

pasky · 2018-03-13T20:30:03Z

I have updated the MR based on the review.

grassking100

LGTM

fchollet

LGTM, thanks

…ack-embeddings-from-layer-outputs * upstream/master: (68 commits) fit/evaluate_generator supporting native tensors (keras-team#9816) keras-team#9642 Add kwarg and documentation for dilation_rate to SeparableConvs (keras-team#9844) Document that "same" is inconsistent across backends with strides!=1 (keras-team#9629) Improve tests by designating dtype of sample data (keras-team#9834) Add documentation for 'subset' and interpolation' arguments (ImageDataGenerator) (keras-team#9817) Revert default theme to readthedocs Various docs fixes. Fix conflict Add support for class methods documentation (keras-team#9751) Add missing verbose opt for evaluate_generator (keras-team#9811) Added `data_format` to flatten layer. (keras-team#9696) Allow saving models directly to binary stream (keras-team#9789) Fix ctc_batch_cost() error when batch_size = 1 (keras-team#9775) Fix keras-team#9802 (keras-team#9803) Fix error in ImageDataGenerator documentation (keras-team#9798) fix typo (keras-team#9792) keras-team#9733: Extend RemoteMonitor to send data as application/json (keras-team#9734) Fixed inconsistencies regarding ReduceLROnPlateau (keras-team#9723) Fix doc issue. General stateful metrics fixes (keras-team#9446) ...

* Require stateful metrics layers to be actually stateful * Prevent stateful metrics to leak np.floats to the History object * Progbar: Format stateful metrics values as floats alike other metrics * test_stateful_metrics: Also test validation set evaluation This makes sure the metric is reset before evaluating valset. * Add support for stateful metrics in fit_generator() and evaluate_generator() * Document stateful metrics It would be even better to have full-fledged stateful layers documentations, but I lack the knowledge and experience to explain that well. * evaluate_generator(): Do not leak np.float to History here either * Revert stateful metrics documentation until the API stabilizes * Progbar: Explain stateful metrics handling * Model.evaluate_generator(): More consistent stateful metrics handling Use metrics_names, rather than metrics + index juggling to skip loss. Make loss-only output handling consistent with other Model methods. all_outs -> outs_per_batch to avoid confusion, all_outs has swapped dimensions in predict_generator().

arquolo · 2018-07-02T09:01:46Z

keras/engine/training.py

@@ -2427,7 +2427,7 @@ def evaluate_generator(self, generator, steps=None,
                    averages.append(np.average([out[i] for out in all_outs],
                                               weights=batch_sizes))
                else:
-                    averages.append(all_outs[-1][i])
+                    averages.append(float(all_outs[-1][i]))


Why float (not np.float64)?
When TensorBoard callback is used with stateful metrics it raise

File "...\keras\callbacks.py", line 942, in on_epoch_end summary_value.simple_value = value.item() AttributeError: 'float' object has no attribute 'item'

np.float64 works correctly due to existence of item() class method.

pasky added 7 commits February 21, 2018 17:44

Require stateful metrics layers to be actually stateful

15d36ff

Prevent stateful metrics to leak np.floats to the History object

27e39cc

Progbar: Format stateful metrics values as floats alike other metrics

2e30a38

test_stateful_metrics: Also test validation set evaluation

7f4addc

This makes sure the metric is reset before evaluating valset.

Add support for stateful metrics in fit_generator() and evaluate_gene…

9fc9a2e

…rator()

Document stateful metrics

de23bd5

It would be even better to have full-fledged stateful layers documentations, but I lack the knowledge and experience to explain that well.

evaluate_generator(): Do not leak np.float to History here either

ff2a790

briannemsick reviewed Feb 22, 2018

View reviewed changes

fchollet reviewed Feb 22, 2018

View reviewed changes

Dref360 reviewed Feb 24, 2018

View reviewed changes

briannemsick reviewed Mar 7, 2018

View reviewed changes

pasky added 3 commits March 13, 2018 19:48

Revert stateful metrics documentation until the API stabilizes

9b27400

Progbar: Explain stateful metrics handling

2b91891

fchollet added the Reviewers wanted label Mar 14, 2018

grassking100 approved these changes Mar 22, 2018

View reviewed changes

fchollet approved these changes Mar 22, 2018

View reviewed changes

fchollet merged commit 2f4685c into keras-team:master Mar 22, 2018

fchollet mentioned this pull request Mar 22, 2018

Reset old metric history and keep in line with fit #9730

Closed

This was referenced Mar 31, 2018

evaluate_generator need to support stateful_metrics #9738

Closed

Why fit is not merged into fit_generator since the latter is more generalized? #9667

Closed

nisargjhaveri mentioned this pull request Apr 12, 2018

Fix stateful metrics when passing dict to compile #9894

Merged

brge17 mentioned this pull request May 9, 2018

[tf.keras] Bug with Stateful Metrics & Fit Generator tensorflow/tensorflow#19186

Closed

arquolo reviewed Jul 2, 2018

View reviewed changes

rak5216 mentioned this pull request Jan 20, 2020

Pull in branch - Rak/resolve issue #41 per class metrics and #47 global metrics mit-quest/necstlab-damage-segmentation#46

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

General stateful metrics fixes #9446

General stateful metrics fixes #9446

pasky commented Feb 21, 2018

briannemsick Feb 22, 2018 •

edited

Loading

fchollet Feb 22, 2018

pasky Mar 13, 2018

fchollet Feb 22, 2018

fchollet Feb 22, 2018

pasky Mar 13, 2018

fchollet Feb 22, 2018

pasky Mar 13, 2018

fchollet Feb 22, 2018

pasky Mar 13, 2018

fchollet commented Feb 22, 2018

Dref360 Feb 24, 2018

pasky Mar 13, 2018

briannemsick Mar 7, 2018

briannemsick Mar 13, 2018

pasky Mar 13, 2018

briannemsick Mar 13, 2018

pasky commented Mar 13, 2018

grassking100 left a comment

fchollet left a comment

arquolo Jul 2, 2018

General stateful metrics fixes #9446

General stateful metrics fixes #9446

Conversation

pasky commented Feb 21, 2018

briannemsick Feb 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fchollet commented Feb 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pasky commented Mar 13, 2018

grassking100 left a comment

Choose a reason for hiding this comment

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

briannemsick Feb 22, 2018 •

edited

Loading