[WIP] Support all early stopping options: LambdaEarlyStopping

[carmocca]

What does this PR do?

Fixes #6795

Pitch

inf = torch.tensor(np.Inf)  # our stop fn maximizes

def bounds_fn(score: float, _ = None) -> Tuple[bool, str]:
    return not (10 < score <= 100), "Out of bounds"

def stop_fn(score: float, best: float = inf) -> bool:
    # maximizing - did not improve
    return score < best

# stateful class example
class StdDevUnder:
    def __init__(self, threshold: float, num_values_to_keep: int) -> None:
        assert threshold > 0, "Standard deviation should be a positive value"
        assert num_values_to_keep > 1, (
            "The number of values to keep must be greater than 1 to compute "
            "the standard deviation"
        )
        self._threshold = threshold
        self._num_values_to_keep = num_values_to_keep
        self._values = []
        self._nval = 0

    def __call__(self, score: float, _ = None) -> Union[bool, Tuple[bool, str]]:
        # Add last value to the values
        if self._num_values_to_keep > len(self._values):
            self._values.append(value)
        else:
            self._values[self._nval] = value
            self._nval = (self._nval + 1) % self._num_values_to_keep

        # If not enough values are kept, return False
        if len(self._values) < self._num_values_to_keep:
            return False

        std = torch.std(torch.stack(self._values))
        if std < self._threshold:
            reason = f"Standard deviation {std} < Threshold {self._threshold}"
            return True, reason
        return False

Trainer(callbacks=[
    LambdaEarlyStopping(stop_fn),
    # stop immediately when the target cost is reached or the run failed completely
    LambdaEarlyStopping(bounds_fn, patience=0),
    # stop when the stddev over the last 10 values is lower than a threshold
    LambdaEarlyStopping(StdDevUnder(0.1, 10)),
])

There is no need to pass mode=min|max. The initial best value is provided by the user with the best keyword argument.

The Callable can optionally return a string with the stopping reason. The callable function name is also shown in the "report".

The Callable can be a class callable and keep a state.

Notes

This does not replace the PR in #6868 as we still want to provide that functionality out-of-the-box.

In a future PR

• Refactor EarlyStopping so it subclasses LambdaEarlyStopping to reuse code: The objective of EarlyStopping will be to provide out-of-the-box stopping functions.

TODO

• Code: Opened this PR to allow people to read the pitch
• Docs
• Tests

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

[pep8speaks]

Hello @carmocca! Thanks for opening this PR.

• In the file pytorch_lightning/callbacks/lambda_early_stopping.py:

Line 154:13: W503 line break before binary operator

Do see the Hitchhiker's guide to code style

[Borda]

seem this quite old PR with several handerts of commits behind the master, consider finishing it or closing as most likely the conflicts will make the PR challenging to finish... 🐰