Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

To introduce MetricGroup class #3264

Closed
sadra-barikbin opened this issue Jul 16, 2024 · 1 comment · Fixed by #3266
Closed

To introduce MetricGroup class #3264

sadra-barikbin opened this issue Jul 16, 2024 · 1 comment · Fixed by #3266

Comments

@sadra-barikbin
Copy link
Collaborator

🚀 Feature

Hi there!
It might be useful to group some metrics together to have less code and more coherence. My use case was using some ignite metrics in HuggingFace Trainer API which expects a single callable for computing metrics. Having this feature, that could be easily realizable with Ignite. MetricGroup could be something like this:

from typing import Any, Dict
from ignite.metrics import Metric


class MetricGroup(Metric):
  _state_dict_all_req_keys = ('metrics',)

  def __init__(self, metrics:Dict[str, Metric]):
    self.metrics = metrics
    super(MetricGroup, self).__init__()

  def reset(self):
    for m in self.metrics.values():
      m.reset()

  def update(self, output):
    for m in self.metrics.values():
      m.update(m._output_transform(output))

  def compute(self) -> Dict[str, Any]:
    return {k: m.compute() for k,m in self.metrics.items()}

Usage:

metric = MetricGroup({'acc': Accuracy(), 'perplexity': Perplexity()})
metric.attach(engine) # or sth in HF Trainer API
@vfdev-5
Copy link
Collaborator

vfdev-5 commented Jul 17, 2024

OK, why not adding this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants