-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Utility function for computing EWE #102
Comments
Yes, it's definitely a good idea to have a utility function for EWE. But maybe it would better fit into audmetric? |
Hm, it doesn't fit the standard API there ( On the other hand, it's very specific to emotion recognition (or tasks where the labels come from subjective ratings) so it doesn't fit well into So I'm not sure..I'd still vote for |
@ChristianGeng started working on a package that has annotation related functions as well, e.g. compute rater agreement. So maybe it would make sense to add the |
I was hoping for a public implementation :-) |
Sounds good! |
Maybe the new package could also be made public? |
We will not add this functionality to |
The topic of computing the EWE has come up several times. As its computation is straightforward but not trivial, I would offer to add a utility function here to compute it, then a user can automatically compute it for their dataset and easily add it to a conversion script.
The approach I have in mind is to add two functions, one to compute annotator confidence, and the other to compute the EWE. Those will roughly look as follows:
@hagenw @frankenjoe what do you think about this?
The text was updated successfully, but these errors were encountered: