-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Insertion Deletion AUC metric #56
Insertion Deletion AUC metric #56
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #56 +/- ##
===========================================
+ Coverage 92.43% 92.64% +0.21%
===========================================
Files 20 22 +2
Lines 1308 1373 +65
===========================================
+ Hits 1209 1272 +63
- Misses 99 101 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the update. I have a few comments below.
Besides, one alternative design idea came up in my mind. Could you please consider the following idea and my comments?
class Metric(ABC):
def __init__(
self,
score_fn: Callable[[np.ndarray], np.ndarray], # image -> score vector
# metric specific static parameters (like 'step' in AUC)
*args,
**kwargs,
):
...
def __call__( # Single image
self,
saliency_map: np.ndarray,
class_idx: int,
input_image: np.ndarray,
) -> Dict[str, float]:
...
def evaluate( # Dataset
self,
explanations: List[Explanation],
input_images: List[np.ndarray],
) -> Dict[str, float]:
...
To me, setting model, preproc, postproc functions looks too complicated.
In above way, users could set arbitrary inference function that computes score vector from image. Explainer.model_forward()
could also be reused in this way.
Thank you for your suggestions, Songki. I implemented
Provably, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's do the refactoring afterwards. Thank you for the update!
Added Insertion Deletion AUC metric that calculates the accuracy drop/increase during deletion/insertion of important pixels
Added parent BasicMetric