Add masks to boundaries #7704

bhack · 2023-06-27T21:25:55Z

How do you would impl the test against:
https://github.com/bowenc0221/boundary-iou-api/blob/master/boundary_iou/utils/boundary_utils.py#L12-L30

I suppose we don't want to add python OpenCV as a test dependecy.

pytorch-bot · 2023-06-27T21:25:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7704

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2023-06-27T21:26:00Z

Hi @bhack!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

facebook-github-bot · 2023-06-27T22:58:33Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

NicolasHug

Thanks for the PR @bhack , I'll take a deeper look later.

How do you would impl the test against

Yeah, we can't have openCV on the tests suite. Maybe we can create a custom tests where we draw simple masks e.g. circles or squares, fill them in, and then assert in the test that the output of masks_to_boundaries corresponds to the contour shape?

NicolasHug · 2023-06-28T10:20:59Z

torchvision/ops/boxes.py

@@ -382,7 +382,39 @@ def _box_diou_iou(boxes1: Tensor, boxes2: Tensor, eps: float = 1e-7) -> Tuple[Te
    # distance between boxes' centers squared.
    return iou - (centers_distance_squared / diagonal_distance_squared), iou

+def masks_to_boundaries(masks: torch.Tensor, dilation_ratio: float = 0.02) -> torch.Tensor:


I guess it's OK to have the implementation in this file even though this isn't related to boxed. However, I don't think we should expose it here. I think we should just expose it in from the torchvision.ops namespace (otherwise the implementation will always have to stay in this file for BC, and that may lock us).

We probably just need to rename this to _masks_to_boundaries and the expose it in torchvision.ops.__init__.py like

from .boxes import import _masks_to_boundaries as masks_to_boundaries

Any other suggestion @pmeier @vfdev-5 @oke-aditya ?

I guess it's OK to have the implementation in this file even though this isn't related to boxed.

No strong opinion, but could we maybe also have a new _masks.py module or move it into the misc.py one?

👍 for only exposing it in the torchvision.ops namespace.

Tbh there is demand for mask_utils. Several of them, #4415 . Candidate utils like convert_masks_format, paste_masks_in_images, etc. Maybe it's time to create new files mask_utils.py and make future extensions possible?

we can always create an ops.mask* namespace at any time. We should only do that when we know for sure we need it, i.e. when we start having 2+ mask utils. Alls ops are exposed in the ops. namespace anyway so there's no need to rush and create a file which will only have one single util in it ATM.

I'm OK with creating _mask.py as well (and we can rename it into mask.py later if we want to).

I'm OK with creating _mask.py as well (and we can rename it into mask.py later if we want to).

This sounds best solution! We can avoid the bloat inside this file as well as keep them private 😄

vfdev-5 · 2023-06-28T11:21:11Z

torchvision/ops/boxes.py

+    n, h, w = masks.shape
+    img_diag = math.sqrt(h ** 2 + w ** 2)
+    dilation = int(round(dilation_ratio * img_diag))
+    selem_size = dilation * 2 + 1
+    selem = torch.ones((n, 1, selem_size, selem_size), device=masks.device)

+    # Compute the boundaries for each mask
+    masks = masks.float().unsqueeze(1)
+    eroded_masks = F.conv2d(masks, selem, padding=dilation, groups=n)
+    eroded_masks = (eroded_masks == selem.view(n, -1).sum(1, keepdim=True)).byte()  # Make the output binary
+
+    contours = masks.byte() - eroded_masks
+
+    return contours.squeeze(1)


I do not think this code works as expected. Here is my test example and it fails in multiple places:

import torch import numpy as np from PIL import ImageDraw, Image mask = torch.zeros(4, 32, 32, dtype=torch.bool) mask[0, 1:10, 1:10] = True mask[0, 12:20, 12:20] = True mask[0, 15:18, 20:32] = True mask[1, 15:23, 15:23] = True mask[1, 22:33, 22:33] = True mask[2, 1:5, 22:30] = True mask[2, 5:14, 25:27] = True pil_img = Image.new("L", (32, 32)) draw = ImageDraw.Draw(pil_img) draw.ellipse([2, 7, 26, 26], fill=1, outline=1, width=1) mask[3, ...] = torch.from_numpy(np.asarray(pil_img)) import math from torch.nn import functional as F dilation_ratio = 0.05 masks = mask.clone() n, h, w = masks.shape img_diag = math.sqrt(h ** 2 + w ** 2) dilation = int(round(dilation_ratio * img_diag)) selem_size = dilation * 2 + 1 selem = torch.ones((n, 1, selem_size, selem_size), device=masks.device) # Compute the boundaries for each mask masks = masks.float().unsqueeze(1) eroded_masks = F.conv2d(masks, selem, padding=dilation, groups=n) eroded_masks = (eroded_masks == selem.view(n, -1).sum(1, keepdim=True)).byte() # Make the output binary contours = masks.byte() - eroded_masks contours. = contours.squeeze(1)

Error:

---> 17 eroded_masks = (eroded_masks == selem.view(n, -1).sum(1, keepdim=True)).byte() # Make the output binary RuntimeError: The size of tensor a (32) must match the size of tensor b (4) at non-singleton dimension 2

Masks:

Error is related to masks = masks.float().unsqueeze(1) where we may need to unsqueeze(0) instead.
But if fixed like that, the next line does not make much sense IMO:

eroded_masks = (eroded_masks == selem.view(n, -1).sum(1, keepdim=True)).byte()

as eroded_masks shape wont match the size of conv weights...

Sorry, if I'm missing something...

What do you think about:

import torch import numpy as np from PIL import ImageDraw, Image import math from torch.nn import functional as F import matplotlib.pyplot as plt mask = torch.zeros(4, 32, 32, dtype=torch.bool) mask[0, 1:10, 1:10] = True mask[0, 12:20, 12:20] = True mask[0, 15:18, 20:32] = True mask[1, 15:23, 15:23] = True mask[1, 22:33, 22:33] = True mask[2, 1:5, 22:30] = True mask[2, 5:14, 25:27] = True pil_img = Image.new("L", (32, 32)) draw = ImageDraw.Draw(pil_img) draw.ellipse([2, 7, 26, 26], fill=1, outline=1, width=1) mask[3, ...] = torch.from_numpy(np.asarray(pil_img)) dilation_ratio = 0.05 masks = mask.clone() n, h, w = masks.shape img_diag = math.sqrt(h ** 2 + w ** 2) dilation = int(round(dilation_ratio * img_diag)) selem_size = dilation * 2 + 1 selem = torch.ones((1, 1, selem_size, selem_size), device=masks.device) # Compute the boundaries for each mask masks = masks.float().unsqueeze(1) eroded_masks = torch.zeros_like(masks) #for i in range(n): # eroded_masks[i] = F.conv2d(masks[i].unsqueeze(0), selem, padding=dilation) eroded_masks = F.conv2d(masks, selem, padding=dilation) eroded_masks = (eroded_masks == selem.view(-1).sum()).byte() # Make the output binary contours = masks.byte() - eroded_masks contours = contours.squeeze(1) # Visualize the results fig, ax = plt.subplots(n, 3, figsize=(10, 10)) for i in range(n): ax[i, 0].imshow(mask[i], cmap='gray') ax[i, 1].imshow(eroded_masks[i].squeeze(), cmap='gray') ax[i, 2].imshow(contours[i], cmap='gray') plt.show()

@bhack why do we need dilation_ratio ? I think we can do the following without extra parametrization:

masks = masks.float().unsqueeze(1) w_size = 3 w = torch.ones((1, 1, w_size, w_size), device=masks.device) / (w_size ** 2) eroded_masks = F.conv2d(masks, w, padding=1) contours = (masks - eroded_masks) > 0 contours = contours.squeeze(1)

what do you think ?

It is in the paper official implementation

https://github.com/bowenc0221/boundary-iou-api/blob/master/boundary_iou/utils/boundary_utils.py#L12

But also in the more classical F score (Davis dataset/challenge official eval kit).

https://github.com/davisvideochallenge/davis2017-evaluation/blob/master/davis2017/metrics.py#L57

As this is often a preprocessing step used in the boundary overlapping metrics (BoundaryIOU/Boundary F-Score) the dilate will give the control over the tolerance of the exact boundaries overlapping of the boundaries.

In both the papers they talked about bipartite graph matching but then they have always approximated with morphological ops.

Of you see the F/Davis case impl there is also an option where the tolerance/dilate Is defined by the input resolution.

Thanks for the links. According to https://github.com/davisvideochallenge/davis2017-evaluation/blob/master/davis2017/metrics.py#L57 code, mask to boundary is done without using any parameters, see _seg2bmap:
https://github.com/davisvideochallenge/davis2017-evaluation/blob/ac7c43fca936f9722837b7fbd337d284ba37004b/davis2017/metrics.py#L122
Anyway, I see why they have dilation_ratio arg.

However, previously I missed the issue description and the context for this PR:

A mask to boundary API is useful for implementing many segmentation metrics used in many dataset and challenges (Davis F score, BoundaryIOU, etc..).
It could be also used more generally for visualization tasks.

In this case, I'm not very sure about torchvision's interest in following line by line what does https://github.com/bowenc0221/boundary-iou-api as 1) IMO we wont be able to reproduce cv2.erode behaviour and 2) as such helper function can be used within a metric implementation, it should be carefully tested vs ref implementation in a lot of corner cases etc (and this is not the role of torchvision, IMO).

In general, a method to produce mask to edges (sort of edge detector) could make sense like mask to bboxes.

Thanks for the links. According to https://github.com/davisvideochallenge/davis2017-evaluation/blob/master/davis2017/metrics.py#L57 code, mask to boundary is done without using any parameters, see _seg2bmap:
https://github.com/davisvideochallenge/davis2017-evaluation/blob/ac7c43fca936f9722837b7fbd337d284ba37004b/davis2017/metrics.py#L122

Yes but cause in F they are dilating in an extra post-processing step in the metric instead of the BoundariesIOU approach (see dilate disk param)
https://github.com/davisvideochallenge/davis2017-evaluation/blob/master/davis2017/metrics.py#L77

In this case, I'm not very sure about torchvision's interest in following line by line what does https://github.com/bowenc0221/boundary-iou-api as 1) IMO we wont be able to reproduce cv2.erode behaviour and 2) as such helper function can be used within a metric implementation, it should be carefully tested vs ref implementation in a lot of corner cases etc (and this is not the role of torchvision, IMO).

I've tested another early implementation with some inputs but the Boundary IOU paper reference impl doesn't have a test suite.

In general, a method to produce mask to edges (sort of edge detector) could make sense like mask to bboxes.

Let me know as I am mainly interested to achieve the metric and eventually to contribute also an intermediate function here in the case it could be compatible and useful for other contexts/domain.

I see you are also a member of the MONAI project so you have already something similar but it still rely on a non-Pytorch implementation:
https://github.com/Project-MONAI/MetricsReloaded/blob/main/MetricsReloaded/metrics/pairwise_measures.py#L963

Add dummy test

bhack · 2023-07-01T13:08:41Z

import torch
import numpy as np
from PIL import ImageDraw, Image
import math
from torch.nn import functional as F
import matplotlib.pyplot as plt

# Create masks
mask = torch.zeros(4, 32, 32, dtype=torch.bool)
mask[0, 1:10, 1:10] = True
mask[0, 12:20, 12:20] = True
mask[0, 15:18, 20:32] = True
mask[1, 15:23, 15:23] = True
mask[1, 22:33, 22:33] = True
mask[2, 1:5, 22:30] = True
mask[2, 5:14, 25:27] = True
pil_img = Image.new("L", (32, 32))
draw = ImageDraw.Draw(pil_img)
draw.ellipse([2, 7, 26, 26], fill=1, outline=1, width=1)
mask[3, ...] = torch.from_numpy(np.asarray(pil_img))

# Define dilation_ratio
dilation_ratio = 0.02

# Clone masks
masks = mask.clone()

# Get the dimensions
n, h, w = masks.shape

# Compute img_diag, dilation, selem_size and selem
img_diag = math.sqrt(h ** 2 + w ** 2)
dilation = int(round(dilation_ratio * img_diag))
selem_size = dilation * 2 + 1
selem = torch.ones((n, 1, selem_size, selem_size), device=masks.device)

# Compute the boundaries for each mask
masks = masks.float().unsqueeze(1)
eroded_masks = F.conv2d(masks, selem, padding=dilation)
eroded_masks = (eroded_masks == selem.view(n, -1).sum(-1).view(n, 1, 1, 1)).byte()  # Make the output binary

contours = masks.byte() - eroded_masks

# Squeeze the contours tensor
contours = contours.squeeze(1)

# Visualize the results
fig, ax = plt.subplots(n, 3, figsize=(10, 10))
for i in range(n):
    ax[i, 0].imshow(mask[i], cmap='gray')
    ax[i, 1].imshow(eroded_masks[i, 0].cpu(), cmap='gray')
    ax[i, 2].imshow(contours[i, 0].cpu(), cmap='gray')

plt.show()

test/test_ops.py

bhack · 2023-07-11T10:43:58Z

docs/source/ops.rst

@@ -22,6 +22,7 @@ The below operators perform pre-processing as well as post-processing required i

    batched_nms
    masks_to_boxes
+    masks_to_boudnaries


Fixed. So what we want to do?

bhack · 2023-11-13T08:51:40Z

Any news on this? Are you still interested?

bhack · 2024-02-15T00:48:01Z

Gently ping

vfdev-5

Thanks for the updates, but the implementation still has some problems. I left comments in the code.

torchvision/ops/boxes.py

Refactor test and add debug image util Refactor implementation

bhack · 2024-03-05T14:16:11Z

@NicolasHug Gently ping.

bhack · 2024-04-29T17:07:16Z

Let me know if we want to close this as we are at the 10th month.

bhack · 2024-09-15T16:50:38Z

Ping again, we are over 1 year.

Add masks to boundaries

79dcbb1

facebook-github-bot added the cla signed label Jun 27, 2023

NicolasHug reviewed Jun 28, 2023

View reviewed changes

vfdev-5 reviewed Jun 28, 2023

View reviewed changes

bhack added 4 commits June 28, 2023 13:24

Doesn't expose directly the def

d171ffd

change erosion

9d41c0a

Add dummy test

e277308

Merge pull request #1 from bhack/patch-2

330301c

Add dummy test

bhack commented Jul 1, 2023

View reviewed changes

test/test_ops.py Outdated Show resolved Hide resolved

Update ops.rst

a8bd95c

bhack commented Jul 11, 2023

View reviewed changes

Merge branch 'main' into patch-1

7311956

bhack requested review from vfdev-5, oke-aditya, pmeier and NicolasHug December 30, 2023 12:49

bhack marked this pull request as ready for review December 30, 2023 12:51

Merge branch 'main' into patch-1

08485c0

vfdev-5 reviewed Feb 16, 2024

View reviewed changes

torchvision/ops/boxes.py Outdated Show resolved Hide resolved

torchvision/ops/boxes.py Outdated Show resolved Hide resolved

torchvision/ops/boxes.py Outdated Show resolved Hide resolved

bhack added 2 commits February 17, 2024 01:05

Add debug image option

59fb72c

Refactor test and add debug image util Refactor implementation

Merge branch 'main' into patch-1

091f3fb

bhack requested a review from vfdev-5 February 17, 2024 01:08

bhack added 2 commits March 5, 2024 14:43

Merge branch 'main' into patch-1

fa68881

Merge branch 'main' into patch-1

c2d8074

bhack added 5 commits March 7, 2024 14:23

Merge branch 'main' into patch-1

aa4b2e3

Merge branch 'main' into patch-1

293e436

Merge branch 'main' into patch-1

cf07bc0

Merge branch 'main' into patch-1

7abbc3b

Merge branch 'main' into patch-1

762992f

bhack added 2 commits August 30, 2024 23:58

Merge branch 'main' into patch-1

0991f93

Merge branch 'main' into patch-1

4de4913

bhack added 7 commits October 18, 2024 23:51

Merge branch 'main' into patch-1

91df477

Merge branch 'main' into patch-1

ebee25e

Merge branch 'main' into patch-1

9fc12a9

Merge branch 'main' into patch-1

080fa0d

Merge branch 'main' into patch-1

e526765

Merge branch 'main' into patch-1

1ec78df

Merge branch 'main' into patch-1

78062c0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add masks to boundaries #7704

Add masks to boundaries #7704

bhack commented Jun 27, 2023

pytorch-bot bot commented Jun 27, 2023 •

edited

Loading

facebook-github-bot commented Jun 27, 2023

facebook-github-bot commented Jun 27, 2023

NicolasHug left a comment

NicolasHug Jun 28, 2023

pmeier Jun 28, 2023

oke-aditya Jun 30, 2023

NicolasHug Jun 30, 2023

oke-aditya Jun 30, 2023

vfdev-5 Jun 28, 2023

bhack Jun 28, 2023

vfdev-5 Jun 29, 2023

bhack Jun 29, 2023

vfdev-5 Jun 29, 2023

bhack Jun 29, 2023

bhack Jun 29, 2023 •

edited

Loading

bhack commented Jul 1, 2023

bhack Jul 11, 2023

bhack commented Nov 13, 2023

bhack commented Feb 15, 2024

vfdev-5 left a comment

bhack commented Mar 5, 2024

bhack commented Apr 29, 2024

bhack commented Sep 15, 2024

Add masks to boundaries #7704

Are you sure you want to change the base?

Add masks to boundaries #7704

Conversation

bhack commented Jun 27, 2023

pytorch-bot bot commented Jun 27, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7704

facebook-github-bot commented Jun 27, 2023

Action Required

Process

facebook-github-bot commented Jun 27, 2023

NicolasHug left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhack Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

bhack commented Jul 1, 2023

Choose a reason for hiding this comment

bhack commented Nov 13, 2023

bhack commented Feb 15, 2024

vfdev-5 left a comment

Choose a reason for hiding this comment

bhack commented Mar 5, 2024

bhack commented Apr 29, 2024

bhack commented Sep 15, 2024

pytorch-bot bot commented Jun 27, 2023 •

edited

Loading

bhack Jun 29, 2023 •

edited

Loading