`masks_to_bounding_boxes` op #4290

0x00b1 · 2021-08-18T17:19:46Z

This (draft) pull request resolves #3960. I created a draft to kickoff new contributor on-boarding (e.g. CLA).

I'm working on a gallery example now. I'll also add test against different dtypes.

facebook-github-bot · 2021-08-18T17:19:50Z

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

facebook-github-bot · 2021-08-18T17:46:04Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

oke-aditya · 2021-08-19T03:45:57Z

Sorry for an early poke at this PR.

I think it would be nice to place the test in test_ops.py

I'm not sure of how it should be tested. My initially were we could manually create boolean masks and test them like the ones done in utils.draw_segmentation_masks test?

Also the code can be kept directly in boxes ? No strong opinion here though.

0x00b1 · 2021-08-19T18:35:03Z

Hi, @oke-aditya! I appreciate any and all feedback! 😄 My code organization was my preference but I am more than happy to adopt whatever convention you or any other maintainers prefer.

I will parametrize the dtype in the fixtures but other than that I think the unit test does its job. If you're curious, I used the random_shapes function from skimage.draw to create the fixtures. I was the original author of that function, or the code that would become that function, and its purpose was to solve the exact issue of writing unit tests for object localization methods. Ideally, in the future, if torchvision starts adding more of these types of operators it would be nice to port over that function and similar data generators from scikit-image to simplify writing these types of tests (e.g. extending the functionality to videos and volumes or arbitrary color spaces). It would also allow for some basic fuzzing.

oke-aditya · 2021-08-19T19:22:16Z

if torchvision starts adding more of these types of operators it would be nice to port over that function and similar data generators from scikit-image to simplify writing these types of tests

Not sure if they fall into same category, but there too we hardcoded some of the boxes and masks for images to test them.
Actually we have bunch of tests, from here (these are operators used for plotting boxes, masks) Also a many operators like box_iou, box_area, where the boxes are predefined (but these are mostly mathematical so probably ok). You can have a look at these and I would be glad to hear your thoughts!

My initial thought was that this function too would be tested in similar way instead of drawing random shapes / boxes on an image.

P.S. I'm just a small contributor to the library (also a novice developer) so please don't mind.

NicolasHug

Thanks a lot for the PR @0x00b1 and @oke-aditya for the review. I made a few comments and will look at the rest once this isn't draft anymore. Thanks for the initiative of writing a gallery example!!

test/test_masks_to_bounding_boxes.py

torchvision/ops/_masks_to_bounding_boxes.py

NicolasHug · 2021-08-20T07:18:33Z

test/test_masks_to_bounding_boxes.py

+
+@pytest.fixture
+def masks() -> torch.Tensor:
+    with PIL.Image.open(os.path.join(ASSETS_DIRECTORY, "masks.tiff")) as image:


Do you think it would be possible to write a test without the need for new images and hard-coded coordinates?

Ideally, we could generate random masks and have a super simple version of masks_to_boxes which we could use as the reference implementation?

Yep. I wrote about this elsewhere in the thread. I'd love to add a generator for various outputs similar to the function @goldsborough and I wrote for scikit-image (skimage.draw.random_shapes). However, would you mind if I did this in a follow-up commit?

@NicolasHug a friendly bump

datumbox · 2021-08-26T09:12:24Z

@0x00b1 Just checking that you still plan to complete the PR. Please let me know :)

0x00b1 · 2021-08-31T15:05:54Z

@datumbox Yep! I was on vacation last week (it was lovely) and started on-boarding at Facebook yesterday. I'll finish this today or tomorrow. Thanks, @NicolasHug for the comments!

RylanSchaeffer · 2021-08-31T15:35:19Z

Can you generalize this to 3D images?

datumbox · 2021-08-31T16:15:29Z

@RylanSchaeffer it's definitely worth discussing it on a new issue. I would prefer if we did this on a separate PR to avoid blocking it for longer.

RylanSchaeffer · 2021-08-31T16:21:23Z

@datumbox I have two opinions. On one hand, I agree that not blocking this PR is good. On the other, a half solution means a fix to the complete problem will probably be delayed.

NicolasHug · 2021-08-31T16:44:09Z

a half solution means a fix to the complete problem will probably be delayed

This PR isn't half a solution, it's a complete solution to a complete problem: 2D images.

3D images are a different problem which we'll be happy to tackle at a future time once this PR is merged, as @datumbox suggested

RylanSchaeffer · 2021-08-31T16:49:07Z

This PR isn't half a solution, it's a complete solution to a complete problem: 2D images.

That's a strange way to think about things. Imagine someone submitted a cross entropy loss implementation for a 2D array. By your metric, it's a complete solution to a complete problem i.e. 1-dimensional classification. But look at the cross entropy loss implementation: it works for arbitrary dimensions, not just one, because we shouldn't be limited to a 2D array.

On this topic, the real problem is more general than 2D images. For us, the real problem is: given a N-dimensional segmentation mask, how to convert the mask to N-dimensional bounding boxes? A solution for 2D is a partial solution.

NicolasHug · 2021-08-31T17:04:19Z

@RylanSchaeffer , just because we can generalize a problem doesn't mean that one problem is less "real" or "complete" than the other. 2D masks are a normal use-case that a lot of people have and solving this will be valuable on its own.

When it comes to software development, a merged PR that solves one problem is worth more than an unmerged PR that solves 2 problems.

Again, we'll be happy to consider an extension if the use-case is compelling.

RylanSchaeffer · 2021-08-31T17:11:46Z

Ok issue opened! #4339

datumbox · 2021-08-31T20:01:27Z

@RylanSchaeffer In addition to what Nicolas said, and for full transparency, here are some reasons for why we often choose not to go straight for the most generic/complicated implementation:

We often create bite-sized issues to help on-board new contributors and new members of the team to the code-base. Limiting the scope can help keep the work manageable.
We might have urgent need to cover a specific limited use-case or there is a time constraint to release a feature.
We might be not certain about some technical parts of the generic implementation and require additional discussions.

I think it's worth continuing the discussion of how this can be made generic on the new issue that you opened.

@0x00b1 Welcome back. Sounds great, if you have any issues with the CI let us know and we can help.

0x00b1 · 2021-08-31T21:07:51Z

@RylanSchaeffer I agree that this should come in a future PR. But don't worry, 3D or n-D images are something I care about too so I'm more than happy to do that work.

Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com>

oke-aditya · 2021-09-20T19:35:47Z

gallery/plot_repurposing_annotations.py

+Repurposing annotations
+=======================
+
+The following example illustrates the operations available in :ref:`the torchvision.ops module <ops>` for repurposing


After some debugging I found out the reason for build_docs CI failure. The problem is torchvision.ops does not have a nice index on right side (basically a html link to #ops like transforms has). This causes CI failure.

We need to remove the ref, and it will work fine. This is slightly hacky fix, but works fine.
I tried running it locally. I could build the gallery example. It looks nice.

Suggested change

The following example illustrates the operations available in :ref:`the torchvision.ops module <ops>` for repurposing

The following example illustrates the operations available in the torchvision.ops module for repurposing

Nice! I appreciate the debugging.

oke-aditya

Hey Allen you need to add docs to docs/ops.rst where you can use.

.. autofunction:: masks_to_boxes

This will add docs for this code.

datumbox

@0x00b1 sorry for the back and forth. Adding an operator is possibly one of the most complex things as one needs to add many things across many files. I think we are almost there to merge. Let me summarize the comments that I think remain unresolved:

Address the docs failure as described here: https://github.com/pytorch/vision/pull/4290/files#r712457642
We missed one use of numpy vs torch. Just copy paste what you got on the examples and we should be good to go. https://github.com/pytorch/vision/pull/4290/files#r712884767
Add the masks_to_boxes in docs as described here: masks_to_bounding_boxes op #4290 (review)

0x00b1 · 2021-09-21T17:19:51Z

Hey Allen you need to add docs to docs/ops.rst where you can use.
.. autofunction:: masks_to_boxes
This will add docs for this code.

Nice catch! Fixed.

0x00b1 · 2021-09-21T17:22:36Z

@0x00b1 sorry for the back and forth. Adding an operator is possibly one of the most complex things as one needs to add many things across many files. I think we are almost there to merge. Let me summarize the comments that I think remain unresolved:

Address the docs failure as described here: https://github.com/pytorch/vision/pull/4290/files#r712457642

We missed one use of numpy vs torch. Just copy paste what you got on the examples and we should be good to go. https://github.com/pytorch/vision/pull/4290/files#r712884767

Add the masks_to_boxes in docs as described here: #4290 (review)

It's no problem dude! I sincerely appreciate your and @oke-aditya's patience! In the future, it might be worth investigating whether someone should add a cookiecutter or cookiecutter-like method for generating op scaffolding.

0x00b1 · 2021-09-21T17:26:52Z

@datumbox OK. Everything has been addressed. Hopefully we don't see any CI failures!

I would also be more than happy to squash these commits down.

torchvision/ops/boxes.py

datumbox

LGTM, thanks a lot @0x00b1. Congrats on your first contribution. :)

NicolasHug · 2021-09-22T07:51:21Z

Hi @0x00b1 ,

Thank you for the great work on this PR!
I'm sorry I wasn't able to make a last pass, I think I missed the point where this PR got un-drafted #4290 (review).

I only have 2 remaining comments at this point:

would it be possible to not rely on a hard-coded image for the tests, as suggested in masks_to_bounding_boxes op #4290 (comment) (sorry again I missed your ping)? It would help make the tests more robust, and also avoid storing files in the repo which can end up bloated. The PR was merged already so it will be included anyway I guess, but this can still help when we make shallow clones of the repo.
Would it be possible to use the draw_segmentation_masks and draw_bounding_box utilities in the example, as suggested by @oke-aditya in masks_to_bounding_boxes op #4290 (comment)? It would likely simplify the example and trim it down to its essential part: the new masks_to_boxes operator, instead of having lots of plotting code. It would also help users discover these plotting tools that they might come useful in other scenarios.

Would you or @oke-aditya be interested in a follow-up PR with these? The first point might be a bit trickier, but the second one should be reasonably simple. We can do them in separate PRs. Thanks!

oke-aditya · 2021-09-22T07:53:37Z

I'm fine with either. I will leave choice to @0x00b1

oke-aditya · 2021-09-23T05:27:11Z

Also another thought about the gallery example.
Another example that can be added is to show how a simple Segmentation dataset can be rewritten to detection dataset.
As pointed out in #3960. This is a common use of masks_to_boxes.

Adding this example will help users to convert PenFudan / Panopatic datasets easily to detection.
This might look simple, but let's keep an example to help users.

from torchvision.ops import masks_to_boxes, box_convert

class SegmentationToDetectionDataset(Dataset):
    def __getitem__(self, idx):
          boxes_xyxy = masks_to_boxes(segmentation_masks)

         # Now for any change of boxes to COCO Format.
          boxes_xywh = box_convert(boxes_xyxy, in_fmt="xyxy", out_fmt="xywh")
          return boxes_xywh

oke-aditya · 2021-09-23T05:31:25Z

torchvision/ops/boxes.py

+
+    n = masks.shape[0]
+
+    bounding_boxes = torch.zeros((n, 4), device=masks.device, dtype=torch.int)


My initial thought was dtype should be torch.float. Since all other ops follow float dtype.

cc @datumbox @NicolasHug

Agreed, also the above zeros needs to have a device:
torch.zeros((0, 4), device=masks.device)

Could you please send a PR that fixes these 2 issues? The rest of the doc/test improvements discussed here can happen on a separate PR.

Summary: * ops.masks_to_bounding_boxes * test fixtures * unit test * ignore lint e201 and e202 for in-lined matrix * ignore e121 and e241 linting rules for in-lined matrix * draft gallery example text * removed type annotations from pytest fixtures * inlined fixture * renamed masks_to_bounding_boxes to masks_to_boxes * reformat inline array * import cleanup * moved masks_to_boxes into boxes module * docstring cleanup * updated docstring * fix formatting issue * gallery example * use torch * use torch * use torch * use torch * updated docs and test * cleanup * updated import * use torch * Update gallery/plot_repurposing_annotations.py * Update gallery/plot_repurposing_annotations.py * Update gallery/plot_repurposing_annotations.py * Autodoc * use torch instead of numpy in tests * fix build_docs failure * Closing quotes. Reviewed By: datumbox Differential Revision: D31268025 fbshipit-source-id: 65f88779516ff0a411600a25b783f00369d56719 Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com> Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com> Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com> Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com> Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com>

0x00b1 added 3 commits August 17, 2021 15:38

ops.masks_to_bounding_boxes

cf51379

test fixtures

c67e035

unit test

3830dd1

0x00b1 added 3 commits August 18, 2021 13:23

Merge branch 'master' into issues/3960

926d444

ignore lint e201 and e202 for in-lined matrix

f777416

ignore e121 and e241 linting rules for in-lined matrix

cd46aa7

facebook-github-bot added the cla signed label Aug 18, 2021

draft gallery example text

712131e

NicolasHug reviewed Aug 20, 2021

View reviewed changes

0x00b1 added 3 commits August 31, 2021 16:42

removed type annotations from pytest fixtures

b6f5c42

inlined fixture

b555c68

renamed masks_to_bounding_boxes to masks_to_boxes

fc26f3a

0x00b1 added 2 commits August 31, 2021 18:35

reformat inline array

c4d3045

import cleanup

4589951

0x00b1 and others added 4 commits September 20, 2021 14:56

Update gallery/plot_repurposing_annotations.py

140e429

Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com>

Update gallery/plot_repurposing_annotations.py

8f2cd4a

Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com>

Update gallery/plot_repurposing_annotations.py

7252723

Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com>

Merge branch 'main' into issues/3960

26f68af

oke-aditya reviewed Sep 20, 2021

View reviewed changes

datumbox reviewed Sep 21, 2021

View reviewed changes

Autodoc

2c2d5dd

0x00b1 added 3 commits September 21, 2021 13:23

use torch instead of numpy in tests

3a91957

fix build_docs failure

e24805c

Merge branch 'main' into issues/3960

65404e9

datumbox reviewed Sep 21, 2021

View reviewed changes

torchvision/ops/boxes.py Outdated Show resolved Hide resolved

Closing quotes.

6c89be7

0x00b1 marked this pull request as ready for review September 21, 2021 18:42

datumbox approved these changes Sep 21, 2021

View reviewed changes

Merge branch 'main' into issues/3960

b2a907c

datumbox merged commit f0422e7 into pytorch:main Sep 21, 2021

0x00b1 deleted the issues/3960 branch September 21, 2021 23:15

oke-aditya reviewed Sep 23, 2021

View reviewed changes

oke-aditya mentioned this pull request Sep 23, 2021

Rewrite test and fix masks_to_boxes implementation #4469

Merged

NicolasHug added module: ops new feature labels Sep 24, 2021

oke-aditya mentioned this pull request Sep 27, 2021

[feature request] [discussion] mask utils in core #4415

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`masks_to_bounding_boxes` op #4290

`masks_to_bounding_boxes` op #4290

0x00b1 commented Aug 18, 2021 •

edited

Loading

facebook-github-bot commented Aug 18, 2021

facebook-github-bot commented Aug 18, 2021

oke-aditya commented Aug 19, 2021

0x00b1 commented Aug 19, 2021 •

edited

Loading

oke-aditya commented Aug 19, 2021

NicolasHug left a comment

NicolasHug Aug 20, 2021

0x00b1 Sep 1, 2021

0x00b1 Sep 15, 2021

datumbox commented Aug 26, 2021

0x00b1 commented Aug 31, 2021 •

edited

Loading

RylanSchaeffer commented Aug 31, 2021

datumbox commented Aug 31, 2021

RylanSchaeffer commented Aug 31, 2021

NicolasHug commented Aug 31, 2021

RylanSchaeffer commented Aug 31, 2021 •

edited

Loading

NicolasHug commented Aug 31, 2021

RylanSchaeffer commented Aug 31, 2021

datumbox commented Aug 31, 2021

0x00b1 commented Aug 31, 2021

oke-aditya Sep 20, 2021 •

edited

Loading

0x00b1 Sep 21, 2021

oke-aditya left a comment •

edited

Loading

datumbox left a comment

0x00b1 commented Sep 21, 2021

0x00b1 commented Sep 21, 2021

0x00b1 commented Sep 21, 2021 •

edited

Loading

datumbox left a comment

NicolasHug commented Sep 22, 2021 •

edited

Loading

oke-aditya commented Sep 22, 2021

oke-aditya commented Sep 23, 2021

oke-aditya Sep 23, 2021 •

edited

Loading

datumbox Sep 23, 2021

	The following example illustrates the operations available in :ref:`the torchvision.ops module <ops>` for repurposing
	The following example illustrates the operations available in the torchvision.ops module for repurposing


		n = masks.shape[0]

		bounding_boxes = torch.zeros((n, 4), device=masks.device, dtype=torch.int)

masks_to_bounding_boxes op #4290

masks_to_bounding_boxes op #4290

Conversation

0x00b1 commented Aug 18, 2021 • edited Loading

facebook-github-bot commented Aug 18, 2021

Action Required

Process

facebook-github-bot commented Aug 18, 2021

oke-aditya commented Aug 19, 2021

0x00b1 commented Aug 19, 2021 • edited Loading

oke-aditya commented Aug 19, 2021

NicolasHug left a comment

Choose a reason for hiding this comment

NicolasHug Aug 20, 2021

Choose a reason for hiding this comment

0x00b1 Sep 1, 2021

Choose a reason for hiding this comment

0x00b1 Sep 15, 2021

Choose a reason for hiding this comment

datumbox commented Aug 26, 2021

0x00b1 commented Aug 31, 2021 • edited Loading

RylanSchaeffer commented Aug 31, 2021

datumbox commented Aug 31, 2021

RylanSchaeffer commented Aug 31, 2021

NicolasHug commented Aug 31, 2021

RylanSchaeffer commented Aug 31, 2021 • edited Loading

NicolasHug commented Aug 31, 2021

RylanSchaeffer commented Aug 31, 2021

datumbox commented Aug 31, 2021

0x00b1 commented Aug 31, 2021

oke-aditya Sep 20, 2021 • edited Loading

Choose a reason for hiding this comment

0x00b1 Sep 21, 2021

Choose a reason for hiding this comment

oke-aditya left a comment • edited Loading

Choose a reason for hiding this comment

datumbox left a comment

Choose a reason for hiding this comment

0x00b1 commented Sep 21, 2021

0x00b1 commented Sep 21, 2021

0x00b1 commented Sep 21, 2021 • edited Loading

datumbox left a comment

Choose a reason for hiding this comment

NicolasHug commented Sep 22, 2021 • edited Loading

oke-aditya commented Sep 22, 2021

oke-aditya commented Sep 23, 2021

oke-aditya Sep 23, 2021 • edited Loading

Choose a reason for hiding this comment

datumbox Sep 23, 2021

Choose a reason for hiding this comment

`masks_to_bounding_boxes` op #4290

`masks_to_bounding_boxes` op #4290

0x00b1 commented Aug 18, 2021 •

edited

Loading

0x00b1 commented Aug 19, 2021 •

edited

Loading

0x00b1 commented Aug 31, 2021 •

edited

Loading

RylanSchaeffer commented Aug 31, 2021 •

edited

Loading

oke-aditya Sep 20, 2021 •

edited

Loading

oke-aditya left a comment •

edited

Loading

0x00b1 commented Sep 21, 2021 •

edited

Loading

NicolasHug commented Sep 22, 2021 •

edited

Loading

oke-aditya Sep 23, 2021 •

edited

Loading