Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CVAT integration] Use pixelwise masks, not polygons, for instance segmentation #4483 #4937

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from

Conversation

NicDionne
Copy link

@NicDionne NicDionne commented Oct 17, 2024

This pull request is currently a draft. The following items are still pending:

  • Code to upload fiftyone to CVAT pixel mask
  • Unit test for uploading to CVAT
  • Unit test for converting CVAT shapes back into the FiftyOne format
  • Implementation for converting CVAT shapes back into FiftyOne format

@ehofesmann

  1. Sorry, I'm not sure how I should go about testing CVAT components with API request ?
  2. I'm also unsure how to incorporate the download from CVAT to Fiftyone. I attempted to use load_annotation_results, but it didn't seem to trigger this particular part of the code. Could you advise on how to proceed?
    label = cvat_shape.to_polyline(closed=True, filled=True)

What changes are proposed in this pull request?

Propose following the issue #4483, to make it possible to export fiftyone mask to CVAT as pixel mask.

How is this patch tested? If it is not, please explain why.

For now, simple upload test with rectangle. I seek guidance regarding how it should be properly tested. For now this pull request is only a draft.

Release Notes

Is this a user-facing change that should be mentioned in the release notes?

Yes,

dataset.annotate(anno_key, label_field='detections', label_type='instances')

The data should now be uploaded in pixel format instead of polygons. This change is good because CVAT does not support polygons with holes, which could create bottlenecks in certain use cases.

  • No. You can skip the rest of this section.
  • Yes. Give a description of this change to be included in the release
    notes for FiftyOne users.

(Details in 1-2 sentences. You can just refer to another PR with a description
if this PR is part of a larger change.)

What areas of FiftyOne does this PR affect?

  • App: FiftyOne application changes
  • Build: Build and test infrastructure changes
  • Core: Core fiftyone Python library changes
  • Documentation: FiftyOne documentation changes
  • Other

Summary by CodeRabbit

  • New Features
    • Introduced a new class for converting between run-length encoded masks and binary image masks.
    • Added methods to facilitate the conversion of masks for better integration with the FiftyOne framework.
    • Enhanced shape conversion capabilities to support instance detection with masks.
  • Bug Fixes
    • Updated the export process to properly handle new mask types during annotation uploads.

… for instance segmentation voxel51#4483

- We can upload mask
[] Missing test
[] Missing download mask
Copy link

codecov bot commented Oct 17, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.18%. Comparing base (bac2d2a) to head (513b633).
Report is 318 commits behind head on develop.

Additional details and impacted files
@@            Coverage Diff            @@
##           develop    #4937    +/-   ##
=========================================
  Coverage    99.17%   99.18%            
=========================================
  Files           49       49            
  Lines        17785    17991   +206     
=========================================
+ Hits         17639    17845   +206     
  Misses         146      146            
Flag Coverage Δ
python 99.18% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@brimoor
Copy link
Contributor

brimoor commented Oct 17, 2024

Amazing, thanks @NicDionne! @ehofesmann will give the PR a review and assist with getting this merged 💪

Copy link
Member

@ehofesmann ehofesmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! Thank you so much for taking this on @NicDionne !!

To your questions:

  1. We don't have a great way to mock the CVAT API at the moment, so most testing is done against a running CVAT instance. In the end, it would be great to add a new test for this to our test suite here.
  2. In order to be able to load the CVAT mask back into FiftyOne, you'd need to add a new check here for if shape_type == "mask", which can then check for instance or instances types and call your new CVATShape.to_instance_detection() method. Let me know if you run into any issues with this!

@NicDionne
Copy link
Author

NicDionne commented Oct 18, 2024

It seems I'm having trouble committing files from cvat_test.py. Pylint is throwing multiple E0401 and E1101 errors, even though everything seems to be working fine.

Here is the code for the unit test I ran:

    def test_pixel_mask_detection(self):
        gt_field = "ground_truth"
        dataset = fo.Dataset()
        dataset.add_samples(
            foz.load_zoo_dataset(
                "coco-2017",
                split="validation",
                label_types=["segmentations"],
            ).head(num_samples=10)
        )

        anno_key = "anno_key_1"
        dataset.annotate(
            anno_key,
            backend="cvat",
            label_field=gt_field,
            label_type="instances",
        )
        test_field = "test"
        dataset.load_annotations(
            anno_key=anno_key, backend="cvat", dest_field=test_field
        )
        for sample in dataset.iter_samples():
            gt_detections = sorted(
                sample[gt_field].detections, key=lambda det: det.id
            )
            cvat_detections = sorted(
                sample[test_field].detections, key=lambda det: det.id
            )
            self.assertEqual(len(gt_detections), len(cvat_detections))
            for gt_detection, cvat_detection in zip(
                gt_detections, cvat_detections
            ):
                self.assertEqual(
                    gt_detection.id,
                    cvat_detection.id,  # If not the case problem with unit test
                )
                np.testing.assert_array_almost_equal(
                    gt_detection.bounding_box,
                    cvat_detection.bounding_box,
                    decimal=2,
                )
                np.testing.assert_equal(gt_detection.mask, cvat_detection.mask)

@NicDionne NicDionne marked this pull request as ready for review October 18, 2024 02:12
Copy link
Contributor

coderabbitai bot commented Oct 18, 2024

Walkthrough

The changes introduce a new class HasCVATBinMask in fiftyone/utils/cvat.py, which includes methods for converting between run-length encoded (RLE) masks and binary image masks. Additionally, the CVATShape class is updated with a new method to_instance_detection for converting shapes to detection labels while handling masks. The export_sample method in CVATVideoAnnotationWriter is also modified to support the new mask type. These changes enhance the handling of segmentation masks in the CVAT format within the FiftyOne framework.

Changes

File Path Change Summary
fiftyone/utils/cvat.py Added class HasCVATBinMask with methods rle_to_binary_image_mask and mask_to_cvat_rle. Updated CVATShape to include to_instance_detection. Modified export_sample in CVATVideoAnnotationWriter to handle new mask type.

Possibly related issues

🐰 In fields of green, I hop with glee,
New masks to see, oh what a spree!
RLE to binary, a dance so fine,
FiftyOne and CVAT, together they shine!
Let's celebrate this change, oh so bright,
With pixelwise joy, our future's in sight! 🌟


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Outside diff range and nitpick comments (4)
fiftyone/utils/cvat.py (4)

1607-1615: Possible Improvement in mask_to_cvat_rle Method

The implementation of mask_to_cvat_rle can be optimized for readability and efficiency. Currently, the method iterates over the grouped elements of the flattened binary mask and constructs the RLE counts.

Consider using np.packbits or other NumPy vectorized operations to optimize the RLE encoding process.

While the current implementation is correct, utilizing NumPy's vectorized functions can improve performance, especially for large masks.


1592-1616: Remove Unused Commented Code

Lines 1602-1604 contain commented-out code that seems to be an alternative implementation of rle_to_binary_image_mask. Unless this code is necessary for future reference, it would improve code cleanliness to remove it.

Remove the commented-out code to enhance readability:

- # mask = np.zeros(mask_width * mask_height, dtype=np.uint8)
- # mask[np.add.accumulate(rle)[::2]] = 1
- # return mask.reshape(mask_width, mask_height)

6439-6443: Swap Width and Height Variables for Consistency

In the calculation of mask_height and mask_width from det.mask.shape, there is a potential mix-up between width and height:

  • Line 6441: mask_height, mask_width = det.mask.shape assigns mask_height to det.mask.shape[0] (which is typically the number of rows, i.e., the height).

This could lead to confusion and inconsistencies later in the code.

Consider assigning mask_height, mask_width as:

- mask_height, mask_width = det.mask.shape
+ mask_width, mask_height = det.mask.shape

Ensure that width corresponds to the number of columns and height corresponds to the number of rows.


1590-1617: Add Docstrings to HasCVATBinMask Methods

The methods rle_to_binary_image_mask and mask_to_cvat_rle lack docstrings explaining their purpose, parameters, and return values.

Add descriptive docstrings to improve code readability and maintainability:

class HasCVATBinMask:
    @staticmethod
    def rle_to_binary_image_mask(rle, mask_width, mask_height) -> np.ndarray:
        """
        Converts a run-length encoded (RLE) mask to a binary image mask.

        Args:
            rle: The run-length encoded mask as a list of integers.
            mask_width: The width of the mask.
            mask_height: The height of the mask.

        Returns:
            A binary mask as a NumPy ndarray.
        """
        ...

    @staticmethod
    def mask_to_cvat_rle(binary_mask: np.ndarray) -> np.array:
        """
        Converts a binary image mask to a CVAT-compatible RLE format.

        Args:
            binary_mask: The binary mask as a NumPy ndarray.

        Returns:
            A list of integers representing the RLE.
        """
        ...
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 1f1d133 and 1df8997.

📒 Files selected for processing (1)
  • fiftyone/utils/cvat.py (4 hunks)
🧰 Additional context used
🔇 Additional comments (2)
fiftyone/utils/cvat.py (2)

5923-5925: Ensure Consistent Handling of 'mask' Shape Types

In the _parse_annotation method, the addition of elif shape_type == "mask": introduces handling for mask shapes.

  • Line 5925: The method cvat_shape.to_instance_detection() is called to convert the mask shape to a detection.

The implementation correctly extends the method to handle mask shapes. Ensure that any invoking code correctly processes the returned detection with a mask.


6436-6458: Handle Cases When det.mask is Missing

In the _create_detection_shapes method, when label_type is "instance" or "instances", the code expects det.mask to be present.

  • Line 6437: Checks if det.mask is None and continues if so.

Confirm that det.mask is always provided for instances of this label type. If det.mask can be None, consider logging a warning or handling the case appropriately.

Ensure that the dataset or upstream code provides the necessary mask data for these instances.

fiftyone/utils/cvat.py Outdated Show resolved Hide resolved
fiftyone/utils/cvat.py Outdated Show resolved Hide resolved
fiftyone/utils/cvat.py Outdated Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
fiftyone/utils/cvat.py (1)

7104-7134: LGTM! New method for converting CVAT shapes to FiftyOne instance detections.

The to_instance_detection method is a valuable addition to the CVATShape class, enabling conversion of CVAT shapes with masks to FiftyOne Detection objects. The implementation correctly handles the conversion process and makes good use of the new HasCVATBinMask class.

Consider adding a brief comment explaining why we add 1 to mask_w and mask_h. For example:

# Add 1 to account for CVAT's inclusive coordinate system
mask_w, mask_h = (
    round(xbr - xtl) + 1,
    round(ybr - ytl) + 1,
)

This would improve code clarity for future maintainers.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 1df8997 and 513b633.

📒 Files selected for processing (1)
  • fiftyone/utils/cvat.py (4 hunks)
🧰 Additional context used
🔇 Additional comments (1)
fiftyone/utils/cvat.py (1)

1590-1613: LGTM! New utility class for handling CVAT binary masks.

The HasCVATBinMask class is a well-implemented utility for converting between run-length encoded (RLE) masks and binary image masks. The use of numpy for array operations is efficient, and the methods are clearly defined with appropriate type hints.

@NicDionne
Copy link
Author

Hi @brimoor,

It looks like the build failed, but strangely it stopped at FFmpeg. I'm not really sure what I might have touched that could cause this. Do you have any ideas?

Thanks,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants