ObjectSeeker Defense Implementation

[f4str]

Description

Implement the ObjectSeeker certifiably robust defense for object detectors. This is an estimator located in art.estimators.certification.object_seeker as the PyTorchObjectSeeker class. It is constructed in a similar way as the PyTorchObjectDetector or PyTorchYolo. Currently, there is only a PyTorch implementation, but a generic and TensorFlow implementation is possible in the future.

Additionally created utility functions for intersection_over_union, intersection_over_area, and non_maximum_supression under art.utils.

Note that unit tests are in tests/estimators/object_detection rather than tests/estimators/certification since a YOLO model is required.

Relevant links:

• Paper: https://arxiv.org/abs/2202.01811
• Code: https://github.com/inspire-group/ObjectSeeker

Fixes #2193

Type of change

Please check all relevant options.

• Improvement (non-breaking)
• Bug fix (non-breaking)
• New feature (non-breaking)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

• Unit tests for ObjectSeeker YOLO
• Unit tests for ObjectSeeker Faster R-CNN

Test Configuration:

• OS
• Python version
• ART version or commit number
• TensorFlow / Keras / PyTorch / MXNet version

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[codecov-commenter]

Codecov Report

Merging #2246 (46af3c2) into dev_1.16.0 (90bf04b) will increase coverage by 0.50%.
The diff coverage is 94.92%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@              Coverage Diff               @@
##           dev_1.16.0    #2246      +/-   ##
==============================================
+ Coverage       85.28%   85.79%   +0.50%     
==============================================
  Files             315      318       +3     
  Lines           28054    28329     +275     
  Branches         5123     5157      +34     
==============================================
+ Hits            23927    24304     +377     
+ Misses           2833     2711     -122     
- Partials         1294     1314      +20     

Files Changed	Coverage Δ
.../estimators/certification/object_seeker/pytorch.py	83.87% <83.87%> (ø)
art/utils.py	73.97% <97.87%> (+1.65%)	⬆️
...ators/certification/object_seeker/object_seeker.py	98.15% <98.15%> (ø)
art/estimators/certification/__init__.py	100.00% <100.00%> (ø)
...estimators/certification/object_seeker/__init__.py	100.00% <100.00%> (ø)

... and 11 files with indirect coverage changes

📢 Have feedback on the report? Share it here.

[beat-buesser]

Hi @f4str Thank you very much for your well prepared pull request implementing ObjectSeeker in ART! I have only a few questions and suggestions, but otherwise the code looks good.

[beat-buesser]

Do we need to include any licence information?

[f4str]

The original code has an MIT license, so we can probably include their license below the ART one.

[beat-buesser]

Is this a fixture that we also could use in other tests and move to contest.py?

[f4str]

Good idea, this can be moved outside to a conftest.pysince it is used by multiple tests now.

[beat-buesser]

Should loss2 be smaller than loss1?

[f4str]

I borrowed the same unit test from test_pytorch_yolo.py. Logically it should be less than, but I believe it was not equal in case of stochasticity that causes the loss to increase. I can look into this and see if it can be changed to less than for both tests.

[f4str]

Hi @beat-buesser thank you for the review. I've addressed the changes by adding the license from the original ObjectSeeker repo and extracting the fixtures into a conftest.py. I've extracted both the YOLO and Faster R-CNN fixtures and created a new unit test for Faster R-CNN. Now ObjectSeeker will test both object detector models.

As for the training loss not-equal vs less-than, it needs to remain a not-equal condition since due to stochasticity, the loss does not always decrease, especially with the "incorrect" dataset. The same applies to the PyTorchYolo and PyTorchFasterRCNN models.

[beat-buesser]

Hi @f4str Thank you very much for the updates as well as checking and documenting! We'll merge this PR as soon as possible.