Added documentation for SAM2 tracker #8820
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThe changes include the addition of a new documentation file, Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (3)
site/content/en/docs/enterprise/segment-anything-2-tracker.md (3)
8-15: Consider adding performance expectationsThe overview effectively explains the feature, but users would benefit from knowing:
- Minimum GPU requirements (memory, compute capability)
- Expected performance differences between GPU and CPU (e.g., average processing time per frame)
73-73: Consider rephrasing for clarityThe current phrase "To do this" could be more specific.
Consider this revision:
-To do this, click the **Menu** button in the annotation view and select the **Run Actions** option: +To track multiple objects, click the **Menu** button in the annotation view and select the **Run Actions** option:🧰 Tools
🪛 LanguageTool
[style] ~73-~73: Consider a more expressive alternative.
Context: ...ack multiple objects simultaneously. To do this, click the Menu button in the ...(DO_ACHIEVE)
95-100: Consider enhancing parameter documentationThe parameters section would benefit from:
- Specifying the valid range for target frame (e.g., maximum recommended tracking distance)
- Explaining any performance implications of shape conversion
- Adding examples of typical usage scenarios for each parameter
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (6)
site/content/en/images/sam2_tracker_run_action.pngis excluded by!**/*.pngsite/content/en/images/sam2_tracker_run_action_modal.pngis excluded by!**/*.pngsite/content/en/images/sam2_tracker_run_action_modal_progress.pngis excluded by!**/*.pngsite/content/en/images/sam2_tracker_run_shape_action.pngis excluded by!**/*.pngsite/content/en/images/sam2_tracker_run_shape_action_modal.pngis excluded by!**/*.pngsite/content/en/images/sam2_tracker_run_shape_action_modal_progress.pngis excluded by!**/*.png
📒 Files selected for processing (2)
site/content/en/docs/enterprise/segment-anything-2-tracker.md(1 hunks)site/content/en/docs/enterprise/social-accounts-configuration.md(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- site/content/en/docs/enterprise/social-accounts-configuration.md
🧰 Additional context used
🪛 LanguageTool
site/content/en/docs/enterprise/segment-anything-2-tracker.md
[style] ~73-~73: Consider a more expressive alternative.
Context: ...ack multiple objects simultaneously. To do this, click the Menu button in the ...
(DO_ACHIEVE)
🔇 Additional comments (1)
site/content/en/docs/enterprise/segment-anything-2-tracker.md (1)
1-6: LGTM!
The front matter is well-structured with appropriate metadata.
Comment on lines
17
to
46
| ## How to install | ||
|
|
||
| > **Note**: This feature is not available in the community CVAT version. | ||
|
|
||
| > **Note**: This feature requires the enhanced actions UI plugin, which is enabled by default. | ||
| Usually, no additional steps are necessary on this. | ||
|
|
||
| ### Docker | ||
|
|
||
| You can use existing scripts from the community repository | ||
| (`./serverless/deploy_cpu.sh` or `./serverless/deploy_gpu.sh`). | ||
| To deploy the feature, simply run: | ||
|
|
||
| ```sh | ||
| ./serverless/deploy_gpu.sh "path/to/the/function" | ||
| ``` | ||
|
|
||
| ### Kubernetes | ||
|
|
||
| - You need to deploy the Nuclio function manually. | ||
| Note that this function requires a Redis client to store the tracking state. | ||
| When running the `nuclio deploy` command, make sure to provide the necessary arguments. | ||
| The minimal command is: | ||
|
|
||
| ```sh | ||
| nuctl deploy "path/to/the/function" | ||
| --env CVAT_FUNCTIONS_REDIS_HOST="<redis_host>" | ||
| --env CVAT_FUNCTIONS_REDIS_PORT="<redis_port>" | ||
| --env CVAT_FUNCTIONS_REDIS_PASSWORD="<redis_password>" # if applicable | ||
| ``` |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Installation instructions need more details
Consider enhancing the installation section with:
- Redis requirements for Docker deployment (if applicable)
- Specific path or location where the function can be found
- A troubleshooting section covering common deployment issues
Comment on lines
1
to
100
| --- | ||
| title: 'Segment Anything 2 Tracker' | ||
| linkTitle: 'Segment Anything 2 Tracker' | ||
| weight: 6 | ||
| description: 'Accelerating video labeling using SAM2 model' | ||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| Segment Anything 2 is a segmentation model that allows fast and precise selection of any object in videos or images. | ||
| For enterprise customers, this model can be installed in their self-hosted solution. To ensure a good experience, | ||
| it is strongly recommended to deploy the model using a GPU. Although it is possible to use a CPU-based version, | ||
| it generally performs much slower and is suitable only for handling a single parallel request. Unlike a regular | ||
| tracking model, the SAM 2 tracker is implemented as an annotation action. This allows it to be applied to existing | ||
| objects (polygons and masks) to track them forward for a specified number of frames. | ||
|
|
||
| ## How to install | ||
|
|
||
| > **Note**: This feature is not available in the community CVAT version. | ||
|
|
||
| > **Note**: This feature requires the enhanced actions UI plugin, which is enabled by default. | ||
| Usually, no additional steps are necessary on this. | ||
|
|
||
| ### Docker | ||
|
|
||
| You can use existing scripts from the community repository | ||
| (`./serverless/deploy_cpu.sh` or `./serverless/deploy_gpu.sh`). | ||
| To deploy the feature, simply run: | ||
|
|
||
| ```sh | ||
| ./serverless/deploy_gpu.sh "path/to/the/function" | ||
| ``` | ||
|
|
||
| ### Kubernetes | ||
|
|
||
| - You need to deploy the Nuclio function manually. | ||
| Note that this function requires a Redis client to store the tracking state. | ||
| When running the `nuclio deploy` command, make sure to provide the necessary arguments. | ||
| The minimal command is: | ||
|
|
||
| ```sh | ||
| nuctl deploy "path/to/the/function" | ||
| --env CVAT_FUNCTIONS_REDIS_HOST="<redis_host>" | ||
| --env CVAT_FUNCTIONS_REDIS_PORT="<redis_port>" | ||
| --env CVAT_FUNCTIONS_REDIS_PASSWORD="<redis_password>" # if applicable | ||
| ``` | ||
|
|
||
| ## Running tracker on an object | ||
|
|
||
| The tracker can be applied to any polygons and masks. To run the tracker on an object, open the object menu and click | ||
| "Run annotation action". | ||
|
|
||
| <img src="/images/sam2_tracker_run_shape_action.png" style="max-width: 200px; padding: 16px;"> | ||
|
|
||
| Alternatively, you can use a hotkey: select the object and press **Ctrl + E** (default shortcut). | ||
| When the modal opened, in "Select action" list, choose **Segment Anything 2: Tracker**: | ||
|
|
||
| <img src="/images/sam2_tracker_run_shape_action_modal.png" style="max-width: 500px; padding: 16px;"> | ||
|
|
||
| Specify the **target frame** until which you want the object to be tracked, | ||
| then click the **Run** button to start tracking. The process begins and may take some time to complete. | ||
| The duration depends on the inference device, and the number of frames where the object will be tracked. | ||
|
|
||
| <img src="/images/sam2_tracker_run_shape_action_modal_progress.png" style="max-width: 500px; padding: 16px;"> | ||
|
|
||
| Once the process is complete, the modal window closes. You can review how the object was tracked. | ||
| If you notice that the tracked shape deteriorates at some point, | ||
| you can adjust the object coordinates and run the tracker again from that frame. | ||
|
|
||
| ## Running tracker on multiple objects | ||
|
|
||
| Instead of tracking each object individually, you can track multiple objects | ||
| simultaneously. To do this, click the **Menu** button in the annotation view and select the **Run Actions** option: | ||
|
|
||
| <img src="/images/sam2_tracker_run_action.png" style="max-width: 200px; padding: 16px;"> | ||
|
|
||
| Alternatively, you can use a hotkey: just press **Ctrl + E** (default shortcut) when there are no objects selected. | ||
| This opens the actions modal. In this case, the tracker will be applied to all visible objects of suitable types | ||
| (polygons and masks). In the action list of the opened model, select **Segment Anything 2: Tracker**: | ||
|
|
||
| <img src="/images/sam2_tracker_run_action_modal.png" style="max-width: 500px; padding: 16px;"> | ||
|
|
||
| Specify the **target frame** until which you want the objects to be tracked, | ||
| then click the **Run** button to start tracking. The process begins and may take some time to complete. | ||
| The duration depends on the inference device, the number of simultaneously tracked objects, | ||
| and the number of frames where the object will be tracked. | ||
|
|
||
| <img src="/images/sam2_tracker_run_action_modal_progress.png" style="max-width: 500px; padding: 16px;"> | ||
|
|
||
| Once the process finishes, you may close the modal and review how the objects were tracked. | ||
| If you notice that the tracked shapes deteriorate, you can adjust their | ||
| coordinates and run the tracker again from that frame (for a single object or for many objects). | ||
|
|
||
|
|
||
| ## Tracker parameters | ||
|
|
||
| - **Target frame**: Objects will be tracked up to this frame. Must be greater than the current frame | ||
| - **Convert polygon shapes to tracks**: When enabled, all visible polygon shapes in the current frame will be converted | ||
| to tracks before tracking begins. Use this option if you need tracks as the final output but started with shapes, | ||
| produced for example by interactors (e.g. SAM2 or another one). |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider adding essential documentation sections
The documentation would benefit from additional sections:
- Model Version & Compatibility
- Supported SAM2 model versions
- CVAT version compatibility
- Known Limitations
- Performance limitations
- Supported object types
- Edge cases where tracking might fail
🧰 Tools
🪛 LanguageTool
[style] ~73-~73: Consider a more expressive alternative.
Context: ...ack multiple objects simultaneously. To do this, click the Menu button in the ...
(DO_ACHIEVE)
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and context
How has this been tested?
Checklist
developbranch(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.
Summary by CodeRabbit
New Features
Bug Fixes