Decorate bounding boxes with element_ids to improve Skyvern accuracy (+ a few more changes) #536
Conversation
<!-- ELLIPSIS_HIDDEN --> | 🚀 | This description was created by [Ellipsis](https://www.ellipsis.dev) for commit 58a439d843b5f612e79e04741b68d72463fa812e | |--------|--------| ### Summary: Enhanced script usage instructions, error handling, and DOM utility functions, including uncommenting a line to remove orphan nodes in `skyvern/webeye/scraper/domUtils.js`. **Key points**: - `scripts/restart_task.py`: Updated script usage instructions for `tasks.csv` file format. - `skyvern/exceptions.py`: Improved error messages and added `FailToClick` exception. - `skyvern/forge/agent.py`: Fixed typo in comment. - `skyvern/forge/prompts/skyvern/extract-action-claude3-sonnet.j2`: Added `action_plan` key to JSON response format. - `skyvern/forge/prompts/skyvern/extract-action.j2`: Added `action_plan` key to JSON response format. - `skyvern/webeye/actions/handler.py`: Added handling for `FailToClick` exception and improved action handling logic. - `skyvern/webeye/scraper/domUtils.js`: Uncommented line to remove orphan nodes from result array, enhanced visibility checks for radio inputs, and improved hint marker creation logic. - `skyvern/webeye/scraper/scraper.py`: Updated element tree building logic and fixed issues with element attribute trimming. ---- Generated with ❤️ by [ellipsis.dev](https://www.ellipsis.dev) <!-- ELLIPSIS_HIDDEN -->
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Changes requested. Incremental review on c62e774 in 1 minute and 17 seconds
More details
- Looked at
251lines of code in7files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_RODiElqkooDjT5Js
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
| @@ -1088,7 +1091,11 @@ async function buildTreeFromBody(frame = "main.frame", open_select = false) { | |||
| const labelElement = document.querySelector( | |||
There was a problem hiding this comment.
The condition to skip adding a label element to the result array is incorrect. The check should include a condition to ensure that the label element does not have a for attribute pointing to another element. This is because a label with a for attribute is not an orphan node but is associated with another element.
There was a problem hiding this comment.
👍 Looks good to me! Reviewed everything up to c62e774 in 1 minute and 18 seconds
More details
- Looked at
252lines of code in7files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. skyvern/webeye/actions/handler.py:794
- Draft comment:
The use of a genericExceptionto represent a click failure inchain_clickmight be misleading. Consider handling specific exceptions or verifying the type of exception before wrapping it withFailToClickto ensure accurate error reporting and handling. - Reason this comment was not posted:
Confidence of 40% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_IfFR9N2DoZFns1za
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Summary:
Enhanced script usage instructions, error handling, DOM utility functions, and added
element_idto bounding boxes.Key points:
scripts/restart_task.py: Updated script usage instructions fortasks.csvfile format.skyvern/exceptions.py: Improved error messages and addedFailToClickexception.skyvern/forge/agent.py: Fixed typo in comment.skyvern/forge/prompts/skyvern/extract-action-claude3-sonnet.j2: Addedaction_plankey to JSON response format.skyvern/forge/prompts/skyvern/extract-action.j2: Addedaction_plankey to JSON response format.skyvern/webeye/actions/handler.py: Added handling forFailToClickexception and improved action handling logic.skyvern/webeye/scraper/domUtils.js: Uncommented line to remove orphan nodes from result array, enhanced visibility checks for radio inputs, and improved hint marker creation logic.skyvern/webeye/scraper/scraper.py: Updated element tree building logic, fixed issues with element attribute trimming, and addedelement_idto bounding boxes.Generated with ❤️ by ellipsis.dev