-
Notifications
You must be signed in to change notification settings - Fork 535
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decorate bounding boxes with element_ids to improve Skyvern accuracy (+ a few more changes) #536
Conversation
<!-- ELLIPSIS_HIDDEN --> | 🚀 | This description was created by [Ellipsis](https://www.ellipsis.dev) for commit 58a439d843b5f612e79e04741b68d72463fa812e | |--------|--------| ### Summary: Enhanced script usage instructions, error handling, and DOM utility functions, including uncommenting a line to remove orphan nodes in `skyvern/webeye/scraper/domUtils.js`. **Key points**: - `scripts/restart_task.py`: Updated script usage instructions for `tasks.csv` file format. - `skyvern/exceptions.py`: Improved error messages and added `FailToClick` exception. - `skyvern/forge/agent.py`: Fixed typo in comment. - `skyvern/forge/prompts/skyvern/extract-action-claude3-sonnet.j2`: Added `action_plan` key to JSON response format. - `skyvern/forge/prompts/skyvern/extract-action.j2`: Added `action_plan` key to JSON response format. - `skyvern/webeye/actions/handler.py`: Added handling for `FailToClick` exception and improved action handling logic. - `skyvern/webeye/scraper/domUtils.js`: Uncommented line to remove orphan nodes from result array, enhanced visibility checks for radio inputs, and improved hint marker creation logic. - `skyvern/webeye/scraper/scraper.py`: Updated element tree building logic and fixed issues with element attribute trimming. ---- Generated with ❤️ by [ellipsis.dev](https://www.ellipsis.dev) <!-- ELLIPSIS_HIDDEN -->
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❌ Changes requested. Incremental review on c62e774 in 1 minute and 17 seconds
More details
- Looked at
251
lines of code in7
files - Skipped
0
files when reviewing. - Skipped posting
0
drafted comments based on config settings.
Workflow ID: wflow_RODiElqkooDjT5Js
Want Ellipsis to fix these issues? Tag @ellipsis-dev
in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet
mode, and more.
@@ -1088,7 +1091,11 @@ async function buildTreeFromBody(frame = "main.frame", open_select = false) { | |||
const labelElement = document.querySelector( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The condition to skip adding a label element to the result array is incorrect. The check should include a condition to ensure that the label element does not have a for
attribute pointing to another element. This is because a label with a for
attribute is not an orphan node but is associated with another element.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 Looks good to me! Reviewed everything up to c62e774 in 1 minute and 18 seconds
More details
- Looked at
252
lines of code in7
files - Skipped
0
files when reviewing. - Skipped posting
1
drafted comments based on config settings.
1. skyvern/webeye/actions/handler.py:794
- Draft comment:
The use of a genericException
to represent a click failure inchain_click
might be misleading. Consider handling specific exceptions or verifying the type of exception before wrapping it withFailToClick
to ensure accurate error reporting and handling. - Reason this comment was not posted:
Confidence of 40% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_IfFR9N2DoZFns1za
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet
mode, and more.
Summary:
Enhanced script usage instructions, error handling, DOM utility functions, and added
element_id
to bounding boxes.Key points:
scripts/restart_task.py
: Updated script usage instructions fortasks.csv
file format.skyvern/exceptions.py
: Improved error messages and addedFailToClick
exception.skyvern/forge/agent.py
: Fixed typo in comment.skyvern/forge/prompts/skyvern/extract-action-claude3-sonnet.j2
: Addedaction_plan
key to JSON response format.skyvern/forge/prompts/skyvern/extract-action.j2
: Addedaction_plan
key to JSON response format.skyvern/webeye/actions/handler.py
: Added handling forFailToClick
exception and improved action handling logic.skyvern/webeye/scraper/domUtils.js
: Uncommented line to remove orphan nodes from result array, enhanced visibility checks for radio inputs, and improved hint marker creation logic.skyvern/webeye/scraper/scraper.py
: Updated element tree building logic, fixed issues with element attribute trimming, and addedelement_id
to bounding boxes.Generated with ❤️ by ellipsis.dev