general autocomplete solution

skyvern/exceptions.py

@@ -464,3 +464,38 @@ def __init__(self, target: str, reason: str | None) -> None:
 class NoElementBoudingBox(SkyvernException):
     def __init__(self, element_id: str) -> None:
         super().__init__(f"Element does not have a bounding box. element_id={element_id}")
+
+
+class NoIncrementalElementFoundForAutoCompletion(SkyvernException):
+    def __init__(self, element_id: str, text: str) -> None:
+        super().__init__(f"No auto completion shown up after fill in [{text}]. element_id={element_id}")
+
+
+class NoSuitableAutoCompleteOption(SkyvernException):
+    def __init__(self, reasoning: str | None, target_value: str) -> None:
+        super().__init__(
+            f"No suitable auto complete option to choose. target_value={target_value}, reasoning={reasoning}"
+        )
+
+
+class NoAutoCompleteOptionMeetCondition(SkyvernException):
+    def __init__(
+        self, reasoning: str | None, required_relevance: float, target_value: str, closest_relevance: float
+    ) -> None:
+        super().__init__(
+            f"No auto complete option meet the condition(relevance_float>{required_relevance}). reasoning={reasoning}, target_value={target_value}, closest_relevance={closest_relevance}"
+        )
+
+
+class ErrEmptyTweakValue(SkyvernException):
+    def __init__(self, reasoning: str | None, current_value: str) -> None:
+        super().__init__(
+            f"Empty tweaked value for the current value. reasoning={reasoning}, current_value={current_value}"
+        )
+
+
+class FailToFindAutocompleteOption(SkyvernException):
+    def __init__(self, current_value: str) -> None:
+        super().__init__(
+            f"Can't find a suitable auto completion for the current value, maybe retry with another reasonable value. current_value={current_value}"
+        )

skyvern/forge/prompts/skyvern/auto-completion-choose-option.j2

@@ -0,0 +1,37 @@
+There is an input element on a HTML page. Based on the context and information you're provided, you have two goals: 
+    - Confirm if there is an auto completion attempt showing up after the user input the current value.
+    - If available auto completion suggestions show up, help user choose the element that's the most relevant to the input value.
+
+You can confirm auto completion attempt based on the following rules:
+    - Several auto completion suggestions show up for the input value. 
+    - Some messages, like "No results", "No match", also indicate an attempt to give auto completion suggestions.
+
+Potential auto completion suggesstion could only be:
+    - Element with ID from "HTML elements". Don't hallucinate any potential option outside "HTML elements".
+
+MAKE SURE YOU OUTPUT VALID JSON. No text before or after JSON, no trailing commas, no comments (//), no unnecessary quotes, etc.
+Each interactable element is tagged with an ID.
+
+Reply in JSON format with the following keys:
+{
+    "auto_completion_attempt": bool, // True if there's any auto completion attempt based on the rules. Otherwise, it should be False.
+    "reasoning": str, // The reasoning behind the decision. Be specific, referencing input value and element ids in your reasoning. Mention why you chose the element id. Keep the reasoning short and to the point.
+    "confidence_float": float, // The confidence of the action. Pick a number between 0.0 and 1.0. 0.0 means no confidence, 1.0 means full confidence.
+    "relevance_float": float, // The relative between the input value and the element. Pick a number between 0.00 and 1.00. 0.00 means no relevance, 1.00 means full relevance, the precision is 0.01.
+    "id": str, // The id of the most relevant and interactable element to take the action. The id must be from "HTML elements". It should be null if no element is relative or there's no auto completion suggestion.
+}
+
+Context:
+```
+{{ context_reasoning }}
+```
+
+Input value:
+```
+{{ filled_value }}
+```
+
+HTML elements:
+```
+{{ elements }}
+```

skyvern/forge/prompts/skyvern/auto-completion-potential-answers.j2

@@ -0,0 +1,29 @@
+You're doing an auto completion input action on HTML page. The current filled value doesn't match any option.
+Based on the context and current value, give ten most potential values with the same meaning as the current value.
+You can provide values like:
+    - Subset or superset meaning from the current value
+    - Summarized from the current value
+    - Remove too detailed information, making more general and concise
+But don't add any extra information to the value.
+
+MAKE SURE YOU OUTPUT VALID JSON. No text before or after JSON, no trailing commas, no comments (//), no unnecessary quotes, etc.
+Reply in JSON format with the following keys:
+{
+    "potential_values": [
+        {
+            "reasoning": str, // the reasoning why you recommend this value, including the relationship between the value you recommend and the current value. Keep the reasoning short and to the point.
+            "relevance_float": float, // The relative between the target value and the element. Pick a number between 0.00 and 1.00. 0.00 means no relevance, 1.00 means full relevance, the precision is 0.01.
+            "value": str, // the value you recommend
+        }
+    ], // The list of potential values. Sorted by the descending order of relevance_float 
+}
+
+Context:
+```
+{{ context_reasoning }}
+```
+
+Current Value:
+```
+{{ current_value }}
+```

skyvern/forge/prompts/skyvern/auto-completion-tweak-value.j2

@@ -0,0 +1,38 @@
+You're doing an auto completion input action on HTML page. User has tried several values, but none of them could find a match.
+Based on the context, current value, tried values, option elements popped up while typing, tweak the value into a reasonable one based on the information.
+You can try to change the value under the following rules:
+    1. the value must be reasonably changed from the current value, like superset, subset of the current value
+    2. If there're popped up elements, find the common concept among all elements, and then tweak the current value into a reasonable value based on the same concept.
+
+Don't add any extra information to the value.
+Don't use any value from the popped up elements.
+
+MAKE SURE YOU OUTPUT VALID JSON. No text before or after JSON, no trailing commas, no comments (//), no unnecessary quotes, etc.
+Reply in JSON format with the following keys:
+{
+    "is_any_popped_up_elements": bool, // if there's any popped up elements to extract the concept
+    "common_concept": str, // Simple words to describe the common concept among all elements. null if there's no popped up elements.
+    "reasoning": str, // The reasoning behind the change. Be specific, referencing tweaked value in your reasoning. Mention why you make this decision. Keep the reasoning short and to the point.
+    "confidence_float": float, // The confidence of the decision. Pick a number between 0.0 and 1.0. 0.0 means no confidence, 1.0 means full confidence
+    "tweaked_value": str, // the value tweaked from current value. If common_concept is not null, the value should also under the same concept
+}
+
+Context:
+```
+{{ context_reasoning }}
+```
+
+Current Value:
+```
+{{ current_value }}
+```
+
+Tried Values:
+```
+{{ tried_values }}
+```
+
+Popped up elements:
+```
+{{ popped_up_elements }}
+```