Merge pull request #4596 from wix/docs/copilot-blogpost

docs(blog): introduce Detox Copilot.

docs/api/copilot.md

@@ -94,4 +94,4 @@ interface PromptHandler {
 
 You can refer to the [Detox Copilot Guide] for an example of implementing a `PromptHandler` for OpenAI's service.
 
-[Detox Copilot Guide]: ../guide/testing-with-copilot
+[Detox Copilot Guide]: ../copilot/testing-with-copilot.md

docs/contributing/code/overview.md

@@ -28,7 +28,6 @@ The other projects are the Detox CLI, the Detox test app, example apps, and the
 Here's a high-level overview of the repository structure:
 
 - 📁 **detox-cli** - The CLI for Detox (e.g., `detox init`, `detox test`, read more about our [CLI docs])
-- 📁 **detox-copilot** - Detox plugin that leverages large language models (LLM) to seamlessly invoke Detox actions (**work in progress**)
 - 📁 **detox** - The Detox framework
   - 📁 **android** - The Android native code, alongside native unit tests
   - 📁 **ios** - The iOS native code, including its native submodules (e.g., DetoxSync)

docs/copilot/best-practices.md

@@ -0,0 +1,69 @@
+# Detox Copilot Best Practices
+
+Detox Copilot allows you to write tests using natural language commands. Each step corresponds to a specific action or assertion within your app. In case you're wondering how to make the most out of this feature, here are some best practices to follow when writing your Copilot intents.
+
+## Step-by-Step Instructions
+
+- **Write Sequential Steps**: Describe your test steps in a clear and sequential manner.
+- **Example**:
+
+```javascript
+it('should navigate and add a product to the cart', async () => {
+  await copilot.perform(
+    'Navigate to the "Products" page',
+    'Tap on the "Add to Cart" button for the first product',
+    'Verify that the "Added to Cart" pop-up is displayed'
+  );
+});
+```
+
+## Be Specific and Clear
+
+- **Provide Clear Instructions**: The clearer your instructions, the better Copilot can interpret them.
+- **Example**:
+  - **Good**: `'Tap on the "Login" button'`
+  - **Better**: `'Tap on the "Login" button located at the top right corner'`
+
+## One Action per Step
+
+- **Avoid Combining Multiple Actions**: Keep each step focused on a single action or assertion.
+- **Example**:
+  - **Avoid**: `'Tap on the "Login" button and enter credentials'`
+  - **Prefer**:
+
+  ```javascript
+  'Tap on the "Login" button',
+  'Enter "user@example.com" into the "Email" field'
+  ```
+
+## Use Exact Labels
+
+- **Refer to UI Elements Precisely**: Use the exact text or identifiers as they appear in the app.
+- **Example**:
+  - **Good**: `'Enter "password123" into the "Password" field'`
+  - **Avoid**: `'Enter password into its field'`
+
+## Keep Assertions Simple
+
+- **Focus on Specific Outcomes**: Make assertions straightforward and specific.
+- **Example**:
+  - **Good**: `'Verify that the "Welcome" message is displayed'`
+  - **Avoid**: `'Check if the welcome message appears correctly on the screen'`
+
+## Leverage Visual Context
+
+- **Utilize Visual Descriptions**: If your LLM supports image snapshots, include visual context in your intents.
+- **Example**: `'Ensure the profile picture is visible at the top of the screen'`
+
+## Avoid Ambiguity
+
+- **Specify Elements Precisely**: If multiple elements could match, provide additional details.
+- **Example**:
+  - **Ambiguous**: `'Tap on the "Submit" button'`
+  - **Specific**: `'Tap on the "Submit" button in the registration form'`
+
+## General Recommendations
+
+- **Flexibility**: While it's best to provide clear instructions, Copilot is designed to interpret a variety of phrasing. Different approaches can work, and you are encouraged to experiment.
+- **Feedback Loop**: Observe how Copilot interprets your instructions and adjust accordingly.
+- **Model Selection**: Choose an LLM model that best suits your application's complexity and language requirements. We recommend advanced models like **Sonnet 3.5** or **GPT-4o** for better performance.

docs/copilot/technical-overview.mdx

@@ -0,0 +1,44 @@
+import OverviewSVG from '../img/copilot/copilot-overview.svg';
+
+# Technical Overview
+
+Detox Copilot integrates seamlessly with your testing environment by combining natural language processing with Detox's robust testing capabilities.
+
+## Building Blocks of Detox Copilot
+
+To enable Detox Copilot to work harmoniously with Detox and your app, it relies on several building blocks:
+
+- **Dynamic Code Generation**: Copilot generates Detox code on-the-fly to perform actions or assertions based on your instructions.
+- **Visual Analysis**: Copilot can analyze the app's screen to verify the presence of specific elements or text, enabling assertions beyond standard UI checks.
+- **App View Hierarchy**: Detox generates an XML representation of the app's view hierarchy, helping Copilot interact with all UI elements, even those not directly visible.
+- **Snapshot Images**: Optional snapshot images provide Copilot with visual context for more precise understanding and analysis.
+- **Injected Test IDs**: When necessary, Detox injects unique test IDs to ensure reliable access to UI elements.
+- **Caching Mechanism**: Copilot caches execution results to optimize performance and reduce unnecessary LLM calls (see [Performance Optimization](#performance-optimization)).
+- **Test Context Awareness**: Copilot maintains awareness of previously executed steps, ensuring continuity and readability in the test flow.
+
+## Copilot's Execution Flow
+
+<OverviewSVG
+  style={{
+    backgroundColor: '#f5f5f5',
+    borderRadius: '10px'
+  }}
+/>
+
+The execution flow of Detox Copilot can be broken down into six main steps:
+
+1. **Gather Context**: Collect relevant app state, view hierarchy, and previous step results.
+2. **Interpret Intent**: Use the LLM to interpret the natural language instruction.
+3. **Generate Code**: Create the appropriate Detox commands.
+4. **Execute Action**: Run the generated Detox code.
+5. **Cache Results**: Store execution results to optimize future runs.
+6. **Provide Feedback**: Return values or confirm actions for subsequent steps.
+
+By combining these steps, Detox Copilot effectively bridges the gap between natural language instructions and concrete test actions.
+
+### Performance Optimization
+
+Detox Copilot is designed to avoid unnecessary calls to the LLM service and optimize performance using static cache that is based on the current state of the app.
+This minimizes the number of calls to the LLM service and reduces latency.
+However, you can optimize your `PromptHandler` implementation to reduce latency and improve response times (e.g., by reducing the image size or implementing a server-side cache).
+We have plans to optimize even further by introducing more advanced caching mechanisms for better performance.

docs/copilot/testing-with-copilot.md

@@ -0,0 +1,185 @@
+# Natural Language Testing with Detox Copilot
+
+In this tutorial, we'll explore how to use **Detox Copilot** to write end-to-end tests using natural language commands. Detox Copilot leverages large language models (LLMs) to translate human-readable instructions into Detox actions and assertions, making test writing more intuitive and accessible.
+
+:::note
+
+Detox Copilot is integrated into Detox and requires no additional installation. For complete API details, refer to our [Detox Copilot API documentation](../api/copilot.md).
+
+:::
+
+:::caution Work in Progress
+
+**Note**: Detox Copilot is in active development. APIs are subject to change in future releases.
+
+:::
+
+## Introduction
+
+Detox Copilot simplifies the process of writing tests by allowing you to describe test steps in natural language.
+It interprets these instructions and translates them into Detox commands. This guide will help you integrate Detox Copilot into your testing workflow and provide best practices for writing effective intents.
+
+![Demo](../img/copilot/copilot-demo.gif)
+
+## Setting Up Detox
+
+Before you begin, ensure that your Detox environment is properly set up.
+If you need assistance with the setup, refer to the [Detox Getting Started Guide](docs/introduction/getting-started/).
+
+## Implementing a `PromptHandler`
+
+The `PromptHandler` is a crucial component that interfaces with your LLM service.
+Below is an example of how to implement a `PromptHandler` using OpenAI's GPT-4 API.
+
+You can adapt this code to work with other LLMs or services as needed. You may also find pre-built `PromptHandler` implementations for popular LLMs in the [detox-copilot repository on GitHub](https://github.com/wix-incubator/detox-copilot).
+
+:::note Open for Contributions
+
+If you have implemented a `PromptHandler` for a specific LLM or service, consider contributing it to the Detox Copilot repository to help the community.
+
+:::
+
+### Example of a `PromptHandler` for OpenAI's GPT-4
+
+```javascript
+const { Configuration, OpenAIApi } = require('openai');
+const path = require('path');
+
+class OpenAIPromptHandler {
+  constructor(apiKey) {
+    const configuration = new Configuration({ apiKey });
+    this.openai = new OpenAIApi(configuration);
+  }
+
+  async runPrompt(prompt, imagePath) {
+    const messages = [
+      { role: 'system', content: 'You are a test automation assistant.' },
+      { role: 'user', content: prompt },
+    ];
+
+    // If an image is provided, "upload" it and include the URL in the prompt
+    if (imagePath && this.isSnapshotImageSupported()) {
+      try {
+        const imageUrl = await this.uploadImage(imagePath);
+        messages.push({
+          role: 'user',
+          content: `Here is an image for reference: ${imageUrl}`,
+        });
+      } catch (error) {
+        console.error('Error uploading image:', error);
+        throw new Error('Failed to upload image');
+      }
+    }
+
+    const response = await this.openai.createChatCompletion({
+      model: 'gpt-4',
+      messages,
+    });
+
+    return response.data.choices[0].message.content;
+  }
+
+  async uploadImage(imagePath) {
+    // Uploads the image and returns the URL
+  }
+
+  isSnapshotImageSupported() {
+    return true; // Set to true to handle image uploads
+  }
+}
+
+module.exports = OpenAIPromptHandler;
+```
+
+**Explanation**:
+
+- **`runPrompt`**: Sends the prompt to the LLM and returns the response.
+- **`isSnapshotImageSupported`**: Indicates whether the LLM can handle snapshot images. If set to `true`, the handler will include image URLs in the prompt and will include them when instructing Detox Copilot.
+
+## Initializing Detox Copilot
+
+Initialize Detox Copilot with your `PromptHandler` before running any tests.
+This is typically done in the `beforeAll` hook or a setup file.
+
+**Example**:
+
+```javascript
+const {copilot} = require('detox/index');
+const OpenAIPromptHandler = require('./OpenAIPromptHandler');
+
+beforeAll(() => {
+  const promptHandler = new OpenAIPromptHandler('YOUR_OPENAI_API_KEY');
+  copilot.init(promptHandler);
+});
+```
+
+## Writing Tests with Detox Copilot
+
+With Detox Copilot initialized, you can now write tests using the `copilot.perform` method.
+
+### Writing Step-by-Step Tests
+
+Detox Copilot allows you to write tests by providing a sequence of natural language instructions. Each instruction corresponds to a single action or assertion.
+
+```javascript
+it('should verify element sizes and button states', async () => {
+  await copilot.perform(
+    'Launch the app with notification permissions enabled',
+    'Navigate to the "Settings" page',
+    'Verify that the "Save" button is disabled',
+    'Locate the profile picture element',
+    'Verify that the profile picture size is 100 x 100 pixels and that the image is available and rendered',
+    'Tap on the "Edit Profile" button',
+    'Verify that the "Save" button is now enabled',
+    'Verify that the "Username" field text is bold'
+  );
+});
+```
+
+In the example above, Copilot can perform checks that go beyond traditional UI testing, such as verifying element sizes, button states (enabled/disabled), or text styles (e.g., bold). This is thanks to the combination of Detox code-generation and multimodal LLMs that can analyze the snapshots.
+
+- **Step-by-Step Instructions**: Each step is a separate string, representing a single action or assertion.
+- **Sequential Execution**: Steps are executed in order, allowing you to describe complex interactions intuitively.
+
+### Hybrid Tests with Copilot and Detox APIs
+
+You can also combine Copilot commands with traditional Detox APIs for more control.
+
+```javascript
+it('should add an item to the cart', async () => {
+  await copilot.perform(
+    'Launch the app',
+    'Navigate to the "Products" page',
+    'Tap on the "Add to Cart" button for the first product'
+  );
+
+  const cartBadge = element(by.id('cart-badge'));
+  await expect(cartBadge).toHaveText('1');
+
+  await copilot.perform(
+    'Navigate to the "Cart" page',
+    'Verify that the product is listed in the cart'
+  );
+});
+```
+
+### Locating Elements with Copilot
+
+You can also use Copilot to retrieve values, locate elements, or perform advanced checks such as verifying element sizes or button states.
+
+```javascript
+it('should display the correct page title', async () => {
+  const pageTitleElement = await copilot.perform(
+    'Launch the app',
+    'Navigate to the "Profile" page',
+    'Locate the page title element'
+  );
+
+  await expect(pageTitleElement).toHaveText('Profile');
+});
+```
+
+## Contributing to Detox Copilot
+
+Contributions are welcome!
+Visit the [Detox Copilot GitHub Repository](https://github.com/wix-incubator/detox-copilot) to open issues or pull requests if they are relevant to the core-library functionality or open a it under [Detox repository](https://github.com/wix/Detox) if it is related to Detox-Copilot integration or if you are not sure where the issue should be opened.