Merge pull request #2130 from Agenta-AI/mmabrouk/docs/AGE-874-custom-…

…applications-docs

docs(app): AGE-874 Custom workflows documentation

docs/docs/custom-workflows/01-overview.mdx

@@ -0,0 +1,60 @@
+---
+title: "Overview"
+description: "Learn about custom workflows with Agenta"
+---
+
+```mdx-code-block
+import Image from "@theme/IdealImage";
+
+```
+
+<Image
+  style={{ display: "block", margin: "10 auto", marginBottom: "20px" }}
+  img={require("/images/custom-workflows/custom-workflow-illustration.png")}
+  alt="Custom Workflows Overview"
+  loading="lazy"
+/>
+
+Custom workflows let you build a playground for any custom LLM workflow, such as RAG or agents. This playground replicates the logic of your real application, enabling subject matter experts to easily iterate on its parameters and evaluate it end-to-end from the web UI
+
+## The problem with traditional prompt playgrounds
+
+Traditional prompt playgrounds suffer from a major limitation: their output doesn't accurately represent the output of LLM applications. The reason is that LLM applications are more than single prompts—they involve workflows with multiple LLM calls, pre- and post-processing steps, and retrieval steps.
+
+As a result, subject matter experts find it hard to trust the outputs of the playground and make the collaboration with developers hard since they don't share the same context.
+
+## Custom workflows to the rescue
+
+Custom workflows solve this problem by allowing you to create custom playground for the workflow in your LLM application. This way you can accurately represent the output of the LLM applications and make the collaboration easy.
+
+With Agenta's Custom Workflows, you can:
+
+- **Collaborate with Subject Matter Experts**: Allow product teams to experiment with the real production workflow using the same codebase and logic as the live application.
+- **Evaluate the Entire Workflow**: Evaluate your workflow end-to-end directly from the web UI.
+- **Simplify Debugging**: View detailed traces of your calls directly in the playground, making debugging easier.
+- **Track Versions**: Keep version control of the entire application configuration, not just individual prompts.
+
+## How to get started
+
+You can get started quickly using our code templates—often in just a few minutes—or in under an hour when integrating with your existing codebase.
+
+To create a custom workflow, add a few lines to your existing codebase to define the workflow configuration and expose the entry points to Agenta.
+
+Refer to our quick start guide to begin.
+
+## How custom workflows work
+
+Custom workflows create an API endpoint between your code and Agenta, allowing Agenta to run the workflow.
+
+Agenta provides an SDK that makes creating the API endpoint as simple as adding a decorator to your existing code. The CLI takes care of serving the codebase and integrating it with Agenta.
+
+## FAQ
+
+- **How long does it take to set up?**  
+  You can usually get started in about an hour by integrating your existing code—less if you are starting from scratch or using a template.
+
+- **Do I need to host production application in Agenta?**  
+  No, you can keep your production application hosted on your premises and use Agenta for prompt engineering and evaluation. Your application will integrate with Agenta to fetch the latest configuration.
+
+- **Can I use Agenta with my existing framework / model?**  
+  Yes, Agenta works with any framework (e.g., Langchain, Llama Index, Haystack) and any model. The only current requirement is that the code for the workflow needs to be written in Python.

docs/docs/custom-workflows/02-quick-start.mdx

@@ -0,0 +1,237 @@
+---
+title: "Quick Start"
+description: "How to build a workflow with a chain of prompts using Agenta"
+---
+
+```mdx-code-block
+import Image from "@theme/IdealImage";
+
+```
+
+<Image
+  style={{ display: "block", margin: "10 auto", marginBottom: "20px" }}
+  img={require("/images/custom-workflows/workflow-cop.png")}
+  alt="Illustration of the workflow for the chain of prompts application"
+  loading="lazy"
+/>
+
+In this tutorial, you'll learn how to create a custom workflow with two prompts. By the end, you'll have a playground where you can edit and run the chain of prompts and evaluate the overall output.
+
+:::tip
+You can find the complete code for this tutorial [here](https://github.com/Agenta-AI/agenta/tree/main/examples/custom_workflows/chain_of_prompts/).
+:::
+
+## Custom workflows in Agenta
+
+Custom workflows are Python programs you can add to Agenta. Once added, you can use Agenta's playground to interact with them, run evaluations, deploy them, and monitor their performance, all through the Agenta webUI.
+
+You can create custom workflows by writing Python code and deploying them using the Agenta Command Line Interface (CLI).
+
+## 1. Writing the application
+
+We are creating a chain of prompt application. The application will take a blog post, the first prompt will summarize it, and the second prompt will write a tweet based on the summary. The highlighted lines are the ones related to Agenta.
+
+```python
+from openai import OpenAI
+from pydantic import BaseModel, Field
+# highlight-start
+import agenta as ag
+
+ag.init()
+# highlight-end
+
+client = OpenAI()
+prompt1 = "Summarize the following blog post: {blog_post}"
+prompt2 = "Write a tweet based on this: {output_1}"
+
+# highlight-start
+class CoPConfig(BaseModel):
+    prompt1: str = Field(default=prompt1)
+    prompt2: str = Field(default=prompt2)
+# highlight-end
+
+# highlight-next-line
+@ag.route("/", schema=CoPConfig)
+def generate(blog_post: str):
+    # highlight-next-line
+    config = ag.ConfigManager.get_from_route(schema=CoPConfig)
+    formatted_prompt1 = config.prompt1.format(blog_post=blog_post)
+    completion = client.chat.completions.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": formatted_prompt1}])
+    output_1 = completion.choices[0].message.content
+    formatted_prompt2 = config.prompt2.format(output_1=output_1)
+    completion = client.chat.completions.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": formatted_prompt2}])
+    return completion.choices[0].message.content
+```
+
+Let's take a look at the different parts of the code:
+
+### Initialization
+
+```python
+import agenta as ag
+
+ag.init()
+```
+
+Here, we initialize the Agenta SDK. `ag.init()` takes the environment variables `AGENTA_API_KEY` and `AGENTA_HOST` as arguments, which Agenta provides automatically when serving the application.
+
+### Workflow configuration
+
+```python
+class CoPConfig(BaseModel):
+    prompt1: str = Field(default=prompt1)
+    prompt2: str = Field(default=prompt2)
+```
+
+Each workflow has a configuration, which you can iterate on in the playground and version. In this case, the configuration includes the two prompts.
+
+Configurations are defined using [Pydantic](https://docs.pydantic.dev/) models. Each field in the model requires a default value. String fields are shown as text areas in the playground. You can also add other field types, such as integers, floats and booleans (which are shown in the playground as sliders and checkboxes).
+
+:::note
+For simplicity, we're using a simple Pydantic model with two prompts. In practice, you can use a more complex model that includes other parameters (model, temperature, top-k, etc.).
+:::
+
+### Specifying entry points
+
+```python
+@ag.route("/", config_schema=CoPConfig)
+def generate(blog_post: str):
+```
+
+Agenta uses the concept of entry points. Entry points are the functions Agenta uses to communicate with the code. Agenta creates an HTTP API for each entry point, which the playground and evaluation use to communicate with the code.
+
+The `schema` argument to the `@ag.route` decorator specifies the configuration the entry point expects. In this case, it expects a configuration with two prompts.
+
+### Using the configuration in the code
+
+```python
+config = ag.ConfigManager.get_from_route(schema=CoPConfig)
+```
+
+Finally, we modify the function to use the configuration provided by the endpoint. `ag.ConfigManager.get_from_route(schema=CoPConfig)` returns the configuration passed to the endpoint, which is provided by the playground or an evaluation.
+
+## 2. Deploying the Application
+
+### Setting up the folder structure
+
+Before serving the application in Agenta using the CLI, set up the folder structure.
+
+Create a `requirement.txt` file containing all the requirements. In this case, we need to add the Agenta and OpenAI SDKs.
+
+```python requirements.txt
+agenta
+openai
+```
+
+Add a `.env` file with any required environment variables. In this case, add the OpenAI API key.
+
+:::info
+We don't need to set the `AGENTA_API_KEY` environment variable since it's provided by Agenta automatically when serving the application.
+:::
+:::info
+We don't need to explicitly load the environment variables from the `.env` file. The Agenta SDK automatically loads the contents of the `.env` file.
+:::
+
+```bash .env
+OPENAI_API_KEY=sk-...
+```
+
+Both these files need to be in the same folder as the application code.
+
+### Serving the application
+
+To serve the application, initialize the project in Agenta. Run the following command in the folder containing the application code and necessary files.
+
+```bash
+agenta init
+```
+
+This command prompts for the application name, Agenta host (Agenta Cloud), and whether to start from a blank project (select "yes" since we wrote the code) or populate the folder with a template application (select "no" in this case).
+
+After running this command, a new `config.toml` file containing the application's configuration is created in the folder. Additionally, a new empty application is created in the Agenta web UI.
+
+Serve the application by running:
+
+```bash
+agenta variant serve myapp.py
+```
+
+This command serves the application in Agenta. The application is now added to the Agenta web interface and can be used from there.
+
+:::info
+Under the hood, this command builds an image for the application, deploys a container with the image, and exposes a REST API that Agenta uses to communicate.
+:::
+
+:::note
+When serving an application, all the files within the folder will be compressed and sent to the backend. You can create an `.agentaignore` file to ignore files and folders from being sent to the backend.
+:::
+
+## Using the application in Agenta
+
+The application should now be visible in Agenta. A new application variant is always created under the name `<filename>.default`. Variants are always named in the format `<filename>.<variant_name>`, allowing you to determine which source code was used to create the application (`<filename>`). When first created, we always generate a 'default' configuration.
+
+<Image
+  style={{ display: "block", margin: "10 auto" }}
+  img={require("/images/custom-workflows/playground-cop.png")}
+  alt="Screenshot of the playground for the chain of prompts application"
+  loading="lazy"
+/>
+
+## Adding observability (optional)
+
+If you've started using the application, you may have noticed that it's not automatically traced. We might want to add observability so that we can debug the application.
+
+Adding observability in custom workflows follows the same process as for applications running outside of Agenta. For more details, please refer to the [observability documentation](/observability/overview).
+
+As we'll be instrumenting the OpenAI client, we need to add the `opentelemetry.instrumentation.openai` package to the `requirements.txt` file.
+
+Here's how the updated code would look:
+
+```python cop.py
+from openai import OpenAI
+import agenta as ag
+from pydantic import BaseModel, Field
+# highlight-start
+from opentelemetry.instrumentation.openai import OpenAIInstrumentor
+# highlight-end
+
+ag.init()
+
+client = OpenAI()
+prompt1 = "Summarize the following blog post: {blog_post}"
+prompt2 = "Write a tweet based on this: {output_1}"
+
+# highlight-start
+OpenAIInstrumentor().instrument()
+# highlight-end
+
+class CoPConfig(BaseModel):
+    prompt1: str = Field(default=prompt1)
+    prompt2: str = Field(default=prompt2)
+
+@ag.route("/", config_schema=CoPConfig)
+# highlight-start
+@ag.instrument()
+# highlight-end
+def generate(blog_post: str):
+    config = ag.ConfigManager.get_from_route(schema=CoPConfig)
+    formatted_prompt1 = config.prompt1.format(blog_post=blog_post)
+    completion = client.chat.completions.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": formatted_prompt1}])
+    output_1 = completion.choices[0].message.content
+    formatted_prompt2 = config.prompt2.format(output_1=output_1)
+    completion = client.chat.completions.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": formatted_prompt2}])
+    return completion.choices[0].message.content
+```
+
+:::warning
+The `@ag.instrument()` decorator must be placed after the `@ag.route` decorator (called first).
+:::
+
+With these changes, we can now view the traces directly in the playground and debug the application.
+
+<Image
+  style={{ display: "block", margin: "10 auto" }}
+  img={require("/images/custom-workflows/trace-cop.png")}
+  alt="Screenshot of the playground for the chain of prompts application"
+  loading="lazy"
+/>

docs/docs/getting-started/01-introduction.mdx

@@ -20,7 +20,7 @@ management and evaluation**.
 
 ### With Agenta, you can:
 
-1. Rapidly [**experiment** and **compare** prompts](/prompt-management/overview) on [any LLM workflow](/prompt-management/creating-a-custom-template) (chain-of-prompts, Retrieval Augmented Generation (RAG), LLM agents...)
+1. Rapidly [**experiment** and **compare** prompts](/prompt-management/overview) on [any LLM workflow](/custom-workflows/overview) (chain-of-prompts, Retrieval Augmented Generation (RAG), LLM agents...)
 2. Rapidly [**create test sets**](/evaluation/create-test-sets) and **golden datasets** for evaluation
 3. **Evaluate** your application with pre-existing or **custom evaluators**
 4. **Annotate** and **A/B test** your applications with **human feedback**
@@ -35,7 +35,7 @@ Agenta focuses on increasing the speed of the development cycle of LLM applicati
 
 Agenta enables prompt engineering and evaluation on any LLM app architecture, such as **Chain of Prompts**, **RAG**, or **LLM agents**. It is compatible with any framework like **Langchain** or **LlamaIndex**, and works with any model provider, such as **OpenAI**, **Cohere**, or **local models**.
 
-[Jump here](/prompt-management/creating-a-custom-template) to see how to use your own custom application with Agenta and [here](/concepts/architecture) to understand more how Agenta works.
+[Jump here](/custom-workflows/overview) to see how to use your own custom application with Agenta and [here](/concepts/architecture) to understand more how Agenta works.
 
 ### Enable collaboration between developers and product teams
 

docs/docs/getting-started/02-quick-start.mdx

@@ -7,7 +7,7 @@ import Image from "@theme/IdealImage";
 
 :::note
 This tutorial helps users create LLM apps using templates within the UI. For more complex applications involving code
-in Agenta, please refer to Using code in Agenta [Using code in agenta](/prompt-management/creating-a-custom-template){" "}
+in Agenta, please refer to Using code in Agenta [Using code in agenta](//custom-workflows/overview){" "}
 :::
 
 Want a video tutorial instead? We have a 3-minute video for you. [Watch it here](https://youtu.be/N3B_ZOYzjLg).
@@ -49,7 +49,9 @@ Your OpenAI API keys are saved locally on your browser and are sent to the Agent
 1. Click on the "Create New App" button.
 2. Select "Create from Template" and choose the "Single Prompt Template"; then name the application "get_capital."
 
-<Image img={require("/images/getting-started-ui-screenshots/01_create_new_app.png")}/>
+<Image
+  img={require("/images/getting-started-ui-screenshots/01_create_new_app.png")}
+/>
 
 :::info
 {" "}
@@ -66,18 +68,21 @@ The provided application template already includes a prompt that states the capi
 - Click the "Run" button.
 - The result should read, "The capital of France is Paris." If you only want to return "Paris," we'll adjust this with prompt engineering in the next step.
 
-<Image img={require("/images/getting-started-ui-screenshots/03_playground.png")}/>
+<Image
+  img={require("/images/getting-started-ui-screenshots/03_playground.png")}
+/>
 
 ## Step 3: Deploy the application
 
 To deploy the application as an API. You need to click on the button "Publish" in the playground and select an environemnt.
 
 You can now find the API endpoint in the "Endpoints" menu. Copy and paste the code from the "Endpoints" menu to use it in your software.
 
-<Image img={require("/images/getting-started-ui-screenshots/06_deployment.png")}/>
+<Image
+  img={require("/images/getting-started-ui-screenshots/06_deployment.png")}
+/>
 
 :::info
 Congratulations! You've created your first LLM application. Feel free to modify it, explore its parameters, and discover
-Agenta's features. Your next steps could include [building an application using your own code](/prompt-management/creating-a-custom-template),
-or following one of our UI-based tutorials.
+Agenta's features. Your next steps could include [fetching prompts from Agenta](/prompt-management/quick-start), [building a custom workflow](/custom-workflows/quick-start), or [running an evaluation](/evaluation/overview).
 :::

docs/docs/prompt-management/02-quick-start.mdx

@@ -165,4 +165,4 @@ If you need to revert to a previously published commit, click on the deployment
 
 ## Next Steps
 
-Now that you've created and published your first prompt, you can learn how to do [prompt engineering in the playground](/prompt-management/using-the-playground) or dive deeper into [the capabilities of the prompt management SDK](/prompt-management/creating-a-custom-template).
+Now that you've created and published your first prompt, you can learn how to do [prompt engineering in the playground](/prompt-management/using-the-playground).