Multi stage attack

.gitignore

@@ -89,3 +89,4 @@ report.xml
 cmake-build-*/
 */artifacts/
 /examples/chrome-data/
+/venv/

README.md

@@ -1,8 +1,15 @@
 # LLAMATOR
 
-## Description 📖
+Red Teaming python-framework for testing chatbots and LLM-systems
 
-Red teaming python-framework for testing vulnerabilities of chatbots based on large language models (LLM). Supports testing of Russian-language RAG systems.
+---
+
+[![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC_BY--NC--SA_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/llamator)](https://pypi.org/project/llamator)
+[![PyPI](https://badge.fury.io/py/llamator.svg)](https://badge.fury.io/py/llamator)
+[![Downloads](https://pepy.tech/badge/llamator)](https://pepy.tech/project/llamator)
+[![Downloads](https://pepy.tech/badge/llamator/month)](https://pepy.tech/project/llamator)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
 ## Install 🚀
 
@@ -26,19 +33,26 @@ Documentation Link: [https://romiconez.github.io/llamator](https://romiconez.git
 
 * 🌐 All LangChain clients
 * 🧠 OpenAI-like API
-* ⚙️ Custom Class (Telegram, Selenium, etc.)
+* ⚙️ Custom Class (Telegram, WhatsApp, Selenium, etc.)
 
 ## Unique Features 🌟
 
-* 🛡️ Support for custom attacks from the user
-* 📊 Results of launching each attack in CSV format
-* 📈 Report with attack requests and responses for all tests in Excel format
-* 📄 Test report document available in DOCX format
+* ️🗡 Support for custom attacks from the user
+* 👜 Large selection of attacks on RAG / Agent / Prompt in English and Russian
+* 🛡 Custom configuration of chat clients
+* 📊 History of attack requests and responses in Excel and CSV format
+* 📄 Test report document in DOCX format
+
+## OWASP Classification 🔒
+
+* 💉 [LLM01: Prompt Injection and Jailbreaks](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM01_PromptInjection.md)
+* 🕵 [LLM07: System Prompt Leakage](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM07_SystemPromptLeakage.md)
+* 🎭 [LLM09: Misinformation](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM09_Misinformation.md)
 
 ## License 📜
 
 © Roman Neronov, Timur Nizamov, Nikita Ivanov
 
 This project is licensed under the terms of the **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International** license. See the [LICENSE](LICENSE) file for details.
 
-[![Creative Commons License](https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
+[![Creative Commons License](https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png)](https://creativecommons.org/licenses/by-nc-sa/4.0/)

docs/howtos.md

@@ -8,7 +8,6 @@
 - **WhatsApp bot testing via Selenium** - [GitHub](https://github.com/RomiconEZ/llamator/blob/release/examples/llamator-whatsapp.ipynb)
 - **LangChain client testing with custom attack** - [GitHub](https://github.com/RomiconEZ/llamator/blob/release/examples/llamator-langchain-custom-attack.ipynb)
 
-
 ## Installation Guide
 
 ```bash

docs/project_overview.md

@@ -3,8 +3,11 @@
 LLAMATOR is a framework for testing vulnerabilities of chatbot systems and LLM.
 
 **Key Features**
-- Custom configuration of clients both for carrying out attacks and for testing
-- Large selection of attacks in English and Russian languages
-- Detailed testing results with reports in Excel format
+
+* ️🗡 Support for custom attacks from the user
+* 👜 Large selection of attacks (RAG / Agent / Prompt) in English and Russian
+* 🛡 Custom configuration of chat clients
+* 📊 History of attack requests and responses in Excel and CSV format
+* 📄 Test report document in DOCX format
 
 This project is designed for developers and researchers working in NLP and LLM domains.

src/llamator/attacks/base64_injection.py

@@ -90,7 +90,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant 'random_state' for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         # Generate list of attack prompt variations
         yield StatusUpdate(self.client_config, self.test_name, self.status, "Working", 0, self.num_attempts)

src/llamator/attacks/dan.py

@@ -71,7 +71,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant `random_state` for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         # Generate list of DAN (do anything now) attack prompt variations
         yield StatusUpdate(

src/llamator/attacks/ethical_compliance.py

@@ -108,7 +108,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant `random_state` for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         chat = ChatSession(self.attack_config.attack_client.get_target_client())
         for attack_prompt_index, row in data.iterrows():

src/llamator/attacks/harmful_behavior.py

@@ -86,7 +86,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant `random_state` for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         # Generate list of attack prompt variations
         yield StatusUpdate(self.client_config, self.test_name, self.status, "Generating", 0, self.num_attempts)

src/llamator/attacks/past_tense.py

@@ -67,7 +67,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant `random_state` for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         # Lists to store prompts, responses, and statuses for report generation
         attack_prompts = []

src/llamator/attacks/ru_dan.py

@@ -71,7 +71,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant `random_state` for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         # Generate list of DAN (do anything now) attack prompt variations
         yield StatusUpdate(

src/llamator/attacks/ru_ucar.py

@@ -72,7 +72,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant `random_state` for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         # Generate list of attack prompt variations
         yield StatusUpdate(

src/llamator/attacks/ucar.py

@@ -75,7 +75,11 @@ def run(self) -> Generator[StatusUpdate, None, None]:
             )  # constant `random_state` for better reproducibility
         else:
             # If dataset is smaller than requested number of attack samples, fit the dataset repeating until num_attempts is reached
-            data = pd.concat([data] * (self.num_attempts // len(data) + 1))[:self.num_attempts].sort_index().reset_index(drop=True)
+            data = (
+                pd.concat([data] * (self.num_attempts // len(data) + 1))[: self.num_attempts]
+                .sort_index()
+                .reset_index(drop=True)
+            )
 
         # Generate list of attack prompt variations
         yield StatusUpdate(