Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: more standard test stubs #28202

Merged
merged 11 commits into from
Nov 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/docs/contributing/how_to/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,8 @@

- [**Documentation**](documentation/index.mdx): Help improve our docs, including this one!
- [**Code**](code/index.mdx): Help us write code, fix bugs, or improve our infrastructure.
- [**Integrations**](integrations/index.mdx): Help us integrate with your favorite vendors and tools.

## Integrations

- [**Start Here**](integrations/index.mdx): Help us integrate with your favorite vendors and tools.
- [**Standard Tests**](integrations/standard_tests): Ensure your integration passes an expected set of tests.
10 changes: 4 additions & 6 deletions docs/docs/contributing/how_to/integrations/community.mdx
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
## How to add a community integration (deprecated guide)
## How to add a community integration (not recommended)

:::danger

We are no longer accepting new community integrations. Please see the
[main integration guide](./index.mdx) for more information on contributing new
integrations.
We recommend following the [main integration guide](./index.mdx) to add new integrations instead.

Note that `langchain-community` is **not** deprecated. Only
the process for adding new community integrations is deprecated.
If you follow this guide, there is a high likelihood we will close your PR with the above
guide linked without much discussion.

:::

Expand Down
6 changes: 3 additions & 3 deletions docs/docs/contributing/how_to/integrations/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,10 @@ that will render on this site (https://python.langchain.com/).

As a prerequisite to adding your integration to our documentation, you must:

1. Confirm that your integration is in the list of components we are currently accepting.
1. Confirm that your integration is in the [list of components](#components-to-integrate) we are currently accepting.
2. Ensure that your integration is in a separate package that can be installed with `pip install <your-package>`.
3. Implement the standard tests for your integration and successfully run them.
3. Write documentation for your integration in the `docs/docs/integrations` directory of the LangChain monorepo.
3. [Implement the standard tests](/docs/contributing/how_to/integrations/standard_tests) for your integration and successfully run them.
3. Write documentation for your integration in the `docs/docs/integrations/<component_type>` directory of the LangChain monorepo.
4. Add a provider page for your integration in the `docs/docs/integrations/providers` directory of the LangChain monorepo.

Once you have completed these steps, you can submit a PR to the LangChain monorepo to add your integration to the documentation.
Expand Down
143 changes: 122 additions & 21 deletions docs/docs/contributing/how_to/integrations/standard_tests.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,14 @@
"source": [
"# How to add standard tests to an integration\n",
"\n",
"Implementing standard tests \n",
"\n",
"When creating either a custom class for yourself or a new tool to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to a tool, and the templates for implementing each different kind of integration are linked [at the bottom](#standard-test-templates-per-component).\n",
"When creating either a custom class for yourself or a new tool to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to a tool, and you can **[Skip to the test templates](#standard-test-templates-per-component)** for implementing tests for each integration.\n",
"\n",
"## Setup\n",
"\n",
"First, let's install 2 dependencies:\n",
"\n",
"- `langchain-core` will define the interfaces we want to import to define our custom tool.\n",
"- `langchain-tests==0.3.0` will provide the standard tests we want to use.\n",
"- `langchain-tests==0.3.2` will provide the standard tests we want to use.\n",
"\n",
":::note\n",
"\n",
Expand All @@ -26,7 +24,7 @@
"or in a new release of `langchain-tests`.\n",
"\n",
"Because added tests in new versions of `langchain-tests` will always break your CI/CD pipelines, we recommend pinning the \n",
"version of `langchain-tests==0.3.0` to avoid unexpected changes.\n",
"version of `langchain-tests==0.3.2` to avoid unexpected changes.\n",
"\n",
":::"
]
Expand All @@ -37,7 +35,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install -U langchain-core langchain-tests==0.3.0 pytest pytest-socket"
"%pip install -U langchain-core langchain-tests==0.3.2 pytest pytest-socket"
]
},
{
Expand Down Expand Up @@ -223,7 +221,11 @@
"class TestChatParrotLinkUnit(ChatModelUnitTests):\n",
" @property\n",
" def chat_model_class(self) -> Type[ChatParrotLink]:\n",
" return ChatParrotLink"
" return ChatParrotLink\n",
"\n",
" @property\n",
" def chat_model_params(self) -> dict:\n",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test comment

" return {\"model\": \"bird-brain-001\", \"temperature\": 0}"
]
},
{
Expand Down Expand Up @@ -254,22 +256,54 @@
"metadata": {},
"source": [
"</details>\n",
"\n",
"<div style={{display:\"none\"}}>\n",
"Work in progress:\n",
"<details>\n",
" <summary>Retrievers</summary>\n",
" TODO"
" <summary>Embedding Models</summary>"
]
},
{
"cell_type": "markdown",
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"</details>\n",
"<details>\n",
" <summary>Vector Stores</summary>\n",
" TODO"
"# title=\"tests/unit_tests/test_embeddings.py\"\n",
"from typing import Tuple, Type\n",
"\n",
"from langchain_parrot_link.embeddings import ParrotLinkEmbeddings\n",
"from langchain_standard_tests.unit_tests import EmbeddingsUnitTests\n",
"\n",
"\n",
"class TestParrotLinkEmbeddingsUnit(EmbeddingsUnitTests):\n",
" @property\n",
" def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:\n",
" return ParrotLinkEmbeddings\n",
"\n",
" @property\n",
" def embedding_model_params(self) -> dict:\n",
" return {\"model\": \"nest-embed-001\", \"temperature\": 0}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# title=\"tests/integration_tests/test_embeddings.py\"\n",
"from typing import Type\n",
"\n",
"from langchain_parrot_link.embeddings import ParrotLinkEmbeddings\n",
"from langchain_standard_tests.integration_tests import EmbeddingsIntegrationTests\n",
"\n",
"\n",
"class TestParrotLinkEmbeddingsIntegration(EmbeddingsIntegrationTests):\n",
" @property\n",
" def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:\n",
" return ParrotLinkEmbeddings\n",
"\n",
" @property\n",
" def embedding_model_params(self) -> dict:\n",
" return {\"model\": \"nest-embed-001\", \"temperature\": 0}"
]
},
{
Expand All @@ -278,16 +312,83 @@
"source": [
"</details>\n",
"<details>\n",
" <summary>Embedding Models</summary>\n",
" TODO"
" <summary>Tools/Toolkits</summary>\n",
" Note: The standard tests for tools/toolkits are implemented in the example in the main body of this guide too."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# title=\"tests/unit_tests/test_tools.py\"\n",
"from typing import Type\n",
"\n",
"from langchain_parrot_link.tools import ParrotMultiplyTool\n",
"from langchain_standard_tests.unit_tests import ToolsUnitTests\n",
"\n",
"\n",
"class TestParrotMultiplyToolUnit(ToolsUnitTests):\n",
" @property\n",
" def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n",
" return ParrotMultiplyTool\n",
"\n",
" def tool_constructor_params(self) -> dict:\n",
" # if your tool constructor instead required initialization arguments like\n",
" # `def __init__(self, some_arg: int):`, you would return those here\n",
" # as a dictionary, e.g.: `return {'some_arg': 42}`\n",
" return {}\n",
"\n",
" def tool_invoke_params_example(self) -> dict:\n",
" \"\"\"\n",
" Returns a dictionary representing the \"args\" of an example tool call.\n",
"\n",
" This should NOT be a ToolCall dict - i.e. it should not\n",
" have {\"name\", \"id\", \"args\"} keys.\n",
" \"\"\"\n",
" return {\"a\": 2, \"b\": 3}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# title=\"tests/integration_tests/test_tools.py\"\n",
"from typing import Type\n",
"\n",
"from langchain_parrot_link.tools import ParrotMultiplyTool\n",
"from langchain_standard_tests.integration_tests import ToolsIntegrationTests\n",
"\n",
"\n",
"class TestParrotMultiplyToolIntegration(ToolsIntegrationTests):\n",
" @property\n",
" def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n",
" return ParrotMultiplyTool\n",
"\n",
" def tool_constructor_params(self) -> dict:\n",
" # if your tool constructor instead required initialization arguments like\n",
" # `def __init__(self, some_arg: int):`, you would return those here\n",
" # as a dictionary, e.g.: `return {'some_arg': 42}`\n",
" return {}\n",
"\n",
" def tool_invoke_params_example(self) -> dict:\n",
" \"\"\"\n",
" Returns a dictionary representing the \"args\" of an example tool call.\n",
"\n",
" This should NOT be a ToolCall dict - i.e. it should not\n",
" have {\"name\", \"id\", \"args\"} keys.\n",
" \"\"\"\n",
" return {\"a\": 2, \"b\": 3}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"</details>\n",
"</div>"
"</details>"
]
}
],
Expand Down
1 change: 1 addition & 0 deletions docs/docs/contributing/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ More coming soon! We are working on tutorials to help you make your first contri
- [**Documentation**](how_to/documentation/index.mdx): Help improve our docs, including this one!
- [**Code**](how_to/code/index.mdx): Help us write code, fix bugs, or improve our infrastructure.
- [**Integrations**](how_to/integrations/index.mdx): Help us integrate with your favorite vendors and tools.
- [**Standard Tests**](how_to/integrations/standard_tests): Ensure your integration passes an expected set of tests.

## Reference

Expand Down
Loading