Skip to content

Commit

Permalink
en
Browse files Browse the repository at this point in the history
Signed-off-by: spacewander <spacewanderlzx@gmail.com>
  • Loading branch information
spacewander committed Jul 15, 2024
1 parent 8320f4d commit de88907
Show file tree
Hide file tree
Showing 2 changed files with 125 additions and 2 deletions.
122 changes: 122 additions & 0 deletions site/content/en/docs/developer-guide/get_involved.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,125 @@
---
title: How to develop HTNN to fit your purpose
---

HTNN's functional code is mainly located in the following modules:

* api/: This is the most basic module. It is referenced by other modules and provides interfaces needed to develop HTNN plugins.
* types/: Modules such as the control plane, data plane, and console all depend on this module to provide plugin metadata, CRD definitions, and other internal common data.
* controller/: Control plane
* plugins/: Data plane (plugin hub)

For users who want to develop their functionalities, it is generally only necessary to refer to the implementations in `controller/` and `plugins/`.

## Controller

HTNN in the `controller/` module mainly does the following:

* Reconcile HTNN's CRD
* Provide a Native Plugin framework to allow modifications to the xDS sent to the data plane through plugins.
* Provide a Service Registry mechanism to integrate the external service discovery systems in the form of registries, converting them into upstream information within the data plane.

Plugin and Registry are closely related to developers. It is advisable to read the code in the `plugins/` and `registries/` directories to see how to develop functionalities using HTNN's interfaces.

Traditionally, there are two ways to alter the behavior of Istio:

1. Translate your resources into EnvoyFilter and write them into k8s to let Istio process the EnvoyFilter.
2. Become an MCP server yourself. Reconcile your own resources, then deliver the results to Istio through the MCP protocol.

HTNN does not use the methods above but instead modifies Istio to embed its own process into Istio. The specific changes can be referred to in the `patch/istio` directory's patch files. The reason for not using method 1 is that writing stateful EnvoyFilter leads to observability and high availability becoming more challenging and prone to problems. Method 2 is not used because to ensure the consistency of network policy and network routing, the network routing part also needs to be integrated into the MCP server, otherwise, there will be a situation where the policy is still reconciling and the routing has been published online. Providing network routing configurations in MCP server would undoubtedly be a significant workload.

If you develop your own Native Plugin or Service Registry in a third-party repository and want to use it in HTNN, you can use a patch to introduce that repository into Istio. For details, refer to this patch: https://github.com/mosn/htnn/blob/main/patch/istio/1.21/20240410-htnn-go-mod.patch. When importing the repository, make sure to place it after the HTNN official plugin package to avoid being overwritten by plugins with the same name.

If you want to run the HTNN control plane, you can refer to the implementation of `make e2e-prepare-controller-image` in the `e2e/` directory to see how to package the Istio embedded with HTNN into an image.

## Plugins

HTNN places data plane-running Go Plugins and their shared pkg libraries in the `plugins/` module.

If you want to write a Go Plugin in a third-party repository, you can refer to this example: https://github.com/mosn/htnn/tree/main/examples/dev_your_plugin. Go Plugins are compiled into shared libraries and deployed to the data plane. When you want to integrate a Go Plugin into the data plane, this file can serve as a template: https://github.com/mosn/htnn/blob/main/examples/dev_your_plugin/cmd/libgolang/main.go.

It is recommended to read the existing plugins' code before developing your own plugin, especially plugins that are similar to the one you're developing – at the very least, you should look at the [demo](https://github.com/mosn/htnn/tree/main/plugins/plugins/demo) plugin. Plugin code is located in the `plugins/` directory. For more information on plugin development, please refer to the [Plugin Development](./plugin_development) documentation.

HTNN provides a [Plugin Integration Test Framework](./plugin_integration_test_framework) that allows the testing of Go Plugin logic to run with only the data plane running. The `dev_your_plugin` example also demonstrates how to run the integration test framework in a third-party repository.

Each plugin contains two types of objects: `config` and `filter`. Where `config` is responsible for configuration management and `filter` is responsible for executing request-level logic.

### Config

In plugin development, features closely related to configuration should all be placed in `config`.

The granularity of `config` can be per-route, per-gateway, or per-consumer, depending on the scope of the plugin configuration. For example, configuring the Gateway-level `limitCountRedis`:

```yaml
apiVersion: htnn.mosn.io/v1
kind: FilterPolicy
metadata:
name: policy
namespace: istio-system
spec:
targetRef:
group: networking.istio.io
kind: Gateway
name: default
filters:
limitCountRedis:
config:
address: "redis:6379"
rules:
- count: 1
timeWindow: "60s"
```
When this occurs, a `limitCountRedis.config` object is created and shared among various routes in the Gateway `default`.

The lifecycle of `config` depends on the plugin configuration itself and the object it acts upon. For example, if we configure `limitCountRedis` on route `vs`:

```yaml
apiVersion: htnn.mosn.io/v1
kind: FilterPolicy
metadata:
name: policy
namespace: istio-system
spec:
targetRef:
group: networking.istio.io
kind: VirtualService
name: vs
filters:
limitCountRedis:
config:
address: "redis:6379"
rules:
- count: 1
timeWindow: "60s"
```

In two cases, the `limitCountRedis.config` object will be recreated:
1. Any changes to the routing under any Gateway that `vs` belongs to
2. Any changes into the specification of FilterPolicy that points to routes mentioned in the previous case

This is because new [RouteConfiguration](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route.proto#envoy-v3-api-msg-config-route-v3-routeconfiguration) in the above cases causes the `limitCountRedis.config` object to be re-created.

(TODO: Support incremental update of routes so that only changes to `vs` itself cause the object to be re-created)

As `config` is shared across multiple requests in different Envoy workers, read and write operations to `config` must be locked.

### Filter

In plugin development, features related to each request should be placed in `filter`. Each request creates a `filter` object for each plugin that is executed.

`filter` mainly defines the following methods:

1. DecodeHeaders
2. DecodeData
3. EncodeHeaders
4. EncodeData
5. OnLog

Normally, the above methods are executed from top to bottom. However, there are exceptions:

1. If there is no body, the corresponding DecodeData and EncodeData methods will not be executed.
2. Since the OnLog operation is triggered by the client interrupting the request, OnLog may execute concurrently with other methods if the client interrupts prematurely.
3. In requests like bidirectional streams, it is possible to handle the request body and upstream response at the same time, so DecodeData may execute concurrently with EncodeHeaders or EncodeData.

So, when reading and writing `filter`, there is a risk of concurrent access and lock consideration is necessary.
5 changes: 3 additions & 2 deletions site/content/zh-hans/docs/developer-guide/get_involved.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ HTNN 在 `controller/` 模块主要做了下面的事:
* 提供 Native Plugin 框架,允许以插件的方式修改发送给数据面的 xDS
* 提供 Service Registry 机制,允许以 registry 的方式对接外部服务发现系统,转换成数据面内的上游信息

其中和开发者紧密相关的是 Plugin 和 Registry建议阅读 `plugins/``registries/` 两个目录下的代码,看看如何使用 HTNN 提供的接口开发功能。
其中和开发者紧密相关的是 Plugin 和 Registry建议阅读 `plugins/``registries/` 两个目录下的代码,看看如何使用 HTNN 提供的接口开发功能。

传统上想要修改 istio 的行为有两种方法,

1. 将自己的资源翻译成 EnvoyFilter 写入到 k8s 里,让 istio 处理 EnvoyFilter。
2. 让自己成为 MCP server。调和自己的资源,再把结果通过 MCP 协议给 istio。

Expand All @@ -39,7 +40,7 @@ HTNN 在 `plugins/` 模块下放置在数据面上运行的 Go Plugin 以及它

在开发自己的插件之前建议读一下现有插件的代码,尤其是和你开发的功能相似的插件,至少应该看看 [demo](https://github.com/mosn/htnn/tree/main/plugins/plugins/demo) 插件。插件代码位于 `plugins/` 目录下面。关于插件开发的更多信息,请参考[插件开发](./plugin_development)文档。

HTNN 提供了一个[插件集成测试框架](./plugin_integration_test_framework),允许在只运行数据面的情况下测试 Go Plugin 的逻辑。在 dev_your_plugin 这个范例里也展示了如何在第三方仓库中运行集成测试框架。
HTNN 提供了一个[插件集成测试框架](./plugin_integration_test_framework),允许在只运行数据面的情况下测试 Go Plugin 的逻辑。在 `dev_your_plugin` 这个范例里也展示了如何在第三方仓库中运行集成测试框架。

每个插件都包含两类对象:`config``filter`。其中 `config` 负责配置管理,`filter` 负责执行请求级别的逻辑。

Expand Down

0 comments on commit de88907

Please sign in to comment.