Skip to content

Commit

Permalink
Handle named wildcards (REST path parameters) (opensearch-project#123)
Browse files Browse the repository at this point in the history
* Register REST paths including named wildcards

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Move HelloWorld extension to subpackage

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Update Hello World example with named wildcard example

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Pass consumed params in RestResponse header

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Linelint hates me and web tool exports

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Code Review tweaks

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Update DESIGN.md

Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
Signed-off-by: Daniel Widdis <widdis@gmail.com>

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
  • Loading branch information
2 people authored and kokibas committed Mar 17, 2023
1 parent 3ceb56a commit c353eb2
Show file tree
Hide file tree
Showing 19 changed files with 659 additions and 97 deletions.
16 changes: 9 additions & 7 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@

Plugin architecture enables extending core features of OpenSearch. There are various kinds of plugins which are supported.
But, the architecture has significant problems for OpenSearch customers. Importantly, plugins can fatally impact the cluster
i.e critical workloads like ingestion/search traffic would be impacted because of a non-critical plugin like s3-repository failed with an exception.
i.e., critical workloads like ingestion/search traffic would be impacted because of a non-critical plugin like s3-repository failed with an exception.

This problem is exponentially grows when we would like to run a 3rd Party plugin from the community.
This problem exponentially grows when we would like to run a third Party plugin from the community.
As OpenSearch and plugins run in the same process, it brings in security risk, dependency conflicts and reduces the velocity of releases.

Introducing extensions, a simple and easy way to extend features of OpenSearch. It would support all plugin features and enable them to run in a seperate process or on another node via OpenSearch SDK Java.
Introducing extensions, a simple and easy way to extend features of OpenSearch. It would support all plugin features and enable them to run in a seperate process or on another node via OpenSearch SDK for Java (other SDKs will be developed).

Meta Issue: [Steps to make OpenSearch extensible](https://github.com/opensearch-project/OpenSearch/issues/2447)
Sandboxing: [Step towards modular architecture in OpenSearch](https://github.com/opensearch-project/OpenSearch/issues/1422)
Expand Down Expand Up @@ -84,7 +84,7 @@ The `org.opensearch.sdk.sample` package contains a sample `HelloWorldExtension`

(2, 3, 4) Using the `ExtensionSettings` from the extension, the `ExtensionsRunner` binds to the configured host and port.

(5, 6, 7) Using the `List<ExtensionRestHandler>` from the extension, the `ExtensionsRunner` stores each handler (Rest Action)'s restPath (method+URI) in a map, identifying the action to execute when that combination is received by the extension.
(5, 6, 7) Using the `List<ExtensionRestHandler>` from the extension, the `ExtensionsRunner` stores each handler (Rest Action)'s restPath (method+URI) in the `ExtensionRestPathRegistry`, identifying the action to execute when that combination is received by the extension. This registry internally uses the same `PathTrie` implementation as OpenSearch's `RestController`.

##### OpenSearch Startup, Extension Initialization, and REST Action Registration

Expand All @@ -94,7 +94,7 @@ The `ExtensionsOrchestrator` reads a list of extensions present in `extensions.y

(11, 12) The `ExtensionsOrchestrator` Initializes the extension using an `InitializeExtensionRequest`/`Response`, establishing the two-way transport mechanism.

(13) Each `Extension` retrieves all REST paths from its pathMap (the key set).
(13) Each `Extension` retrieves all of its REST paths from its `ExtensionRestPathRegistry`.

(14, 15, 16) Each `Extension` sends a `RegisterRestActionsRequest` to the `RestActionsRequestHandler`, which registers a `RestSendToExtensionAction` with the `RestController` to handle each REST path (`Route`). These routes rely on a globally unique identifier for the extension which users will use in REST requests, presently the Extension's `uniqueId`.

Expand All @@ -110,9 +110,11 @@ The `ExtensionsOrchestrator` reads a list of extensions present in `extensions.y

(21, 22) The appropriate `ExtensionRestHandler` handles the request, possibly executing complex logic, and eventually providing a response string.

(23, 24) The response string is relayed by the `Extension` to the `RestActionsRequestHandler` which uses it to complete the `RestSendToExtensionAction` by returning a `BytesRestResponse`.
(23, 24) As part of handling some requests, additional actions, such as creating an index, may require further interactions with OpenSearch's `RestController` which are accomplished via the `SDKClient` as required.

(25) The User receives the response.
(25, 26) The response string is relayed by the `Extension` to the `RestActionsRequestHandler` which uses it to complete the `RestSendToExtensionAction` by returning a `BytesRestResponse`.

(27) The User receives the response.

## FAQ

Expand Down
28 changes: 16 additions & 12 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@
- [Run the Sample Extension](#run-the-sample-extension)
- [Create extensions.yml file](#create-extensions-yml-file)
- [Run OpenSearch](#run-opensearch)
- [Publish OpenSearch-SDK to Maven Local](#publish-opensearch-sdk-to-maven-local)
- [Publish OpenSearch SDK for Java to Maven Local](#publish-opensearch-sdk-for-java-to-maven-local)
- [Perform a REST Request on the Extension](#perform-a-rest-request-on-the-extension)
- [Run Tests](#run-tests)
- [Submitting Changes](#submitting-changes)

## Introduction
Opensearch plugins have allowed the extension and ehancements of various core features however, current plugin architecture carries the risk of fatally impacting clusters should they fail. In order to ensure that plugins may run safely without impacting the system, our goal is to effectively isolate plugin interactions with OpenSearch by modularizing the [extension points](https://opensearch.org/blog/technical-post/2021/12/plugins-intro/) to which they hook onto.
OpenSearch plugins have allowed the extension and enhancements of various core features. However, the current plugin architecture carries the risk of fatally impacting clusters should they fail. In order to ensure that plugins may run safely without impacting the system, our goal is to effectively isolate plugin interactions with OpenSearch by modularizing the [extension points](https://opensearch.org/blog/technical-post/2021/12/plugins-intro/) to which they hook onto.

Read more about extensibility [here](https://github.com/opensearch-project/OpenSearch/issues/1422)

Expand All @@ -24,40 +24,42 @@ Read more about extensibility [here](https://github.com/opensearch-project/OpenS
Fork [OpenSearch SDK for Java](https://github.com/opensearch-project/opensearch-sdk-java) and clone locally, e.g. `git clone https://github.com/[your username]/opensearch-sdk-java.git`.

### Git Clone OpenSearch Repo
Fork [OpenSearch](https://github.com/opensearch-project/OpenSearch/), checkout feature/extensions branch, and clone locally, e.g. `git clone https://github.com/[your username]/OpenSearch.git`.
Fork [OpenSearch](https://github.com/opensearch-project/OpenSearch/), clone locally, e.g., `git clone https://github.com/[your username]/OpenSearch.git`, and checkout the `feature/extensions` branch.

## Publish OpenSearch feature/extensions Branch to Maven local
The work done to support the extensions framework is located on the `feature/extensions` branch of the OpenSearch project. It is necessary to publish the dependencies of this branch to your local maven repository prior to running OpenSearch SDK for Java on a seperate process.
The work done to support the extensions framework is located on the `feature/extensions` branch of the OpenSearch project. Until this branch is merged to `main`, it is necessary to publish the dependencies of this branch to your local maven repository prior to running an Extension on a separate process.

- First navigate to the directory that OpenSearch has been cloned to
- Checkout the correct branch, e.g. `git checkout feature/extensions`.
- Run `./gradlew publishToMavenLocal`.
- Run `./gradlew check` to make sure the build is successful.

It is necessary to publish dependencies to a local maven repository until this branch is merged to `main`, at which point all dependencies will be published to Maven central.
- Run `./gradlew publishToMavenLocal`.

## Run the Sample Extension

Navigate to the directory that OpenSearch-SDK-Java has been cloned to and run the Sample Extension's main method using `./gradlew run`.
Navigate to the directory that OpenSearch-SDK-Java has been cloned to.

You can execute just the SDK's `ExtensionsRunner` main method with test settings using `./gradlew run`.

```
./gradlew run
```

This will execute the main script set within the root `build.gradle` file :
You can execute the sample Hello World extension using the `helloWorld` task:

```
mainClassName = 'transportservice.ExtensionsRunner'
./gradlew helloWorld
```

Bound addresses will then be logged to the terminal :

```bash
[main] INFO transportservice.TransportService - publish_address {127.0.0.1:3333}, bound_addresses {[::1]:3333}, {127.0.0.1:3333}
[main] INFO transportservice.TransportService - profile [test]: publish_address {127.0.0.1:5555}, bound_addresses {[::1]:5555}, {127.0.0.1:5555}
```

## Publish OpenSearch-SDK to Maven local
Until we publish this repo to maven central. Publishing to maven local is the way to import the artifacts
## Publish OpenSearch SDK for Java to Maven local

Until we publish this repo to maven central, publishing to maven local is the way for plugins (outside the sample packages) to import the artifacts:
```
./gradlew publishToMavenLocal
```
Expand Down Expand Up @@ -111,6 +113,7 @@ During OpenSearch bootstrap, `ExtensionsOrchestrator` will then discover the ext
OpenSearch SDK terminal will also log all requests and responses it receives from OpenSearch :

TCP HandShake Request :

```
21:30:18.943 [opensearch[extension][transport_worker][T#7]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0x37b22600, L:/127.0.0.1:4532 - R:/127.0.0.1:47766] READ: 55B
+-------------------------------------------------+
Expand All @@ -125,6 +128,7 @@ MESSAGE RECEIVED:E«󀀀internal:tcp/handshake£·A
```

Extension Name Request / Response :

```
21:30:18.992 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] READ: 204B
+-------------------------------------------------+
Expand Down
2 changes: 1 addition & 1 deletion Docs/ExtensionRestActions.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
16 changes: 10 additions & 6 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ apply plugin: 'application'
apply from: 'gradle/formatting.gradle'
apply plugin: 'maven-publish'

mainClassName = 'org.opensearch.sdk.sample.HelloWorldExtension'

mainClassName = 'org.opensearch.sdk.ExtensionsRunner'
group 'org.opensearch.sdk'
version '1.0.0-SNAPSHOT'

Expand Down Expand Up @@ -98,13 +98,17 @@ task javadocStrict(type: Javadoc) {
classpath = sourceSets.main.runtimeClasspath
options.addStringOption('Xdoclint:all', '-quiet')
options.memberLevel = JavadocMemberLevel.PRIVATE

// the netty4 package will eventually be published to mavenCentral
// See https://github.com/opensearch-project/OpenSearch/issues/3118
exclude 'org/opensearch/sdk/netty4'
}
check.dependsOn javadocStrict

// this task runs the helloworld sample extension
task helloWorld(type: JavaExec) {
group = 'Execution'
description = 'Run HelloWorld Extension.'
mainClass = 'org.opensearch.sdk.sample.helloworld.HelloWorldExtension'
classpath = sourceSets.main.runtimeClasspath
}

test {
useJUnitPlatform()
jvmArgs '--enable-preview'
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/org/opensearch/sdk/ExtensionRestHandler.java
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
import org.opensearch.rest.RestHandler.Route;
import org.opensearch.rest.RestRequest;
import org.opensearch.rest.RestRequest.Method;
import org.opensearch.rest.RestResponse;

/**
* This interface defines methods which an extension REST handler (action) must provide.
Expand All @@ -31,10 +30,11 @@ public interface ExtensionRestHandler {
* Handles REST Requests forwarded from OpenSearch for a configured route on an extension.
* Parameters are components of the {@link RestRequest} received from a user.
* This method corresponds to the {@link BaseRestHandler#prepareRequest} method.
* As in that method, consumed parameters must be tracked and returned in the response.
*
* @param method A REST method.
* @param uri The URI to handle.
* @return A {@link RestResponse} to the request.
* @return An {@link ExtensionRestResponse} to the request.
*/
RestResponse handleRequest(Method method, String uri);
ExtensionRestResponse handleRequest(Method method, String uri);
}
70 changes: 70 additions & 0 deletions src/main/java/org/opensearch/sdk/ExtensionRestPathRegistry.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/
package org.opensearch.sdk;

import java.util.ArrayList;
import java.util.List;

import org.opensearch.common.path.PathTrie;
import org.opensearch.rest.RestUtils;
import org.opensearch.rest.RestRequest.Method;

/**
* This class registers REST paths from extension Rest Handlers.
*/
public class ExtensionRestPathRegistry {

// PathTrie to match paths to handlers
private PathTrie<ExtensionRestHandler> pathTrie = new PathTrie<>(RestUtils.REST_DECODER);
// List to return registered handlers
private List<String> registeredPaths = new ArrayList<>();

/**
* Register a REST handler to handle a method and route in this extension's path registry.
*
* @param method The method to register.
* @param uri The URI to register. May include named wildcards.
* @param extensionRestHandler The RestHandler to handle this route
*/
public void registerHandler(Method method, String uri, ExtensionRestHandler extensionRestHandler) {
String restPath = restPathToString(method, uri);
pathTrie.insert(restPath, extensionRestHandler);
registeredPaths.add(restPath);
}

/**
* Get the registered REST handler for the specified method and URI.
*
* @param method the registered method.
* @param uri the registered URI.
* @return The REST handler registered to handle this method and URI combination if found, null otherwise.
*/
public ExtensionRestHandler getHandler(Method method, String uri) {
return pathTrie.retrieve(restPathToString(method, uri));
}

/**
* List the registered routes.
*
* @return A list of strings identifying the registered routes.
*/
public List<String> getRegisteredPaths() {
return registeredPaths;
}

/**
* Converts a REST method and URI to a string.
*
* @param method the method.
* @param uri the URI.
* @return A string appending the method and URI.
*/
public static String restPathToString(Method method, String uri) {
return method.name() + " " + uri;
}
}
93 changes: 93 additions & 0 deletions src/main/java/org/opensearch/sdk/ExtensionRestResponse.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/
package org.opensearch.sdk;

import java.util.List;

import org.opensearch.common.bytes.BytesReference;
import org.opensearch.common.xcontent.XContentBuilder;
import org.opensearch.rest.BytesRestResponse;
import org.opensearch.rest.RestStatus;

/**
* A subclass of {@link BytesRestResponse} which processes the consumed parameters into a custom header.
*/
public class ExtensionRestResponse extends BytesRestResponse {

/**
* Key passed in {@link BytesRestResponse} headers to identify parameters consumed by the handler. For internal use.
*/
static final String CONSUMED_PARAMS_KEY = "extension.consumed.parameters";

/**
* Creates a new response based on {@link XContentBuilder}.
*
* @param status The REST status.
* @param builder The builder for the response.
* @param consumedParams Parameters consumed by the handler.
*/
public ExtensionRestResponse(RestStatus status, XContentBuilder builder, List<String> consumedParams) {
super(status, builder);
addConsumedParamHeader(consumedParams);
}

/**
* Creates a new plain text response.
*
* @param status The REST status.
* @param content A plain text response string.
* @param consumedParams Parameters consumed by the handler.
*/
public ExtensionRestResponse(RestStatus status, String content, List<String> consumedParams) {
super(status, content);
addConsumedParamHeader(consumedParams);
}

/**
* Creates a new plain text response.
*
* @param status The REST status.
* @param contentType The content type of the response string.
* @param content A response string.
* @param consumedParams Parameters consumed by the handler.
*/
public ExtensionRestResponse(RestStatus status, String contentType, String content, List<String> consumedParams) {
super(status, contentType, content);
addConsumedParamHeader(consumedParams);
}

/**
* Creates a binary response.
*
* @param status The REST status.
* @param contentType The content type of the response bytes.
* @param content Response bytes.
* @param consumedParams Parameters consumed by the handler.
*/
public ExtensionRestResponse(RestStatus status, String contentType, byte[] content, List<String> consumedParams) {
super(status, contentType, content);
addConsumedParamHeader(consumedParams);
}

/**
* Creates a binary response.
*
* @param status The REST status.
* @param contentType The content type of the response bytes.
* @param content Response bytes.
* @param consumedParams Parameters consumed by the handler.
*/
public ExtensionRestResponse(RestStatus status, String contentType, BytesReference content, List<String> consumedParams) {
super(status, contentType, content);
addConsumedParamHeader(consumedParams);
}

private void addConsumedParamHeader(List<String> consumedParams) {
consumedParams.stream().forEach(p -> addHeader(CONSUMED_PARAMS_KEY, p));
}
}
Loading

0 comments on commit c353eb2

Please sign in to comment.