Skip to content

Commit

Permalink
[CONNECT] CommandPlugin et al.
Browse files Browse the repository at this point in the history
  • Loading branch information
jaceklaskowski committed Nov 30, 2024
1 parent e1537dd commit ff8c6d6
Show file tree
Hide file tree
Showing 5 changed files with 91 additions and 1 deletion.
21 changes: 20 additions & 1 deletion docs/connect/CommandPlugin.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
# CommandPlugin

`CommandPlugin` is...FIXME
`CommandPlugin` is an [abstraction](#contract) of [command processors](#implementations) in [Spark Connect](./index.md) that can [process a command](#process).

## Contract

### Process Command { #process }

```scala
process(
command: protobuf.Any,
planner: SparkConnectPlanner): Option[Unit]
```

Used when:

* `SparkConnectPlanner` is requested to [handleCommandPlugin](SparkConnectPlanner.md#handleCommandPlugin)

## Implementations

!!! note
No built-in implementations available.
12 changes: 12 additions & 0 deletions docs/connect/ExecuteHolder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# ExecuteHolder

## Creating Instance

`ExecuteHolder` takes the following to be created:

* <span id="request"> `ExecutePlanRequest`
* <span id="sessionHolder"> `SessionHolder`

`ExecuteHolder` is created when:

* `SparkConnectExecutionManager` is requested to [create an ExecuteHolder](SparkConnectExecutionManager.md#createExecuteHolder)
9 changes: 9 additions & 0 deletions docs/connect/ExecuteThreadRunner.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# ExecuteThreadRunner

`ExecuteThreadRunner` is created alongside [ExecuteHolder](ExecuteHolder.md#runner).

## Creating Instance

`ExecuteThreadRunner` takes the following to be created:

* <span id="executeHolder"> `ExecuteHolder`
9 changes: 9 additions & 0 deletions docs/connect/SparkConnectExecutionManager.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# SparkConnectExecutionManager

## Creating Instance

`SparkConnectExecutionManager` takes no arguments to be created.

`SparkConnectExecutionManager` is created when:

* `SparkConnectService` is requested for the [executionManager](SparkConnectService.md#executionManager)
41 changes: 41 additions & 0 deletions docs/connect/SparkConnectPlanner.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,3 +74,44 @@ transformPythonFunction(
`transformPythonFunction` is used when:

* `SparkConnectPlanner` is requested to [transformPythonUDF](#transformPythonUDF) and [handleRegisterPythonUDF](#handleRegisterPythonUDF)

## Process Command { #process }

```scala
process(
command: proto.Command,
responseObserver: StreamObserver[ExecutePlanResponse],
executeHolder: ExecuteHolder): Unit
```

`process` handles the input `Command` based on its type.

Command Type | Handler
-|-
`REGISTER_FUNCTION` | [handleRegisterUserDefinedFunction](#handleRegisterUserDefinedFunction)
`REGISTER_TABLE_FUNCTION` | [handleRegisterUserDefinedTableFunction](#handleRegisterUserDefinedTableFunction)
`WRITE_OPERATION` | [handleWriteOperation](#handleWriteOperation)
`CREATE_DATAFRAME_VIEW` | [handleCreateViewCommand](#handleCreateViewCommand)
`WRITE_OPERATION_V2` | [handleWriteOperationV2](#handleWriteOperationV2)
`EXTENSION` | [handleCommandPlugin](#handleCommandPlugin)
`SQL_COMMAND` | [handleSqlCommand](#handleSqlCommand)
`WRITE_STREAM_OPERATION_START` | [handleWriteStreamOperationStart](#handleWriteStreamOperationStart)
`STREAMING_QUERY_COMMAND` | [handleStreamingQueryCommand](#handleStreamingQueryCommand)
`STREAMING_QUERY_MANAGER_COMMAND` | [handleStreamingQueryManagerCommand](#handleStreamingQueryManagerCommand)
`GET_RESOURCES_COMMAND` | [handleGetResourcesCommand](#handleGetResourcesCommand)

---

`process` is used when:

* `ExecuteThreadRunner` is requested to [handle a command](ExecuteThreadRunner.md#handleCommand)

### handleCommandPlugin { #handleCommandPlugin }

```scala
handleCommandPlugin(
extension: ProtoAny,
executeHolder: ExecuteHolder): Unit
```

`handleCommandPlugin`...FIXME

0 comments on commit ff8c6d6

Please sign in to comment.