Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[doc] add create functions with package URL related docs #11810

Merged
merged 2 commits into from
Aug 30, 2021
Merged

[doc] add create functions with package URL related docs #11810

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f10a915
add package URL related docs
freeznet Aug 27, 2021
3b406e3
address comments
freeznet Aug 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,16 +199,16 @@ abstract class FunctionDetailsCommand extends BaseCommand {
protected String DEPRECATED_className;
@Parameter(names = "--classname", description = "The class name of a Pulsar Function")
protected String className;
@Parameter(names = "--jar", description = "Path to the JAR file for the function (if the function is written in Java). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package.", listConverter = StringConverter.class)
@Parameter(names = "--jar", description = "Path to the JAR file for the function (if the function is written in Java). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package.", listConverter = StringConverter.class)
protected String jarFile;
@Parameter(
names = "--py",
description = "Path to the main Python file/Python Wheel file for the function (if the function is written in Python)",
description = "Path to the main Python file/Python Wheel file for the function (if the function is written in Python). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package.",
listConverter = StringConverter.class)
protected String pyFile;
@Parameter(
names = "--go",
description = "Path to the main Go executable binary for the function (if the function is written in Go)")
description = "Path to the main Go executable binary for the function (if the function is written in Go). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package.")
protected String goFile;
@Parameter(names = {"-i",
"--inputs"}, description = "The input topic or topics (multiple topics can be specified as a comma-separated list) of a Pulsar Function")
Expand Down
18 changes: 9 additions & 9 deletions site2/docs/functions-cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ dead-letter-topic | The topic where all messages that were not processed success
disk | The disk in bytes that need to be allocated per function instance (applicable only to docker runtime). | |
fqfn | The Fully Qualified Function Name (FQFN) for the function. | |
function-config-file | The path to a YAML config file specifying the configuration of a Pulsar Function. | |
go | Path to the main Go executable binary for the function (if the function is written in Go). | |
go | Path to the main Go executable binary for the function (if the function is written in Go). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
hostname-verification-enabled | Enable hostname verification. | false
inputs | The input topic or topics of a Pulsar Function (multiple topics can be specified as a comma-separated list). | |
jar | Path to the jar file for the function (if the function is written in Java). It also supports URL-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package. | |
jar | Path to the jar file for the function (if the function is written in Java). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
instance-id-offset | Start the instanceIds from this offset. | 0
log-topic | The topic to which the logs a Pulsar Function are produced. | |
max-message-retries | How many times should we try to process a message before giving up. | |
Expand All @@ -37,7 +37,7 @@ output | The output topic of a Pulsar Function (If none is specified, no output
output-serde-classname | The SerDe class to be used for messages output by the function. | |
parallelism | The parallelism factor of a Pulsar Function (i.e. the number of function instances to run). | |
processing-guarantees | The processing guarantees (delivery semantics) applied to the function. Available values: [ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE]. | ATLEAST_ONCE
py | Path to the main Python file/Python Wheel file for the function (if the function is written in Python). | |
py | Path to the main Python file/Python Wheel file for the function (if the function is written in Python). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
ram | The ram in bytes that need to be allocated per function instance (applicable only to process/docker runtime). | |
retain-ordering | Function consumes and processes messages in order. | |
schema-type | The builtin schema type or custom schema class name to be used for messages output by the function. | <empty string>
Expand Down Expand Up @@ -71,9 +71,9 @@ dead-letter-topic | The topic where all messages that were not processed success
disk | The disk in bytes that need to be allocated per function instance (applicable only to docker runtime). | |
fqfn | The Fully Qualified Function Name (FQFN) for the function. | |
function-config-file | The path to a YAML config file specifying the configuration of a Pulsar Function. | |
go | Path to the main Go executable binary for the function (if the function is written in Go). | |
go | Path to the main Go executable binary for the function (if the function is written in Go). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
inputs | The input topic or topics of a Pulsar Function (multiple topics can be specified as a comma-separated list). | |
jar | Path to the jar file for the function (if the function is written in Java). It also supports URL-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package. | |
jar | Path to the jar file for the function (if the function is written in Java). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
log-topic | The topic to which the logs of a Pulsar Function are produced. | |
max-message-retries | How many times should we try to process a message before giving up. | |
name | The name of a Pulsar Function. | |
Expand All @@ -82,7 +82,7 @@ output | The output topic of a Pulsar Function (If none is specified, no output
output-serde-classname | The SerDe class to be used for messages output by the function. | |
parallelism | The parallelism factor of a Pulsar Function (i.e. the number of function instances to run). | |
processing-guarantees | The processing guarantees (delivery semantics) applied to the function. Available values: [ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE]. | ATLEAST_ONCE
py | Path to the main Python file/Python Wheel file for the function (if the function is written in Python). | |
py | Path to the main Python file/Python Wheel file for the function (if the function is written in Python). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
ram | The ram in bytes that need to be allocated per function instance (applicable only to process/docker runtime). | |
retain-ordering | Function consumes and processes messages in order. | |
schema-type | The builtin schema type or custom schema class name to be used for messages output by the function. | <empty string>
Expand Down Expand Up @@ -123,9 +123,9 @@ dead-letter-topic | The topic where all messages that were not processed success
disk | The disk in bytes that need to be allocated per function instance (applicable only to docker runtime). | |
fqfn | The Fully Qualified Function Name (FQFN) for the function. | |
function-config-file | The path to a YAML config file specifying the configuration of a Pulsar Function. | |
go | Path to the main Go executable binary for the function (if the function is written in Go). | |
go | Path to the main Go executable binary for the function (if the function is written in Go). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
inputs | The input topic or topics of a Pulsar Function (multiple topics can be specified as a comma-separated list). | |
jar | Path to the jar file for the function (if the function is written in Java). It also supports URL-path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package. | |
jar | Path to the jar file for the function (if the function is written in Java). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
log-topic | The topic to which the logs of a Pulsar Function are produced. | |
max-message-retries | How many times should we try to process a message before giving up. | |
name | The name of a Pulsar Function. | |
Expand All @@ -134,7 +134,7 @@ output | The output topic of a Pulsar Function (If none is specified, no output
output-serde-classname | The SerDe class to be used for messages output by the function. | |
parallelism | The parallelism factor of a Pulsar Function (i.e. the number of function instances to run). | |
processing-guarantees | The processing guarantees (delivery semantics) applied to the function. Available values: [ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE]. | ATLEAST_ONCE
py | Path to the main Python file/Python Wheel file for the function (if the function is written in Python). | |
py | Path to the main Python file/Python Wheel file for the function (if the function is written in Python). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)/function (package URL from packages management service)] from which worker can download the package. | |
ram | The ram in bytes that need to be allocated per function instance (applicable only to process/docker runtime). | |
retain-ordering | Function consumes and processes messages in order. | |
schema-type | The builtin schema type or custom schema class name to be used for messages output by the function. | <empty string>
Expand Down
19 changes: 19 additions & 0 deletions site2/docs/functions-deploy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,25 @@ $ bin/pulsar-admin functions create \
> #### Resources are *per instance*
> The resources that you apply to a given Pulsar Function are applied to each instance of the function. For example, if you apply 8 GB of RAM to a function with a parallelism of 5, you are applying 40 GB of RAM for the function in total. Make sure that you take the parallelism (the number of instances) factor into your resource calculations.

### Use [Package management service](admin-api-packages.md)
freeznet marked this conversation as resolved.
Show resolved Hide resolved

Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system.

To use Package management service, please ensure that the package management service has been enabled in your cluster by setting following properties in `broker.conf`.
freeznet marked this conversation as resolved.
Show resolved Hide resolved

> Note: Package management service is not enabled by default.

```yaml
enablePackagesManagement=true
packagesManagementStorageProvider=org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider
packagesReplicas=1
packagesManagementLedgerRootPath=/ledgers
```

With Package management service enabled, you can upload your function packages by [upload a package](admin-api-packages.md#upload-a-package) to the service and get the [package URL](admin-api-packages.md#package-url).

When you have a ready to use package URL, you can create the function with package URL by setting `--jar`, `--py`, or `--go` to the package URL with `pulsar-admin functions create`.

## Trigger Pulsar Functions

If a Pulsar Function is running in [cluster mode](#cluster-mode), you can **trigger** it at any time using the command line. Triggering a function means that you send a message with a specific value to the function and get the function output (if any) via the command line.
Expand Down
Loading