Skip to content

Commit

Permalink
feat(cmp): Print stderr output from command even on success (#15921) (#…
Browse files Browse the repository at this point in the history 
…15973) (#16124)

* feat(cmp): Print stderr output from command even on success



* docs(cmp): Document logging from cmp sidecard for development purposes



---------

Signed-off-by: Mathias Petermann <mathias.petermann@gmail.com>
Signed-off-by: Thomas Sands <thomas.sands@ly.st>
Co-authored-by: Mathias Petermann <mathias.petermann@gmail.com>
  • Loading branch information
@thomassandslyst @peschmae
thomassandslyst and peschmae authored Oct 26, 2023
1 parent a9c87d2 commit 7415f47
Show file tree
Hide file tree
Showing 2 changed files with 51 additions and 37 deletions.
13 changes: 9 additions & 4 deletions cmpserver/plugin/plugin.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -120,11 +120,16 @@ func runCommand(ctx context.Context, command Command, path string, env []string)
logCtx.Error(err.Error())
return strings.TrimSuffix(output, "\n"), err
}

logCtx = logCtx.WithFields(log.Fields{
"stderr": stderr.String(),
"command": command,
})
if len(output) == 0 {
log.WithFields(log.Fields{
"stderr": stderr.String(),
"command": command,
}).Warn("Plugin command returned zero output")
logCtx.Warn("Plugin command returned zero output")
} else {
// Log stderr even on successfull commands to help develop plugins
logCtx.Info("Plugin command successfull")
}

return strings.TrimSuffix(output, "\n"), nil
Expand Down
75 changes: 42 additions & 33 deletions docs/operator-manual/config-management-plugins.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,33 +43,34 @@ spec:
args: [-c, 'echo "Initializing..."']
# The generate command runs in the Application source directory each time manifests are generated. Standard output
# must be ONLY valid Kubernetes Objects in either YAML or JSON. A non-zero exit code will fail manifest generation.
# To write log messages from the command, write them to stderr, it will always be displayed.
# Error output will be sent to the UI, so avoid printing sensitive information (such as secrets).
generate:
command: [sh, -c]
args:
- |
echo "{\"kind\": \"ConfigMap\", \"apiVersion\": \"v1\", \"metadata\": { \"name\": \"$ARGOCD_APP_NAME\", \"namespace\": \"$ARGOCD_APP_NAMESPACE\", \"annotations\": {\"Foo\": \"$ARGOCD_ENV_FOO\", \"KubeVersion\": \"$KUBE_VERSION\", \"KubeApiVersion\": \"$KUBE_API_VERSIONS\",\"Bar\": \"baz\"}}}"
# The discovery config is applied to a repository. If every configured discovery tool matches, then the plugin may be
# used to generate manifests for Applications using the repository. If the discovery config is omitted then the plugin
# will not match any application but can still be invoked explicitly by specifying the plugin name in the app spec.
# Only one of fileName, find.glob, or find.command should be specified. If multiple are specified then only the
# used to generate manifests for Applications using the repository. If the discovery config is omitted then the plugin
# will not match any application but can still be invoked explicitly by specifying the plugin name in the app spec.
# Only one of fileName, find.glob, or find.command should be specified. If multiple are specified then only the
# first (in that order) is evaluated.
discover:
# fileName is a glob pattern (https://pkg.go.dev/path/filepath#Glob) that is applied to the Application's source
# fileName is a glob pattern (https://pkg.go.dev/path/filepath#Glob) that is applied to the Application's source
# directory. If there is a match, this plugin may be used for the Application.
fileName: "./subdir/s*.yaml"
find:
# This does the same thing as fileName, but it supports double-start (nested directory) glob patterns.
glob: "**/Chart.yaml"
# The find command runs in the repository's root directory. To match, it must exit with status code 0 _and_
# The find command runs in the repository's root directory. To match, it must exit with status code 0 _and_
# produce non-empty output to standard out.
command: [sh, -c, find . -name env.yaml]
# The parameters config describes what parameters the UI should display for an Application. It is up to the user to
# actually set parameters in the Application manifest (in spec.source.plugin.parameters). The announcements _only_
# inform the "Parameters" tab in the App Details page of the UI.
parameters:
# Static parameter announcements are sent to the UI for _all_ Applications handled by this plugin.
# Think of the `string`, `array`, and `map` values set here as "defaults". It is up to the plugin author to make
# Think of the `string`, `array`, and `map` values set here as "defaults". It is up to the plugin author to make
# sure that these default values actually reflect the plugin's behavior if the user doesn't explicitly set different
# values for those parameters.
static:
Expand Down Expand Up @@ -112,13 +113,13 @@ spec:
```
!!! note
While the ConfigManagementPlugin _looks like_ a Kubernetes object, it is not actually a custom resource.
While the ConfigManagementPlugin _looks like_ a Kubernetes object, it is not actually a custom resource.
It only follows kubernetes-style spec conventions.
The `generate` command must print a valid Kubernetes YAML or JSON object stream to stdout. Both `init` and `generate` commands are executed inside the application source directory.

The `discover.fileName` is used as [glob](https://pkg.go.dev/path/filepath#Glob) pattern to determine whether an
application repository is supported by the plugin or not.
application repository is supported by the plugin or not.

```yaml
discover:
Expand Down Expand Up @@ -167,7 +168,7 @@ data:

#### Register the plugin sidecar

To install a plugin, patch argocd-repo-server to run the plugin container as a sidecar, with argocd-cmp-server as its
To install a plugin, patch argocd-repo-server to run the plugin container as a sidecar, with argocd-cmp-server as its
entrypoint. You can use either off-the-shelf or custom-built plugin image as sidecar image. For example:

```yaml
Expand All @@ -187,7 +188,7 @@ containers:
- mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: plugin.yaml
name: my-plugin-config
# Starting with v2.4, do NOT mount the same tmp volume as the repo-server container. The filesystem separation helps
# Starting with v2.4, do NOT mount the same tmp volume as the repo-server container. The filesystem separation helps
# mitigate path traversal attacks.
- mountPath: /tmp
name: cmp-tmp
Expand All @@ -197,7 +198,7 @@ volumes:
name: my-plugin-config
- emptyDir: {}
name: cmp-tmp
```
```

!!! important "Double-check these items"
1. Make sure to use `/var/run/argocd/argocd-cmp-server` as an entrypoint. The `argocd-cmp-server` is a lightweight GRPC service that allows Argo CD to interact with the plugin.
Expand All @@ -222,9 +223,9 @@ Plugin commands have access to
value: bar
- name: REV
value: test-$ARGOCD_APP_REVISION
Before reaching the `init.command`, `generate.command`, and `discover.find.command` commands, Argo CD prefixes all
user-supplied environment variables (#3 above) with `ARGOCD_ENV_`. This prevents users from directly setting

Before reaching the `init.command`, `generate.command`, and `discover.find.command` commands, Argo CD prefixes all
user-supplied environment variables (#3 above) with `ARGOCD_ENV_`. This prevents users from directly setting
potentially-sensitive environment variables.

4. Parameters in the Application spec:
Expand All @@ -240,43 +241,43 @@ Plugin commands have access to
- name: helm-parameters
map:
image.tag: v1.2.3

The parameters are available as JSON in the `ARGOCD_APP_PARAMETERS` environment variable. The example above would
produce this JSON:

[{"name": "values-files", "array": ["values-dev.yaml"]}, {"name": "helm-parameters", "map": {"image.tag": "v1.2.3"}}]

!!! note
Parameter announcements, even if they specify defaults, are _not_ sent to the plugin in `ARGOCD_APP_PARAMETERS`.
Only parameters explicitly set in the Application spec are sent to the plugin. It is up to the plugin to apply
the same defaults as the ones announced to the UI.

The same parameters are also available as individual environment variables. The names of the environment variables
follows this convention:

- name: some-string-param
string: some-string-value
# PARAM_SOME_STRING_PARAM=some-string-value

- name: some-array-param
value: [item1, item2]
# PARAM_SOME_ARRAY_PARAM_0=item1
# PARAM_SOME_ARRAY_PARAM_1=item2

- name: some-map-param
map:
image.tag: v1.2.3
# PARAM_SOME_MAP_PARAM_IMAGE_TAG=v1.2.3
!!! warning "Sanitize/escape user input"

!!! warning "Sanitize/escape user input"
As part of Argo CD's manifest generation system, config management plugins are treated with a level of trust. Be
sure to escape user input in your plugin to prevent malicious input from causing unwanted behavior.

## Using a config management plugin with an Application

You may leave the `name` field
empty in the `plugin` section for the plugin to be automatically matched with the Application based on its discovery rules. If you do mention the name make sure
it is either `<metadata.name>-<spec.version>` if version is mentioned in the `ConfigManagementPlugin` spec or else just `<metadata.name>`. When name is explicitly
empty in the `plugin` section for the plugin to be automatically matched with the Application based on its discovery rules. If you do mention the name make sure
it is either `<metadata.name>-<spec.version>` if version is mentioned in the `ConfigManagementPlugin` spec or else just `<metadata.name>`. When name is explicitly
specified only that particular plugin will be used iff its discovery pattern/command matches the provided application repo.

```yaml
Expand Down Expand Up @@ -305,17 +306,17 @@ If you don't need to set any environment variables, you can set an empty plugin

!!! important
If your CMP command runs too long, the command will be killed, and the UI will show an error. The CMP server
respects the timeouts set by the `server.repo.server.timeout.seconds` and `controller.repo.server.timeout.seconds`
respects the timeouts set by the `server.repo.server.timeout.seconds` and `controller.repo.server.timeout.seconds`
items in `argocd-cm`. Increase their values from the default of 60s.

Each CMP command will also independently timeout on the `ARGOCD_EXEC_TIMEOUT` set for the CMP sidecar. The default
is 90s. So if you increase the repo server timeout greater than 90s, be sure to set `ARGOCD_EXEC_TIMEOUT` on the
sidecar.

!!! note
Each Application can only have one config management plugin configured at a time. If you're converting an existing
plugin configured through the `argocd-cm` ConfigMap to a sidecar, make sure to update the plugin name to either `<metadata.name>-<spec.version>`
if version was mentioned in the `ConfigManagementPlugin` spec or else just use `<metadata.name>`. You can also remove the name altogether
plugin configured through the `argocd-cm` ConfigMap to a sidecar, make sure to update the plugin name to either `<metadata.name>-<spec.version>`
if version was mentioned in the `ConfigManagementPlugin` spec or else just use `<metadata.name>`. You can also remove the name altogether
and let the automatic discovery to identify the plugin.

## Debugging a CMP
Expand All @@ -329,6 +330,14 @@ If you are actively developing a sidecar-installed CMP, keep a few things in min
image. If you're using a different, static tag, set `imagePullPolicy: Always` on the CMP's sidecar container.
3. CMP errors are cached by the repo-server in Redis. Restarting the repo-server Pod will not clear the cache. Always
do a "Hard Refresh" when actively developing a CMP so you have the latest output.
4. Verify your sidecar has started properly by viewing the Pod and seeing that two containers are running `kubectl get pod -l app.kubernetes.io/component=repo-server -n argocd`
5. Write log message to stderr and set the `--loglevel=info` flag in the sidecar. This will print everything written to stderr, even on successfull command execution.


### Other Common Errors
| Error Message | Cause |
| -- | -- |
| `no matches for kind "ConfigManagementPlugin" in version "argoproj.io/v1alpha1"` | The `ConfigManagementPlugin` CRD was deprecated in Argo CD 2.4 and removed in 2.8. This error means you've tried to put the configuration for your plugin directly into Kubernetes as a CRD. Refer to this [section of documentation](#write-the-plugin-configuration-file) for how to write the plugin configuration file and place it properly in the sidecar. |

## Plugin tar stream exclusions

Expand Down Expand Up @@ -391,14 +400,14 @@ spec:

### Write discovery rules for your plugin

Sidecar plugins can use either discovery rules or a plugin name to match Applications to plugins. If the discovery rule is omitted
Sidecar plugins can use either discovery rules or a plugin name to match Applications to plugins. If the discovery rule is omitted
then you have to explicitly specify the plugin by name in the app spec or else that particular plugin will not match any app.

If you want to use discovery instead of the plugin name to match applications to your plugin, write rules applicable to
your plugin [using the instructions above](#1-write-the-plugin-configuration-file) and add them to your configuration
If you want to use discovery instead of the plugin name to match applications to your plugin, write rules applicable to
your plugin [using the instructions above](#1-write-the-plugin-configuration-file) and add them to your configuration
file.

To use the name instead of discovery, update the name in your application manifest to `<metadata.name>-<spec.version>`
To use the name instead of discovery, update the name in your application manifest to `<metadata.name>-<spec.version>`
if version was mentioned in the `ConfigManagementPlugin` spec or else just use `<metadata.name>`. For example:

```yaml
Expand Down

0 comments on commit 7415f47

Please sign in to comment.