Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to configure config extensions for repos #3349

Open
wants to merge 62 commits into
base: main
Choose a base branch
from

Conversation

anbraten
Copy link
Member

@anbraten anbraten commented Feb 7, 2024

closes #783

Allow users to set / use service extensions (config) on a per repo basis. Those extensions are little external webservice which can implement a set of endpoints to replace woodpeckers internal functionality like:

  • pipeline config processing

TODO

  • add config & secrets extension service urls to repo settings
  • allow to change extensions urls for repo
  • add docs
  • restrict http call locations (prevent local ips)

image

@anbraten anbraten marked this pull request as ready for review June 5, 2024 16:26
@qwerty287 qwerty287 added this to the 2.6.0 milestone Jun 6, 2024
server/model/pagination.go Outdated Show resolved Hide resolved
Comment on lines 53 to 54
SecretExtensionEndpoint string `json:"secret_extension_endpoint" xorm:"varchar(500) 'secret_extension_endpoint'"`
ConfigExtensionEndpoint string `json:"config_extension_endpoint" xorm:"varchar(500) 'config_extension_endpoint'"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are sent to all requests, including unauthorized.
Since you should use signing anyways that should be fine, but I just want to make sure this is expected?

server/services/secret/http.go Outdated Show resolved Hide resolved
server/services/secret/http.go Outdated Show resolved Hide resolved
server/services/utils/http.go Outdated Show resolved Hide resolved
}

var blockedError error
if blockList.MatchHostOrIP(host, tcpAddr.IP) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the blocklist used somewhere?

Comment on lines 466 to 474
"extensions": "Extensions",
"extensions_description": "Extensions are HTTP services that can be called by Woodpecker instead of using the builtin ones.",
"secrets_extension_endpoint": "Secrets extension endpoint",
"secrets_extension_alpha_state": "The secret extension is in alpha state and might change in the future.",
"extension_endpoint_placeholder": "e.g. https://my-extension-endpint.tld/api",
"config_extension_endpoint": "Config extension endpoint",
"extensions_signatures_public_key": "Public key for signatures",
"extensions_signatures_public_key_description": "This public key should be used by your extensions to verify webhook calls from Woodpecker.",
"extensions_configuration_saved": "Extensions configuration saved"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put them into their own object?

web/src/assets/locales/en.json Outdated Show resolved Hide resolved
@anbraten anbraten modified the milestones: 2.6.0, 2.7.0 Jun 10, 2024
@anbraten anbraten modified the milestones: 2.7.0, 2.8.0 Jul 13, 2024
@6543 6543 removed this from the 2.8.0 milestone Jul 22, 2024
@qwerty287 qwerty287 added this to the 3.0.0 milestone Jul 24, 2024
@anbraten anbraten changed the title Repo service extensions Allow to configure config extensions for repos Sep 30, 2024
Copy link
Contributor

@zc-devs zc-devs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried about #4341 (comment). Do not see changes in this regards. I might miss something, therefore double-check, please. See also #4341 (comment).

@@ -0,0 +1,164 @@
# Configuration extension

The configuration extension can be used to modify or generate Woodpeckers pipeline configurations. You can configure a HTTP
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The configuration extension can be used to modify or generate Woodpeckers pipeline configurations. You can configure a HTTP
The configuration extension can be used to modify or generate Woodpeckers pipeline configurations. You can configure an HTTP

Using such an extension can be useful if you want to:

<!-- cSpell:words templating,Starlark,Jsonnet -->
- Preprocess the original configuration file with something like go templating
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Preprocess the original configuration file with something like go templating
- Preprocess the original configuration file with something like Go template

- [netrc model](https://github.com/woodpecker-ci/woodpecker/blob/main/server/model/netrc.go)

:::tip
The `netrc` data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) api to get more information about the repository.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The `netrc` data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) api to get more information about the repository.
The `netrc` data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) API to get more information about the repository.

When a pipeline is triggered Woodpecker will fetch the pipeline configuration from the repository, then make a HTTP POST request to the configured extension with a JSON payload containing some data like the repository, pipeline information and the current config files retrieved from the repository. The extension can then send back modified or even new pipeline configurations following Woodpeckers official yaml format that should be used.

:::tip
The netrc data is pretty powerful as it contains credentials to access the repository. You can use this to clone the repository or even use the forge (Github or Gitlab, ...) api to get more information about the repository.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate?


### Response

The extension should respond with a JSON payload containing the new configuration files in Woodpeckers official yaml format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The extension should respond with a JSON payload containing the new configuration files in Woodpeckers official yaml format.
The extension should respond with a JSON payload containing the new configuration files in Woodpecker's official YAML format.

### Response

The extension should respond with a JSON payload containing the new configuration files in Woodpeckers official yaml format.
If the extension wants to keep the existing configuration files, it can respond with **HTTP 204**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If the extension wants to keep the existing configuration files, it can respond with **HTTP 204**.
If the extension wants to keep the existing configuration files, it can respond with HTTP `204 No Content`.


There are currently two types of extensions available:

- [Configuration extension](./40-configuration-extension.md) to modify or generate Woodpeckers pipeline configurations.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [Configuration extension](./40-configuration-extension.md) to modify or generate Woodpeckers pipeline configurations.
- [Configuration extension](./40-configuration-extension.md) to modify or generate Woodpecker pipeline configurations.


Woodpecker allows you to replace internal logic with external extensions by using pre-defined http endpoints.

There are currently two types of extensions available:
Copy link
Contributor

@zc-devs zc-devs Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the second one? Is it possible to configure it globally like config extension?
There are 3 types on screenshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
addon feature add new functionality ui frontend related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Configuration Extension
4 participants