RFC: pipeline compile step

[6543]

Add a own section called compile into pipeline config.

witch use normal workflow logic (default or custom clone) ... but it's output is no log but an pipeline config.

tldr: with way we could support starlark, jsonnet (#1396), drone >= 1.0, ... without needing to add it to woodpecker core

details

we either could make a special compile pipeline, and show it as own entry, witch trigger the actual pipeline with its config
(if we do so, we need to somehow link them UI/UX wise ...)
or we just use our multi workflow approach ... where one workflow (eventually with a different icon) is the compile and all generated workflows from it are also i it ... (as a compile could create one or more workflows)

we need to make sure we dont have a recursion in it -> see permutate example

example:

compile:
  image: wp/jsonnet
  settings:
    file: woodpecker.jsonnet

compile:
  image: wp/replace-image # permutate normal wp pipeline config

pipeline:
  normal-steps:
...

workaround:

use a Configuration Service

[6543]

be aware as we would support with multible workflows multible compiles ... that this get's complicated quite fast

[6543]

we also might add a server flag to ignore this feature

[anbraten]

seems pretty similar to #53

[gapodo]

I like the idea, this would also allow one to "emulate" some of the features of GitHub workflows by basically overriding a step declared with just a name and e.g. "fake" image with a step with all settings prepared (basically a plugin + the necessary config / commands)... which would increase code reusability and also ease updating a step in all repos (e.g. switching to a newer buildx plugin or changing defaults like proxy settings to match the environment)

[lafriks]

would be nice to be able to specify global (or org wide) permutate image that would by applied than to all respective pipelines

[wiktor-k]

This feature looks really great! Even if it was super-simple: generating entire woodpecker.yml using a docker image this would already provide enough tooling to write virtually any integration one can imagine. Thanks for thinking ahead and for your work 👍

[bkleiner]

What would be the input/output scheme of this compile step?
stdin/stdout seems like a obvious choice, however would prevent the compile step outputting logs to the user and would make it rather hard to take advantage of multi-workflows.

maybe an approach to how github workflows handles output could be taken, eg:

the env-var $COMPILE_INPUT holds the regular pipeline for mutation.
$COMPILE_OUTPUT holds the path to a folder where the compile-step can write files to, which then are handled the same way as the .woodpecker folder normally would, basically overwriting what is delivered by the csv.

adding this feature would make woodpicker a viable alternative for us, as we rely on jsonnet for some of our pipelines.

[ambroisie]

stdin/stdout seems like a obvious choice, however would prevent the compile step outputting logs to the user and would make it rather hard to take advantage of multi-workflows.

The standard is to use stderr for user-oriented output in that case.

adding this feature would make woodpicker a viable alternative for us, as we rely on jsonnet for some of our pipelines.

I had a good experience using the matrix builds in tandem with the when.evaluate option to replace jsonnet in my use-case. But that doesn't get you very far.

[bkleiner]

The standard is to use stderr for user-oriented output in that case.

well, yeah, in theory, tho i think you gonna have a very hard time to make all programs a user would potentially want to use comply with that.

[pinpox]

well, yeah, in theory, tho i think you gonna have a very hard time to make all programs a user would potentially want to use comply with that.

Isn't that something that can be solved with redirection for non-complying programs?