docs: restructure test docs (part 1) (cherry-pick #1441)

website/docs/json-schemas.md

@@ -60,4 +60,4 @@ Chainsaw can also export JSON schemas locally if you don't want to reference the
 chainsaw export schemas <local path>
 ```
 
-See [Chainsaw export schemas reference](./commands/chainsaw_export_schemas.md) for more details.
+See [chainsaw export schemas](./commands/chainsaw_export_schemas.md) command documentation for more details.

website/docs/operations/apply.md

@@ -8,8 +8,8 @@ These can be configurations, deployments, services, or any other Kubernetes reso
 
 !!! tip "Reference documentation"
     - The full structure of the `Apply` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Apply).
-    - This operation supports [bindings](../bindings/index.md).
-    - This operation supports [outputs](../bindings/outputs.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
+    - This operation supports [outputs](../tests/common/outputs.md).
 
 ## Usage examples
 

website/docs/operations/assert.md

@@ -17,7 +17,7 @@ For example, after applying resources, you might want to ensure that a particula
 
 !!! tip "Reference documentation"
     - The full structure of the `Assert` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Assert).
-    - This operation supports [bindings](../bindings/index.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
 
 ## Usage examples
 

website/docs/operations/command.md

@@ -6,8 +6,8 @@ The `command` operation provides a means to execute a specific command during th
 
 !!! tip "Reference documentation"
     - The full structure of the `Command` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Command).
-    - This operation supports [bindings](../bindings/index.md).
-    - This operation supports [outputs](../bindings/outputs.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
+    - This operation supports [outputs](../tests/common/outputs.md).
 
 ## Usage examples
 

website/docs/operations/create.md

@@ -7,8 +7,8 @@ These can be configurations, deployments, services, or any other Kubernetes reso
 
 !!! tip "Reference documentation"
     - The full structure of the `Create` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Create).
-    - This operation supports [bindings](../bindings/index.md).
-    - This operation supports [outputs](../bindings/outputs.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
+    - This operation supports [outputs](../tests/common/outputs.md).
 
 !!! warning
 

website/docs/operations/delete.md

@@ -9,7 +9,7 @@ The `delete` operation allows you to specify resources that should be deleted fr
 
 !!! tip "Reference documentation"
     - The full structure of the `Delete` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Delete).
-    - This operation supports [bindings](../bindings/index.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
 
 ## Usage examples
 

website/docs/operations/error.md

@@ -15,7 +15,7 @@ The `error` operation lets you define a set of expected errors for a test step.
 
 !!! tip "Reference documentation"
     - The full structure of the `Error` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Error).
-    - This operation supports [bindings](../bindings/index.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
 
 ## Usage examples
 

website/docs/operations/patch.md

@@ -7,8 +7,8 @@ These can be configurations, deployments, services, or any other Kubernetes reso
 
 !!! tip "Reference documentation"
     - The full structure of the `Patch` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Patch).
-    - This operation supports [bindings](../bindings/index.md).
-    - This operation supports [outputs](../bindings/outputs.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
+    - This operation supports [outputs](../tests/common/outputs.md).
 
 !!! warning
 

website/docs/operations/script.md

@@ -6,8 +6,8 @@ The `script` operation provides a means to run a script during the test step.
 
 !!! tip "Reference documentation"
     - The full structure of the `Script` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Script).
-    - This operation supports [bindings](../bindings/index.md).
-    - This operation supports [outputs](../bindings/outputs.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
+    - This operation supports [outputs](../tests/common/outputs.md).
 
 ## Usage examples
 

website/docs/operations/update.md

@@ -7,8 +7,8 @@ These can be configurations, deployments, services, or any other Kubernetes reso
 
 !!! tip "Reference documentation"
     - The full structure of the `Update` is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Update).
-    - This operation supports [bindings](../bindings/index.md).
-    - This operation supports [outputs](../bindings/outputs.md).
+    - This operation supports [bindings](../tests/common/bindings.md).
+    - This operation supports [outputs](../tests/common/outputs.md).
 
 !!! warning
 

website/docs/quick-start/first-test.md

@@ -1,6 +1,6 @@
 # Create a test
 
-To create a chainsaw test all you need to do is to create one (or more) YAML file(s).
+To create a Chainsaw test all you need to do is to create one (or more) YAML file(s).
 
 The recommended approach is to create one folder per test, with a `chainsaw-test.yaml` file containing one (or more) test definition(s).
 The test definition can reference other files in the same folder or anywhere else on the file system as needed.

website/docs/bindings/index.md → website/docs/tests/common/bindings.md

@@ -2,7 +2,7 @@
 
 Chainsaw has a concept of `bindings` which can be seen as an execution context.
 
-Bindings are used in [assertion trees](../operations/check.md) and [resource templating](../configuration/options/templating.md), as well as when using the `--values` flag when invoking Chainsaw.
+Bindings are used in [assertion trees](../../operations/check.md) and [resource templating](../../configuration/options/templating.md), as well as when using the `--values` flag when invoking Chainsaw.
 
 While some bindings are built-in and provided by Chainsaw, it's possible to define custom bindings at the test, step, or individual operation level.
 Those bindings can in turn be used to create custom environment variables in `command` and `script` operations.
@@ -18,7 +18,7 @@ Effectively, the last one registered in the chain will be used.
 
 A binding `name` supports templating.
 
-The name of a binding can therefore be dynamic and depend on [values](../quick-start/advanced/values.md) or other bindings.
+The name of a binding can therefore be dynamic and depend on [values](../../quick-start/advanced/values.md) or other bindings.
 
 ## Usage
 

website/docs/tests/conventional.md

@@ -0,0 +1,107 @@
+# Conventional approach
+
+!!! warning
+    While Chainsaw supports the conventional approach, we strongly recommend the explicit one.
+
+    If you are new to Chainsaw we suggest you skip this section and jump directly to the [Explicit approach](./explicit.md).
+
+## Introduction
+
+The conventional approach is the simplest and less verbose one.
+
+You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.
+
+### Limitations
+
+While this syntax is simple, it suffers lots of limitations. It doesn't support deletion operations, commands, scripts, and all Chainsaw helpers.
+
+It is also impossible to specify additional configuration per test, step or individual operation (timeouts, additional verifications, etc...), making this approach highly limited.
+
+It also relies a lot on file naming conventions which can be error prone.
+
+Finally, this approach doesn't encourage reusing files across tests and leads to duplication, making maintenance harder.
+
+## File naming convention
+
+Manifest files must follow a specific naming convention:
+```
+<step index>-<name|assert|errors>.yaml
+```
+
+As an example, `00-configmap.yaml`, `01-assert.yaml` and `02-errors.yaml` are valid file names.
+
+### Assembling steps
+
+It's perfectly valid to have multiple files for the same step.
+
+Let's say we have the following files `00-resources.yaml`, `00-more-resources.yaml`, `00-assert.yaml` and `00-errors.yaml`:
+
+- `00-resources.yaml` and `00-more-resources.yaml` contain resources that will be applied in step `00`
+- `00-assert.yaml` contains assert statements in step `00`
+- `00-errors.yaml` contains error statements in step `00`
+
+With the four files above, Chainsaw will assemble a test step made of the combination of all those files.
+
+### Loading process
+
+The logic to determine the content of a step is always:
+
+- The step index is obtained from the beginning of the file name, it must be composed of two numbers between `0` and `9` (from `00` to `99`)
+- The next character acts as a separator and is expected to be `-`
+- The rest of the file name (without extension) is then evaluated
+    - If it is equal to `assert`, the content is considered assertion statements
+    - If it is equal to `error`, the content is considered error statements
+    - Else the content is considered resources to be applied
+- The extension must be `.yaml` or `.yml`
+
+## Example
+
+### 01-configmap.yaml
+
+The manifest below contains a config map in a file called `01-configmap.yaml`.
+Chainsaw will associate this manifest with an apply operation in step `01`.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: chainsaw-quick-start
+data:
+  foo: bar
+```
+
+### 02-assert.yaml
+
+The manifest below contains an assertion statement in a file called `02-assert.yaml`.
+Chainsaw will associate this manifest with an assert operation in step `02`.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: chainsaw-quick-start
+data:
+  foo: bar
+```
+
+### 03-errors.yaml
+
+The manifest below contains an error statement in a file called `03-errors.yaml`.
+Chainsaw will associate this manifest with an error operation in step `03`.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: chainsaw-quick-start
+data:
+  lorem: ipsum
+```
+
+## Conclusion
+
+This test will first create a config map, then assert the content of the config map contains the `foo: bar` data, and then verify that the config map does not contain the `lorem: ipsum` data.
+
+For such a simple test, the conventional approach works reasonably well but will quickly become limited when the test scenarios get more complex.
+
+Look at the [explicit approach](./explicit.md) for a lot more flexible solution.

website/docs/tests/explicit.md

@@ -0,0 +1,77 @@
+# Explicit approach
+
+The explicit is a bit more verbose than the conventional one but offers a lot more flexibility and features:
+
+- It does not rely on file naming conventions for operations ordering
+- It encourages file reuse across tests, reducing duplication and maintenance
+- It offers the flexibility to provide additional configurations like timeouts, complex logic, etc...
+- It supports all [operations](../operations/index.md) without restrictions
+
+## The Test resource
+
+A `Test` resource, like any other Kubernetes resource, has an `apiVersion`, `kind` and `metadata` section.
+
+It also comes with a `spec` section used to declaratively represent the test logic, steps and operations, as well as other configuration elements belonging to the test being defined.
+
+!!! tip "Reference documentation"
+    The full structure of the `Test` resource is documented [here](../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Test).
+
+## Example
+
+### chainsaw-test.yaml
+
+The `Test` below illustrates a simple test. Chainsaw will load the `Test` and steps defined in its `spec` section.
+
+It's worth noting that:
+
+- The test defines its own `timeouts`
+- It also states that this test should not be executed in parallel with other tests
+- It has multiple steps, most of them reference files that can be used in other tests if needed
+- It uses an arbitrary shell script
+
+```yaml
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: example
+spec:
+  # state that this test should not be executed in parallel with other tests
+  concurrent: false
+  # timeouts for this specific test
+  timeouts:
+    apply: 10s
+    assert: 10s
+    error: 10s
+  steps:
+  # step 1
+  # apply a configmap to the cluster
+  # the path to the configmap is relative to the folder
+  # containing the test, hence allow reusing manifests
+  # across multiple tests
+  - try:
+    - apply:
+        file: ../resources/configmap.yaml
+  # step 2
+  # execute assert statements against existing resources
+  # in the cluster
+  - try:
+    - assert:
+        file: ../resources/configmap-assert.yaml
+  # step 3
+  # execute error statements against existing resources
+  # in the cluster
+  - try:
+    - error:
+        file: ../resources/configmap-error.yaml
+  # step 4
+  # execute an arbitrary shell script
+  - try:
+    - script:
+        content: echo "goodbye"
+```
+
+## Conclusion
+
+While this test is simple, it illustrates the differences with the [conventional approach](./conventional.md).
+
+The purpose here is only to present the explicit approach and there are a lot more features to discuss, we will cover them in the next sections.

website/docs/tests/index._md

@@ -0,0 +1,85 @@
+# Writing Chainsaw tests
+
+This documentation focuses on providing a breakdown of the Chainsaw test structure and how to use it.
+
+Chainsaw supports two different test definition syntaxes:
+
+- [Manifests based syntax](./syntax/manifests-based.md)
+- [`Test` based syntax](./test-based.md)
+
+Each syntax has pros and cons, see the descriptions below for more details on advantages and pitfalls.
+
+!!! tip
+    While chainsaw supports [other syntaxes](../tests/index.md), we strongly recommend the explicit approach.
+
+### [Manifests based syntax](./manifests-based.md)
+
+This is the simplest and less verbose syntax.
+
+You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.
+
+While this syntax is extremly simple, not all operations are supported. For example, `delete`, `command`, `script` and `sleep` operations are not.
+
+Another strong limitation is that it is not possible to specify additional configuration per test, step or operation.
+
+Finally, this syntax relies heavily on file naming conventions, it can be error prone and makes it hard to reuse files across multiple tests.
+
+### [`Test` based syntax](./test-based.md)
+
+The `Test` based syntax is a more verbose and explicit syntax.
+
+It does not rely on file naming conventions to determine test steps order and allows to easily reuse files accross multiple tests.
+
+This syntax also comes with no limitations to provide additional configuration at the test, step or operation level.
+
+!!! tip "Making a choice"
+
+    Choosing one syntax over the other is not a trivial choice, every one will have its own preference and/or constraints.
+
+    It's usually easier to start with the manifests based syntax.
+    However, as test suites grow and tests become more complex, it is often necessary to configure options on a per test, step or operation basis and the `Test` based syntax becomes necessary.
+
+## Namespaced resources
+
+Kubernetes organizes resources into two primary scopes: namespaced and cluster-scoped.
+
+While namespaced resources belong to a specific namespace, cluster-scoped resources span across the entire Kubernetes cluster.
+
+### Purpose of the Namespacer Interface
+
+The [Namespacer interface](https://github.com/kyverno/chainsaw/blob/main/pkg/runner/namespacer/namespacer.go) ensures automated and consistent namespace assignment to Kubernetes resources.
+
+- **Automated Namespacing**
+
+    Automatically assign namespaces to resources that don't have one.
+
+- **Ephemeral Namespaces**
+
+    Handles temporary namespaces for specific tasks.
+
+## Cleanup
+
+Unless configured differently, by default Chainsaw will automatically cleanup the resources it created after a test finishes.
+Cleanup happens in reverse order of creation (created last, cleaned up first).
+
+Note that Chainsaw performs a blocking deletion, that is, it will wait the resource is actually not present anymore in the cluster before proceeding with the next resource cleanup.
+
+This is important, especially when the controller being tested makes use of `finalizers`.
+
+!!! tip "Overriding cleanup timeout"
+
+    A global cleanup timeout can be defined at the configuration level or using command line flags.
+
+    It can also be overriden on a per test or per test step basis but not at the operation level.
+
+## Parallel Execution of Tests
+
+While Chainsaw ensures that the steps within a test are executed sequentially, it is designed to run multiple tests in parallel to each other. This parallel execution helps in significantly reducing the overall time required to run an entire suite of tests, making the testing process more efficient, especially in scenarios with a large number of tests.
+
+!!! tip "Parallel execution of tests"
+
+    By default, Chainsaw will run tests in parallel.
+
+    This can be configured at the configuration level or using command line flags. However, individual tests can be configured to run concurrently by setting `Concurrent: true` in their `TestSpec`.
+
+    All non-concurrent tests are executed first, followed by the concurrent tests in parallel.