chore: improve assertions quick start (#1326) (#1329)

Signed-off-by: Charles-Edouard Brétéché <charles.edouard@nirmata.com>
Co-authored-by: Charles-Edouard Brétéché <charles.edouard@nirmata.com>

website/docs/assertion-trees.md

@@ -0,0 +1,120 @@
+# Leverage assertions trees
+
+Whether verifying the scaling behavior of a database cluster or ensuring data consistency across instances, Chainsaw's assertion model provides the precision and flexibility needed for comprehensive validation.
+
+Chainsaw allows declaring complex assertions with a simple and no-code approach, allowing assertions based on comparisons beyond simple equality, working with arrays, and other scenarios that could not be achieved before.
+
+!!! tip
+    Under the hood, Chainsaw uses [kyverno-json assertion trees](https://kyverno.github.io/kyverno-json/latest/intro/). Refer to the assertion trees documentation for more details on the supported syntax.
+
+## Comparisons beyond simple equality
+
+The assertion below will check that the number of replicas for a deployment is **greater than 3**.
+
+Chainsaw doesn't need to know the exact expected number of replicas.
+The `(replicas > 3)` expression will be evaluated until the assertion passes or the operation timeout expires (making the assertion fail).
+
+```yaml
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: example
+spec:
+  steps:
+  - try:
+    # ...
+    - assert:
+        resource:
+          apiVersion: v1
+          kind: Deployment
+          metadata:
+            name: foo
+          spec:
+            (replicas > 3): true
+    # ...
+```
+
+## Working with arrays
+
+Chainsaw query language makes it easy to assert on arrays.
+You can filter and transform arrays to select what you want to assert.
+
+### Filtering 
+
+In the example below we are creating a resource, then we assert that a condition with `type == 'Ready'` exists and has a field matching `status: 'True'`:
+
+```yaml
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: example
+spec:
+  steps:
+  - try:
+    - apply:
+        resource:
+          apiVersion: tempo.grafana.com/v1alpha1
+          kind: TempoStack
+          metadata:
+            name: simplest
+          spec:
+            storage:
+              secret:
+                name: minio
+                type: s3
+            # ...
+    - assert:
+        resource:
+          apiVersion: tempo.grafana.com/v1alpha1
+          kind: TempoStack
+          metadata:
+            name: simplest
+          status:
+            # filter conditions array to keep elements where `type == 'Ready'`
+            # and assert there's a single element matching the filter
+            # and that this element status is `True`
+            (conditions[?type == 'Ready']):
+            - status: 'True'
+```
+
+## Comprehensive reporting
+
+Chainsaw offers detailed resource diffs upon assertion failures. In the example below, the assertion failure message `metadata.annotations.foo: Invalid value: "null": Expected value: "bar"` is augmented with a resource diff.
+
+It provides a clear view of discrepancies between expected and actual resources and gives more context around the specific failure (we can easily identify the owner of the offending pod for example).
+
+```
+| 09:55:50 | deployment | step-1   | ASSERT    | RUN   | v1/Pod @ chainsaw-rare-liger/*
+| 09:56:20 | deployment | step-1   | ASSERT    | ERROR | v1/Pod @ chainsaw-rare-liger/*
+    === ERROR
+    ---------------------------------------------------
+    v1/Pod/chainsaw-rare-liger/example-5477b4ff8c-tnhd9
+    ---------------------------------------------------
+    * metadata.annotations.foo: Invalid value: "null": Expected value: "bar"
+
+    --- expected
+    +++ actual
+    @@ -1,10 +1,16 @@
+      apiVersion: v1
+      kind: Pod
+      metadata:
+    -  annotations:
+    -    foo: bar
+        labels:
+          app: nginx
+    +  name: example-5477b4ff8c-tnhd9
+        namespace: chainsaw-rare-liger
+    +  ownerReferences:
+    +  - apiVersion: apps/v1
+    +    blockOwnerDeletion: true
+    +    controller: true
+    +    kind: ReplicaSet
+    +    name: example-5477b4ff8c
+    +    uid: 118abe16-ec42-4894-83db-64479c4aac6f
+      spec: {}
+| 09:56:20 | deployment | step-1   | TRY       | DONE  |
+```
+
+## Next step
+
+To continue our exploration of the main Chainsaw features, let's look at [bindings and resource templating](./resource-templating.md).

website/docs/first-test.md

@@ -35,7 +35,7 @@ cd chainsaw-quick-start
 
 ### Create a `ConfigMap` manifest
 
-```bash
+```
 # create a ConfigMap
 cat > configmap.yaml << EOF
 apiVersion: v1
@@ -51,7 +51,7 @@ EOF
 
 By default, Chainsaw will look for a file named `chainsaw-test.yaml` in every folder.
 
-```bash
+```
 # create test file
 cat > chainsaw-test.yaml << EOF
 apiVersion: chainsaw.kyverno.io/v1alpha1
@@ -74,4 +74,4 @@ EOF
 
 ## Next step
 
-Now we have created our first test, you can continue to the next section to [execute it](./run-test.md).
+Now we have created our first test, you can continue to the next section to [execute it](./run-tests.md).

website/docs/multi-cluster.md

@@ -0,0 +1,3 @@
+# Multi-cluster setup
+
+TODO

website/docs/next-steps.md

@@ -0,0 +1,29 @@
+# Next steps
+
+We covered the main features of Chainsaw in this **Getting started** sections.
+
+While this should help you understand Chainsaw better, there are a lot of other things Chainsaw can do for you.
+
+!!! tip
+    If there's anything you would like to be improved, please [reach out](./community.md), we will be happy to discuss and improve as much as we can.
+
+To continue exploring the capabilities of Chainsaw:
+
+- Browse advanced use case discovery guides
+- Look at the Chainsaw configuration options
+- Consult the reference documentation for details about something specific
+- Familiarize with examples
+- Engage with our community and start contributing
+
+---
+
+## Additional resources
+
+Resources, blog posts and videos talking about Chainsaw:
+
+- [Kyverno Chainsaw - The ultimate end-to-end testing tool!](https://kyverno.io/blog/2023/12/12/kyverno-chainsaw-the-ultimate-end-to-end-testing-tool/)
+- [Kyverno Chainsaw - Exploring the Power of Assertion Trees!](https://kyverno.io/blog/2023/12/13/kyverno-chainsaw-exploring-the-power-of-assertion-trees/)
+- [Nirmata Office Hours for Kyverno- Episode 9- Demonstrate Kyverno Chainsaw](https://www.youtube.com/watch?v=IrIteTTjlbU)
+- [Kubebuilder Community Meeting - February 1, 2024](https://www.youtube.com/watch?v=Ejof-wtAdQM)
+- [Kyverno Chainsaw 0.1.4 - Awesome new features!](https://kyverno.io/blog/2024/02/15/kyverno-chainsaw-0.1.4-awesome-new-features/)
+- [Mastering Kubernetes Testing with Kyverno Chainsaw!](https://youtu.be/hQJWGzogIiI)

website/docs/operation-outputs.md

@@ -4,4 +4,6 @@ TODO
 
 ## Next step
 
-TODO
+At this point, we covered the main Chainsaw features.
+
+Look at the [next steps](./next-steps.md) section to find out what to do next.

website/docs/quick-start.md

@@ -1,157 +1,3 @@
-# Quick start
-
-To use Chainsaw you will need a [Kubernetes](https://kybernetes.io) cluster, Chainsaw won't create one for you.
-
-In this Quick start we will use [kind](https://kind.sigs.k8s.io) but feel free to use the tool of your choice.
-
-!!! warning "Not a cluster management tool"
-
-    We consider this is not the responsibility of Chainsaw to manage clusters.
-
-    There are plenty of solutions to create and manage local clusters that will do that better than Chainsaw.
-
-## Create a [kind](https://kind.sigs.k8s.io) cluster
-
-Please refer to the [kind install docs](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) to install it locally.
-
-Once [kind](https://kind.sigs.k8s.io) creating a local cluster is as simple as running:
-
-```bash
-# define kind image
-export KIND_IMAGE="kindest/node:v1.28.0"
-
-# create cluster
-kind create cluster --image $KIND_IMAGE
-```
-
-## Writing tests
-
-A Chainsaw test is [made of YAML files in a folder](./tests/index.md).
-
-YAML files can contain raw manifests with a special file naming convention to identify the step operations.
-This is useful to create tests quickly but doesn't allow great flexibility.
-
-Another option is to have a `chainsaw-test.yaml` file containing one (or more) `Test` resource. While more verbose, this offers full flexibility over the test and test steps configuration.
-
-### Quick start
-
-For this quick start, we will create a `Test` with one step and two operations:
-
-1. Create a `ConfigMap` from a manifest
-1. Verify the `ConfigMap` was created and contains the expected data
-
-### Create the test folder
-
-```bash
-# create test folder
-mkdir chainsaw-quick-start
-
-# enter test folder
-cd chainsaw-quick-start
-```
-
-### Create a `ConfigMap`
-
-```bash
-# create a ConfigMap
-cat > configmap.yaml << EOF
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: chainsaw-quick-start
-data:
-  foo: bar
-EOF
-```
-
-### Create the test
-
-By default, Chainsaw will look for a file named `chainsaw-test.yaml` in every folder.
-
-```bash
-# create test file
-cat > chainsaw-test.yaml << EOF
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: quick-start
-spec:
-  steps:
-  - try:
-    # first operation: create the config map
-    - apply:
-        # file is relative to the test folder
-        file: configmap.yaml
-    # second operation: verify the config map exists and contains the expected data
-    - assert:
-        # file is relative to the test folder
-        file: configmap.yaml
-EOF
-```
-
-## Run Chainsaw
-
-We finished writing our first test, now we can run Chainsaw to verify the test runs as expected:
-
-```bash
-chainsaw test
-
-Version: (devel)
-Loading default configuration...
-- Using test file: chainsaw-test.yaml
-- TestDirs [.]
-- SkipDelete false
-- FailFast false
-- ReportFormat ''
-- ReportName 'chainsaw-report'
-- Namespace ''
-- FullName false
-- IncludeTestRegex ''
-- ExcludeTestRegex ''
-- ApplyTimeout 5s
-- AssertTimeout 30s
-- CleanupTimeout 30s
-- DeleteTimeout 15s
-- ErrorTimeout 30s
-- ExecTimeout 5s
-Loading tests...
-- quick-start (.)
-Running tests...
-=== RUN   chainsaw
-=== PAUSE chainsaw
-=== CONT  chainsaw
-=== RUN   chainsaw/quick-start
-=== PAUSE chainsaw/quick-start
-=== CONT  chainsaw/quick-start
-    | 10:44:26 | quick-start | @setup   | CREATE    | OK    | v1/Namespace @ chainsaw-immense-jay
-    | 10:44:26 | quick-start | step-1   | TRY       | RUN   |
-    | 10:44:26 | quick-start | step-1   | APPLY     | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | step-1   | CREATE    | OK    | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | step-1   | APPLY     | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | step-1   | ASSERT    | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | step-1   | ASSERT    | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | step-1   | TRY       | DONE  |
-    | 10:44:26 | quick-start | @cleanup | DELETE    | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | @cleanup | DELETE    | OK    | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | @cleanup | DELETE    | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start
-    | 10:44:26 | quick-start | @cleanup | DELETE    | RUN   | v1/Namespace @ chainsaw-immense-jay
-    | 10:44:26 | quick-start | @cleanup | DELETE    | OK    | v1/Namespace @ chainsaw-immense-jay
-    | 10:44:31 | quick-start | @cleanup | DELETE    | DONE  | v1/Namespace @ chainsaw-immense-jay
---- PASS: chainsaw (0.00s)
-    --- PASS: chainsaw/quick-start (5.25s)
-PASS
-Tests Summary...
-- Passed  tests 1
-- Failed  tests 0
-- Skipped tests 0
-Done.
-```
-### Chainsaw Charaterstics 
-
-**Resource diff in assertion failures:** Chainsaw offers detailed resource diffs upon assertion failures. It provides a clear view of discrepancies between expected and actual resource.
-
-**Resource templating support:** Chainsaw simplifies dynamic resource configuration with it's templating support. Instead on relying on `envsubst` for dynamic substitution of env-variable. This eliminates the need of any preprocessing step.
-
 ## Chainsaw Characteristics
 
 **Simplified Testing of Kubernetes Operators**

website/docs/resource-templating.md

@@ -15,7 +15,7 @@ You can think of bindings as a side context where you can store and retrieve dat
 Chainsaw offers some built-in bindings you can use. You can also create your own bindings and use outputs to pass information from one operation to the next.
 
 !!! note
-    Under the hood, Chainsaw uses the jmespath language, bindings are implemented using [lexical scoping](https://github.com/jmespath-community/jmespath.spec/blob/main/jep-011a-lexical-scope.md).
+    Under the hood, Chainsaw uses the [jmespath](https://jmespath.site/) language, bindings are implemented using [lexical scoping](https://github.com/jmespath-community/jmespath.spec/blob/main/jep-011a-lexical-scope.md).
 
 ## Built-in bindings
 

website/docs/run-test.md → website/docs/run-tests.md

@@ -1,4 +1,4 @@
-# Run your tests
+# Run tests
 
 After [installing chainsaw](./install.md) and [writing tests](./first-test.md), the next natural step is to run chainsaw to execute the tests.
 
@@ -21,8 +21,8 @@ kind create cluster --image "kindest/node:v1.29.4"
 
 Now you can run the `chainsaw test` command.
 
-```bash
-chainsaw test
+```
+> chainsaw test
 
 Version: (devel)
 Loading default configuration...
@@ -80,4 +80,4 @@ Done.
 
 ## Next step
 
-The test above demonstrates the most basic usage of Chainsaw. In the next sections, we will look at the [main features that make Chainsaw a very unique tool](./resource-templating.md).
+The test above demonstrates the most basic usage of Chainsaw. In the next sections, we will look at the [main features that make Chainsaw a very unique tool](./assertion-trees.md).

website/mkdocs.yaml

@@ -7,10 +7,14 @@ nav:
   - install.md
   - completion.md
   - first-test.md
-  - run-test.md
+  - run-tests.md
+  - assertion-trees.md
   - resource-templating.md
   - operation-outputs.md
+  - next-steps.md
   - quick-start.md
+  - Advanced use cases:
+    - multi-cluster.md
   - gh-action.md
 - Writing tests:
   - writing-tests.md