move quickstart to examples/compose

.github/secret_scanning.yml

@@ -1,4 +1,4 @@
 paths-ignore:
-  - "quickstart/exampledata/config/credentials.yml"
-  - "quickstart/exampledata/config/nginx/mtls.conf.d/*.key"
-  - "quickstart/exampledata/config/nginx/mtls.conf.d/*.crt"
+  - "examples/exampledata/config/credentials.yml"
+  - "examples/exampledata/config/nginx/mtls.conf.d/*.key"
+  - "examples/exampledata/config/nginx/mtls.conf.d/*.crt"

CHANGELOG.md

@@ -315,7 +315,7 @@
 
 * add possibility to convert hex to int in `calculator` processor with new added function `from_hex`
 * add metrics on rule level
-* add grafana example dashboards under `quickstart/exampledata/config/grafana/dashboards`
+* add grafana example dashboards under `examples/exampledata/config/grafana/dashboards`
 * add new configuration field `id` for all rules to identify rules in metrics and logs
   - if no `id` is given, the `id` will be generated in a stable way
   - add verification of rule `id` uniqueness on processor level over both rule trees to ensure metrics are counted correctly on rule level

README.md

@@ -29,7 +29,7 @@ allowing further applications besides log handling.
 This readme provides basic information about the following topics:
 - [About Logprep](#about-logprep)
 - [Getting Started](https://logprep.readthedocs.io/en/latest/getting_started.html)
-- [Docker Quickstart](https://logprep.readthedocs.io/en/latest/getting_started.html#logprep-quickstart-environment)
+- [Deployment Examples](https://logprep.readthedocs.io/en/latest/examples/index.html)
 - [Event Generation](https://logprep.readthedocs.io/en/latest/user_manual/execution.html#event-generation)
 - [Documentation](https://logprep.readthedocs.io/en/latest)
 - [Contributing](#contributing)

doc/source/development/programaticly_start_logprep.rst

@@ -23,10 +23,10 @@ An example with input connector and preprocessors could look like this:
                 "predetector": {
                     "type": "pre_detector",
                     "specific_rules": [
-                        "quickstart/exampledata/rules/pre_detector/specific"
+                        "examples/exampledata/rules/pre_detector/specific"
                     ],
                     "generic_rules": [
-                        "quickstart/exampledata/rules/pre_detector/generic"
+                        "examples/exampledata/rules/pre_detector/generic"
                     ],
                     "pre_detector_topic": "output_topic"
                 }
@@ -61,10 +61,10 @@ An example without input connector and preprocessors could look like this:
                 "predetector": {
                     "type": "pre_detector",
                     "specific_rules": [
-                        "quickstart/exampledata/rules/pre_detector/specific"
+                        "examples/exampledata/rules/pre_detector/specific"
                     ],
                     "generic_rules": [
-                        "quickstart/exampledata/rules/pre_detector/generic"
+                        "examples/exampledata/rules/pre_detector/generic"
                     ],
                     "pre_detector_topic": "output_topic"
                 }

doc/source/examples/compose.rst

@@ -5,7 +5,7 @@ Docker Compose Example Deployment
 To demonstrate the functionality of logprep this repo comes with a complete `kafka`, `logprep` and
 `opensearch` stack.
 To get it running `docker` with compose support must be first installed.
-The docker compose file is located in the directory `quickstart`.
+The docker compose file is located in the directory `examples/compose`.
 A prerequisite is to run `sysctl -w vm.max_map_count=262144`, otherwise Opensearch might not
 properly start.
 
@@ -14,7 +14,7 @@ The environment can either be started with a Logprep container or without one:
 Run without Logprep Container (default)
 ---------------------------------------
 
-  1. Run from within the `quickstart` directory:
+  1. Run from within the `examples/compose` directory:
 
      .. code-block:: bash
 
@@ -25,22 +25,22 @@ Run without Logprep Container (default)
 
      .. code-block:: bash
 
-      logprep run quickstart/exampledata/config/pipeline.yml
+      logprep run examples/exampledata/config/pipeline.yml
 
     If logprep is run with the metrics enabled, the necessary environment variable has to be set first:
 
     .. code-block:: bash
 
       export PROMETHEUS_MULTIPROC_DIR="tmp/logprep"
-      logprep run quickstart/exampledata/config/pipeline.yml
+      logprep run examples/exampledata/config/pipeline.yml
 
 
 
 
 Run with Logprep Container
 --------------------------
 
-  * Run from within the `quickstart` directory:
+  * Run from within the `examples/compose` directory:
 
     .. code-block:: bash
 
@@ -50,7 +50,7 @@ Run with Logprep Container
 Run with getting config from http server with basic authentication
 ------------------------------------------------------------------
 
-  * Run from within the `quickstart` directory:
+  * Run from within the `examples/compose` directory:
 
     .. code-block:: bash
 
@@ -60,14 +60,14 @@ Run with getting config from http server with basic authentication
 
     .. code-block:: bash
 
-      export LOGPREP_CREDENTIALS_FILE="quickstart/exampledata/config/credentials.yml"
+      export LOGPREP_CREDENTIALS_FILE="examples/exampledata/config/credentials.yml"
       logprep run http://localhost:8081/config/pipeline.yml
 
 
 Run with getting config from http server with mTLS authentication
 -----------------------------------------------------------------
 
-  * Run from within the `quickstart` directory:
+  * Run from within the `examples/compose` directory:
 
     .. code-block:: bash
 
@@ -77,12 +77,12 @@ Run with getting config from http server with mTLS authentication
 
     .. code-block:: bash
 
-      export LOGPREP_CREDENTIALS_FILE="quickstart/exampledata/config/credentials.yml"
+      export LOGPREP_CREDENTIALS_FILE="examples/exampledata/config/credentials.yml"
       logprep run https://localhost:8082/config/pipeline.yml
 
 
-Interacting with the Quickstart Environment
--------------------------------------------
+Interacting with the Compose Environment
+----------------------------------------
 
 The start up takes a few seconds to complete, but once everything is up
 and running it is possible to write JSON events into Kafka and read the processed events in
@@ -111,9 +111,9 @@ UCL Postgres:          `localhost:5432`  ucl                       ucl
 ====================== ================= ========================  =======================
 
 The example rules that are used in the docker instance of Logprep can be found
-in `quickstart/exampledata/rules`.
+in `examples/exampledata/rules`.
 Example events that trigger for the example rules can be found in
-`quickstart/exampledata/input_logdata/logclass/test_input.jsonl`.
+`examples/exampledata/input_logdata/logclass/test_input.jsonl`.
 These events can be added to Kafka with the following command:
 
 .. code-block:: bash
@@ -133,12 +133,12 @@ Utilizing FDA and UCL
 If you want to try out the FDA and UCL you first have to do some preparations.
 
 
-0. Run the quickstart setup with the :code:`oauth2` profile:
+0. Run the example compose setup with the :code:`oauth2` profile:
    :code:`docker compose --profile oauth2 up -d`.
 1. Sign into the keycloak admin panel and create a logprep user in the :code:`logprep` realm.
    Make sure that the user is part of the :code:`logprep-admin` group and has a password. If you
    choose a password other than :code:`logprep` you have to update the credentials file
-   :code:`quickstart/exampledata/config/credentials.yml`, such that the password of
+   :code:`examples/exampledata/config/credentials.yml`, such that the password of
    :code:`http://localhost:3001` and :code:`http://localhost:3002` reflects your choice.
 2. You have to login to the FDA with the previously created user and create a release, as well
    as your first logclass. It is also necessary to add an example event to this logclass in order
@@ -148,7 +148,7 @@ If you want to try out the FDA and UCL you first have to do some preparations.
    your created logprep user and then configure required Use-Cases.
    At the current moment these configuration are not yet processed by logprep though, as the ucl
    only provides a mock endpoint which doesn't contain your Use-Case configurations.
-4. Set the env :code:`LOGPREP_CREDENTIALS_FILE` to :code:`quickstart/exampledata/config/credentials.yml`
+4. Set the env :code:`LOGPREP_CREDENTIALS_FILE` to :code:`examples/exampledata/config/credentials.yml`
 
 Once you have set everything up you can run logprep with the following command.
 Just consider that the first :code:`pipeline.yml` argument is used to define a proper :code:`input`
@@ -157,10 +157,10 @@ you should ensure that the :code:`stage` and :code:`loglcass` are set properly.
 
 .. code-block:: bash
 
-     logprep run quickstart/exampledata/config/pipeline.yml "http://localhost:3002/api/v1/pipelines?stage=prod&logclass=ExampleClass" "http://localhost:3001/api/v1/general-predetection"
+     logprep run examples/exampledata/config/pipeline.yml "http://localhost:3002/api/v1/pipelines?stage=prod&logclass=ExampleClass" "http://localhost:3001/api/v1/general-predetection"
 
 .. note::
 
-     If you did use the quickstart setup before and run into problems it is advised to first pull
+     If you did use the example compose setup before and run into problems it is advised to first pull
      all images again to update them to the latest version:
-     :code:`docker compose -f ./quickstart/docker-compose.yml pull`.
+     :code:`docker compose -f ./example/compose/docker-compose.yml pull`.

quickstart/docker-compose.yml → examples/compose/docker-compose.yml

@@ -100,21 +100,21 @@ services:
       - kafka
       - opensearch
     volumes:
-      - ../quickstart/:/home/logprep/quickstart/
+      - ../../examples/:/home/logprep/examples/
     tmpfs:
       - /tmp/logprep/prometheus_multiproc
     entrypoint:
       - logprep
       - run
-      - /home/logprep/quickstart/exampledata/config/pipeline.yml
+      - /home/logprep/examples/exampledata/config/pipeline.yml
   grafana:
     image: bitnami/grafana:latest
     container_name: grafana
     ports:
       - "3000:3000"
     volumes:
-      - ../quickstart/exampledata/config/grafana/datasources:/opt/bitnami/grafana/conf/provisioning/datasources
-      - ../quickstart/exampledata/config/grafana/dashboards:/opt/bitnami/grafana/conf/provisioning/dashboards
+      - ../../examples/exampledata/config/grafana/datasources:/opt/bitnami/grafana/conf/provisioning/datasources
+      - ../../examples/exampledata/config/grafana/dashboards:/opt/bitnami/grafana/conf/provisioning/dashboards
     networks:
       - prometheus
   prometheus:
@@ -123,7 +123,7 @@ services:
     ports:
       - "9090:9090"
     volumes:
-      - ../quickstart/exampledata/config/prometheus/prometheus.yml:/opt/bitnami/prometheus/conf/prometheus.yml
+      - ../../examples/exampledata/config/prometheus/prometheus.yml:/opt/bitnami/prometheus/conf/prometheus.yml
     networks:
       - kafka
       - prometheus
@@ -138,9 +138,9 @@ services:
     ports:
       - "8081:8081"
     volumes:
-      - ../quickstart/exampledata:/usr/share/nginx/html:ro
-      - ../quickstart/exampledata/config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
-      - ../quickstart/exampledata/config/nginx/conf.d:/etc/nginx/conf.d:ro
+      - ../../examples/exampledata:/usr/share/nginx/html:ro
+      - ../../examples/exampledata/config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ../../examples/exampledata/config/nginx/conf.d:/etc/nginx/conf.d:ro
   mtls-config:
     image: nginx:latest
     container_name: mtls-config
@@ -149,9 +149,9 @@ services:
     ports:
       - "8082:8082"
     volumes:
-      - ../quickstart/exampledata:/usr/share/nginx/html:ro
-      - ../quickstart/exampledata/config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
-      - ../quickstart/exampledata/config/nginx/mtls.conf.d:/etc/nginx/conf.d:ro
+      - ../../examples/exampledata:/usr/share/nginx/html:ro
+      - ../../examples/exampledata/config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ../../examples/exampledata/config/nginx/mtls.conf.d:/etc/nginx/conf.d:ro
   keycloak:
     image: bitnami/keycloak:latest
     container_name: keycloak
@@ -160,7 +160,7 @@ services:
     profiles:
       - oauth2
     volumes:
-      - ../quickstart/exampledata/config/keycloak:/opt/keycloak/data/import
+      - ../../examples/exampledata/config/keycloak:/opt/keycloak/data/import
     environment:
       KEYCLOAK_ADMIN_USER: admin
       KEYCLOAK_ADMIN_PASSWORD: admin
@@ -263,8 +263,8 @@ services:
     ports:
       - "3002:80"
     volumes:
-      - ../quickstart/exampledata/config/fda/config.json:/usr/share/nginx/html/assets/config/keycloak/config.json
-      - ../quickstart/exampledata/config/fda/default.conf:/etc/nginx/conf.d/default.conf
+      - ../../examples/exampledata/config/fda/config.json:/usr/share/nginx/html/assets/config/keycloak/config.json
+      - ../../examples/exampledata/config/fda/default.conf:/etc/nginx/conf.d/default.conf
     networks:
       - fda
   ucl-api:
@@ -338,9 +338,9 @@ services:
     ports:
       - "3001:80"
     volumes:
-      - ../quickstart/exampledata/config/ucl/nginx.example.conf:/etc/nginx/conf.d/default.conf
-      - ../quickstart/exampledata/config/ucl/general.example.json:/usr/share/nginx/html/assets/config/general/config.json
-      - ../quickstart/exampledata/config/ucl/keycloak.example.json:/usr/share/nginx/html/assets/config/keycloak/config.json
+      - ../../examples/exampledata/config/ucl/nginx.example.conf:/etc/nginx/conf.d/default.conf
+      - ../../examples/exampledata/config/ucl/general.example.json:/usr/share/nginx/html/assets/config/general/config.json
+      - ../../examples/exampledata/config/ucl/keycloak.example.json:/usr/share/nginx/html/assets/config/keycloak/config.json
     networks:
       - ucl
 

quickstart/exampledata/config/credentials.yml → examples/exampledata/config/credentials.yml

@@ -15,14 +15,14 @@ getter:
     username: user
     password: password
   "https://localhost:8082":
-    client_key: quickstart/exampledata/config/nginx/mtls.conf.d/client.key
-    cert: quickstart/exampledata/config/nginx/mtls.conf.d/client.crt
-    ca_cert: quickstart/exampledata/config/nginx/mtls.conf.d/ca.crt
+    client_key: examples/exampledata/config/nginx/mtls.conf.d/client.key
+    cert: examples/exampledata/config/nginx/mtls.conf.d/client.crt
+    ca_cert: examples/exampledata/config/nginx/mtls.conf.d/ca.crt
 input:
   endpoints:
     /auth-json:
       username: user
-      password_file: quickstart/exampledata/config/user_password.txt
+      password_file: examples/exampledata/config/user_password.txt
     /lab/123/ABC/auditlog:
       username: user
-      password_file: quickstart/exampledata/config/user_password.txt
+      password_file: examples/exampledata/config/user_password.txt

quickstart/exampledata/config/pipeline.yml → examples/exampledata/config/pipeline.yml

@@ -15,26 +15,26 @@ metrics:
 pipeline:
   - labelername:
       type: labeler
-      schema: quickstart/exampledata/rules/labeler/schema.json
+      schema: examples/exampledata/rules/labeler/schema.json
       include_parent_labels: true
       specific_rules:
-        - quickstart/exampledata/rules/labeler/specific
+        - examples/exampledata/rules/labeler/specific
       generic_rules:
-        - quickstart/exampledata/rules/labeler/generic
+        - examples/exampledata/rules/labeler/generic
 
   - dissector:
       type: dissector
       specific_rules:
-        - quickstart/exampledata/rules/dissector/specific/
+        - examples/exampledata/rules/dissector/specific/
       generic_rules:
-        - quickstart/exampledata/rules/dissector/generic/
+        - examples/exampledata/rules/dissector/generic/
 
   - dropper:
       type: dropper
       specific_rules:
-        - quickstart/exampledata/rules/dropper/specific
+        - examples/exampledata/rules/dropper/specific
       generic_rules:
-        - quickstart/exampledata/rules/dropper/generic
+        - examples/exampledata/rules/dropper/generic
         - filter: "test_dropper"
           dropper:
             drop:
@@ -44,37 +44,37 @@ pipeline:
   - pre_detector:
       type: pre_detector
       specific_rules:
-        - quickstart/exampledata/rules/pre_detector/specific
+        - examples/exampledata/rules/pre_detector/specific
       generic_rules:
-        - quickstart/exampledata/rules/pre_detector/generic
+        - examples/exampledata/rules/pre_detector/generic
       outputs:
         - opensearch: sre
-      tree_config: quickstart/exampledata/rules/pre_detector/tree_config.json
-      alert_ip_list_path: quickstart/exampledata/rules/pre_detector/alert_ips.yml
+      tree_config: examples/exampledata/rules/pre_detector/tree_config.json
+      alert_ip_list_path: examples/exampledata/rules/pre_detector/alert_ips.yml
 
   - amides:
       type: amides
       specific_rules:
-        - quickstart/exampledata/rules/amides/specific
+        - examples/exampledata/rules/amides/specific
       generic_rules:
-        - quickstart/exampledata/rules/amides/generic
-      models_path: quickstart/exampledata/models/model.zip
+        - examples/exampledata/rules/amides/generic
+      models_path: examples/exampledata/models/model.zip
       num_rule_attributions: 10
       max_cache_entries: 1000000
       decision_threshold: 0.32
 
   - pseudonymizer:
       type: pseudonymizer
-      pubkey_analyst: quickstart/exampledata/rules/pseudonymizer/example_analyst_pub.pem
-      pubkey_depseudo: quickstart/exampledata/rules/pseudonymizer/example_depseudo_pub.pem
-      regex_mapping: quickstart/exampledata/rules/pseudonymizer/regex_mapping.yml
+      pubkey_analyst: examples/exampledata/rules/pseudonymizer/example_analyst_pub.pem
+      pubkey_depseudo: examples/exampledata/rules/pseudonymizer/example_depseudo_pub.pem
+      regex_mapping: examples/exampledata/rules/pseudonymizer/regex_mapping.yml
       hash_salt: a_secret_tasty_ingredient
       outputs:
         - opensearch: pseudonyms
       specific_rules:
-        - quickstart/exampledata/rules/pseudonymizer/specific/
+        - examples/exampledata/rules/pseudonymizer/specific/
       generic_rules:
-        - quickstart/exampledata/rules/pseudonymizer/generic/
+        - examples/exampledata/rules/pseudonymizer/generic/
       max_cached_pseudonyms: 1000000
 
   - calculator:

logprep/connector/http/input.py

@@ -45,7 +45,7 @@
       endpoints:
         /firstendpoint:
           username: user
-          password_file: quickstart/exampledata/config/user_password.txt
+          password_file: examples/exampledata/config/user_password.txt
         /second*:
           username: user
           password: secret_password