remove parse_http action

README.md

@@ -42,7 +42,7 @@ TBD: throughput on production servers.
 
 **Input**: [dmesg](plugin/input/dmesg/README.md), [fake](plugin/input/fake/README.md), [file](plugin/input/file/README.md), [http](plugin/input/http/README.md), [journalctl](plugin/input/journalctl/README.md), [k8s](plugin/input/k8s/README.md), [kafka](plugin/input/kafka/README.md)
 
-**Action**: [add_file_name](plugin/action/add_file_name/README.md), [add_host](plugin/action/add_host/README.md), [convert_date](plugin/action/convert_date/README.md), [convert_log_level](plugin/action/convert_log_level/README.md), [convert_utf8_bytes](plugin/action/convert_utf8_bytes/README.md), [debug](plugin/action/debug/README.md), [discard](plugin/action/discard/README.md), [flatten](plugin/action/flatten/README.md), [join](plugin/action/join/README.md), [join_template](plugin/action/join_template/README.md), [json_decode](plugin/action/json_decode/README.md), [json_encode](plugin/action/json_encode/README.md), [json_extract](plugin/action/json_extract/README.md), [keep_fields](plugin/action/keep_fields/README.md), [mask](plugin/action/mask/README.md), [modify](plugin/action/modify/README.md), [move](plugin/action/move/README.md), [parse_es](plugin/action/parse_es/README.md), [parse_http](plugin/action/parse_http/README.md), [parse_re2](plugin/action/parse_re2/README.md), [remove_fields](plugin/action/remove_fields/README.md), [rename](plugin/action/rename/README.md), [set_time](plugin/action/set_time/README.md), [split](plugin/action/split/README.md), [throttle](plugin/action/throttle/README.md)
+**Action**: [add_file_name](plugin/action/add_file_name/README.md), [add_host](plugin/action/add_host/README.md), [convert_date](plugin/action/convert_date/README.md), [convert_log_level](plugin/action/convert_log_level/README.md), [convert_utf8_bytes](plugin/action/convert_utf8_bytes/README.md), [debug](plugin/action/debug/README.md), [discard](plugin/action/discard/README.md), [flatten](plugin/action/flatten/README.md), [join](plugin/action/join/README.md), [join_template](plugin/action/join_template/README.md), [json_decode](plugin/action/json_decode/README.md), [json_encode](plugin/action/json_encode/README.md), [json_extract](plugin/action/json_extract/README.md), [keep_fields](plugin/action/keep_fields/README.md), [mask](plugin/action/mask/README.md), [modify](plugin/action/modify/README.md), [move](plugin/action/move/README.md), [parse_es](plugin/action/parse_es/README.md), [parse_re2](plugin/action/parse_re2/README.md), [remove_fields](plugin/action/remove_fields/README.md), [rename](plugin/action/rename/README.md), [set_time](plugin/action/set_time/README.md), [split](plugin/action/split/README.md), [throttle](plugin/action/throttle/README.md)
 
 **Output**: [clickhouse](plugin/output/clickhouse/README.md), [devnull](plugin/output/devnull/README.md), [elasticsearch](plugin/output/elasticsearch/README.md), [file](plugin/output/file/README.md), [gelf](plugin/output/gelf/README.md), [kafka](plugin/output/kafka/README.md), [postgres](plugin/output/postgres/README.md), [s3](plugin/output/s3/README.md), [splunk](plugin/output/splunk/README.md), [stdout](plugin/output/stdout/README.md)
 

_sidebar.md

@@ -40,7 +40,6 @@
     - [modify](plugin/action/modify/README.md)
     - [move](plugin/action/move/README.md)
     - [parse_es](plugin/action/parse_es/README.md)
-    - [parse_http](plugin/action/parse_http/README.md)
     - [parse_re2](plugin/action/parse_re2/README.md)
     - [remove_fields](plugin/action/remove_fields/README.md)
     - [rename](plugin/action/rename/README.md)

charts/filed/README.md

@@ -35,7 +35,7 @@ curl -X GET "http://localhost:9200/k8s-logs/_search/?size=10" -H 'Content-Type:
 ```shell
 helm upgrade --install web-logs . -f values.minikube.web-logs.yaml
 
-curl --resolve "web-logs.local:80:$( minikube ip )" -H 'Content-Type: application/json' http://web-logs.local -d \
+curl --resolve "web-logs.local:80:$( minikube ip )" -H 'Content-Type: application/json' 'http://web-logs.local/?env=cli' -d \
 '{"message": "Test event", "level": "info"}'
 
 kubectl port-forward svc/web-logs-elasticsearch 9201:9200

charts/filed/values.minikube.web-logs.yaml

@@ -30,13 +30,36 @@ elasticsearch:
 input:
   type: http
   address: ":80"
+  meta:
+    remote_addr: "{{ .remote_addr }}"
+    user_agent: '{{ index (index .request.Header "User-Agent") 0}}'
+    env: '{{ index (index .params "env") 0}}'
+  cors:
+      allowed_headers:
+        - DNT
+        - X-CustomHeader
+        - Keep-Alive
+        - User-Agent
+        - X-Requested-With
+        - If-Modified-Since
+        - Cache-Control
+        - Content-Type
+        - Authorization
+      allowed_origins:
+        - http://localhost:8090
 actions:
   - type: set_time
   - type: convert_log_level
     field: level
     style: number
     default_level: info
     remove_on_fail: true
+  - type: mask
+    metric_name: errors_total
+    metric_skip_status: true
+    metric_labels:
+      - env
+      - level
 
 resourceType: Deployment
 replicas: 1

cmd/file.d/file.d.go

@@ -33,7 +33,6 @@ import (
 	_ "github.com/ozontech/file.d/plugin/action/modify"
 	_ "github.com/ozontech/file.d/plugin/action/move"
 	_ "github.com/ozontech/file.d/plugin/action/parse_es"
-	_ "github.com/ozontech/file.d/plugin/action/parse_http"
 	_ "github.com/ozontech/file.d/plugin/action/parse_re2"
 	_ "github.com/ozontech/file.d/plugin/action/remove_fields"
 	_ "github.com/ozontech/file.d/plugin/action/rename"

docs/installation.md

@@ -17,6 +17,10 @@ Mount config from /my-config.yaml and start `file.d` with this config: <br>
 
 Do not forget to set [the actual tag](https://github.com/ozontech/file.d/pkgs/container/file.d) of the image.
 
+## Helm-chart
+
+[Helm-chart](/charts/filed/README.md) and examples for Minikube
+
 ## Precompiled binaries
 
 Precompiled binaries for released versions are available in the

e2e/http_file/config.yml

@@ -10,10 +10,6 @@ pipelines:
         match_fields:
           should_drop: ok
         match_mode: or
-      - type: add_file_name
-        field: source_name
-      - type: parse_http
-        field: source_name
       - type: join
         field: log
         start: '/^(panic:)|(http: panic serving)/'

e2e/start_work_test.go

@@ -33,7 +33,6 @@ import (
 	_ "github.com/ozontech/file.d/plugin/action/mask"
 	_ "github.com/ozontech/file.d/plugin/action/modify"
 	_ "github.com/ozontech/file.d/plugin/action/parse_es"
-	_ "github.com/ozontech/file.d/plugin/action/parse_http"
 	_ "github.com/ozontech/file.d/plugin/action/parse_re2"
 	_ "github.com/ozontech/file.d/plugin/action/remove_fields"
 	_ "github.com/ozontech/file.d/plugin/action/rename"

plugin/README.md

@@ -611,87 +611,6 @@ It parses HTTP input using Elasticsearch `/_bulk` API format. It converts source
 > Check out the details in [Elastic Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html).
 
 [More details...](plugin/action/parse_es/README.md)
-## parse_http
-It adds a field containing the source name to the event and extracts remote_ip and auth login.
-It is only applicable for input plugin http.
-You need add action add_file_name before it.
-
-**Example:**
-Service for receiving events from a static page:
-```yaml
-pipelines:
-  example_http_pipeline:
-    input:
-      # define input type.
-      type: http
-      # define http port.
-      address: ":9400"
-      auth:
-        strategy: basic
-        secrets:
-          frontend: 398fc645-e660-45f4-96bb-53b7a2b120e4
-      cors:
-          allowed_headers:
-            - DNT
-            - X-CustomHeader
-            - Keep-Alive
-            - User-Agent
-            - X-Requested-With
-            - If-Modified-Since
-            - Cache-Control
-            - Content-Type
-            - Authorization
-          allowed_origins:
-            - http://localhost:8090
-    actions:
-    - type: add_file_name
-      field: source_name
-      # parse http info
-    - type: parse_http
-      field: source_name
-      allowed_params:
-        - env
-    - type: convert_log_level
-      field: level
-      style: number
-      default_level: info
-      remove_on_fail: true
-    - type: mask
-      metric_name: errors_total
-      metric_skip_status: true
-      metric_labels:
-        - login
-        - level
-    - type: remove_fields
-      fields:
-        - source_name
-    output:
-      type: stdout
-      # Or we can write to file:
-      # type: file
-      # target_file: "./output.txt"
-```
-
-Setup:
-```bash
-# run server.
-# config.yaml should contains yaml config above.
-go run ./cmd/file.d --config=config.yaml
-
-# now do requests.
-curl "127.0.0.1:9400/?env=cli" -H 'Content-Type: application/json' -H 'Authorization: Basic ZnJvbnRlbmQ6Mzk4ZmM2NDUtZTY2MC00NWY0LTk2YmItNTNiN2EyYjEyMGU0' -d \
-'{"message": "Test event", "level": "info"}'
-
-# run nginx with static page
-docker run -p 8090:80 -v `pwd`/plugin/action/parse_http:/usr/share/nginx/html -it --rm --name my-static-html-server nginx:alpine
-
-# open http://localhost:8090 and click "Send Log Request" button
-
-# check metric
-curl localhost:9000/metrics | grep "file_d_pipeline_example_http_pipeline_errors_total_events_count_total"
-```
-
-[More details...](plugin/action/parse_http/README.md)
 ## parse_re2
 It parses string from the event field using re2 expression with named subgroups and merges the result with the event root.
 

plugin/action/README.md

@@ -442,87 +442,6 @@ It parses HTTP input using Elasticsearch `/_bulk` API format. It converts source
 > Check out the details in [Elastic Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html).
 
 [More details...](plugin/action/parse_es/README.md)
-## parse_http
-It adds a field containing the source name to the event and extracts remote_ip and auth login.
-It is only applicable for input plugin http.
-You need add action add_file_name before it.
-
-**Example:**
-Service for receiving events from a static page:
-```yaml
-pipelines:
-  example_http_pipeline:
-    input:
-      # define input type.
-      type: http
-      # define http port.
-      address: ":9400"
-      auth:
-        strategy: basic
-        secrets:
-          frontend: 398fc645-e660-45f4-96bb-53b7a2b120e4
-      cors:
-          allowed_headers:
-            - DNT
-            - X-CustomHeader
-            - Keep-Alive
-            - User-Agent
-            - X-Requested-With
-            - If-Modified-Since
-            - Cache-Control
-            - Content-Type
-            - Authorization
-          allowed_origins:
-            - http://localhost:8090
-    actions:
-    - type: add_file_name
-      field: source_name
-      # parse http info
-    - type: parse_http
-      field: source_name
-      allowed_params:
-        - env
-    - type: convert_log_level
-      field: level
-      style: number
-      default_level: info
-      remove_on_fail: true
-    - type: mask
-      metric_name: errors_total
-      metric_skip_status: true
-      metric_labels:
-        - login
-        - level
-    - type: remove_fields
-      fields:
-        - source_name
-    output:
-      type: stdout
-      # Or we can write to file:
-      # type: file
-      # target_file: "./output.txt"
-```
-
-Setup:
-```bash
-# run server.
-# config.yaml should contains yaml config above.
-go run ./cmd/file.d --config=config.yaml
-
-# now do requests.
-curl "127.0.0.1:9400/?env=cli" -H 'Content-Type: application/json' -H 'Authorization: Basic ZnJvbnRlbmQ6Mzk4ZmM2NDUtZTY2MC00NWY0LTk2YmItNTNiN2EyYjEyMGU0' -d \
-'{"message": "Test event", "level": "info"}'
-
-# run nginx with static page
-docker run -p 8090:80 -v `pwd`/plugin/action/parse_http:/usr/share/nginx/html -it --rm --name my-static-html-server nginx:alpine
-
-# open http://localhost:8090 and click "Send Log Request" button
-
-# check metric
-curl localhost:9000/metrics | grep "file_d_pipeline_example_http_pipeline_errors_total_events_count_total"
-```
-
-[More details...](plugin/action/parse_http/README.md)
 ## parse_re2
 It parses string from the event field using re2 expression with named subgroups and merges the result with the event root.