[Ingest Manager] possible encoding issue in processing data-sources, results in ui error

[EricDavisX]

[Ingest Manager] possible encoding issue in processing data-sources, results in ui error

Kibana version:
7.9.0 BC3 deployed to cloud with BC3 Agent installed to win 7 x64 host

Browser version:
chrome latest on macos

Describe the bug:
use this path for a custom logs entry and get an error:
"C:\Users\zeus\Desktop\test.log"

First seen with the longer path here used in further examples:
"C:\Users\zeus\Desktop\elastic-agent-7.9.0-bc3\elastic-agent-7.8.0-windows-x86_64\data\logs\elastic-agent-json.log"

Steps to reproduce:
here is the full request POST payload:

{"name":"log-1test","description":"","namespace":"default","config_id":"988a4100-ccf6-11ea-a848-5f6d14e28650","enabled":true,"output_id":"","inputs":[{"type":"logs","enabled":true,"streams":[{"id":"logs-log.log","enabled":true,"dataset":{"name":"log.log","type":"logs"},"vars":{"paths":{"type":"text","value":["C:\\Users\\zeus\\Desktop\\elastic-agent-7.9.0-bc3\\elastic-agent-7.8.0-windows-x86_64\\data\\logs\\elastic-agent-json.log"]},"dataset":{"value":"generic","type":"text"},"custom":{"value":"","type":"yaml"}}}]}],"package":{"name":"log","title":"Customs logs","version":"0.1.0"}}

from this post:
Request URL: https://33193cd6940e4f29a4e4b88335969f5b.europe-west1.gcp.cloud.es.io:9243/api/ingest_manager/package_configs

• returns a 500 error

Expected behavior:
this location seems valid to me (as a test) - it should be accepted and custom logs integration should function correctly

Screenshots (if relevant):

Errors in browser console (if relevant):
500 error response

[jen-huang]

@ph This is a package definition issue:
https://epr.elastic.co/package/log/0.1.0/dataset/log/agent/stream/stream.yml.hbs

This part:

{{#each paths}}
  - "{{this}}"
{{/each}}

should not have double quotes. It should use single quotes or omit the quotes entirely:

{{#each paths}}
  - {{this}}
{{/each}}

From looking in other package stream templates with a similar case, most choose to omit the quotes, e.g.: https://epr.elastic.co/package/system/0.3.0/dataset/auth/agent/stream/log.yml.hbs

From a grep in integrations repo, I think https://epr.elastic.co/package/log/0.1.0/dataset/log/agent/stream/stream.yml.hbs is the only stream template that needs to be corrected.

Assigning back to you to assign work.

[ghost]

Hi @EricDavisX

We are in-progress of bad data input validation for integrations and have reported 01 UI bug #73263 for RabbitMQ integration. We will validate the bad data inputs for other pending integrations and share the observations for the same.

Please let us know if anything else is required from our end.

[ghost]

Hi @EricDavisX

We have validated the below integrations with bad data input on Kibana BC4 oblt server https://release-oblt.elastic.dev and validation is working for these integrations.

1. ZooKeeper
2. Windows
3. System
4. AWS
5. Redis

We are in-progress of doing exploratory on bad data input validation for other pending integrations on Kibana BC4 oblt server https://release-oblt.elastic.dev. We will validate them and share the observations for the same.

Please let us know if anything else is missing from our end.

[ph]

created a PR for integration #73075

[ghost]

Hi @EricDavisX

We have validated the below integrations with bad data input on Kibana BC4 oblt server https://release-oblt.elastic.dev :-

1. Apache
2. Cisco
3. IIS
4. Kafka
5. Kubernetes
6. Custom Logs
7. MongoDB
8. MySQL
9. NetFlow
10. Nginx
11. PostgreSQL
12. Prometheus
13. Staging
14. Elastic Endpoint Security

Observations:

1. Validation error is displayed if no value is provided in mandatory field.

Screenshot for Custom Logs-> Log File path field:

2. No validation is displayed for incorrect or long data inserted in fields. Hence we have reported bug #73636 for the same.

Moreover, error is still displayed on saving the Custom Log integration after providing Log file path as 'C:\Users\zeus\Desktop\test.log'

Screenshot:

Logs:
Response under Browser Network logs:
{"statusCode":500,"error":"Internal Server Error","message":"expected hexadecimal character at line 3, column 10:\n - "C:\Users\zeus\Desktop\test.log"\n ^"}

Please let us know if anything else is missing from our end.

[ghost]

Bug Conversion:

Test-Case is not required for this ticket.