Migrate apache integration

[andresrc]

Package / Dataset creation or update checklist

This checklist is intended for Devs which create or update a package to make sure they are consistent.

If the change targets a specific ES / Kibana / Agent version, uncomment this line and specify version.

• Required Kibana version set to target version: 7.11.0 We don't need to require Kibana 7.11.0 for this package.

All Changes

• Change follows development guidelines
• Supported versions of the subject being monitored are documented
• Supported operating systems are documented (if applicable)
• System tests exist
• Documentation
• Fields follow ECS and naming conventions
• At least a manual test with ES / Kibana / Agent has been performed. (covered by system tests)
• The required Kibana version is set to the lowest version used in the manual test.

Dashboards

• Dashboards exists (if applicable)
• Screenshots of added / updated dashboards
• Datastream filters added to visualizations

Log datasets

• Pipeline tests exist (if applicable)
• Generated output for at least 1 log file exists

Metric datasets

This entry is currently recommended. It will be mandatory once we provide better support for it.

• Sample event (sample_event.json) exists

New Packages

• Screenshot of the Fleet "Add Integration" Page.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[mtojek]

@andresrc I'm not sure if I understand these action items:

Pipeline tests exist (if applicable)
Test log files exist for the grok patterns
Generated output for at least 1 log file exists

Pipeline tests use log files as test input. Once they're processed by ingest pipelines, the result is compared with generated output. I mean, pipeline tests require test log files and generated output, there is no way for them to exist without any of them.

Did you think about sth else?

EDIT:

It looks like it was missed in the PR review: #342 (comment)

[mtojek]

As discussed offline with the team I will try to introduce a condition which will change the Metricbeat's behavior and emit adjusted fields under different names.

[mtojek]

Let's write down all action item in a convenient order:

• Update Docker image to latest version - Apache: refactor integration #420
• Add information about supported OSes - Apache: refactor integration #420
• Update period for the status dataset - Apache: refactor integration #420
• Duplicate field defintions in apache.status to cover new ECS fields (uptime, hostname) - Apache: refactor integration #420
• Test Integrations is parallel, don't let Apache tests block execution of other tests - [CI] Introduce concurrency in package testing #297
• Modify Metricbeat to collect modified fields if running in Fleet mode - Apache status: adjust field collection in fleet mode beats#22821
• Cherry-pick Apache status: adjust field collection in fleet mode beats#22821 - Cherry-pick #22821 to 7.x: Apache status: adjust field collection in fleet mode beats#22882

(once Docker images for fleet are released, dashboards may stop working)

• Update dashboards with latest changes in Metricbeat - Apache: refactor integration (part II) #441
• Remove duplicated field defintions, leave ECS ones - Apache: refactor integration (part II) #441
• Update the sample event for status dataset - Apache: refactor integration (part II) #441
• Save sample events for other dataset - Apache: refactor integration (part II) #441
• Write pipeline tests for Apache - Apache: refactor integration (part II) #441
• Fleet apache: convert status.total_kbytes to status.total_bytes Fleet apache: convert status.total_kbytes to status.total_bytes beats#23022
• Cherry-pick Fleet apache: convert status.total_kbytes to status.total_bytes beats#23022 - Cherry-pick #23022 to 7.x: Fleet apache: convert status.total_kbytes to status.total_bytes beats#23055
• Refactor dashboards according to the instructions in the migration assessment - Apache: refactor dashboards #453
• Fix metrics dashboard to properly visualize total_bytes - Apache: replace total_kbytes with total_bytes #458
• Retake new screenshots of dashboards - Apache: update screenshots #460

[andresrc]

sorry, I'll remove the grok one, thanks!

[mtojek]

Changes partially merged, I will proceed with the next step:

Modify Metricbeat to collect modified fields if running in Fleet mode

[mtojek]

It looks like there were issues in the unified release, which blocked releasing changes in Metricbeat. It will block:

Fix metrics dashboard to properly visualize total_bytes

[mtojek]

All action items done. Resolving.