Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the missing process.name field to System module, Syslog fileset #41354

Merged
merged 1 commit into from
Oct 23, 2024

Conversation

belimawr
Copy link
Contributor

@belimawr belimawr commented Oct 21, 2024

Proposed commit message

This PR adds the missing process.name field to System module, Syslog fileset

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact

## Author's Checklist

How to test this PR locally

  1. Make sure you're testing on a system where system logs are sent to Journald. The Debian 12 Vagrant VM from Beats is a good option.
  2. Package Filebeat (adjust for your system/platform). If you don't package it, dashboards won't be loaded
    DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz" PLATFORMS=linux/amd64 mage -v package
    
  3. Extract it, enable the system module
    ./filebeat modules enable system
    
  4. Edit filebeat/modules.d/system.yml to enable the syslog fileset and ensure Journald input will be used
    - module: system
      syslog:
        enabled: true
        var.use_journald: true
    
  5. Edit filebeat.yml with your ES and Kibana credentials
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      reload.period: 1s
    
    setup.template:
      settings:
        index.number_of_shards: 1
    
    setup.kibana:
      host: "http://kibana:5601"
      username: admin
      password: testing
      ssl.verification_mode: none
    
    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"]
      preset: latency
      protocol: "http"
    
      username: admin
      password: testing
      ssl.verification_mode: none
    
  6. Run the setup command
    ./filebeat setup --modules system
    
  7. Start Filebeat
  8. Ensure the events contain process.name and the [Filebeat System] Syslog dashboard ECS contains data. You might have to adjust the time window

Related issues

## Use cases
## Screenshots
## Logs

@belimawr belimawr added bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Oct 21, 2024
@belimawr belimawr self-assigned this Oct 21, 2024
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 21, 2024
Copy link
Contributor

mergify bot commented Oct 21, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

Copy link
Contributor

mergify bot commented Oct 21, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 21, 2024
@belimawr belimawr added the backport-8.16 Automated backport with mergify label Oct 22, 2024
@@ -36,6 +37,7 @@
"log.syslog.facility.code": 0,
"log.syslog.priority": 6,
"message": "Console: switching to colour frame buffer device 160x50",
"process.name": "",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems most of the examples I got don't have a process.name available 🤦‍♂️ . The tests can be improved on a later PR.

@belimawr belimawr marked this pull request as ready for review October 22, 2024 14:28
@belimawr belimawr requested a review from a team as a code owner October 22, 2024 14:28
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@belimawr belimawr changed the title [WPI] Fix system module dashboards and ingestion with journald Add the missing process.name field to System module, Syslog fileset Oct 22, 2024
@belimawr belimawr merged commit d2796da into elastic:main Oct 23, 2024
30 checks passed
mergify bot pushed a commit that referenced this pull request Oct 23, 2024
This PR adds the missing process.name field to System module, Syslog fileset

(cherry picked from commit d2796da)
mergify bot pushed a commit that referenced this pull request Oct 23, 2024
This PR adds the missing process.name field to System module, Syslog fileset

(cherry picked from commit d2796da)
@pierrehilbert
Copy link
Collaborator

Shouldn't we have additional testing to ensure we won't break it in the future?

belimawr added a commit that referenced this pull request Oct 23, 2024
This PR adds the missing process.name field to System module, Syslog fileset

(cherry picked from commit d2796da)

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
belimawr added a commit that referenced this pull request Oct 23, 2024
This PR adds the missing process.name field to System module, Syslog fileset

(cherry picked from commit d2796da)

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
@belimawr
Copy link
Contributor Author

Shouldn't we have additional testing to ensure we won't break it in the future?

We technically have those tests in place already, when the tests run on CI they ingest some files and compare the final event with a golden file, if fields are different, the test fails.

The problem is that when we add a new source for the test, we don't have a reference, so they're generated. You can see the *-expected.json files that this PR updated. Most of the test data I added for the journald input do not have the process.name, so ad the time I did not realised it was mostly required.

One type of test that do not have and would be nice to have is to have a test dataset that can populate all visualisations from the dashboards and ensure that when ingested, the dashboard is correctly populated. That's not as simple to implement but it would be nice to have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat - System module - Journald] Some fields are missing
4 participants