[filebeat] deployment support feature #964

operatorequals · 2020-12-05T18:50:59Z

This PR is a duplicate of #822, which I lost control of because I deleted my fork by accident

This PR introduces the feature of deploying a Kubernetes deployment
instead of a Daemonset using Filebeat, using a values.yaml syntax
equivalent to Metricbeat.

Specifically, this is used for creation of Filebeat instances not
bound to each Worker, conducting non-Worker-related work, such as
collection of AWS CloudTrail logs as described in [1].

[1] :#821

Chart version not bumped (the versions are all bumped and released at the same time)
README.md updated with any new values or changes
Updated template tests in ${CHART}/tests/*.py
Updated integration tests in ${CHART}/examples/*/test/goss.yaml

Fix #821

This commit indtroduces the feature of deploying a Kubernetes deployment instead of a Daemonset using Filebeat, using a `values.yaml` syntax as below: `values.yaml` --- ```yaml [...] deploymentType: [daemonset|deployment] [...] ``` Specifically, this is used for creation of Filebeat instances not bound to each Worker, conducting non-Worker-related work, such as collection of AWS CloudTrail logs as described in [1]. [1]:elastic#821

This commit adds a default value test for `deploymentType`. Additionally, * `test_deployment_type_deployment` Checks if a `Deployment` is created but NOT a `DaemonSet` * `test_deployment_type_daemonset` Checks if a `DaemonSet` is created but NOT a `Deployment` * `test_deployment_type_case_insensitive` Checks if `deploymentType` value is accepted in a case-insensitive way.

This commit uses the MetricBeat Helm chart to create a Daemonset/Deployment Helm chart for Filebeat. Uses the ```yaml daemonset: [...] deployment: [...] ``` structure falling back to root key defaults.

The value: ```yaml serviceAccount: "" ``` was existing twice in the `filebeat/values.yaml` file.

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com>

elasticmachine · 2020-12-05T18:51:01Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

…atorequals/helm-charts into filebeat-deployment-support-feature

…new values

operatorequals · 2020-12-06T10:47:37Z

Hello @jmlrt !
Just continued my work on filebeat deployment in this PR, as mentioned here:
#822 (comment)

To sum up, I revised all your comments and suggestions (you left in the previous PR: #822), and reworked tests to include both Deployment and Daemonset, by merging the current test script with the Metricbeat's test script.

I hope I eased your life a bit, as these Daemonset/Deployment hybrid Helm Charts are quite tedius :)

$ pytest -v
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.9.0, pytest-4.1.0, py-1.8.0, pluggy-0.13.0 -- [...]
collected 39 items                                                                                                                                                                                             

tests/filebeat_test.py::test_defaults PASSED                                                                                                                                                             [  2%]
tests/filebeat_test.py::test_adding_a_extra_container PASSED                                                                                                                                             [  5%]
tests/filebeat_test.py::test_adding_init_containers_as_yaml PASSED                                                                                                                                       [  7%]
tests/filebeat_test.py::test_adding_a_extra_init_container PASSED                                                                                                                                        [ 10%]
tests/filebeat_test.py::test_adding_envs PASSED                                                                                                                                                          [ 12%]
tests/filebeat_test.py::test_adding_deprecated_envs PASSED                                                                                                                                               [ 15%]
tests/filebeat_test.py::test_adding_image_pull_secrets PASSED                                                                                                                                            [ 17%]
tests/filebeat_test.py::test_adding_host_networking PASSED                                                                                                                                               [ 20%]
tests/filebeat_test.py::test_adding_tolerations PASSED                                                                                                                                                   [ 23%]
tests/filebeat_test.py::test_adding_deprecated_tolerations PASSED                                                                                                                                        [ 25%]
tests/filebeat_test.py::test_override_the_default_update_strategy PASSED                                                                                                                                 [ 28%]
tests/filebeat_test.py::test_setting_a_custom_service_account PASSED                                                                                                                                     [ 30%]
tests/filebeat_test.py::test_self_managing_rbac_resources PASSED                                                                                                                                         [ 33%]
tests/filebeat_test.py::test_setting_pod_security_context PASSED                                                                                                                                         [ 35%]
tests/filebeat_test.py::test_setting_deprecated_pod_security_context PASSED                                                                                                                              [ 38%]
tests/filebeat_test.py::test_adding_in_filebeat_config PASSED                                                                                                                                            [ 41%]
tests/filebeat_test.py::test_adding_in_deprecated_filebeat_config PASSED                                                                                                                                 [ 43%]
tests/filebeat_test.py::test_adding_a_secret_mount PASSED                                                                                                                                                [ 46%]
tests/filebeat_test.py::test_adding_a_deprecated_secret_mount PASSED                                                                                                                                     [ 48%]
tests/filebeat_test.py::test_adding_a_extra_volume_with_volume_mount PASSED                                                                                                                              [ 51%]
tests/filebeat_test.py::test_adding_a_deprecated_extra_volume_with_volume_mount PASSED                                                                                                                   [ 53%]
tests/filebeat_test.py::test_adding_a_node_selector PASSED                                                                                                                                               [ 56%]
tests/filebeat_test.py::test_adding_deprecated_node_selector PASSED                                                                                                                                      [ 58%]
tests/filebeat_test.py::test_adding_an_affinity_rule PASSED                                                                                                                                              [ 61%]
tests/filebeat_test.py::test_priority_class_name PASSED                                                                                                                                                  [ 64%]
tests/filebeat_test.py::test_adding_deprecated_labels PASSED                                                                                                                                             [ 66%]
tests/filebeat_test.py::test_adding_daemonset_labels PASSED                                                                                                                                              [ 69%]
tests/filebeat_test.py::test_adding_daemonset_labels_surpasses_root_labels PASSED                                                                                                                        [ 71%]
tests/filebeat_test.py::test_adding_deployment_labels PASSED                                                                                                                                             [ 74%]
tests/filebeat_test.py::test_adding_deployment_labels_surpasses_root_labels PASSED                                                                                                                       [ 76%]
tests/filebeat_test.py::test_adding_serviceaccount_annotations PASSED                                                                                                                                    [ 79%]
tests/filebeat_test.py::test_adding_env_from PASSED                                                                                                                                                      [ 82%]
tests/filebeat_test.py::test_adding_deprecated_env_from PASSED                                                                                                                                           [ 84%]
tests/filebeat_test.py::test_overriding_resources PASSED                                                                                                                                                 [ 87%]
tests/filebeat_test.py::test_adding_deprecated_resources PASSED                                                                                                                                          [ 89%]
tests/filebeat_test.py::test_setting_fullnameOverride PASSED                                                                                                                                             [ 92%]
tests/filebeat_test.py::test_adding_annotations PASSED                                                                                                                                                   [ 94%]
tests/filebeat_test.py::test_disable_daemonset PASSED                                                                                                                                                    [ 97%]
tests/filebeat_test.py::test_disable_deployment PASSED                                                                                                                                                   [100%]

jmlrt · 2020-12-07T09:00:40Z

Great! I'll to take a look this week. Again thanks for your work here 👍

…he conf file

jmlrt · 2020-12-08T09:25:21Z

jenkins test this please

jmlrt · 2020-12-08T10:16:49Z

jenkins test this please

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com>

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com> Co-authored-by: John Torakis <john.torakis@gmail.com>

elastic/helm-charts#964 introduced the feature of deploying a Kubernetes deployment instead of a Daemonset using Filebeat, using a values.yaml syntax equivalent to Metricbeat

* chore: bump helm charts version * chore: bump helm version * chore: bump kind version * fix: support filebeat deployment elastic/helm-charts#964 introduced the feature of deploying a Kubernetes deployment instead of a Daemonset using Filebeat, using a values.yaml syntax equivalent to Metricbeat * chore: update version to 7.11.2

* chore: bump helm charts version * chore: bump helm version * chore: bump kind version * fix: support filebeat deployment elastic/helm-charts#964 introduced the feature of deploying a Kubernetes deployment instead of a Daemonset using Filebeat, using a values.yaml syntax equivalent to Metricbeat * chore: update version to 7.11.2 # Conflicts: # .ci/Jenkinsfile

John Torakis and others added 12 commits September 24, 2020 13:08

Similar to Metricbeat Templating

959abbf

This commit uses the MetricBeat Helm chart to create a Daemonset/Deployment Helm chart for Filebeat. Uses the ```yaml daemonset: [...] deployment: [...] ``` structure falling back to root key defaults.

Fix double value in filebeat/values.yaml

481104e

The value: ```yaml serviceAccount: "" ``` was existing twice in the `filebeat/values.yaml` file.

Fix missing parameter in Filebeat Deployment template

eaa12c6

Merge branch 'master' into filebeat-deployment-support-feature

9632ad8

Update filebeat/templates/configmap.yaml

fe60db0

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com>

Update filebeat/templates/configmap.yaml

f4f0734

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com>

Update filebeat/templates/daemonset.yaml

8e3a2a5

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com>

Resolving comments for 8e3a2a5

717cbe5

Add explanation for deployment/daemonset values scheme

dca3974

Set toleration defaults to empty list

332188a

operatorequals and others added 4 commits December 5, 2020 18:51

Merge branch 'master' into filebeat-deployment-support-feature

afd58f2

Added replicas Value for deployment

0935040

Fixed all suggestions

a8af38b

updated daemonset hostNetworking

f4346a6

operatorequals mentioned this pull request Dec 5, 2020

[Filebeat] Support for Deployment Kubernetes resource #822

Closed

4 tasks

Merge branch 'filebeat-deployment-support-feature' of github.com:oper…

d82ad0d

…atorequals/helm-charts into filebeat-deployment-support-feature

operatorequals force-pushed the filebeat-deployment-support-feature branch from 1a5d368 to d82ad0d Compare December 5, 2020 23:21

John Torakis added 4 commits December 6, 2020 12:33

Add Labels in deployment pods

f6eb5f9

rename mountpoint of daemonset data

8404054

Made deployment non-root, enabled by default to pass tests

3448e07

Refactored tests to pass on default-ON Deployment and daemonset with …

34f339d

…new values

Changed Filebeat deployment UserID to 0 from 1001 as it cannot read t…

4c463bf

…he conf file

Fixed python linting

6618cf5

jmlrt added a commit to jmlrt/helm-charts that referenced this pull request Jan 5, 2021

Filebeat deployment support feature (elastic#964)

900595d

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com>

jmlrt added a commit to jmlrt/helm-charts that referenced this pull request Jan 5, 2021

Filebeat deployment support feature (elastic#964)

c4a37fd

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com>

jmlrt added a commit that referenced this pull request Jan 5, 2021

Filebeat deployment support feature (#964) (#1004)

a160dee

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com> Co-authored-by: John Torakis <john.torakis@gmail.com>

jmlrt added backported and removed need-backport labels Jan 5, 2021

jmlrt added a commit that referenced this pull request Jan 5, 2021

Filebeat deployment support feature (#964) (#1005)

d1f1b59

Co-authored-by: Julien Mailleret <8582351+jmlrt@users.noreply.github.com> Co-authored-by: John Torakis <john.torakis@gmail.com>

jmlrt mentioned this pull request Jan 28, 2021

[filebeat] variable maxUnavailable updateStrategy #809

Merged

jmlrt mentioned this pull request Feb 8, 2021

backport 1046 7x #1051

Closed

This was referenced Feb 12, 2021

7.11.0 release changelog #1063

Closed

6.8.14 release changelog #1067

Merged

jmlrt mentioned this pull request Feb 18, 2021

7.11.1 changelog #1071

Merged

mdelapenya mentioned this pull request Mar 11, 2021

chore: bump helm versions elastic/e2e-testing#874

Merged

9 tasks

jmlrt mentioned this pull request Mar 15, 2021

7.12 #1101

Closed

jmlrt mentioned this pull request Mar 18, 2021

7.x #1113

Closed

jmlrt mentioned this pull request May 25, 2021

7.x bump #1199

Closed

This was referenced Dec 14, 2021

[meta] initiate 7.16 branch #1475

Merged

[kibana] optionally disable SA token automount (#1301) #1481

Merged

6.8.21 bump #1488

Merged

jmlrt mentioned this pull request Mar 8, 2022

7.17.1 bump #1609

Merged

jmlrt mentioned this pull request Apr 21, 2022

7.17.3 bump #1649

Merged

This was referenced Sep 14, 2022

[meta] fix yaml indent #1698

Merged

[meta] follow-up of #1698 #1700

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[filebeat] deployment support feature #964

[filebeat] deployment support feature #964

operatorequals commented Dec 5, 2020 •

edited

Loading

elasticmachine commented Dec 5, 2020

operatorequals commented Dec 6, 2020

jmlrt commented Dec 7, 2020

jmlrt commented Dec 8, 2020

jmlrt commented Dec 8, 2020

[filebeat] deployment support feature #964

[filebeat] deployment support feature #964

Conversation

operatorequals commented Dec 5, 2020 • edited Loading

elasticmachine commented Dec 5, 2020

operatorequals commented Dec 6, 2020

jmlrt commented Dec 7, 2020

jmlrt commented Dec 8, 2020

jmlrt commented Dec 8, 2020

operatorequals commented Dec 5, 2020 •

edited

Loading